System

Abstract

A system is provided.SOLUTION: A system comprising: means for recording a screen operation; means for extracting information from the recorded video; means for analyzing a voice explanation of an operator; means for integrating the extracted information and the analyzed voice information to construct a work flow; means for generating a procedure based on the work flow; means for adjusting the generated procedure by the operator; and means for executing the adjusted procedure on a schedule.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's work environment, employees often perform repetitive, routine tasks that consume a lot of time and effort. This increases the likelihood of errors and makes it difficult for employees to concentrate on the creative work they should be doing. This invention aims to automate these routine tasks, thereby improving work accuracy and saving time, and providing an environment where employees can focus on more valuable work. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means.

[0006] The system includes a means for recording screen operations, a means for extracting information from the recorded video, a means for analyzing the worker's audio commentary, a means for constructing a workflow by integrating the extracted information and the analyzed audio information, a means for generating procedures based on the workflow, a means for the worker to adjust the generated procedures, and a means for executing the adjusted procedures on a schedule. This system automates routine tasks, improving the efficiency and accuracy of daily work.

[0007] A "means for recording screen operations" is a function that records a series of operations performed on the screen of a device such as a computer or tablet operated by a user in the form of a video or screenshots.

[0008] "Means for extracting information from recorded video" refers to a device that has the function of analyzing and extracting information such as text, images, and operating procedures from recorded video data.

[0009] The "means for analyzing the voice commentary of the worker" has a function of analyzing the voice data of the explanation given by the worker and extracting the content as text or meaning.

[0010] The "means for integrating the extracted information and analyzed audio information to construct a workflow" has the function of combining the extracted video information and audio information to form a certain logical procedure or workflow.

[0011] The "means for generating a procedure based on the workflow" has a function for generating specific steps and procedures that can be automated based on the constructed workflow.

[0012] "Means for workers to adjust generated procedures" refers to a function that allows workers to check procedures generated by AI or a system, and correct, supplement, or optimize unnecessary parts.

[0013] The "means for scheduling and executing the adjusted procedure" has a function of automatically executing the final adjusted procedure according to a specific time and date. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a system for automating routine tasks, and can be implemented based on the following settings.

[0036] This system involves a terminal, a server, and a user. First, the user records routine tasks. The recording is performed on the terminal operated by the user, and screen operations are recorded. All actions performed on the terminal (e.g., mouse clicks, keyboard input, screen capture) are saved in video format.

[0037] The recorded video is uploaded to a server. The server is equipped with a function to extract information from the recorded video. Specifically, the server uses OCR (optical character recognition) technology to analyze the text and data in the video and extract the necessary information. This process identifies the specific operating steps performed by the user.

[0038] Next, the user records an audio commentary about the task. The recording is done using the device's microphone, and the resulting audio file is uploaded to the server. The server analyzes the audio file and obtains the user's explanation as text data. This analysis is performed using voice recognition technology.

[0039] The server integrates the analyzed video and audio information to construct a detailed workflow. This workflow concretely shows the sequential steps of the user's operations. Based on this workflow, a procedure that can be automated is then generated. The server presents the generated procedure in list form and allows the user to confirm it.

[0040] The user can review the proposed procedure and make any necessary modifications, such as deleting unnecessary steps or adding additional comments. The optimized procedure is then saved back to the server.

[0041] Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, a series of operations can be performed automatically, such as opening a specific Excel file at 9:00 every morning, processing the data, and sending the results by email. This scheduled execution function frees users from routine tasks, allowing them to focus on more creative work.

[0042] As a concrete example, the following scenario can be considered.

[0043] A user's daily task is to process specific data in Excel and send the results to their boss by email. The user first records this series of operations. Next, they record an audio commentary about the work and upload it to the server. The server analyzes this data, generates an automatable procedure, and presents it to the user. After the user checks and modifies the procedure, the server schedules the procedure to be executed every morning at 9:00 AM. As a result, the user is freed from the hassle of manually processing and sending data every morning.

[0044] In this way, the system automates routine tasks, saving time and effort while also increasing the accuracy of the work.

[0045] The processing flow will be explained below.

[0046] Step 1:

[0047] The user uses the device to record screen activity. The device uses screen capture software to record all user activity (e.g., mouse clicks, keyboard input) and saves it as a video file.

[0048] Step 2:

[0049] The user stops recording and uploads the recorded video file from the terminal to the server, and the terminal transfers the video file to the server via the network.

[0050] Step 3:

[0051] The server receives the uploaded video file and uses OCR technology to analyze the text information and operation details in the video, extracting the analyzed information as text data.

[0052] Step 4:

[0053] The user uses the device to record an audio commentary of the work they are doing, and the recorded audio file is saved on the device.

[0054] Step 5:

[0055] A user uploads an audio file from a terminal to a server, and the terminal transfers the audio file to the server via a network.

[0056] Step 6:

[0057] The server receives the uploaded audio file and converts the audio content into text data using voice recognition technology. The text data obtained through voice analysis accurately reflects the user's commentary.

[0058] Step 7:

[0059] The server integrates the video and audio analysis data to construct a detailed workflow, which combines the user's operational steps with corresponding audio commentary.

[0060] Step 8:

[0061] The server generates automatable procedures based on the constructed workflow, and the generated procedures are organized in a list format and presented to the user.

[0062] Step 9:

[0063] The user checks the generated procedure and makes corrections or adjustments as necessary. The user uses the terminal to change the content of the procedure and arrange it in the optimal form.

[0064] Step 10:

[0065] The modified and adjusted procedures are saved back to the server, which stores these procedures in a database for scheduled execution.

[0066] Step 11:

[0067] A user uses a terminal to schedule an automated procedure to run, for example, at a specific time each day.

[0068] Step 12:

[0069] At the set time, the server executes the automated procedures according to the schedule. The server executes each procedure in sequence and automatically performs the necessary operations (e.g., processing data in Excel, sending emails).

[0070] Example 1

[0071] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0072] In modern business activities, many routine tasks occur on a daily basis. These tasks require time and effort, and there are issues with efficiency and accuracy. Conventional automation systems have difficulty automating a series of operations at once, and human intervention is required, especially for complex business processes. Therefore, reducing labor costs and improving work accuracy have become important issues.

[0073] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0074] In this invention, the server includes means for a user to record work on a terminal, means for uploading the recorded data to the server, means for the server to analyze the video, means for the user to record an audio commentary on the work content, means for uploading the recorded data to the server, means for the server to analyze the audio file, means for the server to integrate the video information and the audio information, means for the server to generate an automation procedure, means for the user to check and modify the procedure, and means for the server to schedule and execute the procedure. This makes it possible to automate routine business processes efficiently and with high accuracy.

[0075] A "user" is an operator who uses the system to automate routine tasks.

[0076] A "terminal" is a hardware device that a user uses to perform work, and includes a personal computer, a smartphone, etc.

[0077] "Recording" is the process of recording operations performed on a terminal in video format.

[0078] A "server" is a computer or cloud service that is the core of a system, and is a device that analyzes data and generates and executes procedures.

[0079] "Uploading" is the process of sending files or data generated on a device to a server.

[0080] "Video analysis" is the process of extracting text information and operating procedures from recorded video using OCR technology.

[0081] "Audio commentary" is information in which the user verbally explains the content of the work.

[0082] An "audio file" is a file that stores audio data recorded by a user.

[0083] "Speech analysis" is the process of obtaining text information from an audio file using speech recognition technology.

[0084] A "workflow" is information that indicates the sequential steps of a user's operation procedure, and is constructed by integrating video information and audio information.

[0085] An "automation procedure" is a procedure script created based on a workflow to automatically execute a series of operations.

[0086] "Procedure verification and correction" is the process in which the user verifies the automated procedures presented by the server and makes any necessary corrections.

[0087] "Scheduled execution" is the process of automatically executing a procedure at a specified time.

[0088] The present invention relates to a system for automating routine tasks, and involves a user, a terminal, and a server.

[0089] The system uses the following hardware and software:

[0090] Terminal: A device operated by a user, such as a computer or smartphone.

[0091] Servers: High-performance computers, data centers, and cloud services (e.g., AWS (registered trademark), Azure (registered trademark)).

[0092] OCR technology: Software for extracting text data (e.g., Google® Vision API).

[0093] Speech recognition technology: Software that converts speech into text (e.g., Google Speech-to-Text API).

[0094] Recording software: A recording tool that runs on your device (e.g., OBS Studio).

[0095] Audio recording software: A tool to record audio from your microphone input (e.g., Audacity).

[0096] As a specific example of operation, the process below shows how to automate a task in which a user processes specific data in Excel every day and sends the results to their supervisor by email.

[0097] Operating Procedure

[0098] 1. The user launches OBS Studio on their device and clicks the "Record" button to begin recording their work. All on-screen actions (e.g., opening Excel and entering data into specific cells) will be recorded.

[0099] 2. Once recording is complete, download the recording file from the OBS Studio file menu and upload it to the server using the dedicated upload form.

[0100] 3. The server uses the Google Vision API to analyze the uploaded video file and extract the cell contents in Excel and the user interface operation procedures as text data.

[0101] 4. Next, the user uses Audacity to record an audio description of the task, generating an audio file containing instructions such as, "To perform this task, first enter sales data into Excel cell A1."

[0102] 5. After recording is complete, the audio file is uploaded to the server, which uses the Google Speech-to-Text API to extract the text from the audio file.

[0103] 6. The server combines the information extracted from the video file and the information extracted from the audio file to construct a detailed workflow, which shows the steps taken by the user.

[0104] 7. The server generates automated procedures based on the workflow and presents them to the user in list form.

[0105] 8. The user reviews the proposed steps and modifies them as necessary, for example, by inserting additional steps or deleting unnecessary steps.

[0106] 9. Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, an Excel file is opened at 9:00 every morning, the data is processed, and the results are sent by email.

[0107] Specific examples

[0108] If a user wants to automate a daily task of processing certain data in Excel and emailing the results to their boss, they can use a prompt like this:

[0109] "I want an automated task that opens new sales data in Excel every morning, processes it into a specific format, and emails the results to my boss."

[0110] In this way, the system efficiently automates routine tasks, freeing users from routine tasks, saving time and effort, and improving work accuracy.

[0111] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0112] Step 1:

[0113] A user records their work on a device. To do this, the user launches recording software (e.g., OBS Studio) and clicks the "Record" button to begin recording their work. The input is the user's screen operations, and the output is a recorded video file. This video file records the user's operations (e.g., opening Excel, selecting cells, entering data, etc.).

[0114] Step 2:

[0115] The user uploads the recorded data to the server. The user retrieves the file from OBS Studio and sends the data to the server using a dedicated upload form. The input is the recorded video file, and the output is a video file saved on the server. Specifically, the user opens a browser, accesses the upload form, selects the recorded file, and completes the upload.

[0116] Step 3:

[0117] The server analyzes the video. It uses the Google Vision API to analyze the video file frame by frame and extracts text information from the video and user interface operation procedures. The input is the uploaded video file, and the output is text information and operation procedures. Specifically, the server loads the video file and runs OCR on each frame to extract the text data.

[0118] Step 4:

[0119] A user records an audio description of a task. The user launches Audacity and clicks the "Record" button to begin recording. The input is the user's verbal description, and the output is an audio file. Specifically, the user explains into the microphone, for example, "In this operation, first enter sales data into Excel cell A1," and then ends the recording.

[0120] Step 5:

[0121] The user uploads the recorded data to the server. The user sends the audio file saved in Audacity to the server using a dedicated upload form. The input is the recorded audio file, and the output is the audio file saved on the server. Specifically, the user opens a browser, accesses the upload form, selects the audio file, and completes the upload.

[0122] Step 6:

[0123] The server analyzes the audio file. The server converts the audio file into text using the Google Speech-to-Text API. The input is the uploaded audio file, and the output is text data. Specifically, the server loads the audio file, runs it through speech recognition, and extracts the text data.

[0124] Step 7:

[0125] The server integrates the video and audio information. The server then integrates the text data from the analyzed video frames with the text data obtained through speech recognition to construct a detailed workflow. The input is text and audio information, and the output is an integrated workflow. Specifically, the server analyzes the text information stored in the database and organizes it into operational steps to create a workflow.

[0126] Step 8:

[0127] The server generates the automation steps. The server generates a script based on the workflow to create an automatable step. The input is the detailed workflow, and the output is the automation step script. Specifically, the server uses a programming language (e.g., Python) to generate code to perform a specific operation.

[0128] Step 9:

[0129] The user checks and modifies the procedure. The user checks the automation procedure presented by the server and modifies it as necessary. The input is the automation procedure from the server, and the output is the modified procedure. Specifically, the user checks the procedure using an interface on the browser, deletes unnecessary steps, or inserts additional steps.

[0130] Step 10:

[0131] The server executes the procedure on a scheduled basis. The server executes the modified procedure based on the set schedule. The input is the modified automation procedure, and the output is the result of executing the automated operation. Specifically, the server executes the procedure at the set time, for example, opening an Excel file, processing the data, and sending the results by email.

[0132] (Application example 1)

[0133] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] Logistics centers have many routine tasks that need to be performed efficiently. In particular, routine tasks such as inventory checks, stocktaking, and picking require time and effort, and are prone to human error. Therefore, there is a need to automate these tasks to improve efficiency and accuracy.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0136] In this invention, the server includes means for recording screen operations, means for extracting information from the recorded video, means for analyzing the worker's audio commentary, means for building a workflow by integrating the extracted information and the analyzed audio information, means for generating procedures based on the workflow, means for the worker to adjust the generated procedures, means for scheduling and executing the adjusted procedures, means for visually recording the work procedures using a head-mounted display, means for extracting text data from the recorded video using optical character recognition technology, and means for converting the audio commentary into text using voice recognition technology, thereby enabling the automation and efficiency of routine tasks in a logistics center.

[0137] The "means for recording screen operations" refers to a means for recording the operations performed by the user on the screen, such as mouse clicks and keyboard input, as video.

[0138] "Means for extracting information from recorded video" refers to means for extracting necessary data or text information from recorded video.

[0139] The "means for analyzing the voice commentary of the worker" is a means for analyzing the voice recorded by the worker and acquiring the content as text data.

[0140] The "means for integrating the extracted information and analyzed voice information to construct a workflow" is a means for combining information extracted from the video with the voice recognition results to create a workflow that details the steps of the work.

[0141] The "means for generating a procedure based on the workflow" refers to a means for generating an automatable procedure based on the constructed workflow.

[0142] The "means for the worker to adjust the generated procedure" refers to a means for the user to check the generated procedure, delete unnecessary steps, and add additional comments.

[0143] The "means for scheduling and executing the adjusted procedure" is a means for automatically executing the adjusted procedure at a time designated by the user.

[0144] "Means for visually recording work procedures using a head-mounted display" refers to a means in which a worker wears a head-mounted display and records the work procedures from that viewpoint.

[0145] The "means for extracting text data from recorded video using optical character recognition technology" refers to a means for extracting character information from recorded video frames using OCR technology and obtaining it as text data.

[0146] "Means for converting audio commentary into text using speech recognition technology" refers to means for analyzing recorded audio commentary using speech recognition technology and converting it into text data.

[0147] The present invention is a system for automating routine tasks in a logistics center. The system includes a means for recording screen operations, a means for extracting information from recorded video, a means for analyzing audio commentary by workers, a means for integrating the extracted information and the analyzed audio information to create a workflow, a means for workers to adjust the generated procedures, a means for scheduling and executing the adjusted procedures, a means for visually recording the work procedures using a head-mounted display, a means for extracting text data from the recorded video using OCR technology, and a means for converting the audio commentary into text using speech recognition technology.

[0148] Hardware and Software Configuration

[0149] Hardware

[0150] Head-mounted display (HMD): A device worn by a worker to record work procedures from their point of view.

[0151] Terminal (smartphone or tablet): A device for collecting video and audio data and uploading it to a server.

[0152] Server: A computer with powerful processing power that analyzes information and generates procedures.

[0153] software

[0154] OCR Technology: PaddleOCR is used to extract text data from recorded video frames.

[0155] Speech Recognition Technology: Technology for converting recorded audio commentary into text data using the SpeechRecognition library.

[0156] Scheduling library: A system for executing automated procedures at user-specified times using the schedule library.

[0157] Operation overview

[0158] First, the user puts on the head-mounted display and records the procedure while performing routine tasks at a logistics center (e.g., inventory checks, stocktaking, picking). Once the recording is complete, an audio commentary on the procedure is recorded. This data is then uploaded to the server via the device.

[0159] The server first extracts the necessary information from the video data using OCR technology. In parallel, it converts the recorded audio commentary into text data using speech recognition technology. It then integrates the extracted information with the commentary text to construct a detailed workflow. Based on the constructed workflow, automatable steps are generated and presented to the user.

[0160] The user can review the presented procedures and adjust them by deleting unnecessary steps or adding comments. The adjusted procedures are then saved back to the server. Finally, the server automatically executes the adjusted procedures based on a schedule. For example, an inventory check can be automatically performed every morning at 9:00 and the results sent to the person in charge via email.

[0161] Examples of concrete examples and prompts

[0162] Specific examples

[0163] This process involves automatically checking inventory every morning at 9:00 and sending the results to the person in charge via email. The person wearing the head-mounted display records their viewpoint, and the data with recorded audio commentary is uploaded to the server. The server analyzes this data, generates and adjusts the automated procedures, and executes them at the specified time.

[0164] Prompt Sentence Examples

[0165] Example: Write a Python program to automate inventory checks at a distribution center every morning at 9:00. The program should include video recording, voice recognition, OCR analysis, and schedule execution.

[0166] Note: A prompt statement that automates logistics center operations based on the system described in the invention.

[0167] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0168] Step 1:

[0169] Data recording

[0170] A user wears a head-mounted display and performs routine tasks in a logistics center. The head-mounted display records the work procedure on video. The user also records audio commentary about the work procedure. This allows visual and audio information to be collected.

[0171] Input: User instructions and audio description

[0172] Output: Video and audio files

[0173] Step 2:

[0174] Uploading data

[0175] The device uploads the recorded video and audio files to a server, an automated operation that is performed through the device's interface.

[0176] Input: Video and audio files

[0177] Output: Video and audio data on the server

[0178] Step 3:

[0179] Data extraction with OCR

[0180] The server analyzes the uploaded video data and extracts the necessary information using optical character recognition (OCR) technology. Specifically, it identifies and extracts text data from video frames. PaddleOCR is used.

[0181] Input: Video data on the server

[0182] Output: Extracted text data

[0183] Step 4:

[0184] Data analysis using voice recognition

[0185] The server analyzes the uploaded audio data and converts the audio commentary into text data using speech recognition technology, primarily using the SpeechRecognition library.

[0186] Input: Audio data on the server

[0187] Output: Converted text data

[0188] Step 5:

[0189] Integrating information and building workflows

[0190] The server integrates the text data obtained from OCR and speech recognition to construct a detailed workflow, which is based on the user's specific operating steps and audio commentary.

[0191] Input: Extracted and converted text data

[0192] Output: The completed workflow

[0193] Step 6:

[0194] Generating automation procedures

[0195] The server generates automatable procedures based on the constructed workflow, and presents the procedures to the user as a list of steps.

[0196] Input: Completed Workflow

[0197] Output: Generated list of automation steps

[0198] Step 7:

[0199] Adjusting procedures

[0200] The user can review the generated automated procedure and modify it as needed, for example by deleting unnecessary steps or adding additional comments. After this adjustment, the procedure is saved back to the server.

[0201] Input: Generated list of automated procedures

[0202] Output: Modified list of automated steps

[0203] Step 8:

[0204] Schedule execution

[0205] The server schedules and executes the adjusted procedures at the specified time. For example, you can automatically execute an inventory check at 9:00 AM every day and send the results to the person in charge by email.

[0206] Input: Modified automated procedure list

[0207] Output: The results of the automated procedure that was executed (e.g., an inventory checklist)

[0208] Examples of concrete examples and prompts

[0209] Specific examples

[0210] This process involves automatically checking inventory every morning at 9:00 and sending the results to the person in charge via email. The person wearing the head-mounted display records their viewpoint, and the data with recorded audio commentary is uploaded to the server. The server analyzes this data, generates and adjusts the automated procedures, and executes them at the specified time.

[0211] Prompt Sentence Examples

[0212] Example: Write a Python program to automate inventory checks at a distribution center every morning at 9:00. The program should include video recording, voice recognition, OCR analysis, and schedule execution.

[0213] Note: A prompt statement that automates logistics center operations based on the system described in the invention.

[0214] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0215] The present invention relates to a system for automating routine tasks, and also combines it with an emotion engine that recognizes and reflects the user's emotions. It can be implemented based on the following settings.

[0216] This system involves a terminal, a server, and a user. First, the user records routine tasks. The recording is performed on the terminal operated by the user, and screen operations are recorded. All actions performed on the terminal (e.g., mouse clicks, keyboard input, screen capture) are saved in video format.

[0217] The recorded video is uploaded to a server. The server is equipped with a function to extract information from the recorded video. Specifically, the server uses OCR (optical character recognition) technology to analyze the text and data in the video and extract the necessary information. This process identifies the specific operating steps performed by the user.

[0218] Next, the user records an audio commentary about the task. The recording is done using the device's microphone, and the resulting audio file is uploaded to the server. The server analyzes the audio file and obtains the user's explanation as text data. This analysis is performed using voice recognition technology.

[0219] In addition, the system uses an emotion engine to recognize the user's emotions. Emotional information is acquired by analyzing the user's facial expressions and tone of voice using the device's camera and microphone. This emotion engine can grasp the user's emotions in real time while they are operating the device.

[0220] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. This workflow concretely shows the sequential steps of the user's operations, and by incorporating emotional information, it provides a procedure that takes into account the user's stress and motivation.

[0221] Furthermore, a procedure that can be automated is generated based on this workflow. The server presents the generated procedure in list form and allows the user to review it. The user can review the presented procedure and make corrections as necessary. For example, they can delete unnecessary steps or add additional comments. The optimized procedure is then saved back to the server.

[0222] Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, a series of operations can be performed automatically, such as opening a specific Excel file at 9:00 every morning, processing the data, and sending the results by email. This scheduled execution function frees users from routine tasks, allowing them to focus on more creative work.

[0223] As a concrete example, the following scenario can be considered.

[0224] A user's daily work involves processing specific data in Excel and emailing the results to their boss. The user first records this series of operations. Next, they record an audio commentary about the work and upload it to the server. The server analyzes this data, generates automatable steps, and presents them to the user. Furthermore, if the emotion engine detects that the user's stress level is high, it suggests modifications to simplify some of the steps. After the user checks the content and makes modifications, the server schedules this procedure to be executed every morning at 9:00 AM. As a result, the user is freed from the hassle of manually processing and sending data every morning.

[0225] In this way, the system automates routine tasks, saving time and effort, and improves the quality of tasks by taking into account the user's emotions.

[0226] The processing flow will be explained below.

[0227] Step 1:

[0228] The user uses the device to record screen activity. The device uses screen capture software to record all user activity (e.g., mouse clicks, keyboard input) and saves it as a video file.

[0229] Step 2:

[0230] The user stops recording and uploads the recorded video file from the terminal to the server, and the terminal transfers the video file to the server via the network.

[0231] Step 3:

[0232] The server receives the uploaded video file and uses OCR technology to analyze the text information and operation details in the video, extracting the analyzed information as text data.

[0233] Step 4:

[0234] The user uses the device to record an audio commentary of the work they are doing, and the recorded audio file is saved on the device.

[0235] Step 5:

[0236] A user uploads an audio file from a terminal to a server, and the terminal transfers the audio file to the server via a network.

[0237] Step 6:

[0238] The server receives the uploaded audio file and converts the audio content into text data using voice recognition technology. The text data obtained through voice analysis accurately reflects the user's commentary.

[0239] Step 7:

[0240] The user uses the device's camera and microphone to collect emotional data while working. The device then analyzes the user's facial expressions and tone of voice in real time and obtains emotional information through an emotion engine.

[0241] Step 8:

[0242] The server integrates the video analysis data, audio analysis data, and emotional information to construct a detailed workflow, which reflects the user's operation steps and the corresponding audio commentary and emotional state.

[0243] Step 9:

[0244] The server generates automatable procedures based on the constructed workflow, and the generated procedures are organized in a list format and presented to the user.

[0245] Step 10:

[0246] The user can review the generated instructions and make corrections or adjustments as necessary. The user can then use their device to modify the content of the instructions, taking into account feedback from the emotion engine, to optimize them.

[0247] Step 11:

[0248] The modified and adjusted procedures are saved back to the server, which stores these procedures in a database for scheduled execution.

[0249] Step 12:

[0250] A user uses a terminal to schedule an automated procedure to run, for example, at a specific time each day.

[0251] Step 13:

[0252] At the set time, the server executes the automated procedures according to the schedule. The server executes each procedure in sequence and automatically performs the necessary operations (e.g., processing data in Excel, sending emails).

[0253] Example 2

[0254] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0255] Conventional automation systems for routine tasks can automate user work procedures, but they are unable to consider the user's emotions, which often results in inefficient automation. Furthermore, because they are unable to consider the user's stress or motivation, it is difficult to improve the quality of work.

[0256] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0257] In this invention, the server includes means for recording screen operations, means for extracting information from the recorded video, means for analyzing the worker's audio commentary, means for recognizing emotions, means for constructing a workflow by integrating the extracted information with the analyzed audio information and emotional information, means for generating a procedure based on the workflow, means for the worker to adjust the generated procedure, and means for scheduling and executing the adjusted procedure, thereby enabling the efficiency and quality of work performed by the user to be improved.

[0258] The "means for recording screen operations" refers to a device or program that records the operations performed by a user on the screen in video format.

[0259] "Means for extracting information from recorded video" refers to devices or programs that use OCR technology or other methods to analyze on-screen text and data from recorded video and obtain the necessary information.

[0260] The "means for analyzing the worker's audio commentary" refers to a device or program that converts the audio file recorded by the worker into text data using voice recognition technology and analyzes the content.

[0261] "Means for recognizing emotions" refers to a device or program that uses the device's camera or microphone to analyze the user's facial expressions and tone of voice and identify the user's emotional state.

[0262] The "means for constructing a workflow by integrating the extracted information with the analyzed audio information and emotional information" refers to a device or program that integrates the results of video analysis, audio analysis, and emotional analysis to organize the user's work procedures in chronological order and construct a detailed workflow.

[0263] The "means for generating a procedure based on the workflow" refers to a device or program that generates an automatable procedure based on the constructed workflow.

[0264] The "means for an operator to adjust the generated procedure" refers to an interface or device that allows a user to check the generated automated procedure and make corrections or additions as necessary.

[0265] The "means for scheduling and executing the adjusted procedure" refers to a device or program with a scheduling function that automatically executes the procedure modified by the user at a specified time.

[0266] The present invention relates to a system that automates routine tasks and recognizes and reflects the user's emotions. This system involves a terminal, a server, and a user. A specific embodiment of the system is shown below.

[0267] First, the user starts a routine task. The device launches screen recording software (e.g., OBS Studio) and records the user's screen operations in video format. Specific actions recorded include mouse clicks, keyboard input, and window switching. All on-screen actions are then saved as a video file (e.g., .mp4 format).

[0268] Next, the device uploads the recorded video file to the server. This can be done by dragging and dropping it from a file browser. The server receives the uploaded file and analyzes the text and data in the video using OCR (Optical Character Recognition) technology (e.g., Google Cloud Vision API). Specifically, it analyzes the video frames and extracts text and numerical data.

[0269] The user provides an audio description of the work they are doing. This audio description explains the steps and purpose of the work, and includes specific content such as "Enter sales data into this cell and click this button to create a graph." The device records the audio through a microphone and uploads the audio file (e.g., .wav format) to the server. The server receives the audio file and converts it into text data using voice recognition technology (e.g., Google Cloud Speech-to-Text). Specifically, the audio data is analyzed and corresponding text information is generated.

[0270] Furthermore, the device uses a camera and microphone to capture the user's facial expressions and tone of voice in real time. For example, it records the moment when the user smiles or the tone of their voice becomes lower. The server analyzes this data using an emotion engine (e.g., Microsoft® Azure Emotion API) to identify the user's emotional state. For example, it obtains information such as "the user is feeling stressed."

[0271] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. Specifically, it organizes the work content for each step and the corresponding emotional data in chronological order. This workflow specifically shows the sequential steps of the user's operations, and by incorporating emotional information, it provides a procedure that takes into account the user's stress and motivation.

[0272] The server then presents a list of automatable steps for the user to review. For example, the steps are displayed sequentially through a web interface. The user reviews the presented workflow and makes corrections as necessary. Specific actions include deleting unnecessary steps and adding new comments. The user can also review the generated steps and make corrections or additions, and the optimized steps are then saved back to the server.

[0273] Finally, the server schedules the revised procedure to run at a specified time. Specific actions include automatically executing a series of operations, such as "opening an Excel file at 9:00 every morning, processing specific data, and sending the results by email." This frees users from this routine work, allowing them to focus on more creative tasks.

[0274] Prompt Sentence Examples

[0275] "I want to automate the daily task of opening an Excel file, processing the data, and sending it to my boss by email. I've recorded the steps and provided audio explanations of the work. Please automate this procedure in the system and set it to run every morning at 9:00."

[0276] In this way, this system can automate routine tasks and significantly improve the efficiency and quality of work for users. Furthermore, by taking into account the user's emotions, it reduces stress and improves the comfort and effectiveness of work.

[0277] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0278] Step 1:

[0279] The user starts a routine task. The device launches the screen recording software, which records the user's screen operations in video format. Specifically, mouse clicks, keyboard input, window switching, etc. are recorded. The input is the user's screen operations, and the output is a video file (e.g., .mp4 format).

[0280] Step 2:

[0281] The device uploads the recorded video file to the server. The user uploads it by dragging and dropping it from the file browser. The input is the video file, and the output is a file saved on the server. Specifically, the user selects the video file and clicks the upload button.

[0282] Step 3:

[0283] The server receives the uploaded video file. It uses OCR technology to analyze the characters and data in the video and obtain the necessary information. The input is the video file, and the output is the analyzed text information. Specifically, OCR technology recognizes characters for each frame of the video and generates text data.

[0284] Step 4:

[0285] The user gives an audio commentary about the work they are doing. The device uses a microphone to record the audio and uploads the audio file (e.g., .wav format) to the server. The input is the user's voice and the output is an audio file. Specifically, the user presses the record button and explains the work they are doing.

[0286] Step 5:

[0287] The server receives the uploaded audio file and converts it into text data using speech recognition technology. The input is the audio file and the output is text data. Specifically, the audio data is analyzed and the corresponding text information is generated.

[0288] Step 6:

[0289] The device uses a camera and microphone to capture the user's facial expressions and tone of voice in real time. The server uses an emotion engine to analyze this data and identify the user's emotional state. The input is the user's facial expressions and tone of voice, and the output is data on the user's emotional state. Specifically, it uses facial expression recognition algorithms and voice tone analysis technology.

[0290] Step 7:

[0291] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. The input is the analyzed data, and the output is the integrated workflow. Specifically, the work content for each step and the corresponding emotional data are organized in chronological order.

[0292] Step 8:

[0293] The server presents a list of automatable steps to the user for review. The input is the workflow, and the output is a list of steps presented to the user, with each step displayed sequentially through a web interface.

[0294] Step 9:

[0295] The user checks the presented workflow and makes corrections as necessary. The input is a list of steps, and the output is the corrected steps. Specifically, the user deletes unnecessary steps and adds new comments.

[0296] Step 10:

[0297] The server re-saves the modified procedure. The input is the modified procedure, and the output is the procedure data saved on the server. Specifically, the procedure data that reflects the user's modification information is saved on the server.

[0298] Step 11:

[0299] The server schedules the execution of the modified procedure at a specified time. The input is the modified procedure data, and the output is the scheduled execution of the work. Specifically, for example, a series of operations such as "open an Excel file every morning at 9:00, process specific data, and send the results by email" is automatically executed.

[0300] In this way, the system can automate routine tasks, improve efficiency and work quality, and provide a flexible work environment that takes users' emotions into consideration.

[0301] (Application example 2)

[0302] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0303] In logistics centers, employees perform a large number of routine tasks, which not only consume a significant amount of time and effort, but also cause stress and fatigue in workers. Furthermore, because employees' emotional states have a direct impact on work efficiency and quality, it is important to develop an optimized workflow that takes into account employees' emotions. Therefore, there is a need to automate routine tasks while providing a workflow that reflects the emotions of workers.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording screen operations, means for extracting information from recorded video, means for analyzing the worker's audio commentary, means for building a workflow by integrating the extracted information and the analyzed audio information, means for generating procedures based on the workflow, means for the worker to adjust the generated procedures, means for scheduling and executing the adjusted procedures, and means for analyzing the worker's emotions and reflecting them in the workflow. This makes it possible to efficiently automate employees' routine tasks and provide an optimal workflow that takes employees' emotions into consideration.

[0305] The "means for recording screen operations" refers to a device or software that records the screen operations performed by the worker in video format.

[0306] "Means for extracting information from recorded video" refers to technology or equipment that has the ability to analyze and extract useful text or data from recorded video.

[0307] The "means for analyzing the worker's voice commentary" is a voice recognition technology that converts the worker's voice commentary into text data.

[0308] The "means for constructing a workflow by integrating the extracted information and analyzed audio information" is an algorithm that integrates video and audio data to generate detailed work procedures.

[0309] The "means for generating a procedure based on the workflow" is a system that generates an automatable procedure based on the constructed workflow.

[0310] "Means for operators to adjust the generated procedures" refers to interfaces and tools that allow operators to check and correct the automated procedures.

[0311] The "means for scheduling and executing the adjusted procedure" is a function for automatically executing the procedure adjusted by the worker at a specified time.

[0312] "Means for analyzing workers' emotions and reflecting them in the workflow" refers to algorithms and technologies that recognize and analyze the emotional state of workers in real time and reflect the results in the workflow.

[0313] The present invention relates to an automated system that improves the efficiency of routine work in a logistics center and takes into account the emotions of workers. This system is implemented in the following manner.

[0314] First, capture software and a webcam are used to record terminal operations. While workers perform routine tasks (e.g., product picking and packing) at the distribution center on the terminal screen, the operations are recorded in video format. This recorded video is then uploaded to a server.

[0315] The server uses OCR (optical character recognition) technology (such as pytesseract) to extract text and data from the recorded video. The server also uses speech recognition technology (such as Google Speech Recognition) to convert the operator's audio commentary into text. This allows the operation procedures and their explanations to be obtained as data.

[0316] The server then uses an emotion engine to analyze the worker's emotions. This process involves using the device's camera and microphone to analyze the worker's facial expressions and tone of voice in real time. The emotion engine uses technology to extract features from voice data and classify emotions using an SVM model.

[0317] The server integrates the analyzed video, audio, and emotional information to construct a workflow. This workflow concretely shows the sequential steps of the user's operations and incorporates emotional information to provide a procedure that takes into account the user's stress and motivation.

[0318] Based on this workflow, a procedure that can be automated is generated. The server presents the generated procedure in list form to the worker, who can then modify it as needed. For example, it is possible to delete unnecessary steps or add additional comments. Once the procedure has been modified, it is saved on the server and automatically executed at the scheduled time.

[0319] As a concrete example, consider a task in which a worker processes specific data every day and reports the results. The worker first records this series of operations, and then records an audio commentary of the operations. The video and audio are uploaded to a server for analysis. If the emotion engine detects that the worker is in a high stress state, it suggests simplifying some of the steps. After the worker reviews and makes corrections, the procedure is set to be executed automatically every morning at 9:00.

[0320] Below is an example of a prompt sentence to input to the generative AI model.

[0321] Example prompt for a generative AI model:

[0322] You are designing a system to improve the work efficiency of employees at a distribution center. Code the following steps in Python:

[0323] 1. Record and save the work steps.

[0324] 2. Record and save an audio description of your work.

[0325] 3. Extract text data from the recorded video using OCR.

[0326] 4. Convert the recorded audio into text using speech recognition.

[0327] 5. Analyze emotional information using an emotion engine and optimize work procedures accordingly.

[0328] 6. Set up a schedule to automatically execute tasks at specific times.

[0329] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0330] Step 1:

[0331] The routine tasks performed by users at the logistics center are recorded using capture software and a webcam on the terminal. Specifically, the user performs work steps on the terminal, and the operations are saved in video format. This recorded data becomes the input.

[0332] Step 2:

[0333] The recorded video data is uploaded to a server, where the server uses OCR technology (such as pytesseract) to extract text and data from the video. The input is the recorded data, and the output is a list of text and data. This data extraction identifies specific operating procedures and tool usage.

[0334] Step 3:

[0335] The user uses the microphone on their device to record an audio commentary of their work. The recorded audio file is uploaded to the server. This audio file is used as input and converted into text data using speech recognition technology (such as Google Speech Recognition). The output is the text data of the audio commentary.

[0336] Step 4:

[0337] The server uses an emotion engine to analyze the user's emotions. Features are extracted from the user's facial expression data and tone of voice data acquired from the device's camera and microphone. These features are then fed into an SVM model to classify emotions. The output is the user's emotional information.

[0338] Step 5:

[0339] The server integrates the information extracted in step 2, the text data of the audio commentary obtained in step 3, and the emotional information analyzed in step 4 to construct a workflow. Based on this integrated data, a detailed workflow is created that incorporates the specific operational steps performed by the user and their emotional information. The output is the integrated workflow.

[0340] Step 6:

[0341] The server generates automatable steps based on the constructed workflow. The input is the integrated workflow, and the output is a list of automatable steps, which lists tasks to be performed at specific times.

[0342] Step 7:

[0343] The generated procedure list is presented to the user, who then checks and modifies it. At this stage, necessary adjustments are made, such as deleting unnecessary steps or adding additional comments. The input is the pre-generated procedure list, and the output is the procedure list modified by the user.

[0344] Step 8:

[0345] The server executes the procedures modified by the user based on a schedule. This procedure list is registered in the schedule and automatically executed at the specified time. The input is the modified procedure list and schedule information, and the output is the execution result of the automatically executed task.

[0346] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0347] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0348] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0349] [Second embodiment]

[0350] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0351] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0352] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0353] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0354] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0355] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0356] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0357] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0358] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0359] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0360] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0361] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0362] The present invention relates to a system for automating routine tasks, and can be implemented based on the following settings.

[0363] This system involves a terminal, a server, and a user. First, the user records routine tasks. The recording is performed on the terminal operated by the user, and screen operations are recorded. All actions performed on the terminal (e.g., mouse clicks, keyboard input, screen capture) are saved in video format.

[0364] The recorded video is uploaded to a server. The server is equipped with a function to extract information from the recorded video. Specifically, the server uses OCR (optical character recognition) technology to analyze the text and data in the video and extract the necessary information. This process identifies the specific operating steps performed by the user.

[0365] Next, the user records an audio commentary about the task. The recording is done using the device's microphone, and the resulting audio file is uploaded to the server. The server analyzes the audio file and obtains the user's explanation as text data. This analysis is performed using voice recognition technology.

[0366] The server integrates the analyzed video and audio information to construct a detailed workflow. This workflow concretely shows the sequential steps of the user's operations. Based on this workflow, a procedure that can be automated is then generated. The server presents the generated procedure in list form and allows the user to confirm it.

[0367] The user can review the proposed procedure and make any necessary modifications, such as deleting unnecessary steps or adding additional comments. The optimized procedure is then saved back to the server.

[0368] Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, a series of operations can be performed automatically, such as opening a specific Excel file at 9:00 every morning, processing the data, and sending the results by email. This scheduled execution function frees users from routine tasks, allowing them to focus on more creative work.

[0369] As a concrete example, the following scenario can be considered.

[0370] A user's daily task is to process specific data in Excel and send the results to their boss by email. The user first records this series of operations. Next, they record an audio commentary about the work and upload it to the server. The server analyzes this data, generates an automatable procedure, and presents it to the user. After the user checks and modifies the procedure, the server schedules the procedure to be executed every morning at 9:00 AM. As a result, the user is freed from the hassle of manually processing and sending data every morning.

[0371] In this way, the system automates routine tasks, saving time and effort while also increasing the accuracy of the work.

[0372] The processing flow will be explained below.

[0373] Step 1:

[0374] The user uses the device to record screen activity. The device uses screen capture software to record all user activity (e.g., mouse clicks, keyboard input) and saves it as a video file.

[0375] Step 2:

[0376] The user stops recording and uploads the recorded video file from the terminal to the server, and the terminal transfers the video file to the server via the network.

[0377] Step 3:

[0378] The server receives the uploaded video file and uses OCR technology to analyze the text information and operation details in the video, extracting the analyzed information as text data.

[0379] Step 4:

[0380] The user uses the device to record an audio commentary of the work they are doing, and the recorded audio file is saved on the device.

[0381] Step 5:

[0382] A user uploads an audio file from a terminal to a server, and the terminal transfers the audio file to the server via a network.

[0383] Step 6:

[0384] The server receives the uploaded audio file and converts the audio content into text data using voice recognition technology. The text data obtained through voice analysis accurately reflects the user's commentary.

[0385] Step 7:

[0386] The server integrates the video and audio analysis data to construct a detailed workflow, which combines the user's operational steps with corresponding audio commentary.

[0387] Step 8:

[0388] The server generates automatable procedures based on the constructed workflow, and the generated procedures are organized in a list format and presented to the user.

[0389] Step 9:

[0390] The user checks the generated procedure and makes corrections or adjustments as necessary. The user uses the terminal to change the content of the procedure and arrange it in the optimal form.

[0391] Step 10:

[0392] The modified and adjusted procedures are saved back to the server, which stores these procedures in a database for scheduled execution.

[0393] Step 11:

[0394] A user uses a terminal to schedule an automated procedure to run, for example, at a specific time each day.

[0395] Step 12:

[0396] At the set time, the server executes the automated procedures according to the schedule. The server executes each procedure in sequence and automatically performs the necessary operations (e.g., processing data in Excel, sending emails).

[0397] Example 1

[0398] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0399] In modern business activities, many routine tasks occur on a daily basis. These tasks require time and effort, and there are issues with efficiency and accuracy. Conventional automation systems have difficulty automating a series of operations at once, and human intervention is required, especially for complex business processes. Therefore, reducing labor costs and improving work accuracy have become important issues.

[0400] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0401] In this invention, the server includes means for a user to record work on a terminal, means for uploading the recorded data to the server, means for the server to analyze the video, means for the user to record an audio commentary on the work content, means for uploading the recorded data to the server, means for the server to analyze the audio file, means for the server to integrate the video information and the audio information, means for the server to generate an automation procedure, means for the user to check and modify the procedure, and means for the server to schedule and execute the procedure. This makes it possible to automate routine business processes efficiently and with high accuracy.

[0402] A "user" is an operator who uses the system to automate routine tasks.

[0403] A "terminal" is a hardware device that a user uses to perform work, and includes a personal computer, a smartphone, etc.

[0404] "Recording" is the process of recording operations performed on a terminal in video format.

[0405] A "server" is a computer or cloud service that is the core of a system, and is a device that analyzes data and generates and executes procedures.

[0406] "Uploading" is the process of sending files or data generated on a device to a server.

[0407] "Video analysis" is the process of extracting text information and operating procedures from recorded video using OCR technology.

[0408] "Audio commentary" is information in which the user verbally explains the content of the work.

[0409] An "audio file" is a file that stores audio data recorded by a user.

[0410] "Speech analysis" is the process of obtaining text information from an audio file using speech recognition technology.

[0411] A "workflow" is information that indicates the sequential steps of a user's operation procedure, and is constructed by integrating video information and audio information.

[0412] An "automation procedure" is a procedure script created based on a workflow to automatically execute a series of operations.

[0413] "Procedure verification and correction" is the process in which the user verifies the automated procedures presented by the server and makes any necessary corrections.

[0414] "Scheduled execution" is the process of automatically executing a procedure at a specified time.

[0415] The present invention relates to a system for automating routine tasks, and involves a user, a terminal, and a server.

[0416] The system uses the following hardware and software:

[0417] Terminal: A device operated by a user, such as a computer or smartphone.

[0418] Servers: High-performance computers, data centers, and cloud services (e.g., AWS, Azure).

[0419] OCR technology: Software for extracting text data (e.g., Google Vision API).

[0420] Speech recognition technology: Software that converts speech into text (e.g., Google Speech-to-Text API).

[0421] Recording software: A recording tool that runs on your device (e.g., OBS Studio).

[0422] Audio recording software: A tool to record audio from your microphone input (e.g., Audacity).

[0423] As a specific example of operation, the process below shows how to automate a task in which a user processes specific data in Excel every day and sends the results to their supervisor by email.

[0424] Operating Procedure

[0425] 1. The user launches OBS Studio on their device and clicks the "Record" button to begin recording their work. All on-screen actions (e.g., opening Excel and entering data into specific cells) will be recorded.

[0426] 2. Once recording is complete, download the recording file from the OBS Studio file menu and upload it to the server using the dedicated upload form.

[0427] 3. The server uses the Google Vision API to analyze the uploaded video file and extract the cell contents in Excel and the user interface operation procedures as text data.

[0428] 4. Next, the user uses Audacity to record an audio description of the task, generating an audio file containing instructions such as, "To perform this task, first enter sales data into Excel cell A1."

[0429] 5. After recording is complete, the audio file is uploaded to the server, which uses the Google Speech-to-Text API to extract the text from the audio file.

[0430] 6. The server combines the information extracted from the video file and the information extracted from the audio file to construct a detailed workflow, which shows the steps taken by the user.

[0431] 7. The server generates automated procedures based on the workflow and presents them to the user in list form.

[0432] 8. The user reviews the proposed steps and modifies them as necessary, for example, by inserting additional steps or deleting unnecessary steps.

[0433] 9. Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, an Excel file is opened at 9:00 every morning, the data is processed, and the results are sent by email.

[0434] Specific examples

[0435] If a user wants to automate a daily task of processing certain data in Excel and emailing the results to their boss, they can use a prompt like this:

[0436] "I want an automated task that opens new sales data in Excel every morning, processes it into a specific format, and emails the results to my boss."

[0437] In this way, the system efficiently automates routine tasks, freeing users from routine tasks, saving time and effort, and improving work accuracy.

[0438] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0439] Step 1:

[0440] A user records their work on a device. To do this, the user launches recording software (e.g., OBS Studio) and clicks the "Record" button to begin recording their work. The input is the user's screen operations, and the output is a recorded video file. This video file records the user's operations (e.g., opening Excel, selecting cells, entering data, etc.).

[0441] Step 2:

[0442] The user uploads the recorded data to the server. The user retrieves the file from OBS Studio and sends the data to the server using a dedicated upload form. The input is the recorded video file, and the output is a video file saved on the server. Specifically, the user opens a browser, accesses the upload form, selects the recorded file, and completes the upload.

[0443] Step 3:

[0444] The server analyzes the video. It uses the Google Vision API to analyze the video file frame by frame and extracts text information from the video and user interface operation procedures. The input is the uploaded video file, and the output is text information and operation procedures. Specifically, the server loads the video file and runs OCR on each frame to extract the text data.

[0445] Step 4:

[0446] A user records an audio description of a task. The user launches Audacity and clicks the "Record" button to begin recording. The input is the user's verbal description, and the output is an audio file. Specifically, the user explains into the microphone, for example, "In this operation, first enter sales data into Excel cell A1," and then ends the recording.

[0447] Step 5:

[0448] The user uploads the recorded data to the server. The user sends the audio file saved in Audacity to the server using a dedicated upload form. The input is the recorded audio file, and the output is the audio file saved on the server. Specifically, the user opens a browser, accesses the upload form, selects the audio file, and completes the upload.

[0449] Step 6:

[0450] The server analyzes the audio file. The server converts the audio file into text using the Google Speech-to-Text API. The input is the uploaded audio file, and the output is text data. Specifically, the server loads the audio file, runs it through speech recognition, and extracts the text data.

[0451] Step 7:

[0452] The server integrates the video and audio information. The server then integrates the text data from the analyzed video frames with the text data obtained through speech recognition to construct a detailed workflow. The input is text and audio information, and the output is an integrated workflow. Specifically, the server analyzes the text information stored in the database and organizes it into operational steps to create a workflow.

[0453] Step 8:

[0454] The server generates the automation steps. The server generates a script based on the workflow to create an automatable step. The input is the detailed workflow, and the output is the automation step script. Specifically, the server uses a programming language (e.g., Python) to generate code to perform a specific operation.

[0455] Step 9:

[0456] The user checks and modifies the procedure. The user checks the automation procedure presented by the server and modifies it as necessary. The input is the automation procedure from the server, and the output is the modified procedure. Specifically, the user checks the procedure using an interface on the browser, deletes unnecessary steps, or inserts additional steps.

[0457] Step 10:

[0458] The server executes the procedure on a scheduled basis. The server executes the modified procedure based on the set schedule. The input is the modified automation procedure, and the output is the result of executing the automated operation. Specifically, the server executes the procedure at the set time, for example, opening an Excel file, processing the data, and sending the results by email.

[0459] (Application example 1)

[0460] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0461] Logistics centers have many routine tasks that need to be performed efficiently. In particular, routine tasks such as inventory checks, stocktaking, and picking require time and effort, and are prone to human error. Therefore, there is a need to automate these tasks to improve efficiency and accuracy.

[0462] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0463] In this invention, the server includes means for recording screen operations, means for extracting information from the recorded video, means for analyzing the worker's audio commentary, means for building a workflow by integrating the extracted information and the analyzed audio information, means for generating procedures based on the workflow, means for the worker to adjust the generated procedures, means for scheduling and executing the adjusted procedures, means for visually recording the work procedures using a head-mounted display, means for extracting text data from the recorded video using optical character recognition technology, and means for converting the audio commentary into text using voice recognition technology, thereby enabling the automation and efficiency of routine tasks in a logistics center.

[0464] The "means for recording screen operations" refers to a means for recording the operations performed by the user on the screen, such as mouse clicks and keyboard input, as video.

[0465] "Means for extracting information from recorded video" refers to means for extracting necessary data or text information from recorded video.

[0466] The "means for analyzing the voice commentary of the worker" is a means for analyzing the voice recorded by the worker and acquiring the content as text data.

[0467] The "means for integrating the extracted information and analyzed voice information to construct a workflow" is a means for combining information extracted from the video with the voice recognition results to create a workflow that details the steps of the work.

[0468] The "means for generating a procedure based on the workflow" refers to a means for generating an automatable procedure based on the constructed workflow.

[0469] The "means for the worker to adjust the generated procedure" refers to a means for the user to check the generated procedure, delete unnecessary steps, and add additional comments.

[0470] The "means for scheduling and executing the adjusted procedure" is a means for automatically executing the adjusted procedure at a time designated by the user.

[0471] "Means for visually recording work procedures using a head-mounted display" refers to a means in which a worker wears a head-mounted display and records the work procedures from that viewpoint.

[0472] The "means for extracting text data from recorded video using optical character recognition technology" refers to a means for extracting character information from recorded video frames using OCR technology and obtaining it as text data.

[0473] "Means for converting audio commentary into text using speech recognition technology" refers to means for analyzing recorded audio commentary using speech recognition technology and converting it into text data.

[0474] The present invention is a system for automating routine tasks in a logistics center. The system includes a means for recording screen operations, a means for extracting information from recorded video, a means for analyzing audio commentary by workers, a means for integrating the extracted information and the analyzed audio information to create a workflow, a means for workers to adjust the generated procedures, a means for scheduling and executing the adjusted procedures, a means for visually recording the work procedures using a head-mounted display, a means for extracting text data from the recorded video using OCR technology, and a means for converting the audio commentary into text using speech recognition technology.

[0475] Hardware and Software Configuration

[0476] Hardware

[0477] Head-mounted display (HMD): A device worn by a worker to record work procedures from their point of view.

[0478] Terminal (smartphone or tablet): A device for collecting video and audio data and uploading it to a server.

[0479] Server: A computer with powerful processing power that analyzes information and generates procedures.

[0480] software

[0481] OCR Technology: PaddleOCR is used to extract text data from recorded video frames.

[0482] Speech Recognition Technology: Technology for converting recorded audio commentary into text data using the SpeechRecognition library.

[0483] Scheduling library: A system for executing automated procedures at user-specified times using the schedule library.

[0484] Operation overview

[0485] First, the user puts on the head-mounted display and records the procedure while performing routine tasks at a logistics center (e.g., inventory checks, stocktaking, picking). Once the recording is complete, an audio commentary on the procedure is recorded. This data is then uploaded to the server via the device.

[0486] The server first extracts the necessary information from the video data using OCR technology. In parallel, it converts the recorded audio commentary into text data using speech recognition technology. It then integrates the extracted information with the commentary text to construct a detailed workflow. Based on the constructed workflow, automatable steps are generated and presented to the user.

[0487] The user can review the presented procedures and adjust them by deleting unnecessary steps or adding comments. The adjusted procedures are then saved back to the server. Finally, the server automatically executes the adjusted procedures based on a schedule. For example, an inventory check can be automatically performed every morning at 9:00 and the results sent to the person in charge via email.

[0488] Examples of concrete examples and prompts

[0489] Specific examples

[0490] This process involves automatically checking inventory every morning at 9:00 and sending the results to the person in charge via email. The person wearing the head-mounted display records their viewpoint, and the data with recorded audio commentary is uploaded to the server. The server analyzes this data, generates and adjusts the automated procedures, and executes them at the specified time.

[0491] Prompt Sentence Examples

[0492] Example: Write a Python program to automate inventory checks at a distribution center every morning at 9:00. The program should include video recording, voice recognition, OCR analysis, and schedule execution.

[0493] Note: A prompt statement that automates logistics center operations based on the system described in the invention.

[0494] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0495] Step 1:

[0496] Data recording

[0497] A user wears a head-mounted display and performs routine tasks in a logistics center. The head-mounted display records the work procedure on video. The user also records audio commentary about the work procedure. This allows visual and audio information to be collected.

[0498] Input: User instructions and audio description

[0499] Output: Video and audio files

[0500] Step 2:

[0501] Uploading data

[0502] The device uploads the recorded video and audio files to a server, an automated operation that is performed through the device's interface.

[0503] Input: Video and audio files

[0504] Output: Video and audio data on the server

[0505] Step 3:

[0506] Data extraction with OCR

[0507] The server analyzes the uploaded video data and extracts the necessary information using optical character recognition (OCR) technology. Specifically, it identifies and extracts text data from video frames. PaddleOCR is used.

[0508] Input: Video data on the server

[0509] Output: Extracted text data

[0510] Step 4:

[0511] Data analysis using voice recognition

[0512] The server analyzes the uploaded audio data and converts the audio commentary into text data using speech recognition technology, primarily using the SpeechRecognition library.

[0513] Input: Audio data on the server

[0514] Output: Converted text data

[0515] Step 5:

[0516] Integrating information and building workflows

[0517] The server integrates the text data obtained from OCR and speech recognition to construct a detailed workflow, which is based on the user's specific operating steps and audio commentary.

[0518] Input: Extracted and converted text data

[0519] Output: The completed workflow

[0520] Step 6:

[0521] Generating automation procedures

[0522] The server generates automatable procedures based on the constructed workflow, and presents the procedures to the user as a list of steps.

[0523] Input: Completed Workflow

[0524] Output: Generated list of automation steps

[0525] Step 7:

[0526] Adjusting procedures

[0527] The user can review the generated automated procedure and modify it as needed, for example by deleting unnecessary steps or adding additional comments. After this adjustment, the procedure is saved back to the server.

[0528] Input: Generated list of automated procedures

[0529] Output: Modified list of automated steps

[0530] Step 8:

[0531] Schedule execution

[0532] The server schedules and executes the adjusted procedures at the specified time. For example, you can automatically execute an inventory check at 9:00 AM every day and send the results to the person in charge by email.

[0533] Input: Modified automated procedure list

[0534] Output: The results of the automated procedure that was executed (e.g., an inventory checklist)

[0535] Examples of concrete examples and prompts

[0536] Specific examples

[0537] This process involves automatically checking inventory every morning at 9:00 and sending the results to the person in charge via email. The person wearing the head-mounted display records their viewpoint, and the data with recorded audio commentary is uploaded to the server. The server analyzes this data, generates and adjusts the automated procedures, and executes them at the specified time.

[0538] Prompt Sentence Examples

[0539] Example: Write a Python program to automate inventory checks at a distribution center every morning at 9:00. The program should include video recording, voice recognition, OCR analysis, and schedule execution.

[0540] Note: A prompt statement that automates logistics center operations based on the system described in the invention.

[0541] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0542] The present invention relates to a system for automating routine tasks, and also combines it with an emotion engine that recognizes and reflects the user's emotions. It can be implemented based on the following settings.

[0543] This system involves a terminal, a server, and a user. First, the user records routine tasks. The recording is performed on the terminal operated by the user, and screen operations are recorded. All actions performed on the terminal (e.g., mouse clicks, keyboard input, screen capture) are saved in video format.

[0544] The recorded video is uploaded to a server. The server is equipped with a function to extract information from the recorded video. Specifically, the server uses OCR (optical character recognition) technology to analyze the text and data in the video and extract the necessary information. This process identifies the specific operating steps performed by the user.

[0545] Next, the user records an audio commentary about the task. The recording is done using the device's microphone, and the resulting audio file is uploaded to the server. The server analyzes the audio file and obtains the user's explanation as text data. This analysis is performed using voice recognition technology.

[0546] In addition, the system uses an emotion engine to recognize the user's emotions. Emotional information is acquired by analyzing the user's facial expressions and tone of voice using the device's camera and microphone. This emotion engine can grasp the user's emotions in real time while they are operating the device.

[0547] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. This workflow concretely shows the sequential steps of the user's operations, and by incorporating emotional information, it provides a procedure that takes into account the user's stress and motivation.

[0548] Furthermore, a procedure that can be automated is generated based on this workflow. The server presents the generated procedure in list form and allows the user to review it. The user can review the presented procedure and make corrections as necessary. For example, they can delete unnecessary steps or add additional comments. The optimized procedure is then saved back to the server.

[0549] Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, a series of operations can be performed automatically, such as opening a specific Excel file at 9:00 every morning, processing the data, and sending the results by email. This scheduled execution function frees users from routine tasks, allowing them to focus on more creative work.

[0550] As a concrete example, the following scenario can be considered.

[0551] A user's daily work involves processing specific data in Excel and emailing the results to their boss. The user first records this series of operations. Next, they record an audio commentary about the work and upload it to the server. The server analyzes this data, generates automatable steps, and presents them to the user. Furthermore, if the emotion engine detects that the user's stress level is high, it suggests modifications to simplify some of the steps. After the user checks the content and makes modifications, the server schedules this procedure to be executed every morning at 9:00 AM. As a result, the user is freed from the hassle of manually processing and sending data every morning.

[0552] In this way, the system automates routine tasks, saving time and effort, and improves the quality of tasks by taking into account the user's emotions.

[0553] The processing flow will be explained below.

[0554] Step 1:

[0555] The user uses the device to record screen activity. The device uses screen capture software to record all user activity (e.g., mouse clicks, keyboard input) and saves it as a video file.

[0556] Step 2:

[0557] The user stops recording and uploads the recorded video file from the terminal to the server, and the terminal transfers the video file to the server via the network.

[0558] Step 3:

[0559] The server receives the uploaded video file and uses OCR technology to analyze the text information and operation details in the video, extracting the analyzed information as text data.

[0560] Step 4:

[0561] The user uses the device to record an audio commentary of the work they are doing, and the recorded audio file is saved on the device.

[0562] Step 5:

[0563] A user uploads an audio file from a terminal to a server, and the terminal transfers the audio file to the server via a network.

[0564] Step 6:

[0565] The server receives the uploaded audio file and converts the audio content into text data using voice recognition technology. The text data obtained through voice analysis accurately reflects the user's commentary.

[0566] Step 7:

[0567] The user uses the device's camera and microphone to collect emotional data while working. The device then analyzes the user's facial expressions and tone of voice in real time and obtains emotional information through an emotion engine.

[0568] Step 8:

[0569] The server integrates the video analysis data, audio analysis data, and emotional information to construct a detailed workflow, which reflects the user's operation steps and the corresponding audio commentary and emotional state.

[0570] Step 9:

[0571] The server generates automatable procedures based on the constructed workflow, and the generated procedures are organized in a list format and presented to the user.

[0572] Step 10:

[0573] The user can review the generated instructions and make corrections or adjustments as necessary. The user can then use their device to modify the content of the instructions, taking into account feedback from the emotion engine, to optimize them.

[0574] Step 11:

[0575] The modified and adjusted procedures are saved back to the server, which stores these procedures in a database for scheduled execution.

[0576] Step 12:

[0577] A user uses a terminal to schedule an automated procedure to run, for example, at a specific time each day.

[0578] Step 13:

[0579] At the set time, the server executes the automated procedures according to the schedule. The server executes each procedure in sequence and automatically performs the necessary operations (e.g., processing data in Excel, sending emails).

[0580] Example 2

[0581] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0582] Conventional automation systems for routine tasks can automate user work procedures, but they are unable to consider the user's emotions, which often results in inefficient automation. Furthermore, because they are unable to consider the user's stress or motivation, it is difficult to improve the quality of work.

[0583] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0584] In this invention, the server includes means for recording screen operations, means for extracting information from the recorded video, means for analyzing the worker's audio commentary, means for recognizing emotions, means for constructing a workflow by integrating the extracted information with the analyzed audio information and emotional information, means for generating a procedure based on the workflow, means for the worker to adjust the generated procedure, and means for scheduling and executing the adjusted procedure, thereby enabling the efficiency and quality of work performed by the user to be improved.

[0585] The "means for recording screen operations" refers to a device or program that records the operations performed by a user on the screen in video format.

[0586] "Means for extracting information from recorded video" refers to devices or programs that use OCR technology or other methods to analyze on-screen text and data from recorded video and obtain the necessary information.

[0587] The "means for analyzing the worker's audio commentary" refers to a device or program that converts the audio file recorded by the worker into text data using voice recognition technology and analyzes the content.

[0588] "Means for recognizing emotions" refers to a device or program that uses the device's camera or microphone to analyze the user's facial expressions and tone of voice and identify the user's emotional state.

[0589] The "means for constructing a workflow by integrating the extracted information with the analyzed audio information and emotional information" refers to a device or program that integrates the results of video analysis, audio analysis, and emotional analysis to organize the user's work procedures in chronological order and construct a detailed workflow.

[0590] The "means for generating a procedure based on the workflow" refers to a device or program that generates an automatable procedure based on the constructed workflow.

[0591] The "means for an operator to adjust the generated procedure" refers to an interface or device that allows a user to check the generated automated procedure and make corrections or additions as necessary.

[0592] The "means for scheduling and executing the adjusted procedure" refers to a device or program with a scheduling function that automatically executes the procedure modified by the user at a specified time.

[0593] The present invention relates to a system that automates routine tasks and recognizes and reflects the user's emotions. This system involves a terminal, a server, and a user. A specific embodiment of the system is shown below.

[0594] First, the user starts a routine task. The device launches screen recording software (e.g., OBS Studio) and records the user's screen operations in video format. Specific actions recorded include mouse clicks, keyboard input, and window switching. All on-screen actions are then saved as a video file (e.g., .mp4 format).

[0595] Next, the device uploads the recorded video file to the server. This can be done by dragging and dropping it from a file browser. The server receives the uploaded file and analyzes the text and data in the video using OCR (Optical Character Recognition) technology (e.g., Google Cloud Vision API). Specifically, it analyzes the video frames and extracts text and numerical data.

[0596] The user provides an audio description of the work they are doing. This audio description explains the steps and purpose of the work, and includes specific content such as "Enter sales data into this cell and click this button to create a graph." The device records the audio through a microphone and uploads the audio file (e.g., .wav format) to the server. The server receives the audio file and converts it into text data using voice recognition technology (e.g., Google Cloud Speech-to-Text). Specifically, the audio data is analyzed and corresponding text information is generated.

[0597] Furthermore, the device uses a camera and microphone to capture the user's facial expressions and tone of voice in real time. For example, it records the moment when the user smiles or their voice tone becomes lower. The server analyzes this data using an emotion engine (e.g., Microsoft Azure Emotion API) to identify the user's emotional state. For example, it obtains information such as "the user is feeling stressed."

[0598] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. Specifically, it organizes the work content for each step and the corresponding emotional data in chronological order. This workflow specifically shows the sequential steps of the user's operations, and by incorporating emotional information, it provides a procedure that takes into account the user's stress and motivation.

[0599] The server then presents a list of automatable steps for the user to review. For example, the steps are displayed sequentially through a web interface. The user reviews the presented workflow and makes corrections as necessary. Specific actions include deleting unnecessary steps and adding new comments. The user can also review the generated steps and make corrections or additions, and the optimized steps are then saved back to the server.

[0600] Finally, the server schedules the revised procedure to run at a specified time. Specific actions include automatically executing a series of operations, such as "opening an Excel file at 9:00 every morning, processing specific data, and sending the results by email." This frees users from this routine work, allowing them to focus on more creative tasks.

[0601] Prompt Sentence Examples

[0602] "I want to automate the daily task of opening an Excel file, processing the data, and sending it to my boss by email. I've recorded the steps and provided audio explanations of the work. Please automate this procedure in the system and set it to run every morning at 9:00."

[0603] In this way, this system can automate routine tasks and significantly improve the efficiency and quality of work for users. Furthermore, by taking into account the user's emotions, it reduces stress and improves the comfort and effectiveness of work.

[0604] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0605] Step 1:

[0606] The user starts a routine task. The device launches the screen recording software, which records the user's screen operations in video format. Specifically, mouse clicks, keyboard input, window switching, etc. are recorded. The input is the user's screen operations, and the output is a video file (e.g., .mp4 format).

[0607] Step 2:

[0608] The device uploads the recorded video file to the server. The user uploads it by dragging and dropping it from the file browser. The input is the video file, and the output is a file saved on the server. Specifically, the user selects the video file and clicks the upload button.

[0609] Step 3:

[0610] The server receives the uploaded video file. It uses OCR technology to analyze the characters and data in the video and obtain the necessary information. The input is the video file, and the output is the analyzed text information. Specifically, OCR technology recognizes characters for each frame of the video and generates text data.

[0611] Step 4:

[0612] The user gives an audio commentary about the work they are doing. The device uses a microphone to record the audio and uploads the audio file (e.g., .wav format) to the server. The input is the user's voice and the output is an audio file. Specifically, the user presses the record button and explains the work they are doing.

[0613] Step 5:

[0614] The server receives the uploaded audio file and converts it into text data using speech recognition technology. The input is the audio file and the output is text data. Specifically, the audio data is analyzed and the corresponding text information is generated.

[0615] Step 6:

[0616] The device uses a camera and microphone to capture the user's facial expressions and tone of voice in real time. The server uses an emotion engine to analyze this data and identify the user's emotional state. The input is the user's facial expressions and tone of voice, and the output is data on the user's emotional state. Specifically, it uses facial expression recognition algorithms and voice tone analysis technology.

[0617] Step 7:

[0618] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. The input is the analyzed data, and the output is the integrated workflow. Specifically, the work content for each step and the corresponding emotional data are organized in chronological order.

[0619] Step 8:

[0620] The server presents a list of automatable steps to the user for review. The input is the workflow, and the output is a list of steps presented to the user, with each step displayed sequentially through a web interface.

[0621] Step 9:

[0622] The user checks the presented workflow and makes corrections as necessary. The input is a list of steps, and the output is the corrected steps. Specifically, the user deletes unnecessary steps and adds new comments.

[0623] Step 10:

[0624] The server re-saves the modified procedure. The input is the modified procedure, and the output is the procedure data saved on the server. Specifically, the procedure data that reflects the user's modification information is saved on the server.

[0625] Step 11:

[0626] The server schedules the execution of the modified procedure at a specified time. The input is the modified procedure data, and the output is the scheduled execution of the work. Specifically, for example, a series of operations such as "open an Excel file every morning at 9:00, process specific data, and send the results by email" is automatically executed.

[0627] In this way, the system can automate routine tasks, improve efficiency and work quality, and provide a flexible work environment that takes users' emotions into consideration.

[0628] (Application example 2)

[0629] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0630] In logistics centers, employees perform a large number of routine tasks, which not only consume a significant amount of time and effort, but also cause stress and fatigue in workers. Furthermore, because employees' emotional states have a direct impact on work efficiency and quality, it is important to develop an optimized workflow that takes into account employees' emotions. Therefore, there is a need to automate routine tasks while providing a workflow that reflects the emotions of workers.

[0631] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording screen operations, means for extracting information from recorded video, means for analyzing the worker's audio commentary, means for building a workflow by integrating the extracted information and the analyzed audio information, means for generating procedures based on the workflow, means for the worker to adjust the generated procedures, means for scheduling and executing the adjusted procedures, and means for analyzing the worker's emotions and reflecting them in the workflow. This makes it possible to efficiently automate employees' routine tasks and provide an optimal workflow that takes employees' emotions into consideration.

[0632] The "means for recording screen operations" refers to a device or software that records the screen operations performed by the worker in video format.

[0633] "Means for extracting information from recorded video" refers to technology or equipment that has the ability to analyze and extract useful text or data from recorded video.

[0634] The "means for analyzing the worker's voice commentary" is a voice recognition technology that converts the worker's voice commentary into text data.

[0635] The "means for constructing a workflow by integrating the extracted information and analyzed audio information" is an algorithm that integrates video and audio data to generate detailed work procedures.

[0636] The "means for generating a procedure based on the workflow" is a system that generates an automatable procedure based on the constructed workflow.

[0637] "Means for operators to adjust the generated procedures" refers to interfaces and tools that allow operators to check and correct the automated procedures.

[0638] The "means for scheduling and executing the adjusted procedure" is a function for automatically executing the procedure adjusted by the worker at a specified time.

[0639] "Means for analyzing workers' emotions and reflecting them in the workflow" refers to algorithms and technologies that recognize and analyze the emotional state of workers in real time and reflect the results in the workflow.

[0640] The present invention relates to an automated system that improves the efficiency of routine work in a logistics center and takes into account the emotions of workers. This system is implemented in the following manner.

[0641] First, capture software and a webcam are used to record terminal operations. While workers perform routine tasks (e.g., product picking and packing) at the distribution center on the terminal screen, the operations are recorded in video format. This recorded video is then uploaded to a server.

[0642] The server uses OCR (optical character recognition) technology (such as pytesseract) to extract text and data from the recorded video. The server also uses speech recognition technology (such as Google Speech Recognition) to convert the operator's audio commentary into text. This allows the operation procedures and their explanations to be obtained as data.

[0643] The server then uses an emotion engine to analyze the worker's emotions. This process involves using the device's camera and microphone to analyze the worker's facial expressions and tone of voice in real time. The emotion engine uses technology to extract features from voice data and classify emotions using an SVM model.

[0644] The server integrates the analyzed video, audio, and emotional information to construct a workflow. This workflow concretely shows the sequential steps of the user's operations and incorporates emotional information to provide a procedure that takes into account the user's stress and motivation.

[0645] Based on this workflow, a procedure that can be automated is generated. The server presents the generated procedure in list form to the worker, who can then modify it as needed. For example, it is possible to delete unnecessary steps or add additional comments. Once the procedure has been modified, it is saved on the server and automatically executed at the scheduled time.

[0646] As a concrete example, consider a task in which a worker processes specific data every day and reports the results. The worker first records this series of operations, and then records an audio commentary of the operations. The video and audio are uploaded to a server for analysis. If the emotion engine detects that the worker is in a high stress state, it suggests simplifying some of the steps. After the worker reviews and makes corrections, the procedure is set to be executed automatically every morning at 9:00.

[0647] Below is an example of a prompt sentence to input to the generative AI model.

[0648] Example prompt for a generative AI model:

[0649] You are designing a system to improve the work efficiency of employees at a distribution center. Code the following steps in Python:

[0650] 1. Record and save the work steps.

[0651] 2. Record and save an audio description of your work.

[0652] 3. Extract text data from the recorded video using OCR.

[0653] 4. Convert the recorded audio into text using speech recognition.

[0654] 5. Analyze emotional information using an emotion engine and optimize work procedures accordingly.

[0655] 6. Set up a schedule to automatically execute tasks at specific times.

[0656] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0657] Step 1:

[0658] The routine tasks performed by users at the logistics center are recorded using capture software and a webcam on the terminal. Specifically, the user performs work steps on the terminal, and the operations are saved in video format. This recorded data becomes the input.

[0659] Step 2:

[0660] The recorded video data is uploaded to a server, where the server uses OCR technology (such as pytesseract) to extract text and data from the video. The input is the recorded data, and the output is a list of text and data. This data extraction identifies specific operating procedures and tool usage.

[0661] Step 3:

[0662] The user uses the microphone on their device to record an audio commentary of their work. The recorded audio file is uploaded to the server. This audio file is used as input and converted into text data using speech recognition technology (such as Google Speech Recognition). The output is the text data of the audio commentary.

[0663] Step 4:

[0664] The server uses an emotion engine to analyze the user's emotions. Features are extracted from the user's facial expression data and tone of voice data acquired from the device's camera and microphone. These features are then fed into an SVM model to classify emotions. The output is the user's emotional information.

[0665] Step 5:

[0666] The server integrates the information extracted in step 2, the text data of the audio commentary obtained in step 3, and the emotional information analyzed in step 4 to construct a workflow. Based on this integrated data, a detailed workflow is created that incorporates the specific operational steps performed by the user and their emotional information. The output is the integrated workflow.

[0667] Step 6:

[0668] The server generates automatable steps based on the constructed workflow. The input is the integrated workflow, and the output is a list of automatable steps, which lists tasks to be performed at specific times.

[0669] Step 7:

[0670] The generated procedure list is presented to the user, who then checks and modifies it. At this stage, necessary adjustments are made, such as deleting unnecessary steps or adding additional comments. The input is the pre-generated procedure list, and the output is the procedure list modified by the user.

[0671] Step 8:

[0672] The server executes the procedures modified by the user based on a schedule. This procedure list is registered in the schedule and automatically executed at the specified time. The input is the modified procedure list and schedule information, and the output is the execution result of the automatically executed task.

[0673] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0674] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0675] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0676] [Third embodiment]

[0677] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0678] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0679] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0680] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0681] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0682] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0683] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0684] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0685] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0686] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0687] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0688] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0689] The present invention relates to a system for automating routine tasks, and can be implemented based on the following settings.

[0690] This system involves a terminal, a server, and a user. First, the user records routine tasks. The recording is performed on the terminal operated by the user, and screen operations are recorded. All actions performed on the terminal (e.g., mouse clicks, keyboard input, screen capture) are saved in video format.

[0691] The recorded video is uploaded to a server. The server is equipped with a function to extract information from the recorded video. Specifically, the server uses OCR (optical character recognition) technology to analyze the text and data in the video and extract the necessary information. This process identifies the specific operating steps performed by the user.

[0692] Next, the user records an audio commentary about the task. The recording is done using the device's microphone, and the resulting audio file is uploaded to the server. The server analyzes the audio file and obtains the user's explanation as text data. This analysis is performed using voice recognition technology.

[0693] The server integrates the analyzed video and audio information to construct a detailed workflow. This workflow concretely shows the sequential steps of the user's operations. Based on this workflow, a procedure that can be automated is then generated. The server presents the generated procedure in list form and allows the user to confirm it.

[0694] The user can review the proposed procedure and make any necessary modifications, such as deleting unnecessary steps or adding additional comments. The optimized procedure is then saved back to the server.

[0695] Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, a series of operations can be performed automatically, such as opening a specific Excel file at 9:00 every morning, processing the data, and sending the results by email. This scheduled execution function frees users from routine tasks, allowing them to focus on more creative work.

[0696] As a concrete example, the following scenario can be considered.

[0697] A user's daily task is to process specific data in Excel and send the results to their boss by email. The user first records this series of operations. Next, they record an audio commentary about the work and upload it to the server. The server analyzes this data, generates an automatable procedure, and presents it to the user. After the user checks and modifies the procedure, the server schedules the procedure to be executed every morning at 9:00 AM. As a result, the user is freed from the hassle of manually processing and sending data every morning.

[0698] In this way, the system automates routine tasks, saving time and effort while also increasing the accuracy of the work.

[0699] The processing flow will be explained below.

[0700] Step 1:

[0701] The user uses the device to record screen activity. The device uses screen capture software to record all user activity (e.g., mouse clicks, keyboard input) and saves it as a video file.

[0702] Step 2:

[0703] The user stops recording and uploads the recorded video file from the terminal to the server, and the terminal transfers the video file to the server via the network.

[0704] Step 3:

[0705] The server receives the uploaded video file and uses OCR technology to analyze the text information and operation details in the video, extracting the analyzed information as text data.

[0706] Step 4:

[0707] The user uses the device to record an audio commentary of the work they are doing, and the recorded audio file is saved on the device.

[0708] Step 5:

[0709] A user uploads an audio file from a terminal to a server, and the terminal transfers the audio file to the server via a network.

[0710] Step 6:

[0711] The server receives the uploaded audio file and converts the audio content into text data using voice recognition technology. The text data obtained through voice analysis accurately reflects the user's commentary.

[0712] Step 7:

[0713] The server integrates the video and audio analysis data to construct a detailed workflow, which combines the user's operational steps with corresponding audio commentary.

[0714] Step 8:

[0715] The server generates automatable procedures based on the constructed workflow, and the generated procedures are organized in a list format and presented to the user.

[0716] Step 9:

[0717] The user checks the generated procedure and makes corrections or adjustments as necessary. The user uses the terminal to change the content of the procedure and arrange it in the optimal form.

[0718] Step 10:

[0719] The modified and adjusted procedures are saved back to the server, which stores these procedures in a database for scheduled execution.

[0720] Step 11:

[0721] A user uses a terminal to schedule an automated procedure to run, for example, at a specific time each day.

[0722] Step 12:

[0723] At the set time, the server executes the automated procedures according to the schedule. The server executes each procedure in sequence and automatically performs the necessary operations (e.g., processing data in Excel, sending emails).

[0724] Example 1

[0725] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0726] In modern business activities, many routine tasks occur on a daily basis. These tasks require time and effort, and there are issues with efficiency and accuracy. Conventional automation systems have difficulty automating a series of operations at once, and human intervention is required, especially for complex business processes. Therefore, reducing labor costs and improving work accuracy have become important issues.

[0727] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0728] In this invention, the server includes means for a user to record work on a terminal, means for uploading the recorded data to the server, means for the server to analyze the video, means for the user to record an audio commentary on the work content, means for uploading the recorded data to the server, means for the server to analyze the audio file, means for the server to integrate the video information and the audio information, means for the server to generate an automation procedure, means for the user to check and modify the procedure, and means for the server to schedule and execute the procedure. This makes it possible to automate routine business processes efficiently and with high accuracy.

[0729] A "user" is an operator who uses the system to automate routine tasks.

[0730] A "terminal" is a hardware device that a user uses to perform work, and includes a personal computer, a smartphone, etc.

[0731] "Recording" is the process of recording operations performed on a terminal in video format.

[0732] A "server" is a computer or cloud service that is the core of a system, and is a device that analyzes data and generates and executes procedures.

[0733] "Uploading" is the process of sending files or data generated on a device to a server.

[0734] "Video analysis" is the process of extracting text information and operating procedures from recorded video using OCR technology.

[0735] "Audio commentary" is information in which the user verbally explains the content of the work.

[0736] An "audio file" is a file that stores audio data recorded by a user.

[0737] "Speech analysis" is the process of obtaining text information from an audio file using speech recognition technology.

[0738] A "workflow" is information that indicates the sequential steps of a user's operation procedure, and is constructed by integrating video information and audio information.

[0739] An "automation procedure" is a procedure script created based on a workflow to automatically execute a series of operations.

[0740] "Procedure verification and correction" is the process in which the user verifies the automated procedures presented by the server and makes any necessary corrections.

[0741] "Scheduled execution" is the process of automatically executing a procedure at a specified time.

[0742] The present invention relates to a system for automating routine tasks, and involves a user, a terminal, and a server.

[0743] The system uses the following hardware and software:

[0744] Terminal: A device operated by a user, such as a computer or smartphone.

[0745] Servers: High-performance computers, data centers, and cloud services (e.g., AWS, Azure).

[0746] OCR technology: Software for extracting text data (e.g., Google Vision API).

[0747] Speech recognition technology: Software that converts speech into text (e.g., Google Speech-to-Text API).

[0748] Recording software: A recording tool that runs on your device (e.g., OBS Studio).

[0749] Audio recording software: A tool to record audio from your microphone input (e.g., Audacity).

[0750] As a specific example of operation, the process below shows how to automate a task in which a user processes specific data in Excel every day and sends the results to their supervisor by email.

[0751] Operating Procedure

[0752] 1. The user launches OBS Studio on their device and clicks the "Record" button to begin recording their work. All on-screen actions (e.g., opening Excel and entering data into specific cells) will be recorded.

[0753] 2. Once recording is complete, download the recording file from the OBS Studio file menu and upload it to the server using the dedicated upload form.

[0754] 3. The server uses the Google Vision API to analyze the uploaded video file and extract the cell contents in Excel and the user interface operation procedures as text data.

[0755] 4. Next, the user uses Audacity to record an audio description of the task, generating an audio file containing instructions such as, "To perform this task, first enter sales data into Excel cell A1."

[0756] 5. After recording is complete, the audio file is uploaded to the server, which uses the Google Speech-to-Text API to extract the text from the audio file.

[0757] 6. The server combines the information extracted from the video file and the information extracted from the audio file to construct a detailed workflow, which shows the steps taken by the user.

[0758] 7. The server generates automated procedures based on the workflow and presents them to the user in list form.

[0759] 8. The user reviews the proposed steps and modifies them as necessary, for example, by inserting additional steps or deleting unnecessary steps.

[0760] 9. Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, an Excel file is opened at 9:00 every morning, the data is processed, and the results are sent by email.

[0761] Specific examples

[0762] If a user wants to automate a daily task of processing certain data in Excel and emailing the results to their boss, they can use a prompt like this:

[0763] "I want an automated task that opens new sales data in Excel every morning, processes it into a specific format, and emails the results to my boss."

[0764] In this way, the system efficiently automates routine tasks, freeing users from routine tasks, saving time and effort, and improving work accuracy.

[0765] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0766] Step 1:

[0767] A user records their work on a device. To do this, the user launches recording software (e.g., OBS Studio) and clicks the "Record" button to begin recording their work. The input is the user's screen operations, and the output is a recorded video file. This video file records the user's operations (e.g., opening Excel, selecting cells, entering data, etc.).

[0768] Step 2:

[0769] The user uploads the recorded data to the server. The user retrieves the file from OBS Studio and sends the data to the server using a dedicated upload form. The input is the recorded video file, and the output is a video file saved on the server. Specifically, the user opens a browser, accesses the upload form, selects the recorded file, and completes the upload.

[0770] Step 3:

[0771] The server analyzes the video. It uses the Google Vision API to analyze the video file frame by frame and extracts text information from the video and user interface operation procedures. The input is the uploaded video file, and the output is text information and operation procedures. Specifically, the server loads the video file and runs OCR on each frame to extract the text data.

[0772] Step 4:

[0773] A user records an audio description of a task. The user launches Audacity and clicks the "Record" button to begin recording. The input is the user's verbal description, and the output is an audio file. Specifically, the user explains into the microphone, for example, "In this operation, first enter sales data into Excel cell A1," and then ends the recording.

[0774] Step 5:

[0775] The user uploads the recorded data to the server. The user sends the audio file saved in Audacity to the server using a dedicated upload form. The input is the recorded audio file, and the output is the audio file saved on the server. Specifically, the user opens a browser, accesses the upload form, selects the audio file, and completes the upload.

[0776] Step 6:

[0777] The server analyzes the audio file. The server converts the audio file into text using the Google Speech-to-Text API. The input is the uploaded audio file, and the output is text data. Specifically, the server loads the audio file, runs it through speech recognition, and extracts the text data.

[0778] Step 7:

[0779] The server integrates the video and audio information. The server then integrates the text data from the analyzed video frames with the text data obtained through speech recognition to construct a detailed workflow. The input is text and audio information, and the output is an integrated workflow. Specifically, the server analyzes the text information stored in the database and organizes it into operational steps to create a workflow.

[0780] Step 8:

[0781] The server generates the automation steps. The server generates a script based on the workflow to create an automatable step. The input is the detailed workflow, and the output is the automation step script. Specifically, the server uses a programming language (e.g., Python) to generate code to perform a specific operation.

[0782] Step 9:

[0783] The user checks and modifies the procedure. The user checks the automation procedure presented by the server and modifies it as necessary. The input is the automation procedure from the server, and the output is the modified procedure. Specifically, the user checks the procedure using an interface on the browser, deletes unnecessary steps, or inserts additional steps.

[0784] Step 10:

[0785] The server executes the procedure on a scheduled basis. The server executes the modified procedure based on the set schedule. The input is the modified automation procedure, and the output is the result of executing the automated operation. Specifically, the server executes the procedure at the set time, for example, opening an Excel file, processing the data, and sending the results by email.

[0786] (Application example 1)

[0787] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0788] Logistics centers have many routine tasks that need to be performed efficiently. In particular, routine tasks such as inventory checks, stocktaking, and picking require time and effort, and are prone to human error. Therefore, there is a need to automate these tasks to improve efficiency and accuracy.

[0789] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0790] In this invention, the server includes means for recording screen operations, means for extracting information from the recorded video, means for analyzing the worker's audio commentary, means for building a workflow by integrating the extracted information and the analyzed audio information, means for generating procedures based on the workflow, means for the worker to adjust the generated procedures, means for scheduling and executing the adjusted procedures, means for visually recording the work procedures using a head-mounted display, means for extracting text data from the recorded video using optical character recognition technology, and means for converting the audio commentary into text using voice recognition technology, thereby enabling the automation and efficiency of routine tasks in a logistics center.

[0791] The "means for recording screen operations" refers to a means for recording the operations performed by the user on the screen, such as mouse clicks and keyboard input, as video.

[0792] "Means for extracting information from recorded video" refers to means for extracting necessary data or text information from recorded video.

[0793] The "means for analyzing the voice commentary of the worker" is a means for analyzing the voice recorded by the worker and acquiring the content as text data.

[0794] The "means for integrating the extracted information and analyzed voice information to construct a workflow" is a means for combining information extracted from the video with the voice recognition results to create a workflow that details the steps of the work.

[0795] The "means for generating a procedure based on the workflow" refers to a means for generating an automatable procedure based on the constructed workflow.

[0796] The "means for the worker to adjust the generated procedure" refers to a means for the user to check the generated procedure, delete unnecessary steps, and add additional comments.

[0797] The "means for scheduling and executing the adjusted procedure" is a means for automatically executing the adjusted procedure at a time designated by the user.

[0798] "Means for visually recording work procedures using a head-mounted display" refers to a means in which a worker wears a head-mounted display and records the work procedures from that viewpoint.

[0799] The "means for extracting text data from recorded video using optical character recognition technology" refers to a means for extracting character information from recorded video frames using OCR technology and obtaining it as text data.

[0800] "Means for converting audio commentary into text using speech recognition technology" refers to means for analyzing recorded audio commentary using speech recognition technology and converting it into text data.

[0801] The present invention is a system for automating routine tasks in a logistics center. The system includes a means for recording screen operations, a means for extracting information from recorded video, a means for analyzing audio commentary by workers, a means for integrating the extracted information and the analyzed audio information to create a workflow, a means for workers to adjust the generated procedures, a means for scheduling and executing the adjusted procedures, a means for visually recording the work procedures using a head-mounted display, a means for extracting text data from the recorded video using OCR technology, and a means for converting the audio commentary into text using speech recognition technology.

[0802] Hardware and Software Configuration

[0803] Hardware

[0804] Head-mounted display (HMD): A device worn by a worker to record work procedures from their point of view.

[0805] Terminal (smartphone or tablet): A device for collecting video and audio data and uploading it to a server.

[0806] Server: A computer with powerful processing power that analyzes information and generates procedures.

[0807] software

[0808] OCR Technology: PaddleOCR is used to extract text data from recorded video frames.

[0809] Speech Recognition Technology: Technology for converting recorded audio commentary into text data using the SpeechRecognition library.

[0810] Scheduling library: A system for executing automated procedures at user-specified times using the schedule library.

[0811] Operation overview

[0812] First, the user puts on the head-mounted display and records the procedure while performing routine tasks at a logistics center (e.g., inventory checks, stocktaking, picking). Once the recording is complete, an audio commentary on the procedure is recorded. This data is then uploaded to the server via the device.

[0813] The server first extracts the necessary information from the video data using OCR technology. In parallel, it converts the recorded audio commentary into text data using speech recognition technology. It then integrates the extracted information with the commentary text to construct a detailed workflow. Based on the constructed workflow, automatable steps are generated and presented to the user.

[0814] The user can review the presented procedures and adjust them by deleting unnecessary steps or adding comments. The adjusted procedures are then saved back to the server. Finally, the server automatically executes the adjusted procedures based on a schedule. For example, an inventory check can be automatically performed every morning at 9:00 and the results sent to the person in charge via email.

[0815] Examples of concrete examples and prompts

[0816] Specific examples

[0817] This process involves automatically checking inventory every morning at 9:00 and sending the results to the person in charge via email. The person wearing the head-mounted display records their viewpoint, and the data with recorded audio commentary is uploaded to the server. The server analyzes this data, generates and adjusts the automated procedures, and executes them at the specified time.

[0818] Prompt Sentence Examples

[0819] Example: Write a Python program to automate inventory checks at a distribution center every morning at 9:00. The program should include video recording, voice recognition, OCR analysis, and schedule execution.

[0820] Note: A prompt statement that automates logistics center operations based on the system described in the invention.

[0821] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0822] Step 1:

[0823] Data recording

[0824] A user wears a head-mounted display and performs routine tasks in a logistics center. The head-mounted display records the work procedure on video. The user also records audio commentary about the work procedure. This allows visual and audio information to be collected.

[0825] Input: User instructions and audio description

[0826] Output: Video and audio files

[0827] Step 2:

[0828] Uploading data

[0829] The device uploads the recorded video and audio files to a server, an automated operation that is performed through the device's interface.

[0830] Input: Video and audio files

[0831] Output: Video and audio data on the server

[0832] Step 3:

[0833] Data extraction with OCR

[0834] The server analyzes the uploaded video data and extracts the necessary information using optical character recognition (OCR) technology. Specifically, it identifies and extracts text data from video frames. PaddleOCR is used.

[0835] Input: Video data on the server

[0836] Output: Extracted text data

[0837] Step 4:

[0838] Data analysis using voice recognition

[0839] The server analyzes the uploaded audio data and converts the audio commentary into text data using speech recognition technology, primarily using the SpeechRecognition library.

[0840] Input: Audio data on the server

[0841] Output: Converted text data

[0842] Step 5:

[0843] Integrating information and building workflows

[0844] The server integrates the text data obtained from OCR and speech recognition to construct a detailed workflow, which is based on the user's specific operating steps and audio commentary.

[0845] Input: Extracted and converted text data

[0846] Output: The completed workflow

[0847] Step 6:

[0848] Generating automation procedures

[0849] The server generates automatable procedures based on the constructed workflow, and presents the procedures to the user as a list of steps.

[0850] Input: Completed Workflow

[0851] Output: Generated list of automation steps

[0852] Step 7:

[0853] Adjusting procedures

[0854] The user can review the generated automated procedure and modify it as needed, for example by deleting unnecessary steps or adding additional comments. After this adjustment, the procedure is saved back to the server.

[0855] Input: Generated list of automated procedures

[0856] Output: Modified list of automated steps

[0857] Step 8:

[0858] Schedule execution

[0859] The server schedules and executes the adjusted procedures at the specified time. For example, you can automatically execute an inventory check at 9:00 AM every day and send the results to the person in charge by email.

[0860] Input: Modified automated procedure list

[0861] Output: The results of the automated procedure that was executed (e.g., an inventory checklist)

[0862] Examples of concrete examples and prompts

[0863] Specific examples

[0864] This process involves automatically checking inventory every morning at 9:00 and sending the results to the person in charge via email. The person wearing the head-mounted display records their viewpoint, and the data with recorded audio commentary is uploaded to the server. The server analyzes this data, generates and adjusts the automated procedures, and executes them at the specified time.

[0865] Prompt Sentence Examples

[0866] Example: Write a Python program to automate inventory checks at a distribution center every morning at 9:00. The program should include video recording, voice recognition, OCR analysis, and schedule execution.

[0867] Note: A prompt statement that automates logistics center operations based on the system described in the invention.

[0868] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0869] The present invention relates to a system for automating routine tasks, and also combines it with an emotion engine that recognizes and reflects the user's emotions. It can be implemented based on the following settings.

[0870] This system involves a terminal, a server, and a user. First, the user records routine tasks. The recording is performed on the terminal operated by the user, and screen operations are recorded. All actions performed on the terminal (e.g., mouse clicks, keyboard input, screen capture) are saved in video format.

[0871] The recorded video is uploaded to a server. The server is equipped with a function to extract information from the recorded video. Specifically, the server uses OCR (optical character recognition) technology to analyze the text and data in the video and extract the necessary information. This process identifies the specific operating steps performed by the user.

[0872] Next, the user records an audio commentary about the task. The recording is done using the device's microphone, and the resulting audio file is uploaded to the server. The server analyzes the audio file and obtains the user's explanation as text data. This analysis is performed using voice recognition technology.

[0873] In addition, the system uses an emotion engine to recognize the user's emotions. Emotional information is acquired by analyzing the user's facial expressions and tone of voice using the device's camera and microphone. This emotion engine can grasp the user's emotions in real time while they are operating the device.

[0874] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. This workflow concretely shows the sequential steps of the user's operations, and by incorporating emotional information, it provides a procedure that takes into account the user's stress and motivation.

[0875] Furthermore, a procedure that can be automated is generated based on this workflow. The server presents the generated procedure in list form and allows the user to review it. The user can review the presented procedure and make corrections as necessary. For example, they can delete unnecessary steps or add additional comments. The optimized procedure is then saved back to the server.

[0876] Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, a series of operations can be performed automatically, such as opening a specific Excel file at 9:00 every morning, processing the data, and sending the results by email. This scheduled execution function frees users from routine tasks, allowing them to focus on more creative work.

[0877] As a concrete example, the following scenario can be considered.

[0878] A user's daily work involves processing specific data in Excel and emailing the results to their boss. The user first records this series of operations. Next, they record an audio commentary about the work and upload it to the server. The server analyzes this data, generates automatable steps, and presents them to the user. Furthermore, if the emotion engine detects that the user's stress level is high, it suggests modifications to simplify some of the steps. After the user checks the content and makes modifications, the server schedules this procedure to be executed every morning at 9:00 AM. As a result, the user is freed from the hassle of manually processing and sending data every morning.

[0879] In this way, the system automates routine tasks, saving time and effort, and improves the quality of tasks by taking into account the user's emotions.

[0880] The processing flow will be explained below.

[0881] Step 1:

[0882] The user uses the device to record screen activity. The device uses screen capture software to record all user activity (e.g., mouse clicks, keyboard input) and saves it as a video file.

[0883] Step 2:

[0884] The user stops recording and uploads the recorded video file from the terminal to the server, and the terminal transfers the video file to the server via the network.

[0885] Step 3:

[0886] The server receives the uploaded video file and uses OCR technology to analyze the text information and operation details in the video, extracting the analyzed information as text data.

[0887] Step 4:

[0888] The user uses the device to record an audio commentary of the work they are doing, and the recorded audio file is saved on the device.

[0889] Step 5:

[0890] A user uploads an audio file from a terminal to a server, and the terminal transfers the audio file to the server via a network.

[0891] Step 6:

[0892] The server receives the uploaded audio file and converts the audio content into text data using voice recognition technology. The text data obtained through voice analysis accurately reflects the user's commentary.

[0893] Step 7:

[0894] The user uses the device's camera and microphone to collect emotional data while working. The device then analyzes the user's facial expressions and tone of voice in real time and obtains emotional information through an emotion engine.

[0895] Step 8:

[0896] The server integrates the video analysis data, audio analysis data, and emotional information to construct a detailed workflow, which reflects the user's operation steps and the corresponding audio commentary and emotional state.

[0897] Step 9:

[0898] The server generates automatable procedures based on the constructed workflow, and the generated procedures are organized in a list format and presented to the user.

[0899] Step 10:

[0900] The user can review the generated instructions and make corrections or adjustments as necessary. The user can then use their device to modify the content of the instructions, taking into account feedback from the emotion engine, to optimize them.

[0901] Step 11:

[0902] The modified and adjusted procedures are saved back to the server, which stores these procedures in a database for scheduled execution.

[0903] Step 12:

[0904] A user uses a terminal to schedule an automated procedure to run, for example, at a specific time each day.

[0905] Step 13:

[0906] At the set time, the server executes the automated procedures according to the schedule. The server executes each procedure in sequence and automatically performs the necessary operations (e.g., processing data in Excel, sending emails).

[0907] Example 2

[0908] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0909] Conventional automation systems for routine tasks can automate user work procedures, but they are unable to consider the user's emotions, which often results in inefficient automation. Furthermore, because they are unable to consider the user's stress or motivation, it is difficult to improve the quality of work.

[0910] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0911] In this invention, the server includes means for recording screen operations, means for extracting information from the recorded video, means for analyzing the worker's audio commentary, means for recognizing emotions, means for constructing a workflow by integrating the extracted information with the analyzed audio information and emotional information, means for generating a procedure based on the workflow, means for the worker to adjust the generated procedure, and means for scheduling and executing the adjusted procedure, thereby enabling the efficiency and quality of work performed by the user to be improved.

[0912] The "means for recording screen operations" refers to a device or program that records the operations performed by a user on the screen in video format.

[0913] "Means for extracting information from recorded video" refers to devices or programs that use OCR technology or other methods to analyze on-screen text and data from recorded video and obtain the necessary information.

[0914] The "means for analyzing the worker's audio commentary" refers to a device or program that converts the audio file recorded by the worker into text data using voice recognition technology and analyzes the content.

[0915] "Means for recognizing emotions" refers to a device or program that uses the device's camera or microphone to analyze the user's facial expressions and tone of voice and identify the user's emotional state.

[0916] The "means for constructing a workflow by integrating the extracted information with the analyzed audio information and emotional information" refers to a device or program that integrates the results of video analysis, audio analysis, and emotional analysis to organize the user's work procedures in chronological order and construct a detailed workflow.

[0917] The "means for generating a procedure based on the workflow" refers to a device or program that generates an automatable procedure based on the constructed workflow.

[0918] The "means for an operator to adjust the generated procedure" refers to an interface or device that allows a user to check the generated automated procedure and make corrections or additions as necessary.

[0919] The "means for scheduling and executing the adjusted procedure" refers to a device or program with a scheduling function that automatically executes the procedure modified by the user at a specified time.

[0920] The present invention relates to a system that automates routine tasks and recognizes and reflects the user's emotions. This system involves a terminal, a server, and a user. A specific embodiment of the system is shown below.

[0921] First, the user starts a routine task. The device launches screen recording software (e.g., OBS Studio) and records the user's screen operations in video format. Specific actions recorded include mouse clicks, keyboard input, and window switching. All on-screen actions are then saved as a video file (e.g., .mp4 format).

[0922] Next, the device uploads the recorded video file to the server. This can be done by dragging and dropping it from a file browser. The server receives the uploaded file and analyzes the text and data in the video using OCR (Optical Character Recognition) technology (e.g., Google Cloud Vision API). Specifically, it analyzes the video frames and extracts text and numerical data.

[0923] The user provides an audio description of the work they are doing. This audio description explains the steps and purpose of the work, and includes specific content such as "Enter sales data into this cell and click this button to create a graph." The device records the audio through a microphone and uploads the audio file (e.g., .wav format) to the server. The server receives the audio file and converts it into text data using voice recognition technology (e.g., Google Cloud Speech-to-Text). Specifically, the audio data is analyzed and corresponding text information is generated.

[0924] Furthermore, the device uses a camera and microphone to capture the user's facial expressions and tone of voice in real time. For example, it records the moment when the user smiles or their voice tone becomes lower. The server analyzes this data using an emotion engine (e.g., Microsoft Azure Emotion API) to identify the user's emotional state. For example, it obtains information such as "the user is feeling stressed."

[0925] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. Specifically, it organizes the work content for each step and the corresponding emotional data in chronological order. This workflow specifically shows the sequential steps of the user's operations, and by incorporating emotional information, it provides a procedure that takes into account the user's stress and motivation.

[0926] The server then presents a list of automatable steps for the user to review. For example, the steps are displayed sequentially through a web interface. The user reviews the presented workflow and makes corrections as necessary. Specific actions include deleting unnecessary steps and adding new comments. The user can also review the generated steps and make corrections or additions, and the optimized steps are then saved back to the server.

[0927] Finally, the server schedules the revised procedure to run at a specified time. Specific actions include automatically executing a series of operations, such as "opening an Excel file at 9:00 every morning, processing specific data, and sending the results by email." This frees users from this routine work, allowing them to focus on more creative tasks.

[0928] Prompt Sentence Examples

[0929] "I want to automate the daily task of opening an Excel file, processing the data, and sending it to my boss by email. I've recorded the steps and provided audio explanations of the work. Please automate this procedure in the system and set it to run every morning at 9:00."

[0930] In this way, this system can automate routine tasks and significantly improve the efficiency and quality of work for users. Furthermore, by taking into account the user's emotions, it reduces stress and improves the comfort and effectiveness of work.

[0931] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0932] Step 1:

[0933] The user starts a routine task. The device launches the screen recording software, which records the user's screen operations in video format. Specifically, mouse clicks, keyboard input, window switching, etc. are recorded. The input is the user's screen operations, and the output is a video file (e.g., .mp4 format).

[0934] Step 2:

[0935] The device uploads the recorded video file to the server. The user uploads it by dragging and dropping it from the file browser. The input is the video file, and the output is a file saved on the server. Specifically, the user selects the video file and clicks the upload button.

[0936] Step 3:

[0937] The server receives the uploaded video file. It uses OCR technology to analyze the characters and data in the video and obtain the necessary information. The input is the video file, and the output is the analyzed text information. Specifically, OCR technology recognizes characters for each frame of the video and generates text data.

[0938] Step 4:

[0939] The user gives an audio commentary about the work they are doing. The device uses a microphone to record the audio and uploads the audio file (e.g., .wav format) to the server. The input is the user's voice and the output is an audio file. Specifically, the user presses the record button and explains the work they are doing.

[0940] Step 5:

[0941] The server receives the uploaded audio file and converts it into text data using speech recognition technology. The input is the audio file and the output is text data. Specifically, the audio data is analyzed and the corresponding text information is generated.

[0942] Step 6:

[0943] The device uses a camera and microphone to capture the user's facial expressions and tone of voice in real time. The server uses an emotion engine to analyze this data and identify the user's emotional state. The input is the user's facial expressions and tone of voice, and the output is data on the user's emotional state. Specifically, it uses facial expression recognition algorithms and voice tone analysis technology.

[0944] Step 7:

[0945] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. The input is the analyzed data, and the output is the integrated workflow. Specifically, the work content for each step and the corresponding emotional data are organized in chronological order.

[0946] Step 8:

[0947] The server presents a list of automatable steps to the user for review. The input is the workflow, and the output is a list of steps presented to the user, with each step displayed sequentially through a web interface.

[0948] Step 9:

[0949] The user checks the presented workflow and makes corrections as necessary. The input is a list of steps, and the output is the corrected steps. Specifically, the user deletes unnecessary steps and adds new comments.

[0950] Step 10:

[0951] The server re-saves the modified procedure. The input is the modified procedure, and the output is the procedure data saved on the server. Specifically, the procedure data that reflects the user's modification information is saved on the server.

[0952] Step 11:

[0953] The server schedules the execution of the modified procedure at a specified time. The input is the modified procedure data, and the output is the scheduled execution of the work. Specifically, for example, a series of operations such as "open an Excel file every morning at 9:00, process specific data, and send the results by email" is automatically executed.

[0954] In this way, the system can automate routine tasks, improve efficiency and work quality, and provide a flexible work environment that takes users' emotions into consideration.

[0955] (Application example 2)

[0956] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0957] In logistics centers, employees perform a large number of routine tasks, which not only consume a significant amount of time and effort, but also cause stress and fatigue in workers. Furthermore, because employees' emotional states have a direct impact on work efficiency and quality, it is important to develop an optimized workflow that takes into account employees' emotions. Therefore, there is a need to automate routine tasks while providing a workflow that reflects the emotions of workers.

[0958] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording screen operations, means for extracting information from recorded video, means for analyzing the worker's audio commentary, means for building a workflow by integrating the extracted information and the analyzed audio information, means for generating procedures based on the workflow, means for the worker to adjust the generated procedures, means for scheduling and executing the adjusted procedures, and means for analyzing the worker's emotions and reflecting them in the workflow. This makes it possible to efficiently automate employees' routine tasks and provide an optimal workflow that takes employees' emotions into consideration.

[0959] The "means for recording screen operations" refers to a device or software that records the screen operations performed by the worker in video format.

[0960] "Means for extracting information from recorded video" refers to technology or equipment that has the ability to analyze and extract useful text or data from recorded video.

[0961] The "means for analyzing the worker's voice commentary" is a voice recognition technology that converts the worker's voice commentary into text data.

[0962] The "means for constructing a workflow by integrating the extracted information and analyzed audio information" is an algorithm that integrates video and audio data to generate detailed work procedures.

[0963] The "means for generating a procedure based on the workflow" is a system that generates an automatable procedure based on the constructed workflow.

[0964] "Means for operators to adjust the generated procedures" refers to interfaces and tools that allow operators to check and correct the automated procedures.

[0965] The "means for scheduling and executing the adjusted procedure" is a function for automatically executing the procedure adjusted by the worker at a specified time.

[0966] "Means for analyzing workers' emotions and reflecting them in the workflow" refers to algorithms and technologies that recognize and analyze the emotional state of workers in real time and reflect the results in the workflow.

[0967] The present invention relates to an automated system that improves the efficiency of routine work in a logistics center and takes into account the emotions of workers. This system is implemented in the following manner.

[0968] First, capture software and a webcam are used to record terminal operations. While workers perform routine tasks (e.g., product picking and packing) at the distribution center on the terminal screen, the operations are recorded in video format. This recorded video is then uploaded to a server.

[0969] The server uses OCR (optical character recognition) technology (such as pytesseract) to extract text and data from the recorded video. The server also uses speech recognition technology (such as Google Speech Recognition) to convert the operator's audio commentary into text. This allows the operation procedures and their explanations to be obtained as data.

[0970] The server then uses an emotion engine to analyze the worker's emotions. This process involves using the device's camera and microphone to analyze the worker's facial expressions and tone of voice in real time. The emotion engine uses technology to extract features from voice data and classify emotions using an SVM model.

[0971] The server integrates the analyzed video, audio, and emotional information to construct a workflow. This workflow concretely shows the sequential steps of the user's operations and incorporates emotional information to provide a procedure that takes into account the user's stress and motivation.

[0972] Based on this workflow, a procedure that can be automated is generated. The server presents the generated procedure in list form to the worker, who can then modify it as needed. For example, it is possible to delete unnecessary steps or add additional comments. Once the procedure has been modified, it is saved on the server and automatically executed at the scheduled time.

[0973] As a concrete example, consider a task in which a worker processes specific data every day and reports the results. The worker first records this series of operations, and then records an audio commentary of the operations. The video and audio are uploaded to a server for analysis. If the emotion engine detects that the worker is in a high stress state, it suggests simplifying some of the steps. After the worker reviews and makes corrections, the procedure is set to be executed automatically every morning at 9:00.

[0974] Below is an example of a prompt sentence to input to the generative AI model.

[0975] Example prompt for a generative AI model:

[0976] You are designing a system to improve the work efficiency of employees at a distribution center. Code the following steps in Python:

[0977] 1. Record and save the work steps.

[0978] 2. Record and save an audio description of your work.

[0979] 3. Extract text data from the recorded video using OCR.

[0980] 4. Convert the recorded audio into text using speech recognition.

[0981] 5. Analyze emotional information using an emotion engine and optimize work procedures accordingly.

[0982] 6. Set up a schedule to automatically execute tasks at specific times.

[0983] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0984] Step 1:

[0985] The routine tasks performed by users at the logistics center are recorded using capture software and a webcam on the terminal. Specifically, the user performs work steps on the terminal, and the operations are saved in video format. This recorded data becomes the input.

[0986] Step 2:

[0987] The recorded video data is uploaded to a server, where the server uses OCR technology (such as pytesseract) to extract text and data from the video. The input is the recorded data, and the output is a list of text and data. This data extraction identifies specific operating procedures and tool usage.

[0988] Step 3:

[0989] The user uses the microphone on their device to record an audio commentary of their work. The recorded audio file is uploaded to the server. This audio file is used as input and converted into text data using speech recognition technology (such as Google Speech Recognition). The output is the text data of the audio commentary.

[0990] Step 4:

[0991] The server uses an emotion engine to analyze the user's emotions. Features are extracted from the user's facial expression data and tone of voice data acquired from the device's camera and microphone. These features are then fed into an SVM model to classify emotions. The output is the user's emotional information.

[0992] Step 5:

[0993] The server integrates the information extracted in step 2, the text data of the audio commentary obtained in step 3, and the emotional information analyzed in step 4 to construct a workflow. Based on this integrated data, a detailed workflow is created that incorporates the specific operational steps performed by the user and their emotional information. The output is the integrated workflow.

[0994] Step 6:

[0995] The server generates automatable steps based on the constructed workflow. The input is the integrated workflow, and the output is a list of automatable steps, which lists tasks to be performed at specific times.

[0996] Step 7:

[0997] The generated procedure list is presented to the user, who then checks and modifies it. At this stage, necessary adjustments are made, such as deleting unnecessary steps or adding additional comments. The input is the pre-generated procedure list, and the output is the procedure list modified by the user.

[0998] Step 8:

[0999] The server executes the procedures modified by the user based on a schedule. This procedure list is registered in the schedule and automatically executed at the specified time. The input is the modified procedure list and schedule information, and the output is the execution result of the automatically executed task.

[1000] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1001] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1002] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1003] [Fourth embodiment]

[1004] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1005] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1006] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1007] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1008] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1009] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1010] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1011] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1012] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1013] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1014] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1015] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1016] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1017] The present invention relates to a system for automating routine tasks, and can be implemented based on the following settings.

[1018] This system involves a terminal, a server, and a user. First, the user records routine tasks. The recording is performed on the terminal operated by the user, and screen operations are recorded. All actions performed on the terminal (e.g., mouse clicks, keyboard input, screen capture) are saved in video format.

[1019] The recorded video is uploaded to a server. The server is equipped with a function to extract information from the recorded video. Specifically, the server uses OCR (optical character recognition) technology to analyze the text and data in the video and extract the necessary information. This process identifies the specific operating steps performed by the user.

[1020] Next, the user records an audio commentary about the task. The recording is done using the device's microphone, and the resulting audio file is uploaded to the server. The server analyzes the audio file and obtains the user's explanation as text data. This analysis is performed using voice recognition technology.

[1021] The server integrates the analyzed video and audio information to construct a detailed workflow. This workflow concretely shows the sequential steps of the user's operations. Based on this workflow, a procedure that can be automated is then generated. The server presents the generated procedure in list form and allows the user to confirm it.

[1022] The user can review the proposed procedure and make any necessary modifications, such as deleting unnecessary steps or adding additional comments. The optimized procedure is then saved back to the server.

[1023] Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, a series of operations can be performed automatically, such as opening a specific Excel file at 9:00 every morning, processing the data, and sending the results by email. This scheduled execution function frees users from routine tasks, allowing them to focus on more creative work.

[1024] As a concrete example, the following scenario can be considered.

[1025] A user's daily task is to process specific data in Excel and send the results to their boss by email. The user first records this series of operations. Next, they record an audio commentary about the work and upload it to the server. The server analyzes this data, generates an automatable procedure, and presents it to the user. After the user checks and modifies the procedure, the server schedules the procedure to be executed every morning at 9:00 AM. As a result, the user is freed from the hassle of manually processing and sending data every morning.

[1026] In this way, the system automates routine tasks, saving time and effort while also increasing the accuracy of the work.

[1027] The processing flow will be explained below.

[1028] Step 1:

[1029] The user uses the device to record screen activity. The device uses screen capture software to record all user activity (e.g., mouse clicks, keyboard input) and saves it as a video file.

[1030] Step 2:

[1031] The user stops recording and uploads the recorded video file from the terminal to the server, and the terminal transfers the video file to the server via the network.

[1032] Step 3:

[1033] The server receives the uploaded video file and uses OCR technology to analyze the text information and operation details in the video, extracting the analyzed information as text data.

[1034] Step 4:

[1035] The user uses the device to record an audio commentary of the work they are doing, and the recorded audio file is saved on the device.

[1036] Step 5:

[1037] A user uploads an audio file from a terminal to a server, and the terminal transfers the audio file to the server via a network.

[1038] Step 6:

[1039] The server receives the uploaded audio file and converts the audio content into text data using voice recognition technology. The text data obtained through voice analysis accurately reflects the user's commentary.

[1040] Step 7:

[1041] The server integrates the video and audio analysis data to construct a detailed workflow, which combines the user's operational steps with corresponding audio commentary.

[1042] Step 8:

[1043] The server generates automatable procedures based on the constructed workflow, and the generated procedures are organized in a list format and presented to the user.

[1044] Step 9:

[1045] The user checks the generated procedure and makes corrections or adjustments as necessary. The user uses the terminal to change the content of the procedure and arrange it in the optimal form.

[1046] Step 10:

[1047] The modified and adjusted procedures are saved back to the server, which stores these procedures in a database for scheduled execution.

[1048] Step 11:

[1049] A user uses a terminal to schedule an automated procedure to run, for example, at a specific time each day.

[1050] Step 12:

[1051] At the set time, the server executes the automated procedures according to the schedule. The server executes each procedure in sequence and automatically performs the necessary operations (e.g., processing data in Excel, sending emails).

[1052] Example 1

[1053] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1054] In modern business activities, many routine tasks occur on a daily basis. These tasks require time and effort, and there are issues with efficiency and accuracy. Conventional automation systems have difficulty automating a series of operations at once, and human intervention is required, especially for complex business processes. Therefore, reducing labor costs and improving work accuracy have become important issues.

[1055] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1056] In this invention, the server includes means for a user to record work on a terminal, means for uploading the recorded data to the server, means for the server to analyze the video, means for the user to record an audio commentary on the work content, means for uploading the recorded data to the server, means for the server to analyze the audio file, means for the server to integrate the video information and the audio information, means for the server to generate an automation procedure, means for the user to check and modify the procedure, and means for the server to schedule and execute the procedure. This makes it possible to automate routine business processes efficiently and with high accuracy.

[1057] A "user" is an operator who uses the system to automate routine tasks.

[1058] A "terminal" is a hardware device that a user uses to perform work, and includes a personal computer, a smartphone, etc.

[1059] "Recording" is the process of recording operations performed on a terminal in video format.

[1060] A "server" is a computer or cloud service that is the core of a system, and is a device that analyzes data and generates and executes procedures.

[1061] "Uploading" is the process of sending files or data generated on a device to a server.

[1062] "Video analysis" is the process of extracting text information and operating procedures from recorded video using OCR technology.

[1063] "Audio commentary" is information in which the user verbally explains the content of the work.

[1064] An "audio file" is a file that stores audio data recorded by a user.

[1065] "Speech analysis" is the process of obtaining text information from an audio file using speech recognition technology.

[1066] A "workflow" is information that indicates the sequential steps of a user's operation procedure, and is constructed by integrating video information and audio information.

[1067] An "automation procedure" is a procedure script created based on a workflow to automatically execute a series of operations.

[1068] "Procedure verification and correction" is the process in which the user verifies the automated procedures presented by the server and makes any necessary corrections.

[1069] "Scheduled execution" is the process of automatically executing a procedure at a specified time.

[1070] The present invention relates to a system for automating routine tasks, and involves a user, a terminal, and a server.

[1071] The system uses the following hardware and software:

[1072] Terminal: A device operated by a user, such as a computer or smartphone.

[1073] Servers: High-performance computers, data centers, and cloud services (e.g., AWS, Azure).

[1074] OCR technology: Software for extracting text data (e.g., Google Vision API).

[1075] Speech recognition technology: Software that converts speech into text (e.g., Google Speech-to-Text API).

[1076] Recording software: A recording tool that runs on your device (e.g., OBS Studio).

[1077] Audio recording software: A tool to record audio from your microphone input (e.g., Audacity).

[1078] As a specific example of operation, the process below shows how to automate a task in which a user processes specific data in Excel every day and sends the results to their supervisor by email.

[1079] Operating Procedure

[1080] 1. The user launches OBS Studio on their device and clicks the "Record" button to begin recording their work. All on-screen actions (e.g., opening Excel and entering data into specific cells) will be recorded.

[1081] 2. Once recording is complete, download the recording file from the OBS Studio file menu and upload it to the server using the dedicated upload form.

[1082] 3. The server uses the Google Vision API to analyze the uploaded video file and extract the cell contents in Excel and the user interface operation procedures as text data.

[1083] 4. Next, the user uses Audacity to record an audio description of the task, generating an audio file containing instructions such as, "To perform this task, first enter sales data into Excel cell A1."

[1084] 5. After recording is complete, the audio file is uploaded to the server, which uses the Google Speech-to-Text API to extract the text from the audio file.

[1085] 6. The server combines the information extracted from the video file and the information extracted from the audio file to construct a detailed workflow, which shows the steps taken by the user.

[1086] 7. The server generates automated procedures based on the workflow and presents them to the user in list form.

[1087] 8. The user reviews the proposed steps and modifies them as necessary, for example, by inserting additional steps or deleting unnecessary steps.

[1088] 9. Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, an Excel file is opened at 9:00 every morning, the data is processed, and the results are sent by email.

[1089] Specific examples

[1090] If a user wants to automate a daily task of processing certain data in Excel and emailing the results to their boss, they can use a prompt like this:

[1091] "I want an automated task that opens new sales data in Excel every morning, processes it into a specific format, and emails the results to my boss."

[1092] In this way, the system efficiently automates routine tasks, freeing users from routine tasks, saving time and effort, and improving work accuracy.

[1093] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1094] Step 1:

[1095] A user records their work on a device. To do this, the user launches recording software (e.g., OBS Studio) and clicks the "Record" button to begin recording their work. The input is the user's screen operations, and the output is a recorded video file. This video file records the user's operations (e.g., opening Excel, selecting cells, entering data, etc.).

[1096] Step 2:

[1097] The user uploads the recorded data to the server. The user retrieves the file from OBS Studio and sends the data to the server using a dedicated upload form. The input is the recorded video file, and the output is a video file saved on the server. Specifically, the user opens a browser, accesses the upload form, selects the recorded file, and completes the upload.

[1098] Step 3:

[1099] The server analyzes the video. It uses the Google Vision API to analyze the video file frame by frame and extracts text information from the video and user interface operation procedures. The input is the uploaded video file, and the output is text information and operation procedures. Specifically, the server loads the video file and runs OCR on each frame to extract the text data.

[1100] Step 4:

[1101] A user records an audio description of a task. The user launches Audacity and clicks the "Record" button to begin recording. The input is the user's verbal description, and the output is an audio file. Specifically, the user explains into the microphone, for example, "In this operation, first enter sales data into Excel cell A1," and then ends the recording.

[1102] Step 5:

[1103] The user uploads the recorded data to the server. The user sends the audio file saved in Audacity to the server using a dedicated upload form. The input is the recorded audio file, and the output is the audio file saved on the server. Specifically, the user opens a browser, accesses the upload form, selects the audio file, and completes the upload.

[1104] Step 6:

[1105] The server analyzes the audio file. The server converts the audio file into text using the Google Speech-to-Text API. The input is the uploaded audio file, and the output is text data. Specifically, the server loads the audio file, runs it through speech recognition, and extracts the text data.

[1106] Step 7:

[1107] The server integrates the video and audio information. The server then integrates the text data from the analyzed video frames with the text data obtained through speech recognition to construct a detailed workflow. The input is text and audio information, and the output is an integrated workflow. Specifically, the server analyzes the text information stored in the database and organizes it into operational steps to create a workflow.

[1108] Step 8:

[1109] The server generates the automation steps. The server generates a script based on the workflow to create an automatable step. The input is the detailed workflow, and the output is the automation step script. Specifically, the server uses a programming language (e.g., Python) to generate code to perform a specific operation.

[1110] Step 9:

[1111] The user checks and modifies the procedure. The user checks the automation procedure presented by the server and modifies it as necessary. The input is the automation procedure from the server, and the output is the modified procedure. Specifically, the user checks the procedure using an interface on the browser, deletes unnecessary steps, or inserts additional steps.

[1112] Step 10:

[1113] The server executes the procedure on a scheduled basis. The server executes the modified procedure based on the set schedule. The input is the modified automation procedure, and the output is the result of executing the automated operation. Specifically, the server executes the procedure at the set time, for example, opening an Excel file, processing the data, and sending the results by email.

[1114] (Application example 1)

[1115] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1116] Logistics centers have many routine tasks that need to be performed efficiently. In particular, routine tasks such as inventory checks, stocktaking, and picking require time and effort, and are prone to human error. Therefore, there is a need to automate these tasks to improve efficiency and accuracy.

[1117] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1118] In this invention, the server includes means for recording screen operations, means for extracting information from the recorded video, means for analyzing the worker's audio commentary, means for building a workflow by integrating the extracted information and the analyzed audio information, means for generating procedures based on the workflow, means for the worker to adjust the generated procedures, means for scheduling and executing the adjusted procedures, means for visually recording the work procedures using a head-mounted display, means for extracting text data from the recorded video using optical character recognition technology, and means for converting the audio commentary into text using voice recognition technology, thereby enabling the automation and efficiency of routine tasks in a logistics center.

[1119] The "means for recording screen operations" refers to a means for recording the operations performed by the user on the screen, such as mouse clicks and keyboard input, as video.

[1120] "Means for extracting information from recorded video" refers to means for extracting necessary data or text information from recorded video.

[1121] The "means for analyzing the voice commentary of the worker" is a means for analyzing the voice recorded by the worker and acquiring the content as text data.

[1122] The "means for integrating the extracted information and analyzed voice information to construct a workflow" is a means for combining information extracted from the video with the voice recognition results to create a workflow that details the steps of the work.

[1123] The "means for generating a procedure based on the workflow" refers to a means for generating an automatable procedure based on the constructed workflow.

[1124] The "means for the worker to adjust the generated procedure" refers to a means for the user to check the generated procedure, delete unnecessary steps, and add additional comments.

[1125] The "means for scheduling and executing the adjusted procedure" is a means for automatically executing the adjusted procedure at a time designated by the user.

[1126] "Means for visually recording work procedures using a head-mounted display" refers to a means in which a worker wears a head-mounted display and records the work procedures from that viewpoint.

[1127] The "means for extracting text data from recorded video using optical character recognition technology" refers to a means for extracting character information from recorded video frames using OCR technology and obtaining it as text data.

[1128] "Means for converting audio commentary into text using speech recognition technology" refers to means for analyzing recorded audio commentary using speech recognition technology and converting it into text data.

[1129] The present invention is a system for automating routine tasks in a logistics center. The system includes a means for recording screen operations, a means for extracting information from recorded video, a means for analyzing audio commentary by workers, a means for integrating the extracted information and the analyzed audio information to create a workflow, a means for workers to adjust the generated procedures, a means for scheduling and executing the adjusted procedures, a means for visually recording the work procedures using a head-mounted display, a means for extracting text data from the recorded video using OCR technology, and a means for converting the audio commentary into text using speech recognition technology.

[1130] Hardware and Software Configuration

[1131] Hardware

[1132] Head-mounted display (HMD): A device worn by a worker to record work procedures from their point of view.

[1133] Terminal (smartphone or tablet): A device for collecting video and audio data and uploading it to a server.

[1134] Server: A computer with powerful processing power that analyzes information and generates procedures.

[1135] software

[1136] OCR Technology: PaddleOCR is used to extract text data from recorded video frames.

[1137] Speech Recognition Technology: Technology for converting recorded audio commentary into text data using the SpeechRecognition library.

[1138] Scheduling library: A system for executing automated procedures at user-specified times using the schedule library.

[1139] Operation overview

[1140] First, the user puts on the head-mounted display and records the procedure while performing routine tasks at a logistics center (e.g., inventory checks, stocktaking, picking). Once the recording is complete, an audio commentary on the procedure is recorded. This data is then uploaded to the server via the device.

[1141] The server first extracts the necessary information from the video data using OCR technology. In parallel, it converts the recorded audio commentary into text data using speech recognition technology. It then integrates the extracted information with the commentary text to construct a detailed workflow. Based on the constructed workflow, automatable steps are generated and presented to the user.

[1142] The user can review the presented procedures and adjust them by deleting unnecessary steps or adding comments. The adjusted procedures are then saved back to the server. Finally, the server automatically executes the adjusted procedures based on a schedule. For example, an inventory check can be automatically performed every morning at 9:00 and the results sent to the person in charge via email.

[1143] Examples of concrete examples and prompts

[1144] Specific examples

[1145] This process involves automatically checking inventory every morning at 9:00 and sending the results to the person in charge via email. The person wearing the head-mounted display records their viewpoint, and the data with recorded audio commentary is uploaded to the server. The server analyzes this data, generates and adjusts the automated procedures, and executes them at the specified time.

[1146] Prompt Sentence Examples

[1147] Example: Write a Python program to automate inventory checks at a distribution center every morning at 9:00. The program should include video recording, voice recognition, OCR analysis, and schedule execution.

[1148] Note: A prompt statement that automates logistics center operations based on the system described in the invention.

[1149] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1150] Step 1:

[1151] Data recording

[1152] A user wears a head-mounted display and performs routine tasks in a logistics center. The head-mounted display records the work procedure on video. The user also records audio commentary about the work procedure. This allows visual and audio information to be collected.

[1153] Input: User instructions and audio description

[1154] Output: Video and audio files

[1155] Step 2:

[1156] Uploading data

[1157] The device uploads the recorded video and audio files to a server, an automated operation that is performed through the device's interface.

[1158] Input: Video and audio files

[1159] Output: Video and audio data on the server

[1160] Step 3:

[1161] Data extraction with OCR

[1162] The server analyzes the uploaded video data and extracts the necessary information using optical character recognition (OCR) technology. Specifically, it identifies and extracts text data from video frames. PaddleOCR is used.

[1163] Input: Video data on the server

[1164] Output: Extracted text data

[1165] Step 4:

[1166] Data analysis using voice recognition

[1167] The server analyzes the uploaded audio data and converts the audio commentary into text data using speech recognition technology, primarily using the SpeechRecognition library.

[1168] Input: Audio data on the server

[1169] Output: Converted text data

[1170] Step 5:

[1171] Integrating information and building workflows

[1172] The server integrates the text data obtained from OCR and speech recognition to construct a detailed workflow, which is based on the user's specific operating steps and audio commentary.

[1173] Input: Extracted and converted text data

[1174] Output: The completed workflow

[1175] Step 6:

[1176] Generating automation procedures

[1177] The server generates automatable procedures based on the constructed workflow, and presents the procedures to the user as a list of steps.

[1178] Input: Completed Workflow

[1179] Output: Generated list of automation steps

[1180] Step 7:

[1181] Adjusting procedures

[1182] The user can review the generated automated procedure and modify it as needed, for example by deleting unnecessary steps or adding additional comments. After this adjustment, the procedure is saved back to the server.

[1183] Input: Generated list of automated procedures

[1184] Output: Modified list of automated steps

[1185] Step 8:

[1186] Schedule execution

[1187] The server schedules and executes the adjusted procedures at the specified time. For example, you can automatically execute an inventory check at 9:00 AM every day and send the results to the person in charge by email.

[1188] Input: Modified automated procedure list

[1189] Output: The results of the automated procedure that was executed (e.g., an inventory checklist)

[1190] Examples of concrete examples and prompts

[1191] Specific examples

[1192] This process involves automatically checking inventory every morning at 9:00 and sending the results to the person in charge via email. The person wearing the head-mounted display records their viewpoint, and the data with recorded audio commentary is uploaded to the server. The server analyzes this data, generates and adjusts the automated procedures, and executes them at the specified time.

[1193] Prompt Sentence Examples

[1194] Example: Write a Python program to automate inventory checks at a distribution center every morning at 9:00. The program should include video recording, voice recognition, OCR analysis, and schedule execution.

[1195] Note: A prompt statement that automates logistics center operations based on the system described in the invention.

[1196] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1197] The present invention relates to a system for automating routine tasks, and also combines it with an emotion engine that recognizes and reflects the user's emotions. It can be implemented based on the following settings.

[1198] This system involves a terminal, a server, and a user. First, the user records routine tasks. The recording is performed on the terminal operated by the user, and screen operations are recorded. All actions performed on the terminal (e.g., mouse clicks, keyboard input, screen capture) are saved in video format.

[1199] The recorded video is uploaded to a server. The server is equipped with a function to extract information from the recorded video. Specifically, the server uses OCR (optical character recognition) technology to analyze the text and data in the video and extract the necessary information. This process identifies the specific operating steps performed by the user.

[1200] Next, the user records an audio commentary about the task. The recording is done using the device's microphone, and the resulting audio file is uploaded to the server. The server analyzes the audio file and obtains the user's explanation as text data. This analysis is performed using voice recognition technology.

[1201] In addition, the system uses an emotion engine to recognize the user's emotions. Emotional information is acquired by analyzing the user's facial expressions and tone of voice using the device's camera and microphone. This emotion engine can grasp the user's emotions in real time while they are operating the device.

[1202] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. This workflow concretely shows the sequential steps of the user's operations, and by incorporating emotional information, it provides a procedure that takes into account the user's stress and motivation.

[1203] Furthermore, a procedure that can be automated is generated based on this workflow. The server presents the generated procedure in list form and allows the user to review it. The user can review the presented procedure and make corrections as necessary. For example, they can delete unnecessary steps or add additional comments. The optimized procedure is then saved back to the server.

[1204] Finally, the modified procedures are scheduled and executed by the server. The automated procedures are executed sequentially at the time specified by the user. For example, a series of operations can be performed automatically, such as opening a specific Excel file at 9:00 every morning, processing the data, and sending the results by email. This scheduled execution function frees users from routine tasks, allowing them to focus on more creative work.

[1205] As a concrete example, the following scenario can be considered.

[1206] A user's daily work involves processing specific data in Excel and emailing the results to their boss. The user first records this series of operations. Next, they record an audio commentary about the work and upload it to the server. The server analyzes this data, generates automatable steps, and presents them to the user. Furthermore, if the emotion engine detects that the user's stress level is high, it suggests modifications to simplify some of the steps. After the user checks the content and makes modifications, the server schedules this procedure to be executed every morning at 9:00 AM. As a result, the user is freed from the hassle of manually processing and sending data every morning.

[1207] In this way, the system automates routine tasks, saving time and effort, and improves the quality of tasks by taking into account the user's emotions.

[1208] The processing flow will be explained below.

[1209] Step 1:

[1210] The user uses the device to record screen activity. The device uses screen capture software to record all user activity (e.g., mouse clicks, keyboard input) and saves it as a video file.

[1211] Step 2:

[1212] The user stops recording and uploads the recorded video file from the terminal to the server, and the terminal transfers the video file to the server via the network.

[1213] Step 3:

[1214] The server receives the uploaded video file and uses OCR technology to analyze the text information and operation details in the video, extracting the analyzed information as text data.

[1215] Step 4:

[1216] The user uses the device to record an audio commentary of the work they are doing, and the recorded audio file is saved on the device.

[1217] Step 5:

[1218] A user uploads an audio file from a terminal to a server, and the terminal transfers the audio file to the server via a network.

[1219] Step 6:

[1220] The server receives the uploaded audio file and converts the audio content into text data using voice recognition technology. The text data obtained through voice analysis accurately reflects the user's commentary.

[1221] Step 7:

[1222] The user uses the device's camera and microphone to collect emotional data while working. The device then analyzes the user's facial expressions and tone of voice in real time and obtains emotional information through an emotion engine.

[1223] Step 8:

[1224] The server integrates the video analysis data, audio analysis data, and emotional information to construct a detailed workflow, which reflects the user's operation steps and the corresponding audio commentary and emotional state.

[1225] Step 9:

[1226] The server generates automatable procedures based on the constructed workflow, and the generated procedures are organized in a list format and presented to the user.

[1227] Step 10:

[1228] The user can review the generated instructions and make corrections or adjustments as necessary. The user can then use their device to modify the content of the instructions, taking into account feedback from the emotion engine, to optimize them.

[1229] Step 11:

[1230] The modified and adjusted procedures are saved back to the server, which stores these procedures in a database for scheduled execution.

[1231] Step 12:

[1232] A user uses a terminal to schedule an automated procedure to run, for example, at a specific time each day.

[1233] Step 13:

[1234] At the set time, the server executes the automated procedures according to the schedule. The server executes each procedure in sequence and automatically performs the necessary operations (e.g., processing data in Excel, sending emails).

[1235] Example 2

[1236] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1237] Conventional automation systems for routine tasks can automate user work procedures, but they are unable to consider the user's emotions, which often results in inefficient automation. Furthermore, because they are unable to consider the user's stress or motivation, it is difficult to improve the quality of work.

[1238] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1239] In this invention, the server includes means for recording screen operations, means for extracting information from the recorded video, means for analyzing the worker's audio commentary, means for recognizing emotions, means for constructing a workflow by integrating the extracted information with the analyzed audio information and emotional information, means for generating a procedure based on the workflow, means for the worker to adjust the generated procedure, and means for scheduling and executing the adjusted procedure, thereby enabling the efficiency and quality of work performed by the user to be improved.

[1240] The "means for recording screen operations" refers to a device or program that records the operations performed by a user on the screen in video format.

[1241] "Means for extracting information from recorded video" refers to devices or programs that use OCR technology or other methods to analyze on-screen text and data from recorded video and obtain the necessary information.

[1242] The "means for analyzing the worker's audio commentary" refers to a device or program that converts the audio file recorded by the worker into text data using voice recognition technology and analyzes the content.

[1243] "Means for recognizing emotions" refers to a device or program that uses the device's camera or microphone to analyze the user's facial expressions and tone of voice and identify the user's emotional state.

[1244] The "means for constructing a workflow by integrating the extracted information with the analyzed audio information and emotional information" refers to a device or program that integrates the results of video analysis, audio analysis, and emotional analysis to organize the user's work procedures in chronological order and construct a detailed workflow.

[1245] The "means for generating a procedure based on the workflow" refers to a device or program that generates an automatable procedure based on the constructed workflow.

[1246] The "means for an operator to adjust the generated procedure" refers to an interface or device that allows a user to check the generated automated procedure and make corrections or additions as necessary.

[1247] The "means for scheduling and executing the adjusted procedure" refers to a device or program with a scheduling function that automatically executes the procedure modified by the user at a specified time.

[1248] The present invention relates to a system that automates routine tasks and recognizes and reflects the user's emotions. This system involves a terminal, a server, and a user. A specific embodiment of the system is shown below.

[1249] First, the user starts a routine task. The device launches screen recording software (e.g., OBS Studio) and records the user's screen operations in video format. Specific actions recorded include mouse clicks, keyboard input, and window switching. All on-screen actions are then saved as a video file (e.g., .mp4 format).

[1250] Next, the device uploads the recorded video file to the server. This can be done by dragging and dropping it from a file browser. The server receives the uploaded file and analyzes the text and data in the video using OCR (Optical Character Recognition) technology (e.g., Google Cloud Vision API). Specifically, it analyzes the video frames and extracts text and numerical data.

[1251] The user provides an audio description of the work they are doing. This audio description explains the steps and purpose of the work, and includes specific content such as "Enter sales data into this cell and click this button to create a graph." The device records the audio through a microphone and uploads the audio file (e.g., .wav format) to the server. The server receives the audio file and converts it into text data using voice recognition technology (e.g., Google Cloud Speech-to-Text). Specifically, the audio data is analyzed and corresponding text information is generated.

[1252] Furthermore, the device uses a camera and microphone to capture the user's facial expressions and tone of voice in real time. For example, it records the moment when the user smiles or their voice tone becomes lower. The server analyzes this data using an emotion engine (e.g., Microsoft Azure Emotion API) to identify the user's emotional state. For example, it obtains information such as "the user is feeling stressed."

[1253] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. Specifically, it organizes the work content for each step and the corresponding emotional data in chronological order. This workflow specifically shows the sequential steps of the user's operations, and by incorporating emotional information, it provides a procedure that takes into account the user's stress and motivation.

[1254] The server then presents a list of automatable steps for the user to review. For example, the steps are displayed sequentially through a web interface. The user reviews the presented workflow and makes corrections as necessary. Specific actions include deleting unnecessary steps and adding new comments. The user can also review the generated steps and make corrections or additions, and the optimized steps are then saved back to the server.

[1255] Finally, the server schedules the revised procedure to run at a specified time. Specific actions include automatically executing a series of operations, such as "opening an Excel file at 9:00 every morning, processing specific data, and sending the results by email." This frees users from this routine work, allowing them to focus on more creative tasks.

[1256] Prompt Sentence Examples

[1257] "I want to automate the daily task of opening an Excel file, processing the data, and sending it to my boss by email. I've recorded the steps and provided audio explanations of the work. Please automate this procedure in the system and set it to run every morning at 9:00."

[1258] In this way, this system can automate routine tasks and significantly improve the efficiency and quality of work for users. Furthermore, by taking into account the user's emotions, it reduces stress and improves the comfort and effectiveness of work.

[1259] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1260] Step 1:

[1261] The user starts a routine task. The device launches the screen recording software, which records the user's screen operations in video format. Specifically, mouse clicks, keyboard input, window switching, etc. are recorded. The input is the user's screen operations, and the output is a video file (e.g., .mp4 format).

[1262] Step 2:

[1263] The device uploads the recorded video file to the server. The user uploads it by dragging and dropping it from the file browser. The input is the video file, and the output is a file saved on the server. Specifically, the user selects the video file and clicks the upload button.

[1264] Step 3:

[1265] The server receives the uploaded video file. It uses OCR technology to analyze the characters and data in the video and obtain the necessary information. The input is the video file, and the output is the analyzed text information. Specifically, OCR technology recognizes characters for each frame of the video and generates text data.

[1266] Step 4:

[1267] The user gives an audio commentary about the work they are doing. The device uses a microphone to record the audio and uploads the audio file (e.g., .wav format) to the server. The input is the user's voice and the output is an audio file. Specifically, the user presses the record button and explains the work they are doing.

[1268] Step 5:

[1269] The server receives the uploaded audio file and converts it into text data using speech recognition technology. The input is the audio file and the output is text data. Specifically, the audio data is analyzed and the corresponding text information is generated.

[1270] Step 6:

[1271] The device uses a camera and microphone to capture the user's facial expressions and tone of voice in real time. The server uses an emotion engine to analyze this data and identify the user's emotional state. The input is the user's facial expressions and tone of voice, and the output is data on the user's emotional state. Specifically, it uses facial expression recognition algorithms and voice tone analysis technology.

[1272] Step 7:

[1273] The server integrates the analyzed video, audio, and emotional information to construct a detailed workflow. The input is the analyzed data, and the output is the integrated workflow. Specifically, the work content for each step and the corresponding emotional data are organized in chronological order.

[1274] Step 8:

[1275] The server presents a list of automatable steps to the user for review. The input is the workflow, and the output is a list of steps presented to the user, with each step displayed sequentially through a web interface.

[1276] Step 9:

[1277] The user checks the presented workflow and makes corrections as necessary. The input is a list of steps, and the output is the corrected steps. Specifically, the user deletes unnecessary steps and adds new comments.

[1278] Step 10:

[1279] The server re-saves the modified procedure. The input is the modified procedure, and the output is the procedure data saved on the server. Specifically, the procedure data that reflects the user's modification information is saved on the server.

[1280] Step 11:

[1281] The server schedules the execution of the modified procedure at a specified time. The input is the modified procedure data, and the output is the scheduled execution of the work. Specifically, for example, a series of operations such as "open an Excel file every morning at 9:00, process specific data, and send the results by email" is automatically executed.

[1282] In this way, the system can automate routine tasks, improve efficiency and work quality, and provide a flexible work environment that takes users' emotions into consideration.

[1283] (Application example 2)

[1284] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1285] In logistics centers, employees perform a large number of routine tasks, which not only consume a significant amount of time and effort, but also cause stress and fatigue in workers. Furthermore, because employees' emotional states have a direct impact on work efficiency and quality, it is important to develop an optimized workflow that takes into account employees' emotions. Therefore, there is a need to automate routine tasks while providing a workflow that reflects the emotions of workers.

[1286] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording screen operations, means for extracting information from recorded video, means for analyzing the worker's audio commentary, means for building a workflow by integrating the extracted information and the analyzed audio information, means for generating procedures based on the workflow, means for the worker to adjust the generated procedures, means for scheduling and executing the adjusted procedures, and means for analyzing the worker's emotions and reflecting them in the workflow. This makes it possible to efficiently automate employees' routine tasks and provide an optimal workflow that takes employees' emotions into consideration.

[1287] The "means for recording screen operations" refers to a device or software that records the screen operations performed by the worker in video format.

[1288] "Means for extracting information from recorded video" refers to technology or equipment that has the ability to analyze and extract useful text or data from recorded video.

[1289] The "means for analyzing the worker's voice commentary" is a voice recognition technology that converts the worker's voice commentary into text data.

[1290] The "means for constructing a workflow by integrating the extracted information and analyzed audio information" is an algorithm that integrates video and audio data to generate detailed work procedures.

[1291] The "means for generating a procedure based on the workflow" is a system that generates an automatable procedure based on the constructed workflow.

[1292] "Means for operators to adjust the generated procedures" refers to interfaces and tools that allow operators to check and correct the automated procedures.

[1293] The "means for scheduling and executing the adjusted procedure" is a function for automatically executing the procedure adjusted by the worker at a specified time.

[1294] "Means for analyzing workers' emotions and reflecting them in the workflow" refers to algorithms and technologies that recognize and analyze the emotional state of workers in real time and reflect the results in the workflow.

[1295] The present invention relates to an automated system that improves the efficiency of routine work in a logistics center and takes into account the emotions of workers. This system is implemented in the following manner.

[1296] First, capture software and a webcam are used to record terminal operations. While workers perform routine tasks (e.g., product picking and packing) at the distribution center on the terminal screen, the operations are recorded in video format. This recorded video is then uploaded to a server.

[1297] The server uses OCR (optical character recognition) technology (such as pytesseract) to extract text and data from the recorded video. The server also uses speech recognition technology (such as Google Speech Recognition) to convert the operator's audio commentary into text. This allows the operation procedures and their explanations to be obtained as data.

[1298] The server then uses an emotion engine to analyze the worker's emotions. This process involves using the device's camera and microphone to analyze the worker's facial expressions and tone of voice in real time. The emotion engine uses technology to extract features from voice data and classify emotions using an SVM model.

[1299] The server integrates the analyzed video, audio, and emotional information to construct a workflow. This workflow concretely shows the sequential steps of the user's operations and incorporates emotional information to provide a procedure that takes into account the user's stress and motivation.

[1300] Based on this workflow, a procedure that can be automated is generated. The server presents the generated procedure in list form to the worker, who can then modify it as needed. For example, it is possible to delete unnecessary steps or add additional comments. Once the procedure has been modified, it is saved on the server and automatically executed at the scheduled time.

[1301] As a concrete example, consider a task in which a worker processes specific data every day and reports the results. The worker first records this series of operations, and then records an audio commentary of the operations. The video and audio are uploaded to a server for analysis. If the emotion engine detects that the worker is in a high stress state, it suggests simplifying some of the steps. After the worker reviews and makes corrections, the procedure is set to be executed automatically every morning at 9:00.

[1302] Below is an example of a prompt sentence to input to the generative AI model.

[1303] Example prompt for a generative AI model:

[1304] You are designing a system to improve the work efficiency of employees at a distribution center. Code the following steps in Python:

[1305] 1. Record and save the work steps.

[1306] 2. Record and save an audio description of your work.

[1307] 3. Extract text data from the recorded video using OCR.

[1308] 4. Convert the recorded audio into text using speech recognition.

[1309] 5. Analyze emotional information using an emotion engine and optimize work procedures accordingly.

[1310] 6. Set up a schedule to automatically execute tasks at specific times.

[1311] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1312] Step 1:

[1313] The routine tasks performed by users at the logistics center are recorded using capture software and a webcam on the terminal. Specifically, the user performs work steps on the terminal, and the operations are saved in video format. This recorded data becomes the input.

[1314] Step 2:

[1315] The recorded video data is uploaded to a server, where the server uses OCR technology (such as pytesseract) to extract text and data from the video. The input is the recorded data, and the output is a list of text and data. This data extraction identifies specific operating procedures and tool usage.

[1316] Step 3:

[1317] The user uses the microphone on their device to record an audio commentary of their work. The recorded audio file is uploaded to the server. This audio file is used as input and converted into text data using speech recognition technology (such as Google Speech Recognition). The output is the text data of the audio commentary.

[1318] Step 4:

[1319] The server uses an emotion engine to analyze the user's emotions. Features are extracted from the user's facial expression data and tone of voice data acquired from the device's camera and microphone. These features are then fed into an SVM model to classify emotions. The output is the user's emotional information.

[1320] Step 5:

[1321] The server integrates the information extracted in step 2, the text data of the audio commentary obtained in step 3, and the emotional information analyzed in step 4 to construct a workflow. Based on this integrated data, a detailed workflow is created that incorporates the specific operational steps performed by the user and their emotional information. The output is the integrated workflow.

[1322] Step 6:

[1323] The server generates automatable steps based on the constructed workflow. The input is the integrated workflow, and the output is a list of automatable steps, which lists tasks to be performed at specific times.

[1324] Step 7:

[1325] The generated procedure list is presented to the user, who then checks and modifies it. At this stage, necessary adjustments are made, such as deleting unnecessary steps or adding additional comments. The input is the pre-generated procedure list, and the output is the procedure list modified by the user.

[1326] Step 8:

[1327] The server executes the procedures modified by the user based on a schedule. This procedure list is registered in the schedule and automatically executed at the specified time. The input is the modified procedure list and schedule information, and the output is the execution result of the automatically executed task.

[1328] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1329] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1330] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1331] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1332] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1333] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1334] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1335] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1336] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1337] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1338] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1339] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1340] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1341] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1342] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1343] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1344] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1345] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1346] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1347] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1348] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1349] The following is further disclosed regarding the above embodiment.

[1350] (Claim 1)

[1351] A means for recording screen operations;

[1352] means for extracting information from the recorded video;

[1353] means for analyzing the worker's audio commentary;

[1354] a means for integrating the extracted information and the analyzed voice information to construct a workflow;

[1355] means for generating a procedure based on the workflow;

[1356] A means for an operator to adjust the generated procedure;

[1357] means for scheduling and executing the coordinated procedure;

[1358] A system including:

[1359] (Claim 2)

[1360] 10. The system of claim 1, further comprising means for identifying at least one tool usage situation from the recorded video.

[1361] (Claim 3)

[1362] 10. The system of claim 1, further comprising means for automatically executing the scheduled procedures at fixed times.

[1363] "Example 1"

[1364] (Claim 1)

[1365] A means for a user to record their work on the device;

[1366] A means for uploading the recording data to the server;

[1367] A means for the server to analyze the video;

[1368] means for a user to record an audio description of the work;

[1369] A means for uploading the recording data to the server;

[1370] a means for the server to analyze the audio file;

[1371] a means for the server to integrate the video information and the audio information;

[1372] a means for the server to generate the automated procedure;

[1373] a means for the user to review and correct the procedure;

[1374] a means for the server to schedule and execute the procedure;

[1375] A system including:

[1376] (Claim 2)

[1377] 10. The system of claim 1, further comprising means for identifying at least one tool usage situation from the recorded video.

[1378] (Claim 3)

[1379] 10. The system of claim 1, further comprising means for automatically executing the scheduled procedures at fixed times.

[1380] "Application Example 1"

[1381] (Claim 1)

[1382] A means for recording screen operations;

[1383] means for extracting information from the recorded video;

[1384] means for analyzing the worker's audio commentary;

[1385] a means for integrating the extracted information and the analyzed voice information to construct a workflow;

[1386] means for generating a procedure based on the workflow;

[1387] A means for an operator to adjust the generated procedure;

[1388] means for scheduling and executing the coordinated procedure;

[1389] a means for visually recording work procedures using a head-mounted display;

[1390] means for extracting text data from the recorded video using optical character recognition technology;

[1391] a means for converting the audio description into text using speech recognition technology;

[1392] A system including:

[1393] (Claim 2)

[1394] 10. The system of claim 1, further comprising means for identifying at least one tool usage situation from the recorded video.

[1395] (Claim 3)

[1396] 10. The system of claim 1, further comprising means for automatically executing the scheduled procedures at fixed times.

[1397] "Example 2: Combining Emotion Engines"

[1398] (Claim 1)

[1399] A means for recording screen operations;

[1400] means for extracting information from the recorded video;

[1401] means for analyzing the worker's audio commentary;

[1402] A means of recognizing emotions;

[1403] a means for integrating the extracted information with the analyzed voice information and emotion information to construct a workflow;

[1404] means for generating a procedure based on the workflow;

[1405] A means for an operator to adjust the generated procedure;

[1406] means for scheduling and executing the coordinated procedure;

[1407] A system including:

[1408] (Claim 2)

[1409] 10. The system of claim 1, further comprising means for identifying at least one tool usage situation from the recorded video.

[1410] (Claim 3)

[1411] 10. The system of claim 1, further comprising means for automatically executing the scheduled procedures at fixed times.

[1412] "Application example 2 when combining emotion engines"

[1413] (Claim 1)

[1414] A means for recording screen operations;

[1415] means for extracting information from the recorded video;

[1416] means for analyzing the worker's audio commentary;

[1417] a means for integrating the extracted information and the analyzed voice information to construct a workflow;

[1418] means for generating a procedure based on the workflow;

[1419] A means for an operator to adjust the generated procedure;

[1420] means for scheduling and executing the coordinated procedure;

[1421] A means of analyzing the emotions of workers and reflecting them in the work flow;

[1422] A system including:

[1423] (Claim 2)

[1424] 10. The system of claim 1, further comprising means for identifying at least one tool usage situation from the recorded video.

[1425] (Claim 3)

[1426] 10. The system of claim 1, further comprising means for automatically executing the scheduled procedures at fixed times. [Explanation of symbols]

[1427] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for recording screen operations; means for extracting information from the recorded video; means for analyzing the worker's audio commentary; a means for integrating the extracted information and the analyzed voice information to construct a workflow; means for generating a procedure based on the workflow; A means for an operator to adjust the generated procedure; means for scheduling and executing the coordinated procedure; A system including:

2. 10. The system of claim 1, further comprising means for identifying at least one tool usage situation from the recorded video.

3. 10. The system of claim 1, further comprising means for automatically executing the scheduled procedures at fixed times.

Images