System

Abstract

To provide a system capable of maintaining sustainable knowledge of an organization by easily and continuously creating and updating a manual.SOLUTION: Means for collecting work-related interview voice or work video, means for uploading a voice file or a video file to a system, means for performing voice recognition processing on the voice or the video file and converting the voice or the video file into text data, means for analyzing the text data and extracting information necessary for a work process, means for organizing the extracted information and classifying the information for each item, means for automatically generating a manual document on the basis of the classified data, means for displaying the manual document and allowing a user to confirm and correct contents, and means for generating a new version each time the manual is updated, this system includes a means for storing and managing a change history by performing comparison with the past version, a means for periodically confirming the updated manual and receiving feedback, and a means for correcting information on the basis of the received feedback and generating the manual again.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Standardizing business processes and procedures is known to contribute to efficiency and quality improvement, but the reality is that due to the demands of daily work, thorough implementation is often not possible. Furthermore, creating and maintaining a high-quality manual requires a significant amount of man-hours, and it is often neglected. As a result, there is a high risk that the value of organizational knowledge will be lost. Therefore, there is a need for a method to easily and continuously create and update manuals and maintain sustainable organizational knowledge. [Means for solving the problem]

[0005] The present invention solves the above problems by providing a means for collecting work-related interview audio or work video, a means for uploading collected audio and video files to a system, a means for receiving and saving uploaded audio and video files, a means for subjecting received audio and video files to speech recognition processing to convert them into text data, a means for analyzing the text data and extracting information necessary for the business process, a means for organizing the extracted information and classifying it by item, a means for automatically generating a manual document based on the classified data, a means for displaying the manual document and allowing the user to check and modify its contents, a means for generating a new version each time the manual is updated and comparing it with previous versions, a means for saving and managing the change history, a means for periodically checking the updated manual and receiving feedback, and a means for modifying information based on the received feedback and regenerating the manual.

[0006] "Work-related audio interviews or video work" refers to audio or video files collected as part of a work process, which contain interview-style explanations or actual work procedures.

[0007] "Means for uploading to the system" refers to a function that allows users to send audio and video files collected by the user to the system via the Internet and store them.

[0008] "Means for receiving and storing" refers to the function by which the system receives audio and video files uploaded by users and stores them in storage.

[0009] "Means of subjecting voice data to speech recognition processing and converting it into text data" refers to the process of analyzing voice data and converting the voice content into text information.

[0010] "Means of analyzing text data and extracting information necessary for business processes" refers to the process of using natural language processing (NLP) technology to extract information about business processes, such as input, output, tools used, procedures, required skills, and KPIs, from text data.

[0011] "Means for organizing extracted information and classifying it into each item" refers to the process of structuring the analyzed information and organizing it according to specific categories or items.

[0012] "Means for automatically generating manual documents" refers to the process of embedding organized and classified business information into a pre-designed template and generating it as a manual document.

[0013] "Means for displaying the manual document and allowing the user to check and modify the contents" is a function that visually provides the generated manual document to the user, allowing the user to check the contents and modify them as necessary.

[0014] The "means of generating a new version and comparing it with the previous version" is the process of comparing the old and new versions and recording the changes when the manual is updated.

[0015] "Means for saving and managing change history" refers to the process of saving past manual versions and their changes in a database and continuously managing them.

[0016] "Means for receiving feedback" is a function for collecting opinions and requests for corrections from users.

[0017] "Means for regenerating manuals" refers to the process of revising information based on feedback and regenerating new manual documents. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. This system provides a means to generate business manuals by collecting, analyzing, and processing work-related interview audio and video footage.

[0040] Overall system configuration

[0041] This system mainly consists of the following three entities: users, servers, and terminals.

[0042] User

[0043] Users are business personnel and administrators who are responsible for operating the system and providing data. Specifically, they upload work-related interview audio and work videos to the system and check and edit the content.

[0044] server

[0045] The server is the core of the system, and performs voice recognition processing, data analysis, manual generation, and maintenance. The server processes the audio and video data provided by the user and converts it into text data. It then analyzes the text data, extracts and classifies the information required for the business process, and automatically generates manual documents.

[0046] Terminal

[0047] The terminal provides an interface for users to operate the system, displays the generated manual documents to users, and supports them in checking and correcting the contents. It also supports users in easily submitting correction requests through a feedback function.

[0048] Program processing overview

[0049] 1. Audio and Video Collection and Upload

[0050] User: Collects interview audio explaining business processes and actual work videos and uploads them to the system. To do this, the user uses the terminal interface.

[0051] 2. Data Conversion and Analysis

[0052] Server: Receives uploaded audio and video files and converts them into text using speech recognition technology. Based on this text data, natural language processing technology is used to analyze the information required for the job and classify it into categories such as input, output, tools used, procedures, required skills, and KPIs.

[0053] 3. Manual Generation

[0054] Server: Automatically generates manual documents based on the classified information. The manual documents are formatted in a specified format and saved in the specified format (e.g., PDF or Word file).

[0055] 4. Viewing and Modifying the Manual

[0056] Terminal: Provides an interface that displays the generated manual document to the user. Through this interface, the user can check the generated manual and manually correct it if necessary.

[0057] 5. Feedback and Maintenance

[0058] Users: Check that the manual is generated correctly and provide feedback on any necessary corrections or additions. Feedback can be easily provided using the device's functions.

[0059] Server: Receives feedback from users and corrects and updates the manual content accordingly. It also stores and manages the update history and compares the latest version with previous versions.

[0060] Specific examples

[0061] For example, when creating a manual for how to use a new software tool, the user collects and uploads audio files and operation videos that explain the software's operation procedures. The server receives these files and converts them into text using speech recognition processing. Next, natural language processing is used to extract and classify information such as software installation procedures, explanations of key functions, specific operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates a manual document, which the terminal displays to the user. The user then reviews the content and makes any necessary corrections to finalize the manual.

[0062] This allows for efficient standardization of business processes and procedures, maintains organizational knowledge in a sustainable manner, and allows users to provide up-to-date information, contributing to improved efficiency and quality.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] User: To collect job-related audio or video interviews, record workers performing tasks or interviews explaining work procedures.

[0066] Step 2:

[0067] User: To upload collected audio and video files to the system, the user uses the device interface to select the files and clicks the upload button.

[0068] Step 3:

[0069] Server: Receives uploaded audio and video files and saves them to storage. Verifies that the files have been saved successfully.

[0070] Step 4:

[0071] Server: Starts the speech recognition module and converts the received voice data into text data. In the case of video, extracts the audio portion and performs the same process. Saves the conversion result as a text file.

[0072] Step 5:

[0073] Server: Using natural language processing (NLP) technology, the server extracts information necessary for business processes from the converted text data. Specifically, it identifies information such as inputs, outputs, tools used, procedures, required skills, and KPIs, and stores it in a database.

[0074] Step 6:

[0075] Server: Automatically generates manual documents based on the extracted information. It embeds the classified data into a document template, generates the document in the specified format (e.g., PDF, Word), and saves it in storage.

[0076] Step 7:

[0077] Terminal: Displays the generated manual document to the user, who can check the manual contents through the interface.

[0078] Step 8:

[0079] User: Check the contents of the displayed manual document and make any necessary corrections. Enter the corrections in the input field and click the Edit button to save.

[0080] Step 9:

[0081] Server: Receives the user's modifications and updates the manual document, saves the updated version, and records the change history.

[0082] Step 10:

[0083] Server: Every time a manual is updated, a new version is generated and compared with the previous version. Changes are recorded as a history and stored in a database.

[0084] Step 11:

[0085] Server: Receives feedback from users, regenerates the manual based on that feedback, and verifies that the changes based on the feedback are correctly reflected.

[0086] Step 12:

[0087] Terminal: Redisplays the updated manual to the user and provides an interface for review and feedback.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] Conventional business process standardization and manual creation involve a large amount of manual work, requiring time and effort. Furthermore, manual work can lead to errors and information not being updated in a timely manner, resulting in incomplete knowledge management within an organization. This reduces business efficiency and makes training new employees difficult. The objective of this invention is to solve these issues by providing a system that automates business process standardization and creates and updates business manuals in a sustainable manner.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes a means for converting audio data and video data into text data through speech recognition processing, a means for analyzing the text data using natural language processing technology and extracting information necessary for business processes, and a means for organizing the extracted information and classifying it by item, thereby enabling the standardization of business processes and the automation of the creation and updating of business manuals.

[0093] "Audio data" refers to files containing recordings of what was said during business process explanations and interviews.

[0094] "Video data" refers to video files that record business procedures and work procedures.

[0095] "Speech recognition processing" is a technology that converts input voice data into text data.

[0096] "Natural language processing technology" is a technology that analyzes text data and extracts and classifies semantic information.

[0097] A "business process" is a series of steps or procedures for carrying out a specific business operation.

[0098] A "manual document" is a document that describes business processes and procedures and can be referenced by users.

[0099] "Feedback" refers to corrections and additional information provided by users regarding the manual documentation.

[0100] An "update history" is a record of changes that is saved each time a manual document is updated.

[0101] The "specified format" is a format that is predetermined as the output format for the manual document, specifically, PDF or Word file format.

[0102] A "database" is a system for organizing and storing collected and converted information and analysis results.

[0103] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. The core of this system is the collection and analysis of audio and video data, manual generation, display, feedback, and management of update history.

[0104] Hardware and Software Configuration

[0105] Users use devices such as laptops and smartphones to collect audio interviews explaining business processes and videos of actual work. Voice recorders on smartphones and online conferencing tools such as Zoom can be used as audio and video recording devices.

[0106] The terminal provides an interface for users to upload collected audio and video data to the system. This interface has a dedicated upload form, and users can easily upload recorded files by dragging and dropping them.

[0107] The server receives and stores the uploaded data. Using a speech recognition engine (Google Cloud Speech-to-Text API) and a natural language processing engine (SpaCy), the server converts the voice data into text data and analyzes the text data. The analysis results are stored in a database and classified into information required for the business process (input, output, tools used, procedures, required skills, KPIs, etc.).

[0108] The generated text data is automatically formatted into a manual document on the server using LaTeX or Microsoft Word API, and the manual document is saved in the specified format (PDF or Word file format) and provided to the user.

[0109] The terminal displays the generated manual document to the user and provides an interface for the user to check or modify the content as necessary. The generated PDF and Word files can be opened in Adobe Acrobat or Microsoft Word.

[0110] The user checks the contents of the manual document and provides feedback. The feedback is sent to the system through a form on the terminal. The server receives the feedback from the user and regenerates the manual document with necessary corrections.

[0111] Furthermore, the server has the function of storing update history and comparing it with past versions, which allows you to track changes in manual documents and manage them appropriately.

[0112] Specific examples

[0113] For example, when creating a manual for a new software tool, users collect and upload audio files and operation videos that explain the software's operation procedures. The server receives these files and converts the audio to text using the Google Cloud Speech-to-Text API. It then uses SpaCy to analyze the text data and extracts and classifies information such as software installation instructions, descriptions of key features, specific operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates a PDF manual document using LaTeX and provides it to the user via their device. The user then reviews the manual, makes corrections using Adobe Acrobat, and submits their feedback to the system.

[0114] Prompt Sentence Examples

[0115] Simply upload audio and video clips demonstrating how to use a new software tool. The system will convert the audio into text, extract and categorize key information, and automatically generate a manual in PDF format. Users can review and edit the manual and provide feedback.

[0116] This system will standardize business processes and automate the creation and updating of business manuals, enabling efficient and sustainable knowledge management within the organization.

[0117] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0118] Step 1: Collect and upload audio and video

[0119] User: Records interview audio and video of work related to business processes. This is done using a recording device or a recording app on a smartphone. Specifically, the user starts the recording app on the smartphone and records the interview and work procedures. After finishing, the file is saved on the device.

[0120] Input: Audio and video files that explain the business process.

[0121] Output: Audio and video files saved on your device.

[0122] Step 2: Upload your data

[0123] User: Use the device's system interface to upload the recorded audio and video data to the system by dragging and dropping the audio and video files into the device's upload form and clicking the Upload button.

[0124] Input: Audio and video files stored on your device.

[0125] Output: Audio and video files transferred to the server.

[0126] Step 3: Save your data

[0127] Server: Receives uploaded audio and video data and saves it in a specified directory.

[0128] Input: Audio and video files transferred from your device.

[0129] Output: Audio and video files stored on the server.

[0130] Step 4: Convert audio data to text

[0131] Server: Pass the audio file to the Google Cloud Speech-to-Text API and convert it to text data. Send the audio file as an API request and save the returned text data.

[0132] Input: Audio files stored on the server.

[0133] Output: Text data converted from the audio file.

[0134] Step 5: Extract audio from video and convert it to text

[0135] Server: Extracts the audio portion from the video file and performs speech recognition processing. A video analysis tool (e.g., FFmpeg) is used to separate the audio from the video, and the audio is passed to a speech recognition engine for conversion into text data.

[0136] Input: Video files stored on the server.

[0137] Output: Text data of the audio extracted from the video file.

[0138] Step 6: Analyze and classify text data

[0139] Server: Analyzes the text data using a natural language processing engine (SpaCy) to extract and classify the necessary information (input, output, tools used, procedures, required skills, KPIs, etc.). Passes the text data to perform analysis, and stores the data classified into each category in a database.

[0140] Input: Text data converted from audio and video.

[0141] Output: Information required for the classified business process.

[0142] Step 7: Generate manual documentation

[0143] Server: Based on the classified information, manual documents are automatically generated using LaTeX and Microsoft Word API. Information is retrieved from the database, and text data is embedded in templates to generate documents.

[0144] Input: Classified information.

[0145] Output: Manual document in PDF and Word format.

[0146] Step 8: View and correct the manual

[0147] Terminal: The generated manual document is displayed to the user, and an interface is provided for checking and modifying the contents. The generated file is displayed in the terminal viewer, and the contents can be checked using a modification form or PDF editing software.

[0148] Input: Generated manual documentation.

[0149] Output: A manual document that has been checked and corrected by the user.

[0150] Step 9: Provide feedback

[0151] Users: Provide feedback on the quality of the documentation by submitting your corrections or additions using the feedback form.

[0152] Input: User feedback on the manual documentation.

[0153] Output: Feedback data sent to the system.

[0154] Step 10: Manual Updates and History Management

[0155] Server: Updates the manual based on the feedback and generates a new version. A change history is saved for each update, allowing comparison with previous versions.

[0156] Input: Feedback data from users.

[0157] Output: Updated manual documentation and changelog.

[0158] Through these steps, the system automates the standardization of business processes and procedures and provides the ability to create and update manuals in a sustainable manner.

[0159] (Application example 1)

[0160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] Standardizing and maintaining work processes is extremely important in modern industrial workplaces. In particular, accurately and efficiently recording and updating the operating procedures of automated machines and robots directly contributes to improved productivity and quality control. However, creating and updating manuals manually takes time and effort, and is prone to problems such as clerical errors and missed updates. Furthermore, it is necessary to constantly provide the latest information when training new employees or troubleshooting. A system that can efficiently solve these issues is needed.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0163] In this invention, the server includes: means for collecting work-related interview audio or work video; means for uploading the collected audio and video files to the system; means for receiving and saving the uploaded audio and video files; means for converting the received audio and video files into text data through speech recognition; means for analyzing the text data and extracting information necessary for the business process; means for organizing the extracted information and classifying it by category; means for automatically generating a manual document based on the classified data; means for displaying the manual document and allowing the user to check and modify its contents; means for generating a new version of the manual each time it is updated and comparing it with previous versions; means for saving and managing the change history; means for periodically checking the updated manual and receiving feedback; means for correcting the information based on the received feedback and generating a new manual; means for standardizing work procedures for automated machines in factories and displaying them in augmented reality; means for automatically generating operation manuals for factory robots and displaying them on smart glasses; and means for saving the update history using a version management function and comparing it with previous manuals. This automates the standardization and updating of work procedures in industrial sites, enabling the provision of always-updated information.

[0164] The "work-related interview audio" is audio data in which a field worker or manager orally explains work procedures and operation methods in a factory.

[0165] "Work video" is video data that captures the operating procedures of robots and automatic machines on factory floors.

[0166] The "means of collection" refers to a system that includes a device that uses a smartphone or smart glasses to record interview audio and work videos, as well as its operating interface.

[0167] "Uploading means" refers to software and communication means for sending collected audio and video files to a server via a network.

[0168] "Means for receiving and storing" refers to a system by which the server receives uploaded audio and video files and stores them in data storage.

[0169] "Means for processing voice data through speech recognition and converting it into text data" refers to software that analyzes voice data and converts it into text information using speech recognition technology.

[0170] "Means for analyzing text data and extracting information necessary for business processes" refers to a system that uses natural language processing technology to automatically select important business procedures and information from converted text.

[0171] "Means for classifying each item" refers to software that organizes the extracted information and compiles it into categories such as procedures, tools used, and setting methods.

[0172] "Means for automatically generating manual documents" refers to a program for creating manuals in a specified format based on organized and classified information.

[0173] The "means for displaying the manual document and allowing the user to confirm and modify the contents" is an application that provides an interface that allows the user to view the generated manual and to reflect changes as necessary.

[0174] "Means for generating new versions and comparing them with previous versions" refers to a software function that allows you to manage and compare old and new versions when updating a manual.

[0175] "Means for saving and managing change history" is a system that tracks the revision history of a manual and stores that information in a database.

[0176] "Means for receiving feedback" is a feature that provides an interface for collecting improvements and additional information from users.

[0177] The "means for correcting information and regenerating the manual" refers to software that updates and corrects the manual content based on feedback and regenerates it.

[0178] "A means to standardize the work procedures of automated machines in factories and display them in augmented reality" is a function that uniformly records the operating procedures of automated machines and visually presents them using AR technology.

[0179] "A means to automatically generate operating manuals for factory robots and display them on smart glasses" is a technology that automatically creates manuals for robot operating procedures and displays them on smart glasses.

[0180] "A means for saving update history using version control functions and comparing with past manuals" is a system that can track manual versions and compare changes between different versions.

[0181] The present invention is a system for standardizing work procedures for automated machines and robots in factories and automatically generating manuals. This system aims to improve work efficiency and quality by utilizing voice recognition, natural language processing, and augmented reality technologies. Specific embodiments of this system are described in detail below.

[0182] Overall system configuration

[0183] This system is mainly composed of three components: the user, the server, and the terminal. We will explain the specific functions of each component and how they work together.

[0184] User

[0185] The users are factory workers and managers. They use smartphones or smart glasses to collect interview audio and video explaining the robot's work procedures and upload them to the system. They also review the generated manuals, make corrections as necessary, and provide feedback.

[0186] server

[0187] The server is the core of the system, and performs data analysis, manual generation, and maintenance. Specifically, the server has the following functions:

[0188] Speech recognition processing: The collected audio files are converted into text data using the Google Cloud Speech-to-Text API.

[0189] Video analysis: Use Amazon Rekognition to extract visual information from collected video files.

[0190] Natural Language Processing (NLP): Using spaCy and Transformers (Hugging Face), text data is analyzed and information necessary for business operations is extracted.

[0191] Manual generation: A Python template engine is used to generate a manual based on the extracted information.

[0192] Database management: MongoDB is used to store and manage the generated manuals and change history.

[0193] Version control: Provides the ability to save update history and compare with previous versions.

[0194] Terminal

[0195] The terminal provides an interface for users to operate the system. Specifically, it has the following functions:

[0196] Displaying and Editing Manuals: Provides functionality for viewing and editing generated manuals through smart glasses or smartphone interfaces. React Native is used for front-end development.

[0197] Gathering Feedback: Provide an interface that allows users to easily submit requests for corrections.

[0198] Specific examples

[0199] Next, a specific example of this system will be described.

[0200] Example: Creating a manual for routine maintenance procedures for a robot

[0201] 1. Data Collection: A user uses the smart glasses to collect audio and video recordings of the robot explaining the sensor cleaning procedure.

[0202] 2. Data Upload: The collected audio and video data is uploaded to the system.

[0203] 3. Data analysis: The server converts the audio to text using the Google Cloud Speech-to-Text API and analyzes the video using Amazon Rekognition.

[0204] 4. Natural Language Processing: The converted text data is analyzed using spaCy and Transformers to extract and classify the necessary information (procedures, tools used, etc.).

[0205] 5. Manual generation: Based on the extracted data, a manual is generated using a Python template engine and saved in PDF or Word format.

[0206] 6. Manual display and correction: The generated manual is presented to the user via smart glasses, allowing the user to check and correct the contents.

[0207] 7. Feedback and Updates: The manual is updated based on user feedback.

[0208] This allows for standardization of robot operating procedures on factory floors and automatic generation of manuals. It also provides the latest information at all times, contributing to improved efficiency and quality.

[0209] Prompt Sentence Examples

[0210] Analyze audio and video recordings of routine robot maintenance procedures and automatically create a manual based on the content. Extract information such as specific procedures, tools to be used, precautions, and frequency, and output it in PDF format.

[0211] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0212] Step 1:

[0213] The user uses a smartphone or smart glasses to collect interview audio and video explaining the robot's work procedures. The collected data is recorded using the camera and microphone of the smart device. The input is the work audio and video, and the output is these data files.

[0214] Step 2:

[0215] Users upload collected audio and video files to the system using an application on their smart devices. This sends the collected data to the server via the network. The input is the audio and video files, and the output is these data files stored on the server.

[0216] Step 3:

[0217] The server receives the uploaded audio and video files and stores them in an AWS S3 bucket. The specific operation of this step is to store the received data in data storage. The input is the uploaded audio and video files, and the output is the stored data files.

[0218] Step 4:

[0219] The server uses the Google Cloud Speech-to-Text API to analyze the received audio file and convert it into text data. Specifically, it uses speech recognition technology to convert the audio data into text data. The input is an audio file, and the output is text data.

[0220] Step 5:

[0221] The server uses Amazon Rekognition to analyze visual information from the received video file and extract important frames and objects. Specifically, it identifies recognized objects and actions in the video and converts them into text data. The input is the video file, and the output is the analyzed text data.

[0222] Step 6:

[0223] The server analyzes the converted text data using spaCy and Transformers (Hugging Face) and extracts the information necessary for the business (procedures, tools used, etc.). Specifically, it uses natural language processing technology to automatically select the necessary information from the text data. The input is text data, and the output is the extracted information.

[0224] Step 7:

[0225] The server organizes the extracted information and classifies it by item. Specifically, it organizes it into categories such as procedures, tools used, and setting methods. This classification process makes it easier to organize the information. The input is the extracted information, and the output is data classified by category.

[0226] Step 8:

[0227] The server uses a Python template engine to generate a manual based on the classified data. Specifically, it formats the organized information based on a template and creates a manual document. The input is the classified data, and the output is the generated manual document.

[0228] Step 9:

[0229] The terminal displays the generated manual document to the user, who can then check the content through the interface of smart glasses or a smartphone and make corrections as necessary. The input is the manual document, and the output is a modifiable manual displayed to the user.

[0230] Step 10:

[0231] The user checks whether the manual has been generated properly and sends feedback to the server via the feedback function of the terminal. The input is the user's feedback, and the output is the feedback data sent to the server.

[0232] Step 11:

[0233] The server corrects and updates the contents of the manual based on the feedback received from the user, thereby regenerating the manual to reflect the latest information. The input is the feedback data, and the output is the updated manual document.

[0234] Step 12:

[0235] The server uses the version control function to compare the current manual with previous versions and save and manage the change history. Specific operations include saving the update history and comparing it with previous versions. The input is the updated manual document, and the output is the compared version history.

[0236] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0237] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. This system generates business manuals using work-related interview audio and work videos, and also uses an emotion engine to recognize and reflect the user's emotional state.

[0238] Overall system configuration

[0239] This system mainly consists of the following three entities: users, servers, and terminals.

[0240] User

[0241] Users are business personnel or administrators who are responsible for operating the system and providing data. Specifically, they upload work-related interview audio and work videos to the system, and review and edit the content. The system also recognizes the user's emotional state.

[0242] server

[0243] The server is the core of the system, and performs voice recognition processing, data analysis, manual generation, and maintenance. The server also has an emotion engine that recognizes user emotions and stores and utilizes the analysis results.

[0244] 1. Data conversion and analysis

[0245] Server: Using voice recognition technology, it generates text data from the uploaded audio and video, and uses natural language processing technology to extract and classify information necessary for business processes.

[0246] Server: The emotion engine analyzes the user's emotions from their voice and facial expressions, and stores the results in a database.

[0247] Terminal

[0248] The terminal provides an interface for users to operate the system, displays the generated manual document, supports content confirmation and correction, and adjusts the presentation method based on the analysis results of the emotion engine.

[0249] Program processing overview

[0250] 1. Audio and Video Collection and Upload

[0251] User: Collects interview audio and work videos that explain business processes and uploads them to the system using a terminal.

[0252] 2. Data Conversion and Analysis

[0253] Server: Receives the uploaded data and converts it into text using speech recognition processing. Then, natural language processing technology is used to extract and classify the information required for the business process.

[0254] Server: The emotion engine analyzes emotions from the user's voice and facial expressions and stores the results in a database.

[0255] 3. Manual Generation

[0256] Server: Automatically generates manual documents based on the extracted information. The server embeds the classified data into a document template and generates the document in the specified format (PDF, Word, etc.).

[0257] Server: Combines the generated manual documentation with the results of the emotion engine and provides it in a format that is easy for users to read.

[0258] 4. Viewing and Modifying the Manual

[0259] Terminal: Displays the generated manual document to the user and supports confirmation and correction of the contents through the interface.

[0260] Device: Based on the analysis results of the emotion engine, the presentation method is adjusted accordingly. For example, if it determines that the user is confused, it will automatically add detailed annotations.

[0261] 5. Feedback and Maintenance

[0262] Users: Review the manual and provide feedback on any necessary corrections or additional information. Provide feedback and suggestions for improvements to the system through the feedback function.

[0263] Server: Updates the manual based on user feedback and the analysis results of the emotion engine, generates new versions, and stores and manages the update history.

[0264] Specific examples

[0265] For example, when creating a manual for how to use a new software tool, users collect and upload audio files and video instructions explaining how to use the software. The server receives these files and converts them into text using speech recognition. Next, it uses natural language processing to extract and classify information such as software installation instructions, descriptions of key functions, operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates the manual text and also analyzes the user's emotions using an emotion engine to adjust the presentation method as needed.

[0266] This not only efficiently standardizes business processes and procedures, but also provides easy-to-use manuals that reflect the user's feelings. Users can always provide the latest information, contributing to improved efficiency and quality.

[0267] The processing flow will be explained below.

[0268] Step 1:

[0269] Users: Collect job-related audio interviews or video of tasks, either by recording a worker performing a task or by recording an interview in which the worker explains a job procedure.

[0270] Step 2:

[0271] User: Uploads collected audio and video files to the system using the device interface. The user selects the files on the file selection screen and clicks the upload button.

[0272] Step 3:

[0273] Server: Receives uploaded audio and video files and saves them to storage. Checks the file format and confirms that the file was saved successfully.

[0274] Step 4:

[0275] Server: Starts the speech recognition module and converts the received voice data into text data. In the case of video, the audio portion is extracted and processed in the same way. The converted text is temporarily saved.

[0276] Step 5:

[0277] Server: Launches a natural language processing (NLP) engine and extracts information required for business processes (input, output, tools used, procedures, required skills, KPIs, etc.) from the converted text data. The extracted information is categorized and stored in a database.

[0278] Step 6:

[0279] Server: Activates the emotion engine and recognizes and analyzes the user's emotions from the uploaded audio and video data. Specifically, it identifies the user's emotional state using voice tone and facial expression recognition technology and stores the results in a database.

[0280] Step 7:

[0281] Server: Automatically generates manual documents based on the extracted business information and emotion data. Embeds the data in a document template, generates documents in the specified format (PDF, Word, etc.), and saves them in storage.

[0282] Step 8:

[0283] Terminal: Displays the generated manual document to the user. The user can view the document and check its contents through the interface.

[0284] Step 9:

[0285] User: Check the contents of the displayed manual document and make corrections as necessary. Enter the corrections in the terminal interface and click the save button.

[0286] Step 10:

[0287] Server: Receives the user's edits and updates the manual. It saves the updated version and records the change history in a database.

[0288] Step 11:

[0289] Server: Every time a manual is updated, a new version is generated and compared with the previous version. The changes are recorded as a history and made available to users.

[0290] Step 12:

[0291] Device: Based on the results of the emotion engine, the presentation method is adjusted accordingly. For example, if the user is confused, detailed annotations or additional examples are automatically inserted into the manual.

[0292] Step 13:

[0293] Users: Regularly check the updated manual and provide feedback. Submit your thoughts and suggestions for improvement to the system via the feedback function.

[0294] Step 14:

[0295] Server: Regenerates the manual based on user feedback and the analysis results of the emotion engine, and updates the new version. Saves and manages the update history, recording feedback along with previous versions.

[0296] Example 2

[0297] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0298] The conventional manual creation process had the problem of taking a great deal of time and effort to collect business-related information and generate and update the manual. Furthermore, the generated manual did not take into account the user's level of understanding or emotional state, which resulted in a decrease in the effectiveness of user training and work efficiency. Furthermore, the manual update process was cumbersome, and the latest information could not be reflected, which was also an issue.

[0299] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0300] In this invention, the server includes means for collecting business-related audio or video, means for uploading the collected audio and video data to the system, means for receiving and saving the uploaded audio and video data, means for converting the received audio and video data into text data through speech recognition processing, means for analyzing the text data and extracting information necessary for business procedures, means for organizing the extracted information and classifying it by item, means for automatically generating a procedure manual based on the classified data, means for displaying the procedure manual and allowing the user to check and modify its contents, means for generating a new version each time the procedure manual is updated and comparing it with previous versions, means for saving and managing the change history, means for periodically checking the updated procedure manual and receiving feedback, means for modifying the information based on the received feedback and generating a new procedure manual, means for analyzing the user's emotions from their voice and facial expressions, and means for adjusting the presentation method of the procedure manual based on the analysis results. This enables efficient and continuous generation and updating of business manuals, and realizes effective manual provision that corresponds to the user's level of understanding and emotional state.

[0301] "Speech recognition processing" is a technology that analyzes voice data and converts it into text data.

[0302] "Natural language processing" is a technology that analyzes text data to understand its meaning and structure.

[0303] "Business-related audio or video" refers to audio or video data recorded to explain business processes or procedures.

[0304] "Uploading means" refers to a method or device for transmitting collected audio and video data to the system.

[0305] "Means for receiving and storing" refers to a method or device that receives uploaded data and stores it in a database or file system.

[0306] "Means for converting into text data" refers to technology or devices that use voice recognition processing to convert voice data into text information.

[0307] "Information necessary for business procedures" refers to specific steps and important points related to business processes and procedures.

[0308] "Extraction means" refers to a method or device that uses natural language processing technology to extract necessary information from text data.

[0309] "Means for organizing and classifying by item" refers to techniques or devices that logically arrange extracted information and separate it into categories.

[0310] "Means for automatically generating procedure manuals" refers to methods and technologies for creating procedure manuals based on classified data in accordance with templates and formats.

[0311] The "means for displaying the procedure manual and allowing the user to check and modify the contents" refers to a technique or device that allows the user to view the generated procedure manual and provides an interface to enable modification.

[0312] "Means for generating a new version and comparing it with the previous version" refers to a technique or device that creates a new version of an updated procedure manual and compares it with the previous version to confirm the changes.

[0313] "Means for saving and managing change history" refers to technology or devices that record the update history of procedure manuals and make them accessible as needed.

[0314] "Means for receiving feedback" refers to methods or devices that allow users to input and send their opinions and suggestions for improvement.

[0315] "Means for correcting information and regenerating the procedure manual" refers to the technology or device that updates and regenerates the procedure manual based on feedback.

[0316] "Means for analyzing emotions" refers to technology or devices that analyze voice and facial expressions to identify emotional states.

[0317] "Means for adjusting presentation method" refers to technology or devices that change the display or explanation method based on the results of sentiment analysis.

[0318] The present invention provides a system that automates the standardization of business processes and procedures and creates and updates operating manuals in a sustainable manner. This system generates operating manuals using business-related audio and video data, and further uses a sentiment analysis engine to recognize and reflect the user's emotional state. An embodiment of the system is described in detail below.

[0319] System configuration

[0320] This system mainly consists of the following three entities: users, servers, and terminals.

[0321] User

[0322] Users are business personnel or administrators who are responsible for operating the system and providing data. Specifically, they collect work-related audio and video data and upload them to the system. The system also recognizes the user's emotional state.

[0323] server

[0324] The server is the core of the system, and performs voice recognition processing, data analysis, procedure manual generation, and maintenance. The server also has a sentiment analysis engine that recognizes user emotions and stores and utilizes the analysis results. The specific functions of the server are as follows:

[0325] Speech recognition processing: The server converts the voice data into text data using Google Cloud Speech-to-Text.

[0326] Natural language processing: The server uses natural language processing technology such as SpaCy to extract and classify information necessary for business procedures from text data.

[0327] Sentiment analysis: The server uses IBM Watson Emotion Analysis to analyze the user's emotions from audio and video data and stores the results in a database.

[0328] Procedure generation: The server uses LaTeX and Microsoft Word APIs to generate procedures based on the extracted information.

[0329] Information update: The server updates the procedure manual based on user feedback and generates a new version.

[0330] Terminal

[0331] The terminal provides the interface through which the user operates the system and has the following functions:

[0332] Support for displaying and modifying procedures: Provides an interface that displays the generated procedures and allows users to check and modify their contents.

[0333] Reflection of sentiment analysis results: Based on the results of sentiment analysis, the way the instructions are presented is adjusted accordingly. For example, if the user is confused, detailed annotations can be added.

[0334] Example

[0335] The following describes how this system is specifically implemented.

[0336] For example, when creating a manual for how to use a new software tool, a user collects audio files or video instructions explaining the software operation procedures and uploads them to the system. The server uses Google Cloud Speech-to-Text to convert the uploaded audio data into text. It then uses SpaCy to extract and classify the information necessary for the business procedures from the text data. Based on this information, the server automatically generates a PDF manual using LaTeX, and then uses IBM Watson Emotion Analysis to analyze the user's emotional state and adjust the manual accordingly.

[0337] Prompt Sentence Examples

[0338] Below is an example of a prompt sentence to be input into the generative AI model.

[0339] "We have an audio interview and video of an instruction manual explaining how to use a new software tool. From these, please extract installation instructions, explanations of key features, operation instructions, required settings, and troubleshooting information, and automatically generate a PDF instruction manual. Also, please analyze the user's emotional state and provide detailed annotations if the user is confused."

[0340] In this way, by providing concrete examples and prompts, users can clearly understand how the system actually works and how to use it.

[0341] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0342] Step 1:

[0343] Audio and video collection and uploading

[0344] Users collect interview audio and work videos that explain business processes and upload them to the system using a terminal.

[0345] Input: Audio and video files related to business processes

[0346] Specific operation: The user records an interview or activity using a smartphone or camera, transfers the file to a computer, and selects and uploads it from the system's web interface.

[0347] Output: Audio and video files uploaded to the system.

[0348] Step 2:

[0349] Data transformation and analysis

[0350] The server receives the uploaded data and converts it into text using speech recognition processing.

[0351] Input: Uploaded audio and video files

[0352] Specific operation: The server calls the Google Cloud Speech-to-Text API and converts the audio data into text data.

[0353] Output: Text data

[0354] The server applies natural language processing technology to the converted text data to extract and classify the information necessary for business procedures.

[0355] Input: Text data

[0356] Specific operation: The server uses SpaCy to extract noun phrases and perform relationship analysis to extract and classify important points in the business procedures.

[0357] Output: Classified business procedure information

[0358] The server uses an emotion engine to analyze the user's emotions from their voice and facial expressions, and stores the results in a database.

[0359] Input: Audio and video files

[0360] Specific operation: The server uses IBM Watson Emotion Analysis to extract emotional data from audio and video data and stores the analysis results in a database.

[0361] Output: Sentiment analysis result data

[0362] Step 3:

[0363] Procedure generation

[0364] The server automatically generates a procedure manual based on the extracted information.

[0365] Input: Classified operating procedure information

[0366] Specific operation: The server embeds the extracted and classified information into a manual template and generates a PDF procedure manual using LaTeX.

[0367] Output: Instructions in PDF format

[0368] Based on the analysis results of the emotion engine, the server adjusts the instruction manual in a way that is easy for the user to understand.

[0369] Input: PDF format instructions, sentiment analysis result data

[0370] Specific behavior: If the server determines that the user is confused based on the results of sentiment analysis, it adds an annotation to the instruction manual providing detailed explanations.

[0371] Output: Adjusted instructions

[0372] Step 4:

[0373] View and modify procedures

[0374] The terminal displays the generated procedure manual to the user and supports confirmation and correction of the contents.

[0375] Input: Adjusted Procedures

[0376] Specific behavior: Provides an interface that allows users to open PDF documents through a web interface, review the content, and add comments to corrections.

[0377] Output: User-confirmed and corrected instructions

[0378] The device will adjust the presentation method appropriately based on the analysis results of the emotion engine.

[0379] Input: Sentiment analysis result data

[0380] Specific Action: If the user is determined to be confused, provide the ability to add illustrations using visualization tools.

[0381] Output: Adjusted presentation instructions

[0382] Step 5:

[0383] Feedback and Maintenance

[0384] The user checks the contents of the procedure manual and provides feedback on any necessary corrections or additional information.

[0385] Input: User-confirmed and corrected instructions

[0386] What happens: The user fills out a feedback form to submit corrections or additional information to the system.

[0387] Output: Feedback information

[0388] The server updates the recipe based on user feedback and the emotion engine's analysis.

[0389] Input: Feedback information, sentiment analysis result data

[0390] Specific operation: The server analyzes the feedback it receives, automatically updates the relevant part of the procedure manual, and generates a new version.

[0391] Output: Updated instructions, new version generated

[0392] This enables efficient and continuous generation and updating of work procedures, and makes it possible to provide effective procedures that correspond to the user's level of understanding and emotional state.

[0393] (Application example 2)

[0394] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0395] Creating manuals for factory work procedures and business processes takes time and effort, and efficient training methods are required, especially for new employees. Furthermore, if the manuals created do not take into account the reader's feelings and level of understanding, there is a problem of inefficient learning and work execution. Furthermore, delays in updating manuals and reflecting feedback can lead to information not being kept up to date, resulting in inefficiencies in work. However, no effective means have been provided to solve these issues.

[0396] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0397] In this invention, the server includes means for collecting work-related interview audio or work video, means for uploading audio files or video files to the system, means for receiving and saving the uploaded files, means for converting the received files into text data by subjecting them to speech recognition processing, means for analyzing the text data and extracting information necessary for the business process, means for analyzing emotions from the user's voice and facial expressions using an emotion engine and generating emotion data, means for adjusting the presentation method of the manual document based on the emotion data, and means for saving and displaying the generated manual document in a specified format such as PDF or Word.

[0398] This makes it possible to efficiently extract work procedures from audio and video and automatically generate manuals that incorporate emotional data. Furthermore, the generated manuals are kept up to date and are presented appropriately, reflecting the user's emotions, enabling more effective training and business support.

[0399] "Business-related interview audio" refers to audio data obtained from interviews conducted to explain business processes.

[0400] "Work video" refers to video data that records business processes and work procedures.

[0401] "Speech recognition processing" is a technology that analyzes voice data and converts it into text data.

[0402] "Text data" refers to character string data converted by speech recognition processing.

[0403] "Information extraction" refers to the process of extracting information necessary for business processes from text data.

[0404] "Classification" refers to the process of organizing and systematizing extracted information by category.

[0405] "Manual documents" refer to documents that describe business processes and work procedures.

[0406] "Emotion engine" refers to technology for analyzing emotions from voice and facial expressions.

[0407] "Emotional Data" refers to data regarding a user's emotional state analyzed by the emotion engine.

[0408] "Adjusting the presentation method" refers to changing the way the manual document is displayed based on the emotion data.

[0409] "Feedback" refers to receiving opinions and requests from users and reflecting them in the contents of the manual.

[0410] "Application to be installed on a factory robot" refers to application software for executing the above-mentioned system on a factory robot.

[0411] "PDF and Word formats" refer to file formats for saving and displaying manual documents.

[0412] "Comparing with previous versions" refers to the process of comparing the current manual document with the version before the update and checking for changes.

[0413] This invention relates to a system that automatically generates, updates, and manages business process manuals for factory robots. This system efficiently collects business-related interview audio and work videos, and uses an emotion engine to provide manuals that reflect the user's emotional state.

[0414] System Subject

[0415] User

[0416] Users are factory workers and managers who upload work-related audio and video to the system, and are responsible for reviewing and editing the content. The system also recognizes the user's emotional state.

[0417] server

[0418] The server is the core of the system and is responsible for:

[0419] 1. Speech recognition and natural language processing: Uploaded audio and video files are received and stored, and converted into text data using speech recognition technology. The converted text data is then analyzed using natural language processing technology to extract and classify information necessary for business processes.

[0420] 2. Emotion analysis: The emotion engine analyzes the user's emotions from their voice and facial expressions, and stores the results in a database.

[0421] 3. Manual generation: Manual documents are automatically generated based on the extracted information. The generated manual documents are provided in a format that is easy for users to understand, reflecting the analysis results of the emotion engine.

[0422] Terminal

[0423] The terminal is an application installed on the factory robot that provides the following functions:

[0424] 1. Manual display and correction assistance: This system displays the generated manual document and provides an interface for users to check and correct the contents. The display method is adjusted based on the analysis results of the emotion engine.

[0425] 2. Feedback Management: Receive feedback from users and update the manual based on the latest information.

[0426] Program processing overview

[0427] The server uses libraries such as Google Speech Recognition API for speech recognition technology, nltk for natural language processing technology, and Sentiment Intensity Analyzer for emotion engine. Below is an overview of the processing for each function of the system.

[0428] 1. Speech Recognition: Uses Google Speech Recognition API to convert audio files into text data. Audio data is collected from a microphone or audio files.

[0429] 2. Natural Language Processing: Analyze text data using the nltk library to extract and classify information necessary for business processes.

[0430] 3. Sentiment Analysis: A Sentiment Intensity Analyzer is used to extract emotional information from text data. Voice and facial expression data are also analyzed.

[0431] 4. Manual generation: Manual documents are automatically generated based on the extracted and analyzed information, and saved and displayed in PDF or Word format.

[0432] Specific examples

[0433] For example, when creating an operating procedure manual for a factory robot, the user uploads audio files or video files explaining the operating procedures. The server receives these files, performs speech recognition processing, and converts them into text. Natural language processing technology is used to extract and classify information such as installation procedures, explanations of key functions, operation methods, required settings, and troubleshooting. An emotion engine also analyzes the user's emotions and adjusts the presentation method as needed.

[0434] Prompt Sentence Examples

[0435] Below is an example of a prompt sentence to input to the generative AI model.

[0436] text

[0437] "Analyze the audio files below explaining the work procedures and identify the work procedures and emotional state. Based on the results, demonstrate the process for generating a detailed manual."

[0438] In this way, the system can efficiently extract work procedures from audio and video and automatically generate manuals that incorporate emotional data. The generated manuals are kept up to date and are presented appropriately, reflecting the user's emotions, enabling more effective training and work support.

[0439] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0440] Step 1: Collect and upload work-related interview audio and video footage

[0441] The user collects interview audio and video that explain work procedures and business processes in the factory. These audio and video files are uploaded to the system using a terminal. The input is the audio and video files, and the output is the data transferred to the server. Specifically, the user selects the collected files and transfers them to cloud storage using the system's upload function.

[0442] Step 2: Receiving and storing data

[0443] The server receives uploaded audio and video files and saves them in a specified location. The input is the audio or video file from the user, and the output is data saved in a specific folder on the server. Specifically, the server processes the HTTP request and saves the received files in a database or file system.

[0444] Step 3: Speech recognition processing and text conversion

[0445] The server processes the received audio file through speech recognition and converts it into text data. This is done using the Google Speech Recognition API. The input is an audio file, and the output is the converted text data. Specifically, the server sends the audio file to the Google Speech Recognition API and saves the returned text result.

[0446] Step 4: Information extraction using natural language processing

[0447] The server analyzes the text data using natural language processing technology (nltk library) and automatically extracts the information necessary for the business process. The input is text data, and the output is the extracted information. Specifically, the server analyzes the text data and identifies work procedures and important business information.

[0448] Step 5: Sentiment analysis

[0449] The server uses a sentiment engine (Sentiment Intensity Analyzer) to analyze emotions from the user's voice and text. Voice data and text data are input, and emotional data is obtained as output. Specifically, the server inputs this data into the sentiment engine and stores the analysis results in a database.

[0450] Step 6: Automatic generation of manuals

[0451] The server automatically generates a manual document based on the extracted information and emotion data. A document template is used for this purpose. The extracted information and emotion data are input, and the generated manual is obtained as output. Specifically, the server organizes the information appropriately and creates a manual document according to the format.

[0452] Step 7: View and modify the manual

[0453] The terminal displays the generated manual document to the user and provides an interface that allows the user to check and modify the content. The generated manual document is the input, and user feedback is the output. Specifically, the terminal displays the manual document to the user and adjusts the presentation method based on the emotion data.

[0454] Step 8: Updating the manual and managing feedback

[0455] The server modifies the manual based on the feedback received from the user, generates a new version, and saves it. The input is the feedback and the existing manual, and the output is the updated manual. Specifically, the server analyzes the feedback, makes appropriate modifications, and saves the new version.

[0456] Step 9: Managing Change History

[0457] Every time a manual is updated, the server compares the new version with the previous version and saves and manages the change history. The input is the old and new manual document, and the output is the change history. Specifically, the server performs differential analysis, records the changes, and saves them in a history database.

[0458] Through these processing steps, it is possible to efficiently collect, analyze, and generate information for factory robots, and provide manuals that are always kept up to date.

[0459] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0460] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0461] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0462] [Second embodiment]

[0463] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0464] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0465] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0466] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0467] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0468] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0469] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0470] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0471] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0472] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0473] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0474] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0475] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. This system provides a means to generate business manuals by collecting, analyzing, and processing work-related interview audio and video footage.

[0476] Overall system configuration

[0477] This system mainly consists of the following three entities: users, servers, and terminals.

[0478] User

[0479] Users are business personnel and administrators who are responsible for operating the system and providing data. Specifically, they upload work-related interview audio and work videos to the system and check and edit the content.

[0480] server

[0481] The server is the core of the system, and performs voice recognition processing, data analysis, manual generation, and maintenance. The server processes the audio and video data provided by the user and converts it into text data. It then analyzes the text data, extracts and classifies the information required for the business process, and automatically generates manual documents.

[0482] Terminal

[0483] The terminal provides an interface for users to operate the system, displays the generated manual documents to users, and supports them in checking and correcting the contents. It also supports users in easily submitting correction requests through a feedback function.

[0484] Program processing overview

[0485] 1. Audio and Video Collection and Upload

[0486] User: Collects interview audio explaining business processes and actual work videos and uploads them to the system. To do this, the user uses the terminal interface.

[0487] 2. Data Conversion and Analysis

[0488] Server: Receives uploaded audio and video files and converts them into text using speech recognition technology. Based on this text data, natural language processing technology is used to analyze the information required for the job and classify it into categories such as input, output, tools used, procedures, required skills, and KPIs.

[0489] 3. Manual Generation

[0490] Server: Automatically generates manual documents based on the classified information. The manual documents are formatted in a specified format and saved in the specified format (e.g., PDF or Word file).

[0491] 4. Viewing and Modifying the Manual

[0492] Terminal: Provides an interface that displays the generated manual document to the user. Through this interface, the user can check the generated manual and manually correct it if necessary.

[0493] 5. Feedback and Maintenance

[0494] Users: Check that the manual is generated correctly and provide feedback on any necessary corrections or additions. Feedback can be easily provided using the device's functions.

[0495] Server: Receives feedback from users and corrects and updates the manual content accordingly. It also stores and manages the update history and compares the latest version with previous versions.

[0496] Specific examples

[0497] For example, when creating a manual for how to use a new software tool, the user collects and uploads audio files and operation videos that explain the software's operation procedures. The server receives these files and converts them into text using speech recognition processing. Next, natural language processing is used to extract and classify information such as software installation procedures, explanations of key functions, specific operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates a manual document, which the terminal displays to the user. The user then reviews the content and makes any necessary corrections to finalize the manual.

[0498] This allows for efficient standardization of business processes and procedures, maintains organizational knowledge in a sustainable manner, and allows users to provide up-to-date information, contributing to improved efficiency and quality.

[0499] The processing flow will be explained below.

[0500] Step 1:

[0501] User: To collect job-related audio or video interviews, record workers performing tasks or interviews explaining work procedures.

[0502] Step 2:

[0503] User: To upload collected audio and video files to the system, the user uses the device interface to select the files and clicks the upload button.

[0504] Step 3:

[0505] Server: Receives uploaded audio and video files and saves them to storage. Verifies that the files have been saved successfully.

[0506] Step 4:

[0507] Server: Starts the speech recognition module and converts the received voice data into text data. In the case of video, extracts the audio portion and performs the same process. Saves the conversion result as a text file.

[0508] Step 5:

[0509] Server: Using natural language processing (NLP) technology, the server extracts information necessary for business processes from the converted text data. Specifically, it identifies information such as inputs, outputs, tools used, procedures, required skills, and KPIs, and stores it in a database.

[0510] Step 6:

[0511] Server: Automatically generates manual documents based on the extracted information. It embeds the classified data into a document template, generates the document in the specified format (e.g., PDF, Word), and saves it in storage.

[0512] Step 7:

[0513] Terminal: Displays the generated manual document to the user, who can check the manual contents through the interface.

[0514] Step 8:

[0515] User: Check the contents of the displayed manual document and make any necessary corrections. Enter the corrections in the input field and click the Edit button to save.

[0516] Step 9:

[0517] Server: Receives the user's modifications and updates the manual document, saves the updated version, and records the change history.

[0518] Step 10:

[0519] Server: Every time a manual is updated, a new version is generated and compared with the previous version. Changes are recorded as a history and stored in a database.

[0520] Step 11:

[0521] Server: Receives feedback from users, regenerates the manual based on that feedback, and verifies that the changes based on the feedback are correctly reflected.

[0522] Step 12:

[0523] Terminal: Redisplays the updated manual to the user and provides an interface for review and feedback.

[0524] Example 1

[0525] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0526] Conventional business process standardization and manual creation involve a large amount of manual work, requiring time and effort. Furthermore, manual work can lead to errors and information not being updated in a timely manner, resulting in incomplete knowledge management within an organization. This reduces business efficiency and makes training new employees difficult. The objective of this invention is to solve these issues by providing a system that automates business process standardization and creates and updates business manuals in a sustainable manner.

[0527] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0528] In this invention, the server includes a means for converting audio data and video data into text data through speech recognition processing, a means for analyzing the text data using natural language processing technology and extracting information necessary for business processes, and a means for organizing the extracted information and classifying it by item, thereby enabling the standardization of business processes and the automation of the creation and updating of business manuals.

[0529] "Audio data" refers to files containing recordings of what was said during business process explanations and interviews.

[0530] "Video data" refers to video files that record business procedures and work procedures.

[0531] "Speech recognition processing" is a technology that converts input voice data into text data.

[0532] "Natural language processing technology" is a technology that analyzes text data and extracts and classifies semantic information.

[0533] A "business process" is a series of steps or procedures for carrying out a specific business operation.

[0534] A "manual document" is a document that describes business processes and procedures and can be referenced by users.

[0535] "Feedback" refers to corrections and additional information provided by users regarding the manual documentation.

[0536] An "update history" is a record of changes that is saved each time a manual document is updated.

[0537] The "specified format" is a format that is predetermined as the output format for the manual document, specifically, PDF or Word file format.

[0538] A "database" is a system for organizing and storing collected and converted information and analysis results.

[0539] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. The core of this system is the collection and analysis of audio and video data, manual generation, display, feedback, and management of update history.

[0540] Hardware and Software Configuration

[0541] Users use devices such as laptops and smartphones to collect audio interviews explaining business processes and videos of actual work. Voice recorders on smartphones and online conferencing tools such as Zoom can be used as audio and video recording devices.

[0542] The terminal provides an interface for users to upload collected audio and video data to the system. This interface has a dedicated upload form, and users can easily upload recorded files by dragging and dropping them.

[0543] The server receives and stores the uploaded data. Using a speech recognition engine (Google Cloud Speech-to-Text API) and a natural language processing engine (SpaCy), the server converts the voice data into text data and analyzes the text data. The analysis results are stored in a database and classified into information required for the business process (input, output, tools used, procedures, required skills, KPIs, etc.).

[0544] The generated text data is automatically formatted into a manual document on the server using LaTeX or Microsoft Word API, and the manual document is saved in the specified format (PDF or Word file format) and provided to the user.

[0545] The terminal displays the generated manual document to the user and provides an interface for the user to check or modify the content as necessary. The generated PDF and Word files can be opened in Adobe Acrobat or Microsoft Word.

[0546] The user checks the contents of the manual document and provides feedback. The feedback is sent to the system through a form on the terminal. The server receives the feedback from the user and regenerates the manual document with necessary corrections.

[0547] Furthermore, the server has the function of storing update history and comparing it with past versions, which allows you to track changes in manual documents and manage them appropriately.

[0548] Specific examples

[0549] For example, when creating a manual for a new software tool, users collect and upload audio files and operation videos that explain the software's operation procedures. The server receives these files and converts the audio to text using the Google Cloud Speech-to-Text API. It then uses SpaCy to analyze the text data and extracts and classifies information such as software installation instructions, descriptions of key features, specific operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates a PDF manual document using LaTeX and provides it to the user via their device. The user then reviews the manual, makes corrections using Adobe Acrobat, and submits their feedback to the system.

[0550] Prompt Sentence Examples

[0551] Simply upload audio and video clips demonstrating how to use a new software tool. The system will convert the audio into text, extract and categorize key information, and automatically generate a manual in PDF format. Users can review and edit the manual and provide feedback.

[0552] This system will standardize business processes and automate the creation and updating of business manuals, enabling efficient and sustainable knowledge management within the organization.

[0553] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0554] Step 1: Collect and upload audio and video

[0555] User: Records interview audio and video of work related to business processes. This is done using a recording device or a recording app on a smartphone. Specifically, the user starts the recording app on the smartphone and records the interview and work procedures. After finishing, the file is saved on the device.

[0556] Input: Audio and video files that explain the business process.

[0557] Output: Audio and video files saved on your device.

[0558] Step 2: Upload your data

[0559] User: Use the device's system interface to upload the recorded audio and video data to the system by dragging and dropping the audio and video files into the device's upload form and clicking the Upload button.

[0560] Input: Audio and video files stored on your device.

[0561] Output: Audio and video files transferred to the server.

[0562] Step 3: Save your data

[0563] Server: Receives uploaded audio and video data and saves it in a specified directory.

[0564] Input: Audio and video files transferred from your device.

[0565] Output: Audio and video files stored on the server.

[0566] Step 4: Convert audio data to text

[0567] Server: Pass the audio file to the Google Cloud Speech-to-Text API and convert it to text data. Send the audio file as an API request and save the returned text data.

[0568] Input: Audio files stored on the server.

[0569] Output: Text data converted from the audio file.

[0570] Step 5: Extract audio from video and convert it to text

[0571] Server: Extracts the audio portion from the video file and performs speech recognition processing. A video analysis tool (e.g., FFmpeg) is used to separate the audio from the video, and the audio is passed to a speech recognition engine for conversion into text data.

[0572] Input: Video files stored on the server.

[0573] Output: Text data of the audio extracted from the video file.

[0574] Step 6: Analyze and classify text data

[0575] Server: Analyzes the text data using a natural language processing engine (SpaCy) to extract and classify the necessary information (input, output, tools used, procedures, required skills, KPIs, etc.). Passes the text data to perform analysis, and stores the data classified into each category in a database.

[0576] Input: Text data converted from audio and video.

[0577] Output: Information required for the classified business process.

[0578] Step 7: Generate manual documentation

[0579] Server: Based on the classified information, manual documents are automatically generated using LaTeX and Microsoft Word API. Information is retrieved from the database, and text data is embedded in templates to generate documents.

[0580] Input: Classified information.

[0581] Output: Manual document in PDF and Word format.

[0582] Step 8: View and correct the manual

[0583] Terminal: The generated manual document is displayed to the user, and an interface is provided for checking and modifying the contents. The generated file is displayed in the terminal viewer, and the contents can be checked using a modification form or PDF editing software.

[0584] Input: Generated manual documentation.

[0585] Output: A manual document that has been checked and corrected by the user.

[0586] Step 9: Provide feedback

[0587] Users: Provide feedback on the quality of the documentation by submitting your corrections or additions using the feedback form.

[0588] Input: User feedback on the manual documentation.

[0589] Output: Feedback data sent to the system.

[0590] Step 10: Manual Updates and History Management

[0591] Server: Updates the manual based on the feedback and generates a new version. A change history is saved for each update, allowing comparison with previous versions.

[0592] Input: Feedback data from users.

[0593] Output: Updated manual documentation and changelog.

[0594] Through these steps, the system automates the standardization of business processes and procedures and provides the ability to create and update manuals in a sustainable manner.

[0595] (Application example 1)

[0596] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0597] Standardizing and maintaining work processes is extremely important in modern industrial workplaces. In particular, accurately and efficiently recording and updating the operating procedures of automated machines and robots directly contributes to improved productivity and quality control. However, creating and updating manuals manually takes time and effort, and is prone to problems such as clerical errors and missed updates. Furthermore, it is necessary to constantly provide the latest information when training new employees or troubleshooting. A system that can efficiently solve these issues is needed.

[0598] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0599] In this invention, the server includes: means for collecting work-related interview audio or work video; means for uploading the collected audio and video files to the system; means for receiving and saving the uploaded audio and video files; means for converting the received audio and video files into text data through speech recognition; means for analyzing the text data and extracting information necessary for the business process; means for organizing the extracted information and classifying it by category; means for automatically generating a manual document based on the classified data; means for displaying the manual document and allowing the user to check and modify its contents; means for generating a new version of the manual each time it is updated and comparing it with previous versions; means for saving and managing the change history; means for periodically checking the updated manual and receiving feedback; means for correcting the information based on the received feedback and generating a new manual; means for standardizing work procedures for automated machines in factories and displaying them in augmented reality; means for automatically generating operation manuals for factory robots and displaying them on smart glasses; and means for saving the update history using a version management function and comparing it with previous manuals. This automates the standardization and updating of work procedures in industrial sites, enabling the provision of always-updated information.

[0600] The "work-related interview audio" is audio data in which a field worker or manager orally explains work procedures and operation methods in a factory.

[0601] "Work video" is video data that captures the operating procedures of robots and automatic machines on factory floors.

[0602] The "means of collection" refers to a system that includes a device that uses a smartphone or smart glasses to record interview audio and work videos, as well as its operating interface.

[0603] "Uploading means" refers to software and communication means for sending collected audio and video files to a server via a network.

[0604] "Means for receiving and storing" refers to a system by which the server receives uploaded audio and video files and stores them in data storage.

[0605] "Means for processing voice data through speech recognition and converting it into text data" refers to software that analyzes voice data and converts it into text information using speech recognition technology.

[0606] "Means for analyzing text data and extracting information necessary for business processes" refers to a system that uses natural language processing technology to automatically select important business procedures and information from converted text.

[0607] "Means for classifying each item" refers to software that organizes the extracted information and compiles it into categories such as procedures, tools used, and setting methods.

[0608] "Means for automatically generating manual documents" refers to a program for creating manuals in a specified format based on organized and classified information.

[0609] The "means for displaying the manual document and allowing the user to confirm and modify the contents" is an application that provides an interface that allows the user to view the generated manual and to reflect changes as necessary.

[0610] "Means for generating new versions and comparing them with previous versions" refers to a software function that allows you to manage and compare old and new versions when updating a manual.

[0611] "Means for saving and managing change history" is a system that tracks the revision history of a manual and stores that information in a database.

[0612] "Means for receiving feedback" is a feature that provides an interface for collecting improvements and additional information from users.

[0613] The "means for correcting information and regenerating the manual" refers to software that updates and corrects the manual content based on feedback and regenerates it.

[0614] "A means to standardize the work procedures of automated machines in factories and display them in augmented reality" is a function that uniformly records the operating procedures of automated machines and visually presents them using AR technology.

[0615] "A means to automatically generate operating manuals for factory robots and display them on smart glasses" is a technology that automatically creates manuals for robot operating procedures and displays them on smart glasses.

[0616] "A means for saving update history using version control functions and comparing with past manuals" is a system that can track manual versions and compare changes between different versions.

[0617] The present invention is a system for standardizing work procedures for automated machines and robots in factories and automatically generating manuals. This system aims to improve work efficiency and quality by utilizing voice recognition, natural language processing, and augmented reality technologies. Specific embodiments of this system are described in detail below.

[0618] Overall system configuration

[0619] This system is mainly composed of three components: the user, the server, and the terminal. We will explain the specific functions of each component and how they work together.

[0620] User

[0621] The users are factory workers and managers. They use smartphones or smart glasses to collect interview audio and video explaining the robot's work procedures and upload them to the system. They also review the generated manuals, make corrections as necessary, and provide feedback.

[0622] server

[0623] The server is the core of the system, and performs data analysis, manual generation, and maintenance. Specifically, the server has the following functions:

[0624] Speech recognition processing: The collected audio files are converted into text data using the Google Cloud Speech-to-Text API.

[0625] Video analysis: Use Amazon Rekognition to extract visual information from collected video files.

[0626] Natural Language Processing (NLP): Using spaCy and Transformers (Hugging Face), text data is analyzed and information necessary for business operations is extracted.

[0627] Manual generation: A Python template engine is used to generate a manual based on the extracted information.

[0628] Database management: MongoDB is used to store and manage the generated manuals and change history.

[0629] Version control: Provides the ability to save update history and compare with previous versions.

[0630] Terminal

[0631] The terminal provides an interface for users to operate the system. Specifically, it has the following functions:

[0632] Displaying and Editing Manuals: Provides functionality for viewing and editing generated manuals through smart glasses or smartphone interfaces. React Native is used for front-end development.

[0633] Gathering Feedback: Provide an interface that allows users to easily submit requests for corrections.

[0634] Specific examples

[0635] Next, a specific example of this system will be described.

[0636] Example: Creating a manual for routine maintenance procedures for a robot

[0637] 1. Data Collection: A user uses the smart glasses to collect audio and video recordings of the robot explaining the sensor cleaning procedure.

[0638] 2. Data Upload: The collected audio and video data is uploaded to the system.

[0639] 3. Data analysis: The server converts the audio to text using the Google Cloud Speech-to-Text API and analyzes the video using Amazon Rekognition.

[0640] 4. Natural Language Processing: The converted text data is analyzed using spaCy and Transformers to extract and classify the necessary information (procedures, tools used, etc.).

[0641] 5. Manual generation: Based on the extracted data, a manual is generated using a Python template engine and saved in PDF or Word format.

[0642] 6. Manual display and correction: The generated manual is presented to the user via smart glasses, allowing the user to check and correct the contents.

[0643] 7. Feedback and Updates: The manual is updated based on user feedback.

[0644] This allows for standardization of robot operating procedures on factory floors and automatic generation of manuals. It also provides the latest information at all times, contributing to improved efficiency and quality.

[0645] Prompt Sentence Examples

[0646] Analyze audio and video recordings of routine robot maintenance procedures and automatically create a manual based on the content. Extract information such as specific procedures, tools to be used, precautions, and frequency, and output it in PDF format.

[0647] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0648] Step 1:

[0649] The user uses a smartphone or smart glasses to collect interview audio and video explaining the robot's work procedures. The collected data is recorded using the camera and microphone of the smart device. The input is the work audio and video, and the output is these data files.

[0650] Step 2:

[0651] Users upload collected audio and video files to the system using an application on their smart devices. This sends the collected data to the server via the network. The input is the audio and video files, and the output is these data files stored on the server.

[0652] Step 3:

[0653] The server receives the uploaded audio and video files and stores them in an AWS S3 bucket. The specific operation of this step is to store the received data in data storage. The input is the uploaded audio and video files, and the output is the stored data files.

[0654] Step 4:

[0655] The server uses the Google Cloud Speech-to-Text API to analyze the received audio file and convert it into text data. Specifically, it uses speech recognition technology to convert the audio data into text data. The input is an audio file, and the output is text data.

[0656] Step 5:

[0657] The server uses Amazon Rekognition to analyze visual information from the received video file and extract important frames and objects. Specifically, it identifies recognized objects and actions in the video and converts them into text data. The input is the video file, and the output is the analyzed text data.

[0658] Step 6:

[0659] The server analyzes the converted text data using spaCy and Transformers (Hugging Face) and extracts the information necessary for the business (procedures, tools used, etc.). Specifically, it uses natural language processing technology to automatically select the necessary information from the text data. The input is text data, and the output is the extracted information.

[0660] Step 7:

[0661] The server organizes the extracted information and classifies it by item. Specifically, it organizes it into categories such as procedures, tools used, and setting methods. This classification process makes it easier to organize the information. The input is the extracted information, and the output is data classified by category.

[0662] Step 8:

[0663] The server uses a Python template engine to generate a manual based on the classified data. Specifically, it formats the organized information based on a template and creates a manual document. The input is the classified data, and the output is the generated manual document.

[0664] Step 9:

[0665] The terminal displays the generated manual document to the user, who can then check the content through the interface of smart glasses or a smartphone and make corrections as necessary. The input is the manual document, and the output is a modifiable manual displayed to the user.

[0666] Step 10:

[0667] The user checks whether the manual has been generated properly and sends feedback to the server via the feedback function of the terminal. The input is the user's feedback, and the output is the feedback data sent to the server.

[0668] Step 11:

[0669] The server corrects and updates the contents of the manual based on the feedback received from the user, thereby regenerating the manual to reflect the latest information. The input is the feedback data, and the output is the updated manual document.

[0670] Step 12:

[0671] The server uses the version control function to compare the current manual with previous versions and save and manage the change history. Specific operations include saving the update history and comparing it with previous versions. The input is the updated manual document, and the output is the compared version history.

[0672] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0673] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. This system generates business manuals using work-related interview audio and work videos, and also uses an emotion engine to recognize and reflect the user's emotional state.

[0674] Overall system configuration

[0675] This system mainly consists of the following three entities: users, servers, and terminals.

[0676] User

[0677] Users are business personnel or administrators who are responsible for operating the system and providing data. Specifically, they upload work-related interview audio and work videos to the system, and review and edit the content. The system also recognizes the user's emotional state.

[0678] server

[0679] The server is the core of the system, and performs voice recognition processing, data analysis, manual generation, and maintenance. The server also has an emotion engine that recognizes user emotions and stores and utilizes the analysis results.

[0680] 1. Data conversion and analysis

[0681] Server: Using voice recognition technology, it generates text data from the uploaded audio and video, and uses natural language processing technology to extract and classify information necessary for business processes.

[0682] Server: The emotion engine analyzes the user's emotions from their voice and facial expressions, and stores the results in a database.

[0683] Terminal

[0684] The terminal provides an interface for users to operate the system, displays the generated manual document, supports content confirmation and correction, and adjusts the presentation method based on the analysis results of the emotion engine.

[0685] Program processing overview

[0686] 1. Audio and Video Collection and Upload

[0687] User: Collects interview audio and work videos that explain business processes and uploads them to the system using a terminal.

[0688] 2. Data Conversion and Analysis

[0689] Server: Receives the uploaded data and converts it into text using speech recognition processing. Then, natural language processing technology is used to extract and classify the information required for the business process.

[0690] Server: The emotion engine analyzes emotions from the user's voice and facial expressions and stores the results in a database.

[0691] 3. Manual Generation

[0692] Server: Automatically generates manual documents based on the extracted information. The server embeds the classified data into a document template and generates the document in the specified format (PDF, Word, etc.).

[0693] Server: Combines the generated manual documentation with the results of the emotion engine and provides it in a format that is easy for users to read.

[0694] 4. Viewing and Modifying the Manual

[0695] Terminal: Displays the generated manual document to the user and supports confirmation and correction of the contents through the interface.

[0696] Device: Based on the analysis results of the emotion engine, the presentation method is adjusted accordingly. For example, if it determines that the user is confused, it will automatically add detailed annotations.

[0697] 5. Feedback and Maintenance

[0698] Users: Review the manual and provide feedback on any necessary corrections or additional information. Provide feedback and suggestions for improvements to the system through the feedback function.

[0699] Server: Updates the manual based on user feedback and the analysis results of the emotion engine, generates new versions, and stores and manages the update history.

[0700] Specific examples

[0701] For example, when creating a manual for how to use a new software tool, users collect and upload audio files and video instructions explaining how to use the software. The server receives these files and converts them into text using speech recognition. Next, it uses natural language processing to extract and classify information such as software installation instructions, descriptions of key functions, operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates the manual text and also analyzes the user's emotions using an emotion engine to adjust the presentation method as needed.

[0702] This not only efficiently standardizes business processes and procedures, but also provides easy-to-use manuals that reflect the user's feelings. Users can always provide the latest information, contributing to improved efficiency and quality.

[0703] The processing flow will be explained below.

[0704] Step 1:

[0705] Users: Collect job-related audio interviews or video of tasks, either by recording a worker performing a task or by recording an interview in which the worker explains a job procedure.

[0706] Step 2:

[0707] User: Uploads collected audio and video files to the system using the device interface. The user selects the files on the file selection screen and clicks the upload button.

[0708] Step 3:

[0709] Server: Receives uploaded audio and video files and saves them to storage. Checks the file format and confirms that the file was saved successfully.

[0710] Step 4:

[0711] Server: Starts the speech recognition module and converts the received voice data into text data. In the case of video, the audio portion is extracted and processed in the same way. The converted text is temporarily saved.

[0712] Step 5:

[0713] Server: Launches a natural language processing (NLP) engine and extracts information required for business processes (input, output, tools used, procedures, required skills, KPIs, etc.) from the converted text data. The extracted information is categorized and stored in a database.

[0714] Step 6:

[0715] Server: Activates the emotion engine and recognizes and analyzes the user's emotions from the uploaded audio and video data. Specifically, it identifies the user's emotional state using voice tone and facial expression recognition technology and stores the results in a database.

[0716] Step 7:

[0717] Server: Automatically generates manual documents based on the extracted business information and emotion data. Embeds the data in a document template, generates documents in the specified format (PDF, Word, etc.), and saves them in storage.

[0718] Step 8:

[0719] Terminal: Displays the generated manual document to the user. The user can view the document and check its contents through the interface.

[0720] Step 9:

[0721] User: Check the contents of the displayed manual document and make corrections as necessary. Enter the corrections in the terminal interface and click the save button.

[0722] Step 10:

[0723] Server: Receives the user's edits and updates the manual. It saves the updated version and records the change history in a database.

[0724] Step 11:

[0725] Server: Every time a manual is updated, a new version is generated and compared with the previous version. The changes are recorded as a history and made available to users.

[0726] Step 12:

[0727] Device: Based on the results of the emotion engine, the presentation method is adjusted accordingly. For example, if the user is confused, detailed annotations or additional examples are automatically inserted into the manual.

[0728] Step 13:

[0729] Users: Regularly check the updated manual and provide feedback. Submit your thoughts and suggestions for improvement to the system via the feedback function.

[0730] Step 14:

[0731] Server: Regenerates the manual based on user feedback and the analysis results of the emotion engine, and updates the new version. Saves and manages the update history, recording feedback along with previous versions.

[0732] Example 2

[0733] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0734] The conventional manual creation process had the problem of taking a great deal of time and effort to collect business-related information and generate and update the manual. Furthermore, the generated manual did not take into account the user's level of understanding or emotional state, which resulted in a decrease in the effectiveness of user training and work efficiency. Furthermore, the manual update process was cumbersome, and the latest information could not be reflected, which was also an issue.

[0735] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0736] In this invention, the server includes means for collecting business-related audio or video, means for uploading the collected audio and video data to the system, means for receiving and saving the uploaded audio and video data, means for converting the received audio and video data into text data through speech recognition processing, means for analyzing the text data and extracting information necessary for business procedures, means for organizing the extracted information and classifying it by item, means for automatically generating a procedure manual based on the classified data, means for displaying the procedure manual and allowing the user to check and modify its contents, means for generating a new version each time the procedure manual is updated and comparing it with previous versions, means for saving and managing the change history, means for periodically checking the updated procedure manual and receiving feedback, means for modifying the information based on the received feedback and generating a new procedure manual, means for analyzing the user's emotions from their voice and facial expressions, and means for adjusting the presentation method of the procedure manual based on the analysis results. This enables efficient and continuous generation and updating of business manuals, and realizes effective manual provision that corresponds to the user's level of understanding and emotional state.

[0737] "Speech recognition processing" is a technology that analyzes voice data and converts it into text data.

[0738] "Natural language processing" is a technology that analyzes text data to understand its meaning and structure.

[0739] "Business-related audio or video" refers to audio or video data recorded to explain business processes or procedures.

[0740] "Uploading means" refers to a method or device for transmitting collected audio and video data to the system.

[0741] "Means for receiving and storing" refers to a method or device that receives uploaded data and stores it in a database or file system.

[0742] "Means for converting into text data" refers to technology or devices that use voice recognition processing to convert voice data into text information.

[0743] "Information necessary for business procedures" refers to specific steps and important points related to business processes and procedures.

[0744] "Extraction means" refers to a method or device that uses natural language processing technology to extract necessary information from text data.

[0745] "Means for organizing and classifying by item" refers to techniques or devices that logically arrange extracted information and separate it into categories.

[0746] "Means for automatically generating procedure manuals" refers to methods and technologies for creating procedure manuals based on classified data in accordance with templates and formats.

[0747] The "means for displaying the procedure manual and allowing the user to check and modify the contents" refers to a technique or device that allows the user to view the generated procedure manual and provides an interface to enable modification.

[0748] "Means for generating a new version and comparing it with the previous version" refers to a technique or device that creates a new version of an updated procedure manual and compares it with the previous version to confirm the changes.

[0749] "Means for saving and managing change history" refers to technology or devices that record the update history of procedure manuals and make them accessible as needed.

[0750] "Means for receiving feedback" refers to methods or devices that allow users to input and send their opinions and suggestions for improvement.

[0751] "Means for correcting information and regenerating the procedure manual" refers to the technology or device that updates and regenerates the procedure manual based on feedback.

[0752] "Means for analyzing emotions" refers to technology or devices that analyze voice and facial expressions to identify emotional states.

[0753] "Means for adjusting presentation method" refers to technology or devices that change the display or explanation method based on the results of sentiment analysis.

[0754] The present invention provides a system that automates the standardization of business processes and procedures and creates and updates operating manuals in a sustainable manner. This system generates operating manuals using business-related audio and video data, and further uses a sentiment analysis engine to recognize and reflect the user's emotional state. An embodiment of the system is described in detail below.

[0755] System configuration

[0756] This system mainly consists of the following three entities: users, servers, and terminals.

[0757] User

[0758] Users are business personnel or administrators who are responsible for operating the system and providing data. Specifically, they collect work-related audio and video data and upload them to the system. The system also recognizes the user's emotional state.

[0759] server

[0760] The server is the core of the system, and performs voice recognition processing, data analysis, procedure manual generation, and maintenance. The server also has a sentiment analysis engine that recognizes user emotions and stores and utilizes the analysis results. The specific functions of the server are as follows:

[0761] Speech recognition processing: The server converts the voice data into text data using Google Cloud Speech-to-Text.

[0762] Natural language processing: The server uses natural language processing technology such as SpaCy to extract and classify information necessary for business procedures from text data.

[0763] Sentiment analysis: The server uses IBM Watson Emotion Analysis to analyze the user's emotions from audio and video data and stores the results in a database.

[0764] Procedure generation: The server uses LaTeX and Microsoft Word APIs to generate procedures based on the extracted information.

[0765] Information update: The server updates the procedure manual based on user feedback and generates a new version.

[0766] Terminal

[0767] The terminal provides the interface through which the user operates the system and has the following functions:

[0768] Support for displaying and modifying procedures: Provides an interface that displays the generated procedures and allows users to check and modify their contents.

[0769] Reflection of sentiment analysis results: Based on the results of sentiment analysis, the way the instructions are presented is adjusted accordingly. For example, if the user is confused, detailed annotations can be added.

[0770] Example

[0771] The following describes how this system is specifically implemented.

[0772] For example, when creating a manual for how to use a new software tool, a user collects audio files or video instructions explaining the software operation procedures and uploads them to the system. The server uses Google Cloud Speech-to-Text to convert the uploaded audio data into text. It then uses SpaCy to extract and classify the information necessary for the business procedures from the text data. Based on this information, the server automatically generates a PDF manual using LaTeX, and then uses IBM Watson Emotion Analysis to analyze the user's emotional state and adjust the manual accordingly.

[0773] Prompt Sentence Examples

[0774] Below is an example of a prompt sentence to be input into the generative AI model.

[0775] "We have an audio interview and video of an instruction manual explaining how to use a new software tool. From these, please extract installation instructions, explanations of key features, operation instructions, required settings, and troubleshooting information, and automatically generate a PDF instruction manual. Also, please analyze the user's emotional state and provide detailed annotations if the user is confused."

[0776] In this way, by providing concrete examples and prompts, users can clearly understand how the system actually works and how to use it.

[0777] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0778] Step 1:

[0779] Audio and video collection and uploading

[0780] Users collect interview audio and work videos that explain business processes and upload them to the system using a terminal.

[0781] Input: Audio and video files related to business processes

[0782] Specific operation: The user records an interview or activity using a smartphone or camera, transfers the file to a computer, and selects and uploads it from the system's web interface.

[0783] Output: Audio and video files uploaded to the system.

[0784] Step 2:

[0785] Data transformation and analysis

[0786] The server receives the uploaded data and converts it into text using speech recognition processing.

[0787] Input: Uploaded audio and video files

[0788] Specific operation: The server calls the Google Cloud Speech-to-Text API and converts the audio data into text data.

[0789] Output: Text data

[0790] The server applies natural language processing technology to the converted text data to extract and classify the information necessary for business procedures.

[0791] Input: Text data

[0792] Specific operation: The server uses SpaCy to extract noun phrases and perform relationship analysis to extract and classify important points in the business procedures.

[0793] Output: Classified business procedure information

[0794] The server uses an emotion engine to analyze the user's emotions from their voice and facial expressions, and stores the results in a database.

[0795] Input: Audio and video files

[0796] Specific operation: The server uses IBM Watson Emotion Analysis to extract emotional data from audio and video data and stores the analysis results in a database.

[0797] Output: Sentiment analysis result data

[0798] Step 3:

[0799] Procedure generation

[0800] The server automatically generates a procedure manual based on the extracted information.

[0801] Input: Classified operating procedure information

[0802] Specific operation: The server embeds the extracted and classified information into a manual template and generates a PDF procedure manual using LaTeX.

[0803] Output: Instructions in PDF format

[0804] Based on the analysis results of the emotion engine, the server adjusts the instruction manual in a way that is easy for the user to understand.

[0805] Input: PDF format instructions, sentiment analysis result data

[0806] Specific behavior: If the server determines that the user is confused based on the results of sentiment analysis, it adds an annotation to the instruction manual providing detailed explanations.

[0807] Output: Adjusted instructions

[0808] Step 4:

[0809] View and modify procedures

[0810] The terminal displays the generated procedure manual to the user and supports confirmation and correction of the contents.

[0811] Input: Adjusted Procedures

[0812] Specific behavior: Provides an interface that allows users to open PDF documents through a web interface, review the content, and add comments to corrections.

[0813] Output: User-confirmed and corrected instructions

[0814] The device will adjust the presentation method appropriately based on the analysis results of the emotion engine.

[0815] Input: Sentiment analysis result data

[0816] Specific Action: If the user is determined to be confused, provide the ability to add illustrations using visualization tools.

[0817] Output: Adjusted presentation instructions

[0818] Step 5:

[0819] Feedback and Maintenance

[0820] The user checks the contents of the procedure manual and provides feedback on any necessary corrections or additional information.

[0821] Input: User-confirmed and corrected instructions

[0822] What happens: The user fills out a feedback form to submit corrections or additional information to the system.

[0823] Output: Feedback information

[0824] The server updates the recipe based on user feedback and the emotion engine's analysis.

[0825] Input: Feedback information, sentiment analysis result data

[0826] Specific operation: The server analyzes the feedback it receives, automatically updates the relevant part of the procedure manual, and generates a new version.

[0827] Output: Updated instructions, new version generated

[0828] This enables efficient and continuous generation and updating of work procedures, and makes it possible to provide effective procedures that correspond to the user's level of understanding and emotional state.

[0829] (Application example 2)

[0830] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0831] Creating manuals for factory work procedures and business processes takes time and effort, and efficient training methods are required, especially for new employees. Furthermore, if the manuals created do not take into account the reader's feelings and level of understanding, there is a problem of inefficient learning and work execution. Furthermore, delays in updating manuals and reflecting feedback can lead to information not being kept up to date, resulting in inefficiencies in work. However, no effective means have been provided to solve these issues.

[0832] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0833] In this invention, the server includes means for collecting work-related interview audio or work video, means for uploading audio files or video files to the system, means for receiving and saving the uploaded files, means for converting the received files into text data by subjecting them to speech recognition processing, means for analyzing the text data and extracting information necessary for the business process, means for analyzing emotions from the user's voice and facial expressions using an emotion engine and generating emotion data, means for adjusting the presentation method of the manual document based on the emotion data, and means for saving and displaying the generated manual document in a specified format such as PDF or Word.

[0834] This makes it possible to efficiently extract work procedures from audio and video and automatically generate manuals that incorporate emotional data. Furthermore, the generated manuals are kept up to date and are presented appropriately, reflecting the user's emotions, enabling more effective training and business support.

[0835] "Business-related interview audio" refers to audio data obtained from interviews conducted to explain business processes.

[0836] "Work video" refers to video data that records business processes and work procedures.

[0837] "Speech recognition processing" is a technology that analyzes voice data and converts it into text data.

[0838] "Text data" refers to character string data converted by speech recognition processing.

[0839] "Information extraction" refers to the process of extracting information necessary for business processes from text data.

[0840] "Classification" refers to the process of organizing and systematizing extracted information by category.

[0841] "Manual documents" refer to documents that describe business processes and work procedures.

[0842] "Emotion engine" refers to technology for analyzing emotions from voice and facial expressions.

[0843] "Emotional Data" refers to data regarding a user's emotional state analyzed by the emotion engine.

[0844] "Adjusting the presentation method" refers to changing the way the manual document is displayed based on the emotion data.

[0845] "Feedback" refers to receiving opinions and requests from users and reflecting them in the contents of the manual.

[0846] "Application to be installed on a factory robot" refers to application software for executing the above-mentioned system on a factory robot.

[0847] "PDF and Word formats" refer to file formats for saving and displaying manual documents.

[0848] "Comparing with previous versions" refers to the process of comparing the current manual document with the version before the update and checking for changes.

[0849] This invention relates to a system that automatically generates, updates, and manages business process manuals for factory robots. This system efficiently collects business-related interview audio and work videos, and uses an emotion engine to provide manuals that reflect the user's emotional state.

[0850] System Subject

[0851] User

[0852] Users are factory workers and managers who upload work-related audio and video to the system, and are responsible for reviewing and editing the content. The system also recognizes the user's emotional state.

[0853] server

[0854] The server is the core of the system and is responsible for:

[0855] 1. Speech recognition and natural language processing: Uploaded audio and video files are received and stored, and converted into text data using speech recognition technology. The converted text data is then analyzed using natural language processing technology to extract and classify information necessary for business processes.

[0856] 2. Emotion analysis: The emotion engine analyzes the user's emotions from their voice and facial expressions, and stores the results in a database.

[0857] 3. Manual generation: Manual documents are automatically generated based on the extracted information. The generated manual documents are provided in a format that is easy for users to understand, reflecting the analysis results of the emotion engine.

[0858] Terminal

[0859] The terminal is an application installed on the factory robot that provides the following functions:

[0860] 1. Manual display and correction assistance: This system displays the generated manual document and provides an interface for users to check and correct the contents. The display method is adjusted based on the analysis results of the emotion engine.

[0861] 2. Feedback Management: Receive feedback from users and update the manual based on the latest information.

[0862] Program processing overview

[0863] The server uses libraries such as Google Speech Recognition API for speech recognition technology, nltk for natural language processing technology, and Sentiment Intensity Analyzer for emotion engine. Below is an overview of the processing for each function of the system.

[0864] 1. Speech Recognition: Uses Google Speech Recognition API to convert audio files into text data. Audio data is collected from a microphone or audio files.

[0865] 2. Natural Language Processing: Analyze text data using the nltk library to extract and classify information necessary for business processes.

[0866] 3. Sentiment Analysis: A Sentiment Intensity Analyzer is used to extract emotional information from text data. Voice and facial expression data are also analyzed.

[0867] 4. Manual generation: Manual documents are automatically generated based on the extracted and analyzed information, and saved and displayed in PDF or Word format.

[0868] Specific examples

[0869] For example, when creating an operating procedure manual for a factory robot, the user uploads audio files or video files explaining the operating procedures. The server receives these files, performs speech recognition processing, and converts them into text. Natural language processing technology is used to extract and classify information such as installation procedures, explanations of key functions, operation methods, required settings, and troubleshooting. An emotion engine also analyzes the user's emotions and adjusts the presentation method as needed.

[0870] Prompt Sentence Examples

[0871] Below is an example of a prompt sentence to input to the generative AI model.

[0872] text

[0873] "Analyze the audio files below explaining the work procedures and identify the work procedures and emotional state. Based on the results, demonstrate the process for generating a detailed manual."

[0874] In this way, the system can efficiently extract work procedures from audio and video and automatically generate manuals that incorporate emotional data. The generated manuals are kept up to date and are presented appropriately, reflecting the user's emotions, enabling more effective training and work support.

[0875] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0876] Step 1: Collect and upload work-related interview audio and video footage

[0877] The user collects interview audio and video that explain work procedures and business processes in the factory. These audio and video files are uploaded to the system using a terminal. The input is the audio and video files, and the output is the data transferred to the server. Specifically, the user selects the collected files and transfers them to cloud storage using the system's upload function.

[0878] Step 2: Receiving and storing data

[0879] The server receives uploaded audio and video files and saves them in a specified location. The input is the audio or video file from the user, and the output is data saved in a specific folder on the server. Specifically, the server processes the HTTP request and saves the received files in a database or file system.

[0880] Step 3: Speech recognition processing and text conversion

[0881] The server processes the received audio file through speech recognition and converts it into text data. This is done using the Google Speech Recognition API. The input is an audio file, and the output is the converted text data. Specifically, the server sends the audio file to the Google Speech Recognition API and saves the returned text result.

[0882] Step 4: Information extraction using natural language processing

[0883] The server analyzes the text data using natural language processing technology (nltk library) and automatically extracts the information necessary for the business process. The input is text data, and the output is the extracted information. Specifically, the server analyzes the text data and identifies work procedures and important business information.

[0884] Step 5: Sentiment analysis

[0885] The server uses a sentiment engine (Sentiment Intensity Analyzer) to analyze emotions from the user's voice and text. Voice data and text data are input, and emotional data is obtained as output. Specifically, the server inputs this data into the sentiment engine and stores the analysis results in a database.

[0886] Step 6: Automatic generation of manuals

[0887] The server automatically generates a manual document based on the extracted information and emotion data. A document template is used for this purpose. The extracted information and emotion data are input, and the generated manual is obtained as output. Specifically, the server organizes the information appropriately and creates a manual document according to the format.

[0888] Step 7: View and modify the manual

[0889] The terminal displays the generated manual document to the user and provides an interface that allows the user to check and modify the content. The generated manual document is the input, and user feedback is the output. Specifically, the terminal displays the manual document to the user and adjusts the presentation method based on the emotion data.

[0890] Step 8: Updating the manual and managing feedback

[0891] The server modifies the manual based on the feedback received from the user, generates a new version, and saves it. The input is the feedback and the existing manual, and the output is the updated manual. Specifically, the server analyzes the feedback, makes appropriate modifications, and saves the new version.

[0892] Step 9: Managing Change History

[0893] Every time a manual is updated, the server compares the new version with the previous version and saves and manages the change history. The input is the old and new manual document, and the output is the change history. Specifically, the server performs differential analysis, records the changes, and saves them in a history database.

[0894] Through these processing steps, it is possible to efficiently collect, analyze, and generate information for factory robots, and provide manuals that are always kept up to date.

[0895] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0896] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0897] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0898] [Third embodiment]

[0899] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0900] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0901] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0902] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0903] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0904] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0905] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0906] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0907] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0908] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0909] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0910] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0911] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. This system provides a means to generate business manuals by collecting, analyzing, and processing work-related interview audio and video footage.

[0912] Overall system configuration

[0913] This system mainly consists of the following three entities: users, servers, and terminals.

[0914] User

[0915] Users are business personnel and administrators who are responsible for operating the system and providing data. Specifically, they upload work-related interview audio and work videos to the system and check and edit the content.

[0916] server

[0917] The server is the core of the system, and performs voice recognition processing, data analysis, manual generation, and maintenance. The server processes the audio and video data provided by the user and converts it into text data. It then analyzes the text data, extracts and classifies the information required for the business process, and automatically generates manual documents.

[0918] Terminal

[0919] The terminal provides an interface for users to operate the system, displays the generated manual documents to users, and supports them in checking and correcting the contents. It also supports users in easily submitting correction requests through a feedback function.

[0920] Program processing overview

[0921] 1. Audio and Video Collection and Upload

[0922] User: Collects interview audio explaining business processes and actual work videos and uploads them to the system. To do this, the user uses the terminal interface.

[0923] 2. Data Conversion and Analysis

[0924] Server: Receives uploaded audio and video files and converts them into text using speech recognition technology. Based on this text data, natural language processing technology is used to analyze the information required for the job and classify it into categories such as input, output, tools used, procedures, required skills, and KPIs.

[0925] 3. Manual Generation

[0926] Server: Automatically generates manual documents based on the classified information. The manual documents are formatted in a specified format and saved in the specified format (e.g., PDF or Word file).

[0927] 4. Viewing and Modifying the Manual

[0928] Terminal: Provides an interface that displays the generated manual document to the user. Through this interface, the user can check the generated manual and manually correct it if necessary.

[0929] 5. Feedback and Maintenance

[0930] Users: Check that the manual is generated correctly and provide feedback on any necessary corrections or additions. Feedback can be easily provided using the device's functions.

[0931] Server: Receives feedback from users and corrects and updates the manual content accordingly. It also stores and manages the update history and compares the latest version with previous versions.

[0932] Specific examples

[0933] For example, when creating a manual for how to use a new software tool, the user collects and uploads audio files and operation videos that explain the software's operation procedures. The server receives these files and converts them into text using speech recognition processing. Next, natural language processing is used to extract and classify information such as software installation procedures, explanations of key functions, specific operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates a manual document, which the terminal displays to the user. The user then reviews the content and makes any necessary corrections to finalize the manual.

[0934] This allows for efficient standardization of business processes and procedures, maintains organizational knowledge in a sustainable manner, and allows users to provide up-to-date information, contributing to improved efficiency and quality.

[0935] The processing flow will be explained below.

[0936] Step 1:

[0937] User: To collect job-related audio or video interviews, record workers performing tasks or interviews explaining work procedures.

[0938] Step 2:

[0939] User: To upload collected audio and video files to the system, the user uses the device interface to select the files and clicks the upload button.

[0940] Step 3:

[0941] Server: Receives uploaded audio and video files and saves them to storage. Verifies that the files have been saved successfully.

[0942] Step 4:

[0943] Server: Starts the speech recognition module and converts the received voice data into text data. In the case of video, extracts the audio portion and performs the same process. Saves the conversion result as a text file.

[0944] Step 5:

[0945] Server: Using natural language processing (NLP) technology, the server extracts information necessary for business processes from the converted text data. Specifically, it identifies information such as inputs, outputs, tools used, procedures, required skills, and KPIs, and stores it in a database.

[0946] Step 6:

[0947] Server: Automatically generates manual documents based on the extracted information. It embeds the classified data into a document template, generates the document in the specified format (e.g., PDF, Word), and saves it in storage.

[0948] Step 7:

[0949] Terminal: Displays the generated manual document to the user, who can check the manual contents through the interface.

[0950] Step 8:

[0951] User: Check the contents of the displayed manual document and make any necessary corrections. Enter the corrections in the input field and click the Edit button to save.

[0952] Step 9:

[0953] Server: Receives the user's modifications and updates the manual document, saves the updated version, and records the change history.

[0954] Step 10:

[0955] Server: Every time a manual is updated, a new version is generated and compared with the previous version. Changes are recorded as a history and stored in a database.

[0956] Step 11:

[0957] Server: Receives feedback from users, regenerates the manual based on that feedback, and verifies that the changes based on the feedback are correctly reflected.

[0958] Step 12:

[0959] Terminal: Redisplays the updated manual to the user and provides an interface for review and feedback.

[0960] Example 1

[0961] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0962] Conventional business process standardization and manual creation involve a large amount of manual work, requiring time and effort. Furthermore, manual work can lead to errors and information not being updated in a timely manner, resulting in incomplete knowledge management within an organization. This reduces business efficiency and makes training new employees difficult. The objective of this invention is to solve these issues by providing a system that automates business process standardization and creates and updates business manuals in a sustainable manner.

[0963] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0964] In this invention, the server includes a means for converting audio data and video data into text data through speech recognition processing, a means for analyzing the text data using natural language processing technology and extracting information necessary for business processes, and a means for organizing the extracted information and classifying it by item, thereby enabling the standardization of business processes and the automation of the creation and updating of business manuals.

[0965] "Audio data" refers to files containing recordings of what was said during business process explanations and interviews.

[0966] "Video data" refers to video files that record business procedures and work procedures.

[0967] "Speech recognition processing" is a technology that converts input voice data into text data.

[0968] "Natural language processing technology" is a technology that analyzes text data and extracts and classifies semantic information.

[0969] A "business process" is a series of steps or procedures for carrying out a specific business operation.

[0970] A "manual document" is a document that describes business processes and procedures and can be referenced by users.

[0971] "Feedback" refers to corrections and additional information provided by users regarding the manual documentation.

[0972] An "update history" is a record of changes that is saved each time a manual document is updated.

[0973] The "specified format" is a format that is predetermined as the output format for the manual document, specifically, PDF or Word file format.

[0974] A "database" is a system for organizing and storing collected and converted information and analysis results.

[0975] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. The core of this system is the collection and analysis of audio and video data, manual generation, display, feedback, and management of update history.

[0976] Hardware and Software Configuration

[0977] Users use devices such as laptops and smartphones to collect audio interviews explaining business processes and videos of actual work. Voice recorders on smartphones and online conferencing tools such as Zoom can be used as audio and video recording devices.

[0978] The terminal provides an interface for users to upload collected audio and video data to the system. This interface has a dedicated upload form, and users can easily upload recorded files by dragging and dropping them.

[0979] The server receives and stores the uploaded data. Using a speech recognition engine (Google Cloud Speech-to-Text API) and a natural language processing engine (SpaCy), the server converts the voice data into text data and analyzes the text data. The analysis results are stored in a database and classified into information required for the business process (input, output, tools used, procedures, required skills, KPIs, etc.).

[0980] The generated text data is automatically formatted into a manual document on the server using LaTeX or Microsoft Word API, and the manual document is saved in the specified format (PDF or Word file format) and provided to the user.

[0981] The terminal displays the generated manual document to the user and provides an interface for the user to check or modify the content as necessary. The generated PDF and Word files can be opened in Adobe Acrobat or Microsoft Word.

[0982] The user checks the contents of the manual document and provides feedback. The feedback is sent to the system through a form on the terminal. The server receives the feedback from the user and regenerates the manual document with necessary corrections.

[0983] Furthermore, the server has the function of storing update history and comparing it with past versions, which allows you to track changes in manual documents and manage them appropriately.

[0984] Specific examples

[0985] For example, when creating a manual for a new software tool, users collect and upload audio files and operation videos that explain the software's operation procedures. The server receives these files and converts the audio to text using the Google Cloud Speech-to-Text API. It then uses SpaCy to analyze the text data and extracts and classifies information such as software installation instructions, descriptions of key features, specific operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates a PDF manual document using LaTeX and provides it to the user via their device. The user then reviews the manual, makes corrections using Adobe Acrobat, and submits their feedback to the system.

[0986] Prompt Sentence Examples

[0987] Simply upload audio and video clips demonstrating how to use a new software tool. The system will convert the audio into text, extract and categorize key information, and automatically generate a manual in PDF format. Users can review and edit the manual and provide feedback.

[0988] This system will standardize business processes and automate the creation and updating of business manuals, enabling efficient and sustainable knowledge management within the organization.

[0989] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0990] Step 1: Collect and upload audio and video

[0991] User: Records interview audio and video of work related to business processes. This is done using a recording device or a recording app on a smartphone. Specifically, the user starts the recording app on the smartphone and records the interview and work procedures. After finishing, the file is saved on the device.

[0992] Input: Audio and video files that explain the business process.

[0993] Output: Audio and video files saved on your device.

[0994] Step 2: Upload your data

[0995] User: Use the device's system interface to upload the recorded audio and video data to the system by dragging and dropping the audio and video files into the device's upload form and clicking the Upload button.

[0996] Input: Audio and video files stored on your device.

[0997] Output: Audio and video files transferred to the server.

[0998] Step 3: Save your data

[0999] Server: Receives uploaded audio and video data and saves it in a specified directory.

[1000] Input: Audio and video files transferred from your device.

[1001] Output: Audio and video files stored on the server.

[1002] Step 4: Convert audio data to text

[1003] Server: Pass the audio file to the Google Cloud Speech-to-Text API and convert it to text data. Send the audio file as an API request and save the returned text data.

[1004] Input: Audio files stored on the server.

[1005] Output: Text data converted from the audio file.

[1006] Step 5: Extract audio from video and convert it to text

[1007] Server: Extracts the audio portion from the video file and performs speech recognition processing. A video analysis tool (e.g., FFmpeg) is used to separate the audio from the video, and the audio is passed to a speech recognition engine for conversion into text data.

[1008] Input: Video files stored on the server.

[1009] Output: Text data of the audio extracted from the video file.

[1010] Step 6: Analyze and classify text data

[1011] Server: Analyzes the text data using a natural language processing engine (SpaCy) to extract and classify the necessary information (input, output, tools used, procedures, required skills, KPIs, etc.). Passes the text data to perform analysis, and stores the data classified into each category in a database.

[1012] Input: Text data converted from audio and video.

[1013] Output: Information required for the classified business process.

[1014] Step 7: Generate manual documentation

[1015] Server: Based on the classified information, manual documents are automatically generated using LaTeX and Microsoft Word API. Information is retrieved from the database, and text data is embedded in templates to generate documents.

[1016] Input: Classified information.

[1017] Output: Manual document in PDF and Word format.

[1018] Step 8: View and correct the manual

[1019] Terminal: The generated manual document is displayed to the user, and an interface is provided for checking and modifying the contents. The generated file is displayed in the terminal viewer, and the contents can be checked using a modification form or PDF editing software.

[1020] Input: Generated manual documentation.

[1021] Output: A manual document that has been checked and corrected by the user.

[1022] Step 9: Provide feedback

[1023] Users: Provide feedback on the quality of the documentation by submitting your corrections or additions using the feedback form.

[1024] Input: User feedback on the manual documentation.

[1025] Output: Feedback data sent to the system.

[1026] Step 10: Manual Updates and History Management

[1027] Server: Updates the manual based on the feedback and generates a new version. A change history is saved for each update, allowing comparison with previous versions.

[1028] Input: Feedback data from users.

[1029] Output: Updated manual documentation and changelog.

[1030] Through these steps, the system automates the standardization of business processes and procedures and provides the ability to create and update manuals in a sustainable manner.

[1031] (Application example 1)

[1032] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1033] Standardizing and maintaining work processes is extremely important in modern industrial workplaces. In particular, accurately and efficiently recording and updating the operating procedures of automated machines and robots directly contributes to improved productivity and quality control. However, creating and updating manuals manually takes time and effort, and is prone to problems such as clerical errors and missed updates. Furthermore, it is necessary to constantly provide the latest information when training new employees or troubleshooting. A system that can efficiently solve these issues is needed.

[1034] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1035] In this invention, the server includes: means for collecting work-related interview audio or work video; means for uploading the collected audio and video files to the system; means for receiving and saving the uploaded audio and video files; means for converting the received audio and video files into text data through speech recognition; means for analyzing the text data and extracting information necessary for the business process; means for organizing the extracted information and classifying it by category; means for automatically generating a manual document based on the classified data; means for displaying the manual document and allowing the user to check and modify its contents; means for generating a new version of the manual each time it is updated and comparing it with previous versions; means for saving and managing the change history; means for periodically checking the updated manual and receiving feedback; means for correcting the information based on the received feedback and generating a new manual; means for standardizing work procedures for automated machines in factories and displaying them in augmented reality; means for automatically generating operation manuals for factory robots and displaying them on smart glasses; and means for saving the update history using a version management function and comparing it with previous manuals. This automates the standardization and updating of work procedures in industrial sites, enabling the provision of always-updated information.

[1036] The "work-related interview audio" is audio data in which a field worker or manager orally explains work procedures and operation methods in a factory.

[1037] "Work video" is video data that captures the operating procedures of robots and automatic machines on factory floors.

[1038] The "means of collection" refers to a system that includes a device that uses a smartphone or smart glasses to record interview audio and work videos, as well as its operating interface.

[1039] "Uploading means" refers to software and communication means for sending collected audio and video files to a server via a network.

[1040] "Means for receiving and storing" refers to a system by which the server receives uploaded audio and video files and stores them in data storage.

[1041] "Means for processing voice data through speech recognition and converting it into text data" refers to software that analyzes voice data and converts it into text information using speech recognition technology.

[1042] "Means for analyzing text data and extracting information necessary for business processes" refers to a system that uses natural language processing technology to automatically select important business procedures and information from converted text.

[1043] "Means for classifying each item" refers to software that organizes the extracted information and compiles it into categories such as procedures, tools used, and setting methods.

[1044] "Means for automatically generating manual documents" refers to a program for creating manuals in a specified format based on organized and classified information.

[1045] The "means for displaying the manual document and allowing the user to confirm and modify the contents" is an application that provides an interface that allows the user to view the generated manual and to reflect changes as necessary.

[1046] "Means for generating new versions and comparing them with previous versions" refers to a software function that allows you to manage and compare old and new versions when updating a manual.

[1047] "Means for saving and managing change history" is a system that tracks the revision history of a manual and stores that information in a database.

[1048] "Means for receiving feedback" is a feature that provides an interface for collecting improvements and additional information from users.

[1049] The "means for correcting information and regenerating the manual" refers to software that updates and corrects the manual content based on feedback and regenerates it.

[1050] "A means to standardize the work procedures of automated machines in factories and display them in augmented reality" is a function that uniformly records the operating procedures of automated machines and visually presents them using AR technology.

[1051] "A means to automatically generate operating manuals for factory robots and display them on smart glasses" is a technology that automatically creates manuals for robot operating procedures and displays them on smart glasses.

[1052] "A means for saving update history using version control functions and comparing with past manuals" is a system that can track manual versions and compare changes between different versions.

[1053] The present invention is a system for standardizing work procedures for automated machines and robots in factories and automatically generating manuals. This system aims to improve work efficiency and quality by utilizing voice recognition, natural language processing, and augmented reality technologies. Specific embodiments of this system are described in detail below.

[1054] Overall system configuration

[1055] This system is mainly composed of three components: the user, the server, and the terminal. We will explain the specific functions of each component and how they work together.

[1056] User

[1057] The users are factory workers and managers. They use smartphones or smart glasses to collect interview audio and video explaining the robot's work procedures and upload them to the system. They also review the generated manuals, make corrections as necessary, and provide feedback.

[1058] server

[1059] The server is the core of the system, and performs data analysis, manual generation, and maintenance. Specifically, the server has the following functions:

[1060] Speech recognition processing: The collected audio files are converted into text data using the Google Cloud Speech-to-Text API.

[1061] Video analysis: Use Amazon Rekognition to extract visual information from collected video files.

[1062] Natural Language Processing (NLP): Using spaCy and Transformers (Hugging Face), text data is analyzed and information necessary for business operations is extracted.

[1063] Manual generation: A Python template engine is used to generate a manual based on the extracted information.

[1064] Database management: MongoDB is used to store and manage the generated manuals and change history.

[1065] Version control: Provides the ability to save update history and compare with previous versions.

[1066] Terminal

[1067] The terminal provides an interface for users to operate the system. Specifically, it has the following functions:

[1068] Displaying and Editing Manuals: Provides functionality for viewing and editing generated manuals through smart glasses or smartphone interfaces. React Native is used for front-end development.

[1069] Gathering Feedback: Provide an interface that allows users to easily submit requests for corrections.

[1070] Specific examples

[1071] Next, a specific example of this system will be described.

[1072] Example: Creating a manual for routine maintenance procedures for a robot

[1073] 1. Data Collection: A user uses the smart glasses to collect audio and video recordings of the robot explaining the sensor cleaning procedure.

[1074] 2. Data Upload: The collected audio and video data is uploaded to the system.

[1075] 3. Data analysis: The server converts the audio to text using the Google Cloud Speech-to-Text API and analyzes the video using Amazon Rekognition.

[1076] 4. Natural Language Processing: The converted text data is analyzed using spaCy and Transformers to extract and classify the necessary information (procedures, tools used, etc.).

[1077] 5. Manual generation: Based on the extracted data, a manual is generated using a Python template engine and saved in PDF or Word format.

[1078] 6. Manual display and correction: The generated manual is presented to the user via smart glasses, allowing the user to check and correct the contents.

[1079] 7. Feedback and Updates: The manual is updated based on user feedback.

[1080] This allows for standardization of robot operating procedures on factory floors and automatic generation of manuals. It also provides the latest information at all times, contributing to improved efficiency and quality.

[1081] Prompt Sentence Examples

[1082] Analyze audio and video recordings of routine robot maintenance procedures and automatically create a manual based on the content. Extract information such as specific procedures, tools to be used, precautions, and frequency, and output it in PDF format.

[1083] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1084] Step 1:

[1085] The user uses a smartphone or smart glasses to collect interview audio and video explaining the robot's work procedures. The collected data is recorded using the camera and microphone of the smart device. The input is the work audio and video, and the output is these data files.

[1086] Step 2:

[1087] Users upload collected audio and video files to the system using an application on their smart devices. This sends the collected data to the server via the network. The input is the audio and video files, and the output is these data files stored on the server.

[1088] Step 3:

[1089] The server receives the uploaded audio and video files and stores them in an AWS S3 bucket. The specific operation of this step is to store the received data in data storage. The input is the uploaded audio and video files, and the output is the stored data files.

[1090] Step 4:

[1091] The server uses the Google Cloud Speech-to-Text API to analyze the received audio file and convert it into text data. Specifically, it uses speech recognition technology to convert the audio data into text data. The input is an audio file, and the output is text data.

[1092] Step 5:

[1093] The server uses Amazon Rekognition to analyze visual information from the received video file and extract important frames and objects. Specifically, it identifies recognized objects and actions in the video and converts them into text data. The input is the video file, and the output is the analyzed text data.

[1094] Step 6:

[1095] The server analyzes the converted text data using spaCy and Transformers (Hugging Face) and extracts the information necessary for the business (procedures, tools used, etc.). Specifically, it uses natural language processing technology to automatically select the necessary information from the text data. The input is text data, and the output is the extracted information.

[1096] Step 7:

[1097] The server organizes the extracted information and classifies it by item. Specifically, it organizes it into categories such as procedures, tools used, and setting methods. This classification process makes it easier to organize the information. The input is the extracted information, and the output is data classified by category.

[1098] Step 8:

[1099] The server uses a Python template engine to generate a manual based on the classified data. Specifically, it formats the organized information based on a template and creates a manual document. The input is the classified data, and the output is the generated manual document.

[1100] Step 9:

[1101] The terminal displays the generated manual document to the user, who can then check the content through the interface of smart glasses or a smartphone and make corrections as necessary. The input is the manual document, and the output is a modifiable manual displayed to the user.

[1102] Step 10:

[1103] The user checks whether the manual has been generated properly and sends feedback to the server via the feedback function of the terminal. The input is the user's feedback, and the output is the feedback data sent to the server.

[1104] Step 11:

[1105] The server corrects and updates the contents of the manual based on the feedback received from the user, thereby regenerating the manual to reflect the latest information. The input is the feedback data, and the output is the updated manual document.

[1106] Step 12:

[1107] The server uses the version control function to compare the current manual with previous versions and save and manage the change history. Specific operations include saving the update history and comparing it with previous versions. The input is the updated manual document, and the output is the compared version history.

[1108] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1109] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. This system generates business manuals using work-related interview audio and work videos, and also uses an emotion engine to recognize and reflect the user's emotional state.

[1110] Overall system configuration

[1111] This system mainly consists of the following three entities: users, servers, and terminals.

[1112] User

[1113] Users are business personnel or administrators who are responsible for operating the system and providing data. Specifically, they upload work-related interview audio and work videos to the system, and review and edit the content. The system also recognizes the user's emotional state.

[1114] server

[1115] The server is the core of the system, and performs voice recognition processing, data analysis, manual generation, and maintenance. The server also has an emotion engine that recognizes user emotions and stores and utilizes the analysis results.

[1116] 1. Data conversion and analysis

[1117] Server: Using voice recognition technology, it generates text data from the uploaded audio and video, and uses natural language processing technology to extract and classify information necessary for business processes.

[1118] Server: The emotion engine analyzes the user's emotions from their voice and facial expressions, and stores the results in a database.

[1119] Terminal

[1120] The terminal provides an interface for users to operate the system, displays the generated manual document, supports content confirmation and correction, and adjusts the presentation method based on the analysis results of the emotion engine.

[1121] Program processing overview

[1122] 1. Audio and Video Collection and Upload

[1123] User: Collects interview audio and work videos that explain business processes and uploads them to the system using a terminal.

[1124] 2. Data Conversion and Analysis

[1125] Server: Receives the uploaded data and converts it into text using speech recognition processing. Then, natural language processing technology is used to extract and classify the information required for the business process.

[1126] Server: The emotion engine analyzes emotions from the user's voice and facial expressions and stores the results in a database.

[1127] 3. Manual Generation

[1128] Server: Automatically generates manual documents based on the extracted information. The server embeds the classified data into a document template and generates the document in the specified format (PDF, Word, etc.).

[1129] Server: Combines the generated manual documentation with the results of the emotion engine and provides it in a format that is easy for users to read.

[1130] 4. Viewing and Modifying the Manual

[1131] Terminal: Displays the generated manual document to the user and supports confirmation and correction of the contents through the interface.

[1132] Device: Based on the analysis results of the emotion engine, the presentation method is adjusted accordingly. For example, if it determines that the user is confused, it will automatically add detailed annotations.

[1133] 5. Feedback and Maintenance

[1134] Users: Review the manual and provide feedback on any necessary corrections or additional information. Provide feedback and suggestions for improvements to the system through the feedback function.

[1135] Server: Updates the manual based on user feedback and the analysis results of the emotion engine, generates new versions, and stores and manages the update history.

[1136] Specific examples

[1137] For example, when creating a manual for how to use a new software tool, users collect and upload audio files and video instructions explaining how to use the software. The server receives these files and converts them into text using speech recognition. Next, it uses natural language processing to extract and classify information such as software installation instructions, descriptions of key functions, operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates the manual text and also analyzes the user's emotions using an emotion engine to adjust the presentation method as needed.

[1138] This not only efficiently standardizes business processes and procedures, but also provides easy-to-use manuals that reflect the user's feelings. Users can always provide the latest information, contributing to improved efficiency and quality.

[1139] The processing flow will be explained below.

[1140] Step 1:

[1141] Users: Collect job-related audio interviews or video of tasks, either by recording a worker performing a task or by recording an interview in which the worker explains a job procedure.

[1142] Step 2:

[1143] User: Uploads collected audio and video files to the system using the device interface. The user selects the files on the file selection screen and clicks the upload button.

[1144] Step 3:

[1145] Server: Receives uploaded audio and video files and saves them to storage. Checks the file format and confirms that the file was saved successfully.

[1146] Step 4:

[1147] Server: Starts the speech recognition module and converts the received voice data into text data. In the case of video, the audio portion is extracted and processed in the same way. The converted text is temporarily saved.

[1148] Step 5:

[1149] Server: Launches a natural language processing (NLP) engine and extracts information required for business processes (input, output, tools used, procedures, required skills, KPIs, etc.) from the converted text data. The extracted information is categorized and stored in a database.

[1150] Step 6:

[1151] Server: Activates the emotion engine and recognizes and analyzes the user's emotions from the uploaded audio and video data. Specifically, it identifies the user's emotional state using voice tone and facial expression recognition technology and stores the results in a database.

[1152] Step 7:

[1153] Server: Automatically generates manual documents based on the extracted business information and emotion data. Embeds the data in a document template, generates documents in the specified format (PDF, Word, etc.), and saves them in storage.

[1154] Step 8:

[1155] Terminal: Displays the generated manual document to the user. The user can view the document and check its contents through the interface.

[1156] Step 9:

[1157] User: Check the contents of the displayed manual document and make corrections as necessary. Enter the corrections in the terminal interface and click the save button.

[1158] Step 10:

[1159] Server: Receives the user's edits and updates the manual. It saves the updated version and records the change history in a database.

[1160] Step 11:

[1161] Server: Every time a manual is updated, a new version is generated and compared with the previous version. The changes are recorded as a history and made available to users.

[1162] Step 12:

[1163] Device: Based on the results of the emotion engine, the presentation method is adjusted accordingly. For example, if the user is confused, detailed annotations or additional examples are automatically inserted into the manual.

[1164] Step 13:

[1165] Users: Regularly check the updated manual and provide feedback. Submit your thoughts and suggestions for improvement to the system via the feedback function.

[1166] Step 14:

[1167] Server: Regenerates the manual based on user feedback and the analysis results of the emotion engine, and updates the new version. Saves and manages the update history, recording feedback along with previous versions.

[1168] Example 2

[1169] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1170] The conventional manual creation process had the problem of taking a great deal of time and effort to collect business-related information and generate and update the manual. Furthermore, the generated manual did not take into account the user's level of understanding or emotional state, which resulted in a decrease in the effectiveness of user training and work efficiency. Furthermore, the manual update process was cumbersome, and the latest information could not be reflected, which was also an issue.

[1171] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1172] In this invention, the server includes means for collecting business-related audio or video, means for uploading the collected audio and video data to the system, means for receiving and saving the uploaded audio and video data, means for converting the received audio and video data into text data through speech recognition processing, means for analyzing the text data and extracting information necessary for business procedures, means for organizing the extracted information and classifying it by item, means for automatically generating a procedure manual based on the classified data, means for displaying the procedure manual and allowing the user to check and modify its contents, means for generating a new version each time the procedure manual is updated and comparing it with previous versions, means for saving and managing the change history, means for periodically checking the updated procedure manual and receiving feedback, means for modifying the information based on the received feedback and generating a new procedure manual, means for analyzing the user's emotions from their voice and facial expressions, and means for adjusting the presentation method of the procedure manual based on the analysis results. This enables efficient and continuous generation and updating of business manuals, and realizes effective manual provision that corresponds to the user's level of understanding and emotional state.

[1173] "Speech recognition processing" is a technology that analyzes voice data and converts it into text data.

[1174] "Natural language processing" is a technology that analyzes text data to understand its meaning and structure.

[1175] "Business-related audio or video" refers to audio or video data recorded to explain business processes or procedures.

[1176] "Uploading means" refers to a method or device for transmitting collected audio and video data to the system.

[1177] "Means for receiving and storing" refers to a method or device that receives uploaded data and stores it in a database or file system.

[1178] "Means for converting into text data" refers to technology or devices that use voice recognition processing to convert voice data into text information.

[1179] "Information necessary for business procedures" refers to specific steps and important points related to business processes and procedures.

[1180] "Extraction means" refers to a method or device that uses natural language processing technology to extract necessary information from text data.

[1181] "Means for organizing and classifying by item" refers to techniques or devices that logically arrange extracted information and separate it into categories.

[1182] "Means for automatically generating procedure manuals" refers to methods and technologies for creating procedure manuals based on classified data in accordance with templates and formats.

[1183] The "means for displaying the procedure manual and allowing the user to check and modify the contents" refers to a technique or device that allows the user to view the generated procedure manual and provides an interface to enable modification.

[1184] "Means for generating a new version and comparing it with the previous version" refers to a technique or device that creates a new version of an updated procedure manual and compares it with the previous version to confirm the changes.

[1185] "Means for saving and managing change history" refers to technology or devices that record the update history of procedure manuals and make them accessible as needed.

[1186] "Means for receiving feedback" refers to methods or devices that allow users to input and send their opinions and suggestions for improvement.

[1187] "Means for correcting information and regenerating the procedure manual" refers to the technology or device that updates and regenerates the procedure manual based on feedback.

[1188] "Means for analyzing emotions" refers to technology or devices that analyze voice and facial expressions to identify emotional states.

[1189] "Means for adjusting presentation method" refers to technology or devices that change the display or explanation method based on the results of sentiment analysis.

[1190] The present invention provides a system that automates the standardization of business processes and procedures and creates and updates operating manuals in a sustainable manner. This system generates operating manuals using business-related audio and video data, and further uses a sentiment analysis engine to recognize and reflect the user's emotional state. An embodiment of the system is described in detail below.

[1191] System configuration

[1192] This system mainly consists of the following three entities: users, servers, and terminals.

[1193] User

[1194] Users are business personnel or administrators who are responsible for operating the system and providing data. Specifically, they collect work-related audio and video data and upload them to the system. The system also recognizes the user's emotional state.

[1195] server

[1196] The server is the core of the system, and performs voice recognition processing, data analysis, procedure manual generation, and maintenance. The server also has a sentiment analysis engine that recognizes user emotions and stores and utilizes the analysis results. The specific functions of the server are as follows:

[1197] Speech recognition processing: The server converts the voice data into text data using Google Cloud Speech-to-Text.

[1198] Natural language processing: The server uses natural language processing technology such as SpaCy to extract and classify information necessary for business procedures from text data.

[1199] Sentiment analysis: The server uses IBM Watson Emotion Analysis to analyze the user's emotions from audio and video data and stores the results in a database.

[1200] Procedure generation: The server uses LaTeX and Microsoft Word APIs to generate procedures based on the extracted information.

[1201] Information update: The server updates the procedure manual based on user feedback and generates a new version.

[1202] Terminal

[1203] The terminal provides the interface through which the user operates the system and has the following functions:

[1204] Support for displaying and modifying procedures: Provides an interface that displays the generated procedures and allows users to check and modify their contents.

[1205] Reflection of sentiment analysis results: Based on the results of sentiment analysis, the way the instructions are presented is adjusted accordingly. For example, if the user is confused, detailed annotations can be added.

[1206] Example

[1207] The following describes how this system is specifically implemented.

[1208] For example, when creating a manual for how to use a new software tool, a user collects audio files or video instructions explaining the software operation procedures and uploads them to the system. The server uses Google Cloud Speech-to-Text to convert the uploaded audio data into text. It then uses SpaCy to extract and classify the information necessary for the business procedures from the text data. Based on this information, the server automatically generates a PDF manual using LaTeX, and then uses IBM Watson Emotion Analysis to analyze the user's emotional state and adjust the manual accordingly.

[1209] Prompt Sentence Examples

[1210] Below is an example of a prompt sentence to be input into the generative AI model.

[1211] "We have an audio interview and video of an instruction manual explaining how to use a new software tool. From these, please extract installation instructions, explanations of key features, operation instructions, required settings, and troubleshooting information, and automatically generate a PDF instruction manual. Also, please analyze the user's emotional state and provide detailed annotations if the user is confused."

[1212] In this way, by providing concrete examples and prompts, users can clearly understand how the system actually works and how to use it.

[1213] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1214] Step 1:

[1215] Audio and video collection and uploading

[1216] Users collect interview audio and work videos that explain business processes and upload them to the system using a terminal.

[1217] Input: Audio and video files related to business processes

[1218] Specific operation: The user records an interview or activity using a smartphone or camera, transfers the file to a computer, and selects and uploads it from the system's web interface.

[1219] Output: Audio and video files uploaded to the system.

[1220] Step 2:

[1221] Data transformation and analysis

[1222] The server receives the uploaded data and converts it into text using speech recognition processing.

[1223] Input: Uploaded audio and video files

[1224] Specific operation: The server calls the Google Cloud Speech-to-Text API and converts the audio data into text data.

[1225] Output: Text data

[1226] The server applies natural language processing technology to the converted text data to extract and classify the information necessary for business procedures.

[1227] Input: Text data

[1228] Specific operation: The server uses SpaCy to extract noun phrases and perform relationship analysis to extract and classify important points in the business procedures.

[1229] Output: Classified business procedure information

[1230] The server uses an emotion engine to analyze the user's emotions from their voice and facial expressions, and stores the results in a database.

[1231] Input: Audio and video files

[1232] Specific operation: The server uses IBM Watson Emotion Analysis to extract emotional data from audio and video data and stores the analysis results in a database.

[1233] Output: Sentiment analysis result data

[1234] Step 3:

[1235] Procedure generation

[1236] The server automatically generates a procedure manual based on the extracted information.

[1237] Input: Classified operating procedure information

[1238] Specific operation: The server embeds the extracted and classified information into a manual template and generates a PDF procedure manual using LaTeX.

[1239] Output: Instructions in PDF format

[1240] Based on the analysis results of the emotion engine, the server adjusts the instruction manual in a way that is easy for the user to understand.

[1241] Input: PDF format instructions, sentiment analysis result data

[1242] Specific behavior: If the server determines that the user is confused based on the results of sentiment analysis, it adds an annotation to the instruction manual providing detailed explanations.

[1243] Output: Adjusted instructions

[1244] Step 4:

[1245] View and modify procedures

[1246] The terminal displays the generated procedure manual to the user and supports confirmation and correction of the contents.

[1247] Input: Adjusted Procedures

[1248] Specific behavior: Provides an interface that allows users to open PDF documents through a web interface, review the content, and add comments to corrections.

[1249] Output: User-confirmed and corrected instructions

[1250] The device will adjust the presentation method appropriately based on the analysis results of the emotion engine.

[1251] Input: Sentiment analysis result data

[1252] Specific Action: If the user is determined to be confused, provide the ability to add illustrations using visualization tools.

[1253] Output: Adjusted presentation instructions

[1254] Step 5:

[1255] Feedback and Maintenance

[1256] The user checks the contents of the procedure manual and provides feedback on any necessary corrections or additional information.

[1257] Input: User-confirmed and corrected instructions

[1258] What happens: The user fills out a feedback form to submit corrections or additional information to the system.

[1259] Output: Feedback information

[1260] The server updates the recipe based on user feedback and the emotion engine's analysis.

[1261] Input: Feedback information, sentiment analysis result data

[1262] Specific operation: The server analyzes the feedback it receives, automatically updates the relevant part of the procedure manual, and generates a new version.

[1263] Output: Updated instructions, new version generated

[1264] This enables efficient and continuous generation and updating of work procedures, and makes it possible to provide effective procedures that correspond to the user's level of understanding and emotional state.

[1265] (Application example 2)

[1266] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1267] Creating manuals for factory work procedures and business processes takes time and effort, and efficient training methods are required, especially for new employees. Furthermore, if the manuals created do not take into account the reader's feelings and level of understanding, there is a problem of inefficient learning and work execution. Furthermore, delays in updating manuals and reflecting feedback can lead to information not being kept up to date, resulting in inefficiencies in work. However, no effective means have been provided to solve these issues.

[1268] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1269] In this invention, the server includes means for collecting work-related interview audio or work video, means for uploading audio files or video files to the system, means for receiving and saving the uploaded files, means for converting the received files into text data by subjecting them to speech recognition processing, means for analyzing the text data and extracting information necessary for the business process, means for analyzing emotions from the user's voice and facial expressions using an emotion engine and generating emotion data, means for adjusting the presentation method of the manual document based on the emotion data, and means for saving and displaying the generated manual document in a specified format such as PDF or Word.

[1270] This makes it possible to efficiently extract work procedures from audio and video and automatically generate manuals that incorporate emotional data. Furthermore, the generated manuals are kept up to date and are presented appropriately, reflecting the user's emotions, enabling more effective training and business support.

[1271] "Business-related interview audio" refers to audio data obtained from interviews conducted to explain business processes.

[1272] "Work video" refers to video data that records business processes and work procedures.

[1273] "Speech recognition processing" is a technology that analyzes voice data and converts it into text data.

[1274] "Text data" refers to character string data converted by speech recognition processing.

[1275] "Information extraction" refers to the process of extracting information necessary for business processes from text data.

[1276] "Classification" refers to the process of organizing and systematizing extracted information by category.

[1277] "Manual documents" refer to documents that describe business processes and work procedures.

[1278] "Emotion engine" refers to technology for analyzing emotions from voice and facial expressions.

[1279] "Emotional Data" refers to data regarding a user's emotional state analyzed by the emotion engine.

[1280] "Adjusting the presentation method" refers to changing the way the manual document is displayed based on the emotion data.

[1281] "Feedback" refers to receiving opinions and requests from users and reflecting them in the contents of the manual.

[1282] "Application to be installed on a factory robot" refers to application software for executing the above-mentioned system on a factory robot.

[1283] "PDF and Word formats" refer to file formats for saving and displaying manual documents.

[1284] "Comparing with previous versions" refers to the process of comparing the current manual document with the version before the update and checking for changes.

[1285] This invention relates to a system that automatically generates, updates, and manages business process manuals for factory robots. This system efficiently collects business-related interview audio and work videos, and uses an emotion engine to provide manuals that reflect the user's emotional state.

[1286] System Subject

[1287] User

[1288] Users are factory workers and managers who upload work-related audio and video to the system, and are responsible for reviewing and editing the content. The system also recognizes the user's emotional state.

[1289] server

[1290] The server is the core of the system and is responsible for:

[1291] 1. Speech recognition and natural language processing: Uploaded audio and video files are received and stored, and converted into text data using speech recognition technology. The converted text data is then analyzed using natural language processing technology to extract and classify information necessary for business processes.

[1292] 2. Emotion analysis: The emotion engine analyzes the user's emotions from their voice and facial expressions, and stores the results in a database.

[1293] 3. Manual generation: Manual documents are automatically generated based on the extracted information. The generated manual documents are provided in a format that is easy for users to understand, reflecting the analysis results of the emotion engine.

[1294] Terminal

[1295] The terminal is an application installed on the factory robot that provides the following functions:

[1296] 1. Manual display and correction assistance: This system displays the generated manual document and provides an interface for users to check and correct the contents. The display method is adjusted based on the analysis results of the emotion engine.

[1297] 2. Feedback Management: Receive feedback from users and update the manual based on the latest information.

[1298] Program processing overview

[1299] The server uses libraries such as Google Speech Recognition API for speech recognition technology, nltk for natural language processing technology, and Sentiment Intensity Analyzer for emotion engine. Below is an overview of the processing for each function of the system.

[1300] 1. Speech Recognition: Uses Google Speech Recognition API to convert audio files into text data. Audio data is collected from a microphone or audio files.

[1301] 2. Natural Language Processing: Analyze text data using the nltk library to extract and classify information necessary for business processes.

[1302] 3. Sentiment Analysis: A Sentiment Intensity Analyzer is used to extract emotional information from text data. Voice and facial expression data are also analyzed.

[1303] 4. Manual generation: Manual documents are automatically generated based on the extracted and analyzed information, and saved and displayed in PDF or Word format.

[1304] Specific examples

[1305] For example, when creating an operating procedure manual for a factory robot, the user uploads audio files or video files explaining the operating procedures. The server receives these files, performs speech recognition processing, and converts them into text. Natural language processing technology is used to extract and classify information such as installation procedures, explanations of key functions, operation methods, required settings, and troubleshooting. An emotion engine also analyzes the user's emotions and adjusts the presentation method as needed.

[1306] Prompt Sentence Examples

[1307] Below is an example of a prompt sentence to input to the generative AI model.

[1308] text

[1309] "Analyze the audio files below explaining the work procedures and identify the work procedures and emotional state. Based on the results, demonstrate the process for generating a detailed manual."

[1310] In this way, the system can efficiently extract work procedures from audio and video and automatically generate manuals that incorporate emotional data. The generated manuals are kept up to date and are presented appropriately, reflecting the user's emotions, enabling more effective training and work support.

[1311] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1312] Step 1: Collect and upload work-related interview audio and video footage

[1313] The user collects interview audio and video that explain work procedures and business processes in the factory. These audio and video files are uploaded to the system using a terminal. The input is the audio and video files, and the output is the data transferred to the server. Specifically, the user selects the collected files and transfers them to cloud storage using the system's upload function.

[1314] Step 2: Receiving and storing data

[1315] The server receives uploaded audio and video files and saves them in a specified location. The input is the audio or video file from the user, and the output is data saved in a specific folder on the server. Specifically, the server processes the HTTP request and saves the received files in a database or file system.

[1316] Step 3: Speech recognition processing and text conversion

[1317] The server processes the received audio file through speech recognition and converts it into text data. This is done using the Google Speech Recognition API. The input is an audio file, and the output is the converted text data. Specifically, the server sends the audio file to the Google Speech Recognition API and saves the returned text result.

[1318] Step 4: Information extraction using natural language processing

[1319] The server analyzes the text data using natural language processing technology (nltk library) and automatically extracts the information necessary for the business process. The input is text data, and the output is the extracted information. Specifically, the server analyzes the text data and identifies work procedures and important business information.

[1320] Step 5: Sentiment analysis

[1321] The server uses a sentiment engine (Sentiment Intensity Analyzer) to analyze emotions from the user's voice and text. Voice data and text data are input, and emotional data is obtained as output. Specifically, the server inputs this data into the sentiment engine and stores the analysis results in a database.

[1322] Step 6: Automatic generation of manuals

[1323] The server automatically generates a manual document based on the extracted information and emotion data. A document template is used for this purpose. The extracted information and emotion data are input, and the generated manual is obtained as output. Specifically, the server organizes the information appropriately and creates a manual document according to the format.

[1324] Step 7: View and modify the manual

[1325] The terminal displays the generated manual document to the user and provides an interface that allows the user to check and modify the content. The generated manual document is the input, and user feedback is the output. Specifically, the terminal displays the manual document to the user and adjusts the presentation method based on the emotion data.

[1326] Step 8: Updating the manual and managing feedback

[1327] The server modifies the manual based on the feedback received from the user, generates a new version, and saves it. The input is the feedback and the existing manual, and the output is the updated manual. Specifically, the server analyzes the feedback, makes appropriate modifications, and saves the new version.

[1328] Step 9: Managing Change History

[1329] Every time a manual is updated, the server compares the new version with the previous version and saves and manages the change history. The input is the old and new manual document, and the output is the change history. Specifically, the server performs differential analysis, records the changes, and saves them in a history database.

[1330] Through these processing steps, it is possible to efficiently collect, analyze, and generate information for factory robots, and provide manuals that are always kept up to date.

[1331] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1332] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1333] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1334] [Fourth embodiment]

[1335] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1336] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1337] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1338] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1339] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1340] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1341] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1342] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1343] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1344] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1345] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1346] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1347] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1348] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. This system provides a means to generate business manuals by collecting, analyzing, and processing work-related interview audio and work videos.

[1349] Overall system configuration

[1350] This system mainly consists of the following three entities: users, servers, and terminals.

[1351] User

[1352] Users are business personnel and administrators who are responsible for operating the system and providing data. Specifically, they upload work-related interview audio and work videos to the system and check and edit the content.

[1353] server

[1354] The server is the core of the system, and performs voice recognition processing, data analysis, manual generation, and maintenance. The server processes the audio and video data provided by the user and converts it into text data. It then analyzes the text data, extracts and classifies the information required for the business process, and automatically generates manual documents.

[1355] Terminal

[1356] The terminal provides an interface for users to operate the system, displays the generated manual documents to users, and supports them in checking and correcting the contents. It also supports users in easily submitting correction requests through a feedback function.

[1357] Program processing overview

[1358] 1. Audio and Video Collection and Upload

[1359] User: Collects interview audio explaining business processes and actual work videos and uploads them to the system. To do this, the user uses the terminal interface.

[1360] 2. Data Conversion and Analysis

[1361] Server: Receives uploaded audio and video files and converts them into text using speech recognition technology. Based on this text data, natural language processing technology is used to analyze the information required for the job and classify it into categories such as input, output, tools used, procedures, required skills, and KPIs.

[1362] 3. Manual Generation

[1363] Server: Automatically generates manual documents based on the classified information. The manual documents are formatted in a specified format and saved in the specified format (e.g., PDF or Word file).

[1364] 4. Viewing and Modifying the Manual

[1365] Terminal: Provides an interface that displays the generated manual document to the user. Through this interface, the user can check the generated manual and manually correct it if necessary.

[1366] 5. Feedback and Maintenance

[1367] Users: Check that the manual is generated correctly and provide feedback on any necessary corrections or additions. Feedback can be easily provided using the device's functions.

[1368] Server: Receives feedback from users and corrects and updates the manual content accordingly. It also stores and manages the update history and compares the latest version with previous versions.

[1369] Specific examples

[1370] For example, when creating a manual for how to use a new software tool, the user collects and uploads audio files and operation videos that explain the software's operation procedures. The server receives these files and converts them into text using speech recognition processing. Next, natural language processing is used to extract and classify information such as software installation procedures, explanations of key functions, specific operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates a manual document, which the terminal displays to the user. The user then reviews the content and makes any necessary corrections to finalize the manual.

[1371] This allows for efficient standardization of business processes and procedures, maintains organizational knowledge in a sustainable manner, and allows users to provide up-to-date information, contributing to improved efficiency and quality.

[1372] The processing flow will be explained below.

[1373] Step 1:

[1374] User: To collect job-related audio or video interviews, record personnel performing tasks or interviews explaining work procedures.

[1375] Step 2:

[1376] User: To upload collected audio and video files to the system, the user uses the device interface to select the files and clicks the upload button.

[1377] Step 3:

[1378] Server: Receives uploaded audio and video files and saves them to storage. Verifies that the files have been saved successfully.

[1379] Step 4:

[1380] Server: Starts the speech recognition module and converts the received voice data into text data. In the case of video, extracts the audio portion and performs the same process. Saves the conversion result as a text file.

[1381] Step 5:

[1382] Server: Using natural language processing (NLP) technology, the server extracts information necessary for business processes from the converted text data. Specifically, it identifies information such as inputs, outputs, tools used, procedures, required skills, and KPIs, and stores it in a database.

[1383] Step 6:

[1384] Server: Automatically generates manual documents based on the extracted information. It embeds the classified data into a document template, generates the document in the specified format (e.g., PDF, Word), and saves it in storage.

[1385] Step 7:

[1386] Terminal: Displays the generated manual document to the user, who can check the manual contents through the interface.

[1387] Step 8:

[1388] User: Check the contents of the displayed manual document and make any necessary corrections. Enter the corrections in the input field and click the Edit button to save.

[1389] Step 9:

[1390] Server: Receives the user's modifications and updates the manual document, saves the updated version, and records the change history.

[1391] Step 10:

[1392] Server: Every time a manual is updated, a new version is generated and compared with the previous version. Changes are recorded as a history and stored in a database.

[1393] Step 11:

[1394] Server: Receives feedback from users, regenerates the manual based on that feedback, and verifies that the changes based on the feedback are correctly reflected.

[1395] Step 12:

[1396] Terminal: Redisplays the updated manual to the user and provides an interface for review and feedback.

[1397] Example 1

[1398] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1399] Conventional business process standardization and manual creation involve a large amount of manual work, requiring time and effort. Furthermore, manual work can lead to errors and information not being updated in a timely manner, resulting in incomplete knowledge management within an organization. This reduces business efficiency and makes training new employees difficult. The objective of this invention is to solve these issues by providing a system that automates business process standardization and creates and updates business manuals in a sustainable manner.

[1400] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1401] In this invention, the server includes a means for converting audio data and video data into text data through speech recognition processing, a means for analyzing the text data using natural language processing technology and extracting information necessary for business processes, and a means for organizing the extracted information and classifying it by item, thereby enabling the standardization of business processes and the automation of the creation and updating of business manuals.

[1402] "Audio data" refers to files containing recordings of what was said during business process explanations and interviews.

[1403] "Video data" refers to video files that record business procedures and work procedures.

[1404] "Speech recognition processing" is a technology that converts input voice data into text data.

[1405] "Natural language processing technology" is a technology that analyzes text data and extracts and classifies semantic information.

[1406] A "business process" is a series of steps or procedures for carrying out a specific business operation.

[1407] A "manual document" is a document that describes business processes and procedures and can be referenced by users.

[1408] "Feedback" refers to corrections and additional information provided by users regarding the manual documentation.

[1409] An "update history" is a record of changes that is saved each time a manual document is updated.

[1410] The "specified format" is a format that is predetermined as the output format for the manual document, specifically, PDF or Word file format.

[1411] A "database" is a system for organizing and storing collected and converted information and analysis results.

[1412] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. The core of this system is the collection and analysis of audio and video data, manual generation, display, feedback, and management of update history.

[1413] Hardware and Software Configuration

[1414] Users use devices such as laptops and smartphones to collect audio interviews explaining business processes and videos of actual work. Voice recorders on smartphones and online conferencing tools such as Zoom can be used as audio and video recording devices.

[1415] The terminal provides an interface for users to upload collected audio and video data to the system. This interface has a dedicated upload form, and users can easily upload recorded files by dragging and dropping them.

[1416] The server receives and stores the uploaded data. Using a speech recognition engine (Google Cloud Speech-to-Text API) and a natural language processing engine (SpaCy), the server converts the voice data into text data and analyzes the text data. The analysis results are stored in a database and classified into information required for the business process (input, output, tools used, procedures, required skills, KPIs, etc.).

[1417] The generated text data is automatically formatted into a manual document on the server using LaTeX or Microsoft Word API, and the manual document is saved in the specified format (PDF or Word file format) and provided to the user.

[1418] The terminal displays the generated manual document to the user and provides an interface for the user to check or modify the content as necessary. The generated PDF and Word files can be opened in Adobe Acrobat or Microsoft Word.

[1419] The user checks the contents of the manual document and provides feedback. The feedback is sent to the system through a form on the terminal. The server receives the feedback from the user and regenerates the manual document with necessary corrections.

[1420] Furthermore, the server has the function of storing update history and comparing it with past versions, which allows you to track changes in manual documents and manage them appropriately.

[1421] Specific examples

[1422] For example, when creating a manual for a new software tool, users collect and upload audio files and operation videos that explain the software's operation procedures. The server receives these files and converts the audio to text using the Google Cloud Speech-to-Text API. It then uses SpaCy to analyze the text data and extract and classify information such as software installation instructions, descriptions of key features, specific operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates a PDF manual document using LaTeX and provides it to the user via their device. The user then reviews the manual, makes corrections using Adobe Acrobat, and submits feedback to the system.

[1423] Prompt Sentence Examples

[1424] Simply upload audio and video clips demonstrating how to use a new software tool. The system will convert the audio into text, extract and categorize key information, and automatically generate a manual in PDF format. Users can review and edit the manual and provide feedback.

[1425] This system will standardize business processes and automate the creation and updating of business manuals, enabling efficient and sustainable knowledge management within the organization.

[1426] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1427] Step 1: Collect and upload audio and video

[1428] User: Records interview audio and video of work related to business processes. This is done using a recording device or a recording app on a smartphone. Specifically, the user starts the recording app on the smartphone and records the interview and work procedures. After finishing, the file is saved on the device.

[1429] Input: Audio and video files that explain the business process.

[1430] Output: Audio and video files saved on your device.

[1431] Step 2: Upload your data

[1432] User: Use the device's system interface to upload the recorded audio and video data to the system by dragging and dropping the audio and video files into the device's upload form and clicking the Upload button.

[1433] Input: Audio and video files stored on your device.

[1434] Output: Audio and video files transferred to the server.

[1435] Step 3: Save your data

[1436] Server: Receives uploaded audio and video data and saves it in a specified directory.

[1437] Input: Audio and video files transferred from your device.

[1438] Output: Audio and video files stored on the server.

[1439] Step 4: Convert audio data to text

[1440] Server: Pass the audio file to the Google Cloud Speech-to-Text API and convert it to text data. Send the audio file as an API request and save the returned text data.

[1441] Input: Audio files stored on the server.

[1442] Output: Text data converted from the audio file.

[1443] Step 5: Extract audio from video and convert it to text

[1444] Server: Extracts the audio portion from the video file and performs speech recognition processing. A video analysis tool (e.g., FFmpeg) is used to separate the audio from the video, and the audio is passed to a speech recognition engine for conversion into text data.

[1445] Input: Video files stored on the server.

[1446] Output: Text data of the audio extracted from the video file.

[1447] Step 6: Analyze and classify text data

[1448] Server: Analyzes the text data using a natural language processing engine (SpaCy) to extract and classify the necessary information (input, output, tools used, procedures, required skills, KPIs, etc.). Passes the text data to perform analysis, and stores the data classified into each category in a database.

[1449] Input: Text data converted from audio and video.

[1450] Output: Information required for the classified business process.

[1451] Step 7: Generate manual documentation

[1452] Server: Based on the classified information, manual documents are automatically generated using LaTeX and Microsoft Word API. Information is retrieved from the database, and text data is embedded in templates to generate documents.

[1453] Input: Classified information.

[1454] Output: Manual document in PDF and Word format.

[1455] Step 8: View and correct the manual

[1456] Terminal: The generated manual document is displayed to the user, and an interface is provided for checking and modifying the contents. The generated file is displayed in the terminal viewer, and the contents can be checked using a modification form or PDF editing software.

[1457] Input: Generated manual documentation.

[1458] Output: A manual document that has been checked and corrected by the user.

[1459] Step 9: Provide feedback

[1460] Users: Provide feedback on the quality of the documentation by submitting your corrections or additions using the feedback form.

[1461] Input: User feedback on the manual documentation.

[1462] Output: Feedback data sent to the system.

[1463] Step 10: Manual Updates and History Management

[1464] Server: Updates the manual based on the feedback and generates a new version. A change history is saved for each update, allowing comparison with previous versions.

[1465] Input: Feedback data from users.

[1466] Output: Updated manual documentation and changelog.

[1467] Through these steps, the system automates the standardization of business processes and procedures and provides the ability to create and update manuals in a sustainable manner.

[1468] (Application example 1)

[1469] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1470] Standardizing and maintaining work processes is extremely important in modern industrial workplaces. In particular, accurately and efficiently recording and updating the operating procedures of automated machines and robots directly contributes to improved productivity and quality control. However, creating and updating manuals manually takes time and effort, and is prone to problems such as clerical errors and missed updates. Furthermore, it is necessary to constantly provide the latest information when training new employees or troubleshooting. A system that can efficiently solve these issues is needed.

[1471] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1472] In this invention, the server includes: means for collecting work-related interview audio or work video; means for uploading the collected audio and video files to the system; means for receiving and saving the uploaded audio and video files; means for converting the received audio and video files into text data through speech recognition; means for analyzing the text data and extracting information necessary for the business process; means for organizing the extracted information and classifying it by category; means for automatically generating a manual document based on the classified data; means for displaying the manual document and allowing the user to check and modify its contents; means for generating a new version of the manual each time it is updated and comparing it with previous versions; means for saving and managing the change history; means for periodically checking the updated manual and receiving feedback; means for correcting the information based on the received feedback and generating a new manual; means for standardizing work procedures for automated machines in factories and displaying them in augmented reality; means for automatically generating operation manuals for factory robots and displaying them on smart glasses; and means for saving the update history using a version management function and comparing it with previous manuals. This automates the standardization and updating of work procedures in industrial sites, enabling the provision of always-updated information.

[1473] The "work-related interview audio" is audio data in which a field worker or manager orally explains work procedures and operation methods in a factory.

[1474] "Work video" is video data that captures the operating procedures of robots and automatic machines on factory floors.

[1475] The "means of collection" refers to a system that includes a device that uses a smartphone or smart glasses to record interview audio and work videos, as well as its operating interface.

[1476] "Uploading means" refers to software and communication means for sending collected audio and video files to a server via a network.

[1477] "Means for receiving and storing" refers to a system by which the server receives uploaded audio and video files and stores them in data storage.

[1478] "Means for processing voice data through speech recognition and converting it into text data" refers to software that analyzes voice data and converts it into text information using speech recognition technology.

[1479] "Means for analyzing text data and extracting information necessary for business processes" refers to a system that uses natural language processing technology to automatically select important business procedures and information from converted text.

[1480] "Means for classifying each item" refers to software that organizes the extracted information and compiles it into categories such as procedures, tools used, and setting methods.

[1481] "Means for automatically generating manual documents" refers to a program for creating manuals in a specified format based on organized and classified information.

[1482] The "means for displaying the manual document and allowing the user to confirm and modify the contents" is an application that provides an interface that allows the user to view the generated manual and to reflect changes as necessary.

[1483] "Means for generating new versions and comparing them with previous versions" refers to a software function that allows you to manage and compare old and new versions when updating a manual.

[1484] "Means for saving and managing change history" is a system that tracks the revision history of a manual and stores that information in a database.

[1485] "Means for receiving feedback" is a feature that provides an interface for collecting improvements and additional information from users.

[1486] The "means for correcting information and regenerating the manual" refers to software that updates and corrects the manual content based on feedback and regenerates it.

[1487] "A means to standardize the work procedures of automated machines in factories and display them in augmented reality" is a function that uniformly records the operating procedures of automated machines and visually presents them using AR technology.

[1488] "A means to automatically generate operating manuals for factory robots and display them on smart glasses" is a technology that automatically creates manuals for robot operating procedures and displays them on smart glasses.

[1489] "A means for saving update history using version control functions and comparing with past manuals" is a system that can track manual versions and compare changes between different versions.

[1490] The present invention is a system for standardizing work procedures for automated machines and robots in factories and automatically generating manuals. This system aims to improve work efficiency and quality by utilizing voice recognition, natural language processing, and augmented reality technologies. Specific embodiments of this system are described in detail below.

[1491] Overall system configuration

[1492] This system is mainly composed of three components: the user, the server, and the terminal. We will explain the specific functions of each component and how they work together.

[1493] User

[1494] The users are factory workers and managers. They use smartphones or smart glasses to collect interview audio and video explaining the robot's work procedures and upload them to the system. They also review the generated manuals, make corrections as necessary, and provide feedback.

[1495] server

[1496] The server is the core of the system, and performs data analysis, manual generation, and maintenance. Specifically, the server has the following functions:

[1497] Speech recognition processing: The collected audio files are converted into text data using the Google Cloud Speech-to-Text API.

[1498] Video analysis: Use Amazon Rekognition to extract visual information from collected video files.

[1499] Natural Language Processing (NLP): Using spaCy and Transformers (Hugging Face), text data is analyzed and information necessary for business operations is extracted.

[1500] Manual generation: A Python template engine is used to generate a manual based on the extracted information.

[1501] Database management: MongoDB is used to store and manage the generated manuals and change history.

[1502] Version control: Provides the ability to save update history and compare with previous versions.

[1503] Terminal

[1504] The terminal provides an interface for users to operate the system. Specifically, it has the following functions:

[1505] Displaying and Editing Manuals: Provides functionality for viewing and editing generated manuals through smart glasses or smartphone interfaces. React Native is used for front-end development.

[1506] Gathering Feedback: Provide an interface that allows users to easily submit requests for corrections.

[1507] Specific examples

[1508] Next, a specific example of this system will be described.

[1509] Example: Creating a manual for routine maintenance procedures for a robot

[1510] 1. Data Collection: A user uses the smart glasses to collect audio and video recordings of the robot explaining the sensor cleaning procedure.

[1511] 2. Data Upload: The collected audio and video data is uploaded to the system.

[1512] 3. Data analysis: The server converts the audio to text using the Google Cloud Speech-to-Text API and analyzes the video using Amazon Rekognition.

[1513] 4. Natural Language Processing: The converted text data is analyzed using spaCy and Transformers to extract and classify the necessary information (procedures, tools used, etc.).

[1514] 5. Manual generation: Based on the extracted data, a manual is generated using a Python template engine and saved in PDF or Word format.

[1515] 6. Manual display and correction: The generated manual is presented to the user via smart glasses, allowing the user to check and correct the contents.

[1516] 7. Feedback and Updates: The manual is updated based on user feedback.

[1517] This allows for standardization of robot operating procedures on factory floors and automatic generation of manuals. It also provides the latest information at all times, contributing to improved efficiency and quality.

[1518] Prompt Sentence Examples

[1519] Analyze audio and video recordings of routine robot maintenance procedures and automatically create a manual based on the content. Extract information such as specific procedures, tools to be used, precautions, and frequency, and output it in PDF format.

[1520] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1521] Step 1:

[1522] The user uses a smartphone or smart glasses to collect interview audio and video explaining the robot's work procedures. The collected data is recorded using the camera and microphone of the smart device. The input is the work audio and video, and the output is these data files.

[1523] Step 2:

[1524] Users upload collected audio and video files to the system using an application on their smart devices. This sends the collected data to the server via the network. The input is the audio and video files, and the output is these data files stored on the server.

[1525] Step 3:

[1526] The server receives the uploaded audio and video files and stores them in an AWS S3 bucket. The specific operation of this step is to store the received data in data storage. The input is the uploaded audio and video files, and the output is the stored data files.

[1527] Step 4:

[1528] The server uses the Google Cloud Speech-to-Text API to analyze the received audio file and convert it into text data. Specifically, it uses speech recognition technology to convert the audio data into text data. The input is an audio file, and the output is text data.

[1529] Step 5:

[1530] The server uses Amazon Rekognition to analyze visual information from the received video file and extract important frames and objects. Specifically, it identifies recognized objects and actions in the video and converts them into text data. The input is the video file, and the output is the analyzed text data.

[1531] Step 6:

[1532] The server analyzes the converted text data using spaCy and Transformers (Hugging Face) and extracts the information necessary for the business (procedures, tools used, etc.). Specifically, it uses natural language processing technology to automatically select the necessary information from the text data. The input is text data, and the output is the extracted information.

[1533] Step 7:

[1534] The server organizes the extracted information and classifies it by item. Specifically, it organizes it into categories such as procedures, tools used, and setting methods. This classification process makes it easier to organize the information. The input is the extracted information, and the output is data classified by category.

[1535] Step 8:

[1536] The server uses a Python template engine to generate a manual based on the classified data. Specifically, it formats the organized information based on a template and creates a manual document. The input is the classified data, and the output is the generated manual document.

[1537] Step 9:

[1538] The terminal displays the generated manual document to the user, who can then check the content through the interface of smart glasses or a smartphone and make corrections as necessary. The input is the manual document, and the output is a modifiable manual displayed to the user.

[1539] Step 10:

[1540] The user checks whether the manual has been generated properly and sends feedback to the server via the feedback function of the terminal. The input is the user's feedback, and the output is the feedback data sent to the server.

[1541] Step 11:

[1542] The server corrects and updates the contents of the manual based on the feedback received from the user, thereby regenerating the manual to reflect the latest information. The input is the feedback data, and the output is the updated manual document.

[1543] Step 12:

[1544] The server uses the version control function to compare the current manual with previous versions and save and manage the change history. Specific operations include saving the update history and comparing it with previous versions. The input is the updated manual document, and the output is the compared version history.

[1545] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1546] This invention is a system that automates the standardization of business processes and procedures and creates and updates manuals in a sustainable manner. This system generates business manuals using work-related interview audio and work videos, and also uses an emotion engine to recognize and reflect the user's emotional state.

[1547] Overall system configuration

[1548] This system mainly consists of the following three entities: users, servers, and terminals.

[1549] User

[1550] Users are business personnel or administrators who are responsible for operating the system and providing data. Specifically, they upload work-related interview audio and work videos to the system, and review and edit the content. The system also recognizes the user's emotional state.

[1551] server

[1552] The server is the core of the system, and performs voice recognition processing, data analysis, manual generation, and maintenance. The server also has an emotion engine that recognizes user emotions and stores and utilizes the analysis results.

[1553] 1. Data conversion and analysis

[1554] Server: Using voice recognition technology, it generates text data from the uploaded audio and video, and uses natural language processing technology to extract and classify information necessary for business processes.

[1555] Server: The emotion engine analyzes the user's emotions from their voice and facial expressions, and stores the results in a database.

[1556] Terminal

[1557] The terminal provides an interface for users to operate the system, displays the generated manual document, supports content confirmation and correction, and adjusts the presentation method based on the analysis results of the emotion engine.

[1558] Program processing overview

[1559] 1. Audio and Video Collection and Upload

[1560] User: Collects interview audio and work videos that explain business processes and uploads them to the system using a terminal.

[1561] 2. Data Conversion and Analysis

[1562] Server: Receives the uploaded data and converts it into text using speech recognition processing. Then, natural language processing technology is used to extract and classify the information required for the business process.

[1563] Server: The emotion engine analyzes emotions from the user's voice and facial expressions and stores the results in a database.

[1564] 3. Manual Generation

[1565] Server: Automatically generates manual documents based on the extracted information. The server embeds the classified data into a document template and generates the document in the specified format (PDF, Word, etc.).

[1566] Server: Combines the generated manual documentation with the results of the emotion engine and provides it in a format that is easy for users to read.

[1567] 4. Viewing and Modifying the Manual

[1568] Terminal: Displays the generated manual document to the user and supports confirmation and correction of the contents through the interface.

[1569] Device: Based on the analysis results of the emotion engine, the presentation method is adjusted accordingly. For example, if it determines that the user is confused, it will automatically add detailed annotations.

[1570] 5. Feedback and Maintenance

[1571] Users: Review the manual and provide feedback on any necessary corrections or additional information. Provide feedback and suggestions for improvements to the system through the feedback function.

[1572] Server: Updates the manual based on user feedback and the analysis results of the emotion engine, generates new versions, and stores and manages the update history.

[1573] Specific examples

[1574] For example, when creating a manual for how to use a new software tool, users collect and upload audio files and video instructions explaining how to use the software. The server receives these files and converts them into text using speech recognition. Next, it uses natural language processing to extract and classify information such as software installation instructions, descriptions of key functions, operation methods, required settings, and troubleshooting. Based on this information, the server automatically generates the manual text and also analyzes the user's emotions using an emotion engine to adjust the presentation method as needed.

[1575] This not only efficiently standardizes business processes and procedures, but also provides easy-to-use manuals that reflect the user's feelings. Users can always provide the latest information, contributing to improved efficiency and quality.

[1576] The processing flow will be explained below.

[1577] Step 1:

[1578] Users: Collect job-related audio interviews or video of tasks, either by recording a worker performing a task or by recording an interview in which the worker explains a job procedure.

[1579] Step 2:

[1580] User: Uploads collected audio and video files to the system using the device interface. The user selects the files on the file selection screen and clicks the upload button.

[1581] Step 3:

[1582] Server: Receives uploaded audio and video files and saves them to storage. Checks the file format and confirms that the file was saved successfully.

[1583] Step 4:

[1584] Server: Starts the speech recognition module and converts the received voice data into text data. In the case of video, the audio portion is extracted and processed in the same way. The converted text is temporarily saved.

[1585] Step 5:

[1586] Server: Launches a natural language processing (NLP) engine and extracts information required for business processes (input, output, tools used, procedures, required skills, KPIs, etc.) from the converted text data. The extracted information is categorized and stored in a database.

[1587] Step 6:

[1588] Server: Activates the emotion engine and recognizes and analyzes the user's emotions from the uploaded audio and video data. Specifically, it identifies the user's emotional state using voice tone and facial expression recognition technology and stores the results in a database.

[1589] Step 7:

[1590] Server: Automatically generates manual documents based on the extracted business information and emotion data. Embeds the data in a document template, generates documents in the specified format (PDF, Word, etc.), and saves them in storage.

[1591] Step 8:

[1592] Terminal: Displays the generated manual document to the user. The user can view the document and check its contents through the interface.

[1593] Step 9:

[1594] User: Check the contents of the displayed manual document and make corrections as necessary. Enter the corrections in the terminal interface and click the save button.

[1595] Step 10:

[1596] Server: Receives the user's edits and updates the manual. It saves the updated version and records the change history in a database.

[1597] Step 11:

[1598] Server: Every time a manual is updated, a new version is generated and compared with the previous version. The changes are recorded as a history and made available to users.

[1599] Step 12:

[1600] Device: Based on the results of the emotion engine, the presentation method is adjusted accordingly. For example, if the user is confused, detailed annotations or additional examples are automatically inserted into the manual.

[1601] Step 13:

[1602] Users: Regularly check the updated manual and provide feedback. Submit your thoughts and suggestions for improvement to the system via the feedback function.

[1603] Step 14:

[1604] Server: Regenerates the manual based on user feedback and the analysis results of the emotion engine, and updates the new version. Saves and manages the update history, recording feedback along with previous versions.

[1605] Example 2

[1606] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1607] The conventional manual creation process had the problem of taking a great deal of time and effort to collect business-related information and generate and update the manual. Furthermore, the generated manual did not take into account the user's level of understanding or emotional state, which resulted in a decrease in the effectiveness of user training and work efficiency. Furthermore, the manual update process was cumbersome, and the latest information could not be reflected, which was also an issue.

[1608] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1609] In this invention, the server includes means for collecting business-related audio or video, means for uploading the collected audio and video data to the system, means for receiving and saving the uploaded audio and video data, means for converting the received audio and video data into text data through speech recognition processing, means for analyzing the text data and extracting information necessary for business procedures, means for organizing the extracted information and classifying it by item, means for automatically generating a procedure manual based on the classified data, means for displaying the procedure manual and allowing the user to check and modify its contents, means for generating a new version each time the procedure manual is updated and comparing it with previous versions, means for saving and managing the change history, means for periodically checking the updated procedure manual and receiving feedback, means for modifying the information based on the received feedback and generating a new procedure manual, means for analyzing the user's emotions from their voice and facial expressions, and means for adjusting the presentation method of the procedure manual based on the analysis results. This enables efficient and continuous generation and updating of business manuals, and realizes effective manual provision that corresponds to the user's level of understanding and emotional state.

[1610] "Speech recognition processing" is a technology that analyzes voice data and converts it into text data.

[1611] "Natural language processing" is a technology that analyzes text data to understand its meaning and structure.

[1612] "Business-related audio or video" refers to audio or video data recorded to explain business processes or procedures.

[1613] "Uploading means" refers to a method or device for transmitting collected audio and video data to the system.

[1614] "Means for receiving and storing" refers to a method or device that receives uploaded data and stores it in a database or file system.

[1615] "Means for converting into text data" refers to technology or devices that use voice recognition processing to convert voice data into text information.

[1616] "Information necessary for business procedures" refers to specific steps and important points related to business processes and procedures.

[1617] "Extraction means" refers to a method or device that uses natural language processing technology to extract necessary information from text data.

[1618] "Means for organizing and classifying by item" refers to techniques or devices that logically arrange extracted information and separate it into categories.

[1619] "Means for automatically generating procedure manuals" refers to methods and technologies for creating procedure manuals based on classified data in accordance with templates and formats.

[1620] The "means for displaying the procedure manual and allowing the user to check and modify the contents" refers to a technique or device that allows the user to view the generated procedure manual and provides an interface to enable modification.

[1621] "Means for generating a new version and comparing it with the previous version" refers to a technique or device that creates a new version of an updated procedure manual and compares it with the previous version to confirm the changes.

[1622] "Means for saving and managing change history" refers to technology or devices that record the update history of procedure manuals and make them accessible as needed.

[1623] "Means for receiving feedback" refers to methods or devices that allow users to input and send their opinions and suggestions for improvement.

[1624] "Means for correcting information and regenerating the procedure manual" refers to the technology or device that updates and regenerates the procedure manual based on feedback.

[1625] "Means for analyzing emotions" refers to technology or devices that analyze voice and facial expressions to identify emotional states.

[1626] "Means for adjusting presentation method" refers to technology or devices that change the display or explanation method based on the results of sentiment analysis.

[1627] The present invention provides a system that automates the standardization of business processes and procedures and creates and updates operating manuals in a sustainable manner. This system generates operating manuals using business-related audio and video data, and further uses a sentiment analysis engine to recognize and reflect the user's emotional state. An embodiment of the system is described in detail below.

[1628] System configuration

[1629] This system mainly consists of the following three entities: users, servers, and terminals.

[1630] User

[1631] Users are business personnel or administrators who are responsible for operating the system and providing data. Specifically, they collect work-related audio and video data and upload them to the system. The system also recognizes the user's emotional state.

[1632] server

[1633] The server is the core of the system, and performs voice recognition processing, data analysis, procedure manual generation, and maintenance. The server also has a sentiment analysis engine that recognizes user emotions and stores and utilizes the analysis results. The specific functions of the server are as follows:

[1634] Speech recognition processing: The server converts the voice data into text data using Google Cloud Speech-to-Text.

[1635] Natural language processing: The server uses natural language processing technology such as SpaCy to extract and classify information necessary for business procedures from text data.

[1636] Sentiment analysis: The server uses IBM Watson Emotion Analysis to analyze the user's emotions from audio and video data and stores the results in a database.

[1637] Procedure generation: The server uses LaTeX and Microsoft Word APIs to generate procedures based on the extracted information.

[1638] Information update: The server updates the procedure manual based on user feedback and generates a new version.

[1639] Terminal

[1640] The terminal provides the interface through which the user operates the system and has the following functions:

[1641] Support for displaying and modifying procedures: Provides an interface that displays the generated procedures and allows users to check and modify their contents.

[1642] Reflection of sentiment analysis results: Based on the results of sentiment analysis, the way the instructions are presented is adjusted accordingly. For example, if the user is confused, detailed annotations can be added.

[1643] Example

[1644] The following describes how this system is specifically implemented.

[1645] For example, when creating a manual for how to use a new software tool, a user collects audio files or video instructions explaining the software operation procedures and uploads them to the system. The server uses Google Cloud Speech-to-Text to convert the uploaded audio data into text. It then uses SpaCy to extract and classify the information necessary for the business procedures from the text data. Based on this information, the server automatically generates a PDF manual using LaTeX, and then uses IBM Watson Emotion Analysis to analyze the user's emotional state and adjust the manual accordingly.

[1646] Prompt Sentence Examples

[1647] Below is an example of a prompt sentence to be input into the generative AI model.

[1648] "We have an audio interview and video of an instruction manual explaining how to use a new software tool. From these, please extract installation instructions, explanations of key features, operation instructions, required settings, and troubleshooting information, and automatically generate a PDF instruction manual. Also, please analyze the user's emotional state and provide detailed annotations if the user is confused."

[1649] In this way, by providing concrete examples and prompts, users can clearly understand how the system actually works and how to use it.

[1650] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1651] Step 1:

[1652] Audio and video collection and uploading

[1653] Users collect interview audio and work videos that explain business processes and upload them to the system using a terminal.

[1654] Input: Audio and video files related to business processes

[1655] Specific operation: The user records an interview or activity using a smartphone or camera, transfers the file to a computer, and selects and uploads it from the system's web interface.

[1656] Output: Audio and video files uploaded to the system.

[1657] Step 2:

[1658] Data transformation and analysis

[1659] The server receives the uploaded data and converts it into text using speech recognition processing.

[1660] Input: Uploaded audio and video files

[1661] Specific operation: The server calls the Google Cloud Speech-to-Text API and converts the audio data into text data.

[1662] Output: Text data

[1663] The server applies natural language processing technology to the converted text data to extract and classify the information necessary for business procedures.

[1664] Input: Text data

[1665] Specific operation: The server uses SpaCy to extract noun phrases and perform relationship analysis to extract and classify important points in the business procedures.

[1666] Output: Classified business procedure information

[1667] The server uses an emotion engine to analyze the user's emotions from their voice and facial expressions, and stores the results in a database.

[1668] Input: Audio and video files

[1669] Specific operation: The server uses IBM Watson Emotion Analysis to extract emotional data from audio and video data and stores the analysis results in a database.

[1670] Output: Sentiment analysis result data

[1671] Step 3:

[1672] Procedure generation

[1673] The server automatically generates a procedure manual based on the extracted information.

[1674] Input: Classified operating procedure information

[1675] Specific operation: The server embeds the extracted and classified information into a manual template and generates a PDF procedure manual using LaTeX.

[1676] Output: Instructions in PDF format

[1677] Based on the analysis results of the emotion engine, the server adjusts the instruction manual in a way that is easy for the user to understand.

[1678] Input: PDF format instructions, sentiment analysis result data

[1679] Specific behavior: If the server determines that the user is confused based on the results of sentiment analysis, it adds an annotation to the instruction manual providing detailed explanations.

[1680] Output: Adjusted instructions

[1681] Step 4:

[1682] View and modify procedures

[1683] The terminal displays the generated procedure manual to the user and supports confirmation and correction of the contents.

[1684] Input: Adjusted Procedures

[1685] Specific behavior: Provides an interface that allows users to open PDF documents through a web interface, review the content, and add comments to corrections.

[1686] Output: User-confirmed and corrected instructions

[1687] The device will adjust the presentation method appropriately based on the analysis results of the emotion engine.

[1688] Input: Sentiment analysis result data

[1689] Specific Action: If the user is determined to be confused, provide the ability to add illustrations using visualization tools.

[1690] Output: Adjusted presentation instructions

[1691] Step 5:

[1692] Feedback and Maintenance

[1693] The user checks the contents of the procedure manual and provides feedback on any necessary corrections or additional information.

[1694] Input: User-confirmed and corrected instructions

[1695] What happens: The user fills out a feedback form to submit corrections or additional information to the system.

[1696] Output: Feedback information

[1697] The server updates the recipe based on user feedback and the emotion engine's analysis.

[1698] Input: Feedback information, sentiment analysis result data

[1699] Specific operation: The server analyzes the feedback it receives, automatically updates the relevant part of the procedure manual, and generates a new version.

[1700] Output: Updated instructions, new version generated

[1701] This enables efficient and continuous generation and updating of work procedures, and makes it possible to provide effective procedures that correspond to the user's level of understanding and emotional state.

[1702] (Application example 2)

[1703] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1704] Creating manuals for factory work procedures and business processes takes time and effort, and efficient training methods are required, especially for new employees. Furthermore, if the manuals created do not take into account the reader's feelings and level of understanding, there is a problem of inefficient learning and work execution. Furthermore, delays in updating manuals and reflecting feedback can lead to information not being kept up to date, resulting in inefficiencies in work. However, no effective means have been provided to solve these issues.

[1705] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1706] In this invention, the server includes means for collecting work-related interview audio or work video, means for uploading audio files or video files to the system, means for receiving and saving the uploaded files, means for converting the received files into text data by subjecting them to speech recognition processing, means for analyzing the text data and extracting information necessary for the business process, means for analyzing emotions from the user's voice and facial expressions using an emotion engine and generating emotion data, means for adjusting the presentation method of the manual document based on the emotion data, and means for saving and displaying the generated manual document in a specified format such as PDF or Word.

[1707] This makes it possible to efficiently extract work procedures from audio and video and automatically generate manuals that incorporate emotional data. Furthermore, the generated manuals are kept up to date and are presented appropriately, reflecting the user's emotions, enabling more effective training and business support.

[1708] "Business-related interview audio" refers to audio data obtained from interviews conducted to explain business processes.

[1709] "Work video" refers to video data that records business processes and work procedures.

[1710] "Speech recognition processing" is a technology that analyzes voice data and converts it into text data.

[1711] "Text data" refers to character string data converted by speech recognition processing.

[1712] "Information extraction" refers to the process of extracting information necessary for business processes from text data.

[1713] "Classification" refers to the process of organizing and systematizing extracted information by category.

[1714] "Manual documents" refer to documents that describe business processes and work procedures.

[1715] "Emotion engine" refers to technology for analyzing emotions from voice and facial expressions.

[1716] "Emotional Data" refers to data regarding a user's emotional state analyzed by the emotion engine.

[1717] "Adjusting the presentation method" refers to changing the way the manual document is displayed based on the emotion data.

[1718] "Feedback" refers to receiving opinions and requests from users and reflecting them in the contents of the manual.

[1719] "Application to be installed on a factory robot" refers to application software for executing the above-mentioned system on a factory robot.

[1720] "PDF and Word formats" refer to file formats for saving and displaying manual documents.

[1721] "Comparing with previous versions" refers to the process of comparing the current manual document with the version before the update and checking for changes.

[1722] This invention relates to a system that automatically generates, updates, and manages business process manuals for factory robots. This system efficiently collects business-related interview audio and work videos, and uses an emotion engine to provide manuals that reflect the user's emotional state.

[1723] System Subject

[1724] User

[1725] Users are factory workers and managers who upload work-related audio and video to the system, and are responsible for reviewing and editing the content. The system also recognizes the user's emotional state.

[1726] server

[1727] The server is the core of the system and is responsible for:

[1728] 1. Speech recognition and natural language processing: Uploaded audio and video files are received and stored, and converted into text data using speech recognition technology. The converted text data is then analyzed using natural language processing technology to extract and classify information necessary for business processes.

[1729] 2. Emotion analysis: The emotion engine analyzes the user's emotions from their voice and facial expressions, and stores the results in a database.

[1730] 3. Manual generation: Manual documents are automatically generated based on the extracted information. The generated manual documents are provided in a format that is easy for users to understand, reflecting the analysis results of the emotion engine.

[1731] Terminal

[1732] The terminal is an application installed on the factory robot that provides the following functions:

[1733] 1. Manual display and correction assistance: This system displays the generated manual document and provides an interface for users to check and correct the contents. The display method is adjusted based on the analysis results of the emotion engine.

[1734] 2. Feedback Management: Receive feedback from users and update the manual based on the latest information.

[1735] Program processing overview

[1736] The server uses libraries such as Google Speech Recognition API for speech recognition technology, nltk for natural language processing technology, and Sentiment Intensity Analyzer for emotion engine. Below is an overview of the processing for each function of the system.

[1737] 1. Speech Recognition: Uses Google Speech Recognition API to convert audio files into text data. Audio data is collected from a microphone or audio files.

[1738] 2. Natural Language Processing: Analyze text data using the nltk library to extract and classify information necessary for business processes.

[1739] 3. Sentiment Analysis: A Sentiment Intensity Analyzer is used to extract emotional information from text data. Voice and facial expression data are also analyzed.

[1740] 4. Manual generation: Manual documents are automatically generated based on the extracted and analyzed information, and saved and displayed in PDF or Word format.

[1741] Specific examples

[1742] For example, when creating an operating procedure manual for a factory robot, the user uploads audio files or video files explaining the operating procedures. The server receives these files, performs speech recognition processing, and converts them into text. Natural language processing technology is used to extract and classify information such as installation procedures, explanations of key functions, operation methods, required settings, and troubleshooting. An emotion engine also analyzes the user's emotions and adjusts the presentation method as needed.

[1743] Prompt Sentence Examples

[1744] Below is an example of a prompt sentence to input to the generative AI model.

[1745] text

[1746] "Analyze the audio files below explaining the work procedures and identify the work procedures and emotional state. Based on the results, demonstrate the process for generating a detailed manual."

[1747] In this way, the system can efficiently extract work procedures from audio and video and automatically generate manuals that incorporate emotional data. The generated manuals are kept up to date and are presented appropriately, reflecting the user's emotions, enabling more effective training and work support.

[1748] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1749] Step 1: Collect and upload work-related interview audio and video footage

[1750] The user collects interview audio and video that explain work procedures and business processes in the factory. These audio and video files are uploaded to the system using a terminal. The input is the audio and video files, and the output is the data transferred to the server. Specifically, the user selects the collected files and transfers them to cloud storage using the system's upload function.

[1751] Step 2: Receiving and storing data

[1752] The server receives uploaded audio and video files and saves them in a specified location. The input is the audio or video file from the user, and the output is data saved in a specific folder on the server. Specifically, the server processes the HTTP request and saves the received files in a database or file system.

[1753] Step 3: Speech recognition processing and text conversion

[1754] The server processes the received audio file through speech recognition and converts it into text data. This is done using the Google Speech Recognition API. The input is an audio file, and the output is the converted text data. Specifically, the server sends the audio file to the Google Speech Recognition API and saves the returned text result.

[1755] Step 4: Information extraction using natural language processing

[1756] The server analyzes the text data using natural language processing technology (nltk library) and automatically extracts the information necessary for the business process. The input is text data, and the output is the extracted information. Specifically, the server analyzes the text data and identifies work procedures and important business information.

[1757] Step 5: Sentiment analysis

[1758] The server uses a sentiment engine (Sentiment Intensity Analyzer) to analyze emotions from the user's voice and text. Voice data and text data are input, and emotional data is obtained as output. Specifically, the server inputs this data into the sentiment engine and stores the analysis results in a database.

[1759] Step 6: Automatic generation of manuals

[1760] The server automatically generates a manual document based on the extracted information and emotion data. A document template is used for this purpose. The extracted information and emotion data are input, and the generated manual is obtained as output. Specifically, the server organizes the information appropriately and creates a manual document according to the format.

[1761] Step 7: View and modify the manual

[1762] The terminal displays the generated manual document to the user and provides an interface that allows the user to check and modify the content. The generated manual document is the input, and user feedback is the output. Specifically, the terminal displays the manual document to the user and adjusts the presentation method based on the emotion data.

[1763] Step 8: Updating the manual and managing feedback

[1764] The server modifies the manual based on the feedback received from the user, generates a new version, and saves it. The input is the feedback and the existing manual, and the output is the updated manual. Specifically, the server analyzes the feedback, makes appropriate modifications, and saves the new version.

[1765] Step 9: Managing Change History

[1766] Every time a manual is updated, the server compares the new version with the previous version and saves and manages the change history. The input is the old and new manual document, and the output is the change history. Specifically, the server performs differential analysis, records the changes, and saves them in a history database.

[1767] Through these processing steps, it is possible to efficiently collect, analyze, and generate information for factory robots, and provide manuals that are always kept up to date.

[1768] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1769] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1770] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1771] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1772] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions o...

Claims

1. A means of collecting work-related audio interviews or work videos; A means of uploading collected audio and video files to the system; means for receiving and storing uploaded audio and video files; A means for converting received audio and video files into text data through speech recognition processing; A means of analyzing text data and extracting information necessary for business processes; A means of organizing the extracted information and classifying it by each item, A means for automatically generating a manual document based on the classified data; A means for displaying the manual document and allowing the user to review and modify the contents; A means to generate a new version each time the manual is updated and compare it with previous versions, A means of storing and managing change history; A means to regularly review updated documentation and receive feedback; The system includes a means for modifying the information based on the feedback received and generating the manual again.

2. 2. The system according to claim 1, wherein information required for a business process is automatically extracted using speech recognition processing and natural language processing.

3. 2. The system according to claim 1, further comprising means for saving and displaying the generated manual document in a designated format such as PDF or Word format.

Images