System

Abstract

A system is provided.SOLUTION: A system comprising: a voice acquisition unit configured to acquire an instruction voice performed by a veteran worker at a site; a voice analysis unit configured to analyze voice data acquired by the voice acquisition unit and convert the voice data into a natural language; an information extraction unit configured to extract an instruction content and a procedure from text data converted by the voice analysis unit; a manual generation unit configured to automatically generate and update an operation manual based on information extracted by the information extraction unit; and an interface unit configured to allow a user to view and correct the generated operation manual.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the manufacturing and construction industries, the transfer of know-how from experienced veteran employees to the next generation of employees is a challenge. In particular, there are concerns that the tacit knowledge of veteran employees remains unwritten, leading to reduced work efficiency on site and frequent quality issues and accidents. When veteran employees retire en masse due to this situation, their valuable know-how is lost, posing a significant risk to the company. This invention aims to ensure the smooth transfer of know-how by visualizing the tacit knowledge of veteran employees and automatically creating and updating appropriate work manuals. [Means for solving the problem]

[0005] The present invention provides a system including a voice acquisition means for acquiring voice guidance provided by an experienced worker on-site, a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting it into natural language, an information extraction means for extracting instruction content and procedures from the converted text data, a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means, and an interface means for a user to view the generated work manual and provide correction instructions. The system also includes a translation means for translating into multiple languages, making it suitable for use by foreign workers. Furthermore, the voice acquisition means is a headset, which is configured so that experienced workers can naturally acquire daily guidance by wearing it. This means makes the know-how of experienced employees visible and allows it to be smoothly passed on to the next generation of employees.

[0006] "Audio acquisition means" refers to a device that captures audio in real time regarding on-site instruction and work by experienced workers and saves it as data.

[0007] "Speech analysis means" refers to devices or software that execute the process of analyzing acquired voice data and converting it into natural language.

[0008] "Information extraction means" refers to a device or software that automatically extracts specific information such as work content and procedures from the text data converted by the voice analysis means.

[0009] "Manual generation means" refers to a device or software that automatically generates a work manual based on the extracted information and updates existing manuals as necessary.

[0010] The "interface means" refers to a user interface that allows the user to view the generated operation manual and give correction instructions as necessary.

[0011] The "translation means" refers to a device or software that translates the generated operation manual into multiple languages ​​and makes it available to users who speak different languages.

[0012] A "headset" refers to a terminal device worn by an experienced worker to acquire voice data.

[0013] "Text data" refers to character information converted from voice data by voice analysis means.

[0014] "Instruction content" refers to information including specific work procedures and precautions that experienced workers perform on-site. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system for efficiently transferring the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and is configured as follows.

[0037] System configuration

[0038] The system consists of the following main components:

[0039] 1. Voice acquisition means: The terminal acquires the voice of the experienced worker giving instructions. Specifically, the experienced worker wearing a headset captures the voice data of the real-time work instruction and explanations given on-site.

[0040] 2. Voice analysis means: The device transmits the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology.

[0041] 3. Information extraction means: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the voice analysis means. This is done using natural language processing technology.

[0042] 4. Manual generation means: The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This ensures that the latest instruction information is always reflected.

[0043] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[0044] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API or similar.

[0045] Specific examples of program processing

[0046] Operation of the audio acquisition means

[0047] The device (headset) picks up the voice of an experienced worker instructing in real time. For example, an experienced worker may say, "This part needs to be inspected once a week." The device collects this voice and temporarily stores it locally.

[0048] Sending and analyzing voice data

[0049] The device sends voice data to the server, which receives the data and uses a voice analysis tool to convert it into natural language text, such as "This part needs to be inspected once a week."

[0050] Information Extraction and Manual Generation

[0051] The server uses the information extraction means to extract information such as "part inspection," "once a week," and "necessary" from the text data. Based on the extracted information, the server adds a new procedure to the "part inspection manual" page using the manual generation means. The new procedure states, "Inspect part A once a week."

[0052] Manual review and translation

[0053] A user checks a manual through a web interface. For example, a new employee opens the "Parts Inspection Manual" and views the section that says "Part A should be inspected once a week." If a foreign worker wants to view the manual in Spanish, the server uses a translation tool to translate the manual into Spanish. After translation, the user can view the manual in Spanish through the same web interface.

[0054] In this way, the present invention digitally and automatically visualizes the tacit knowledge of experienced workers, enabling them to be constantly updated as the latest work manuals. Furthermore, by translating the knowledge into multiple languages, the system can be used in international work sites. This allows for the smooth transfer of know-how to the next generation of employees, improving work efficiency and safety on-site.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The device receives audio

[0058] An experienced worker wears a headset and provides on-site work instructions. The device captures the worker's voice in real time through a microphone and temporarily stores it locally as audio data.

[0059] Step 2:

[0060] The device sends the voice data to the server

[0061] At regular intervals, the device encrypts the collected voice data and transmits it to the server through a secure channel.

[0062] Step 3:

[0063] The server analyzes the voice data

[0064] The server decompresses the received voice data and passes it to the voice analysis module, which uses natural language processing technology to convert the voice data into text format.

[0065] Step 4:

[0066] The server extracts the information

[0067] The server analyzes the text data and extracts specific keywords (e.g., work procedures, time, location) that correspond to the business context, using natural language processing technology.

[0068] Step 5:

[0069] The server generates the operation manual

[0070] Based on the information extracted by the server, the first edition of the work manual is automatically generated, including new work procedures and precautions. This manual summarizes specific procedures in an easy-to-understand manner using bullet points and diagrams.

[0071] Step 6:

[0072] Server updates manual

[0073] The server analyzes the new voice data, adds or modifies relevant information to the existing work manual, and updates the manual to reflect the latest information.

[0074] Step 7:

[0075] User checks the manual

[0076] Next-generation employees and foreign workers can access and review the generated work manuals through a web interface, and can easily find specific work procedures and precautions using the search function.

[0077] Step 8:

[0078] The server translates the manual

[0079] If the user requests the work manual in a different language, the server invokes the automatic translation module to convert the manual into the specified language, and the translated manual is displayed according to the user's language setting.

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] On-site work instruction often relies on the experience and knowledge of veteran workers, and if this knowledge is not passed on properly, it hinders the improvement of the skills and knowledge of the next generation of workers. Furthermore, if the content of instruction is not standardized, there is a risk of work efficiency and safety declining. Furthermore, language barriers exist in international work sites, requiring multilingual support. To resolve these issues, an efficient and standardized knowledge transfer system is needed.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes a voice analysis means, an information extraction means, and a manual generation means, which automatically converts the knowledge and skills of experienced workers from voice to text, extracts important instructional content, and generates and updates a work manual based on the extracted information.

[0085] "Audio acquisition means" refers to equipment and software for collecting the instructional audio given by experienced workers on-site.

[0086] "Speech analysis means" refers to technology for analyzing captured speech data and converting it into natural language text.

[0087] "Information extraction means" refers to a technology for extracting important information such as instruction content and procedures from the text data converted by the voice analysis means.

[0088] "Manual generation means" refers to a system that automatically generates and updates a work manual based on the extracted information.

[0089] The "interface means" refers to an interface that allows a user to view the generated work manual and give correction instructions.

[0090] "Translation means" refers to the technology or system for translating the generated work manual into multiple languages.

[0091] This invention is a system for efficiently transferring the knowledge and skills of veteran workers to the next generation of employees at manufacturing and construction sites. The system consists of the following main components:

[0092] 1. Voice acquisition method: A method in which the terminal acquires the voice of an experienced worker giving instructions. Specifically, an experienced worker wearing a headset captures real-time work instruction and explanations on-site as voice data.

[0093] 2. Voice analysis: The device sends the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[0094] 3. Information extraction: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the speech analysis method. This is done using natural language processing technology (e.g., spaCy, NLTK).

[0095] 4. Manual generation means: The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This ensures that the latest instruction information is always reflected.

[0096] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[0097] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API (e.g., Google Translate API).

[0098] Specific examples of program processing

[0099] The device receives the voice instructions of the experienced worker through the headset and temporarily stores the voice data locally. The device then sends the voice data to a server. The server receives the voice data and converts it into natural language text using a voice analysis tool. For example, if an experienced worker instructs, "This part needs to be inspected once a week," the voice instructions are converted into text.

[0100] Next, the server uses information extraction means to extract important information from the text data, such as "part inspection," "once a week," and "necessary." Based on this information, the server automatically updates the work manual using manual generation means and adds new procedures. For example, a new procedure might read, "Inspect part A once a week."

[0101] Users can check the updated work manual through a web interface and make corrections or add comments as necessary. The server also uses a translation means to translate the generated manual into multiple languages. For example, if a foreign worker wants to view the manual in Spanish, the server translates it and the user can view the manual in Spanish from the same web interface.

[0102] Prompt Sentence Examples

[0103] ---

[0104] An experienced worker instructed, "This part needs to be inspected once a week." Add a new procedure to the work manual: "Inspect part A once a week."

[0105] ---

[0106] This system digitizes the tacit knowledge of veteran workers and makes it possible to efficiently pass it on to the next generation of employees. Multilingual support allows for smooth use even in international work sites. This improves work efficiency and ensures safety.

[0107] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0108] Step 1: Audio capture

[0109] Subject: Device

[0110] Specific operation: The terminal (experienced worker wearing a headset) receives real-time instruction audio on-site.

[0111] Input: Operator's voice instruction

[0112] Data processing: Using a microphone to capture audio signals and store them as digital audio data.

[0113] Output: Digital audio data

[0114] Step 2: Sending audio data

[0115] Subject: Device

[0116] Specific operation: The device temporarily stores the voice data locally and then transmits it to the server over the network.

[0117] Input: Digital audio data

[0118] Data processing: Converts voice data into a network format and transmits it

[0119] Output: Audio data sent to the server

[0120] Step 3: Audio analysis

[0121] Subject: Server

[0122] What it does: The server converts the received voice data into natural language text using speech analysis techniques, such as a speech recognition API (e.g., Google Cloud Speech-to-Text).

[0123] Input: Audio data sent to the server

[0124] Data processing: Analyzes voice data and converts it into natural language text

[0125] Output: Parsed text data

[0126] Step 4: Information Extraction

[0127] Subject: Server

[0128] Specific operation: The server processes the text data obtained from the speech analysis means using information extraction means to extract important instructional content and work procedures. Natural language processing technology (e.g., spaCy, NLTK) is used.

[0129] Input: Parsed text data

[0130] Data processing: Extracting important information from text

[0131] Output: Extracted information

[0132] Step 5: Manual generation

[0133] Subject: Server

[0134] Specific operation: The server automatically generates and updates the work manual based on the data obtained by the information extraction means.

[0135] Input: Extracted information

[0136] Data processing: Manual creation and updating

[0137] Output: Updated operating manual

[0138] Step 6: Manual review and correction

[0139] Subject: User

[0140] Specific operation: The user uses a web interface to view the generated work manual and add correction instructions or comments as necessary.

[0141] Input: Updated Operation Manual

[0142] Data processing: Adding corrections and comments

[0143] Output: Revised operating manual

[0144] Step 7: Manual Translation

[0145] Subject: Server

[0146] Specific operation: The server uses a translation tool to translate the work manual into multiple languages, using a translation API (e.g., Google Translate API).

[0147] Input: Corrected operation manual

[0148] Data processing: Manual translation

[0149] Output: Translated work manual

[0150] In this way, the system can digitize the knowledge of veteran workers and efficiently pass it on to the next generation. By detailing the processing content for each specific step, the overall picture of the operation can be clearly understood.

[0151] (Application example 1)

[0152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] It is important to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees in on-site work. However, previous methods lacked consistency and accuracy in training content, and were not multilingual, which resulted in impacts on work efficiency and safety. Furthermore, manual recording and transmission methods made it difficult to automate and reference in real time.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0155] In this invention, the server includes a voice acquisition means for acquiring voice guidance provided by an experienced worker on-site, a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting it into natural language, an information extraction means for extracting guidance content and procedures from the text data converted by the voice analysis means, a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means, an interface means for a user to view the generated work manual and give correction instructions, a translation means for translating the work manual into other languages, and a machine operation means for a machine to automatically refer to the work manual and perform work. This makes it possible to efficiently transfer the knowledge and skills of experienced workers, achieve consistency and accuracy in work, and support multiple languages, while enabling work automation and real-time reference.

[0156] "Veteran workers" are workers with long-term experience and deep knowledge in the field.

[0157] "Site" refers to the location where actual work, such as manufacturing or construction, takes place.

[0158] "Instruction voice" is a voice for instruction or explanation given by an experienced worker.

[0159] The "voice acquisition means" refers to a device or system that collects the instruction voice of an experienced worker.

[0160] "Audio data" refers to information in which instructional audio is recorded in digital format.

[0161] "Speech analysis means" refers to a device or software that analyzes acquired speech data and converts it into natural language text.

[0162] "Text data" is character string information converted by speech analysis.

[0163] "Information extraction means" refers to devices or software that extract important instructional content and procedures from text data.

[0164] "Instruction content" refers to specific instructions and explanations such as work procedures and precautions.

[0165] A "procedure" refers to the steps or method for performing a task.

[0166] The "manual generation means" refers to a device or software that automatically creates and updates a work manual based on the extracted information.

[0167] A "work manual" is a document that describes specific work procedures and instructions to be followed on-site.

[0168] "User" refers to the workers and administrators who use this system.

[0169] "Interface means" refers to devices and software that allow a user to view and correct the operation manual.

[0170] "Translation means" refers to devices or software that translates work manuals into multiple languages.

[0171] "Other languages" are languages ​​other than one's native language.

[0172] "Machine operating means" refers to devices and software that allow a machine to automatically perform work based on the created work manual.

[0173] "Machine" refers to an automated device that can perform simple tasks or complex processes.

[0174] The following describes an embodiment of the present invention. This system is designed to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees, and is composed of the following main components:

[0175] System Program

[0176] The audio capture means is used to capture the instruction voice given by experienced workers on-site. For this purpose, audio recording devices, smartphones, and smart headsets are used. The audio of the experienced worker giving instructions such as "This part needs to be inspected once a week" is recorded and saved locally.

[0177] The speech analysis means analyzes the acquired speech data and converts it into natural language text. This process uses the Google Speech Recognition API and Python libraries. The acquired speech is converted into text data, such as "This part needs to be inspected once a week."

[0178] The information extraction means extracts instruction content and procedures from the text data converted by the voice analysis means. This process uses natural language processing technology to extract key keywords (e.g., inspection, once a week, necessary, etc.).

[0179] The manual generation means automatically generates and updates the work manual based on the extracted information. This allows the work manual to always reflect the latest instruction information and be saved in JSON format.

[0180] The interface means allows the user to view the generated work manual and give correction instructions. This interface is web-based and designed to be easily accessible by the user.

[0181] The translation means translates the work manual into other languages ​​(e.g., Spanish). This process uses the Google Translate API. By translating the generated work manual, it can be used by foreign workers.

[0182] The machine operation means allows the robot to automatically refer to the created work manual and perform the work, which allows the robot to act according to the procedure, improving the accuracy and safety of on-site work.

[0183] Specific examples

[0184] For example, in a factory, a robot can capture the voice instructions of an experienced worker in real time and record the weekly part inspection procedure as text data. This data is automatically analyzed and generated as a work manual. The generated manual is translated into Spanish, making it suitable for foreign workers as well. The robot can refer to this manual and perform part inspection work with high precision.

[0185] Prompt Sentence Examples

[0186] "Take audio containing part inspection procedures and convert it to text."

[0187] Extract important keywords from the captured text.

[0188] "Generate a parts inspection manual based on the extracted information."

[0189] "Please translate the generated manual into Spanish."

[0190] In this way, the system of the present invention efficiently transfers the knowledge and skills of experienced workers, achieves consistency and accuracy in work, and supports multiple languages, while enabling automation of work and real-time reference.

[0191] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0192] Step 1:

[0193] Voice acquisition: The terminal acquires the voice of an experienced worker giving on-site instruction. This is done using a voice recorder or smart headset. The input is the voice of the experienced worker, and the voice data is saved locally as output. This voice data is used for subsequent analysis.

[0194] Step 2:

[0195] Speech analysis: The device sends the captured voice data to the server. The server uses the Google Speech Recognition API to convert the voice data into natural language text data. The input is voice data, and the output is text data, which is used for subsequent information extraction.

[0196] Step 3:

[0197] Information extraction: The server uses natural language processing technology to extract instruction content and procedures from the text data converted by the speech analysis means. The input is text data, and keywords for the instruction content and procedures are generated as output. Specifically, text analysis is performed, and keywords such as "inspection," "once a week," and "necessary" are extracted.

[0198] Step 4:

[0199] Manual generation: The server automatically generates and updates a work manual based on the information extracted by the information extraction means. The input is the extracted keywords, and the output is a structured work manual. This manual is saved in JSON format and used for subsequent translation and reference.

[0200] Step 5:

[0201] Translation: The server translates the generated work manual into another language. For example, it uses the Google Translation API to translate a Japanese manual into Spanish. The input is the Japanese work manual, and the output is a translated work manual. This makes it possible to accommodate foreign workers.

[0202] Step 6:

[0203] Manual Reference: Users can view and correct work manuals through a web-based interface. The input is the user's viewing instructions and correction instructions, and the output is the displayed or updated manual. This interface is designed to allow easy access for users.

[0204] Step 7:

[0205] Machine operation: The terminal operates the machine based on the operation manual obtained from the server. The input is the operation manual, and the output is the machine performing the operation. A specific operation would be, for example, a robot inspecting parts according to a procedure. This improves work accuracy and safety.

[0206] Based on these processing steps, the knowledge and skills of experienced workers can be efficiently transferred, ensuring consistency and accuracy of work, and multilingual support, while enabling automation of work and real-time reference.

[0207] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0208] This invention is a system that efficiently transfers the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and improves the effectiveness of learning by recognizing the user's emotions. It consists of the following components:

[0209] System configuration

[0210] The system consists of the following main components:

[0211] 1. Voice acquisition means: The terminal acquires the voice of an experienced worker in real time. Specifically, the experienced worker wearing a headset captures the voice data of real-time work instruction and explanations given on-site.

[0212] 2. Voice analysis means: The device transmits the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology.

[0213] 3. Information extraction: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the voice analysis method. This is done using natural language processing technology.

[0214] 4. Manual generation method: The server automatically generates and updates the work manual based on the extracted information, ensuring that the latest instruction information is always reflected.

[0215] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[0216] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API, etc.

[0217] 7. Emotion Engine: Recognizes the user's emotions and provides manual feedback based on the emotional data. In conjunction with voice analysis, it can analyze the user's emotions in real time.

[0218] Specific examples of program processing

[0219] Operation of the audio acquisition means

[0220] The terminal (headset) picks up the voice of the experienced worker instructing in real time. When the experienced worker instructs, "This part needs to be inspected once a week," the terminal collects this voice and temporarily stores it locally.

[0221] Sending and analyzing voice data

[0222] The device sends voice data to the server, which receives the data and converts it into natural language text using a speech analysis tool. The textual content will be in the form of "This part needs to be inspected once a week."

[0223] Information Extraction and Manual Generation

[0224] The server uses the information extraction means to extract information such as "parts inspection" and "once a week" from the text data. Based on the extracted information, the server uses the manual generation means to add a new procedure to the "parts inspection manual" page. The new procedure will state, "Inspect part A once a week."

[0225] Emotion Engine Operation

[0226] The server uses an emotion engine to analyze the user's emotions in real time as they view the manual. It analyzes the user's facial expressions and tone of voice to detect emotions such as "low level of understanding" or "dissatisfaction." Depending on the results, it adjusts the content of the work manual or displays additional explanations.

[0227] Manual review and translation

[0228] A user checks a manual through a web interface. For example, a new employee opens the "Parts Inspection Manual" and sees the section that says "Part A should be inspected once a week." If a foreign worker wants to view the manual in Spanish, the server uses a translation tool to translate the manual into Spanish. After translation, the user can view the manual in Spanish through the same web interface.

[0229] In this way, the present invention not only digitizes the tacit knowledge of veteran workers and keeps them updated as the latest work manual, but also provides feedback based on the user's emotions. This allows for the smooth transfer of know-how to the next generation of employees, improving not only work efficiency and safety on-site, but also the effectiveness of learning.

[0230] The processing flow will be explained below.

[0231] Step 1:

[0232] The device receives audio

[0233] An experienced worker wears a headset and instructs a new worker at the work site, saying, "This part needs to be inspected once a week." The device (headset) picks up this voice in real time and temporarily stores it locally as voice data.

[0234] Step 2:

[0235] The device sends the voice data to the server

[0236] The device periodically encrypts and transmits the audio data over a secure channel to the server, at intervals optimized to accommodate battery and network usage.

[0237] Step 3:

[0238] The server analyzes the voice data

[0239] The server decompresses the received voice data and passes it to the voice analysis module, which converts the data into text format, resulting in the text "This part needs to be inspected once a week."

[0240] Step 4:

[0241] The server extracts the information

[0242] The server analyzes the text data and uses natural language processing technology to extract important information, including keywords such as "parts," "inspection," and "once a week." This information is used in later steps to generate the manual.

[0243] Step 5:

[0244] The server generates the operation manual

[0245] Based on the data obtained by the information extraction means, the server adds new work procedures to the work manual, specifically including a new procedure such as "Part A should be inspected once a week."

[0246] Step 6:

[0247] Server updates manual

[0248] Each time additional voice data is sent, the server updates the existing work manual, ensuring that the manual always reflects the most up-to-date information.

[0249] Step 7:

[0250] The server analyzes the user's emotions

[0251] When a user browses the work manual through the web interface, an emotion engine installed on the server analyzes the user's voice and facial expressions. If the user is confused, it provides feedback such as displaying additional explanations.

[0252] Step 8:

[0253] User checks the manual

[0254] A user (e.g., a new worker) checks the work manual from the web interface. The user opens the "Parts Inspection Manual" and sees the statement that "Part A should be inspected once a week."

[0255] Step 9:

[0256] The server translates the manual

[0257] If a foreign worker wishes to read the manual in Spanish, the server uses a translation means to convert the manual into Spanish, and the translated manual is displayed on the user's interface.

[0258] Step 10:

[0259] The server provides feedback based on emotional data

[0260] The emotion engine analyzes the user's reactions, and if it determines that the user's level of understanding is low, the server automatically inserts additional explanations, videos, and illustrations into the manual to support the user's learning.

[0261] In this way, the present invention not only digitizes the knowledge of experienced workers and automatically generates and updates work manuals, but also provides feedback according to the user's emotional state, thereby facilitating the transfer of knowledge to the next generation of employees and improving learning effectiveness.

[0262] Example 2

[0263] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0264] Efficiently transferring the knowledge and skills of veteran workers to the next generation of employees and improving on-site work efficiency and safety are important issues at many manufacturing and construction sites. Furthermore, providing appropriate feedback is necessary to improve user learning, but it is essential to consider the user's emotions when providing such feedback. However, with previous systems, it was difficult to analyze the user's emotions in real time and reflect them in the feedback.

[0265] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0266] In this invention, the server includes a means for acquiring the instruction voice of an experienced worker on-site, a means for analyzing the voice data acquired by the voice acquisition means and converting it into natural language, and a means for extracting instruction content and procedures from the text data converted by the voice analysis means. This new technology enables the digitization of the knowledge of experienced workers and smooth knowledge transfer to the next generation of employees. The server also includes an interface means for users to view the generated work manual and issue correction instructions, and an emotion engine that analyzes the user's emotions and provides feedback based on the emotion data, making it possible to provide flexible feedback based on the user's emotions.

[0267] "Audio acquisition means" refers to devices or technology that acquires in real time the instructional audio given by experienced workers on-site.

[0268] "Speech analysis means" refers to devices or technologies that analyze acquired voice data and convert it into natural language.

[0269] "Information extraction means" refers to a device or technology that extracts important information such as instruction content and procedures from the text data converted by the voice analysis means.

[0270] "Manual generation means" refers to a device or technology that automatically generates and updates a work manual based on the extracted information.

[0271] "Interface means" refers to an operation screen or system that allows a user to view the generated work manual and give instructions for correction.

[0272] An "emotion engine" refers to a device or technology that analyzes a user's emotions and provides feedback based on that emotional data.

[0273] "Translation means" refers to devices and technologies that translate work manuals into multiple languages.

[0274] A "headset" refers to a device worn by an experienced worker to capture audio.

[0275] This invention is a system that efficiently transfers the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and improves the effectiveness of learning by recognizing the user's emotions. The components of this system and their specific operation are described below.

[0276] System configuration

[0277] The system consists of the following main components:

[0278] 1. Audio acquisition method

[0279] The device uses a headset to capture the voice of an experienced worker in real time. For example, when an experienced worker wearing a headset provides instruction on-site and explains aloud, "This part needs to be inspected once a week," the device collects this voice and temporarily stores it locally.

[0280] 2. Audio analysis methods

[0281] The device sends the voice data to the server, which then uses voice recognition technology to convert the data into natural language text. Specific software used is a voice recognition API (e.g., Google Speech-to-Text). This process converts the voice data into text data, which is formatted as "This part needs to be inspected once a week."

[0282] 3. Information extraction means

[0283] The server uses natural language processing technology to extract important information from the text data, such as instruction content, work procedures, and points to note. Specifically, it uses a natural language processing library (e.g., NLTK) to extract keywords such as "parts inspection" and "once a week" from the text data.

[0284] 4. Manual Generation Methods

[0285] The server automatically generates and updates the work manual based on the extracted information. This ensures that the latest instruction information is always reflected. A manual page is generated that states, as a new procedure, that "Part A should be inspected once a week."

[0286] 5. Interface Methods

[0287] It provides an interface that allows users to view the generated work manual and provide correction instructions as necessary. The interface is web-based and designed for easy user access. For example, a new employee can open the "Parts Inspection Manual" and view the section that says "Part A should be inspected once a week."

[0288] 6. Emotion Engine

[0289] The server analyzes the user's emotions in real time as they view the manual and provides feedback. Specifically, it uses an emotion analysis API (e.g., Microsoft Azure Emotion API) that uses a webcam and microphone to analyze the user's facial expressions and tone of voice. For example, if the user's facial expression indicates a lack of understanding, the system will display additional explanations.

[0290] 7. Translation Methods

[0291] The server translates the generated work manual into multiple languages, making it available to foreign workers. The specific software used is a translation API (e.g., Google Translate API). For example, if a foreign worker wants to view the manual in Spanish, the server translates the manual into Spanish, and the user can view it from the same web interface.

[0292] Examples of concrete examples and prompts

[0293] For example, if an experienced worker instructs, "This part needs to be inspected once a week," the following sequence of actions will occur:

[0294] 1. The device captures audio through the headset and temporarily stores the audio data locally.

[0295] 2. The device sends the voice data to the server, which converts the voice data into text.

[0296] 3. The server extracts important information from the text data.

[0297] 4. The server adds the new information to the manual.

[0298] 5. The user checks the manual through the web interface and provides correction instructions.

[0299] 6. The server analyzes the user's sentiment and displays additional explanations if necessary.

[0300] 7. The server translates the manual so that users can view it in multiple languages.

[0301] Example prompt sentence:

[0302] "Please explain a system that captures in real time the voice of an experienced worker at a manufacturing site giving instructions on parts that require inspection once a week, converts that content into natural language, and reflects it in an operation manual. As a concrete example, please show instructions on part A."

[0303] In this way, the knowledge of veteran workers can be efficiently digitized and smoothly transferred to the next generation of employees. Also, by providing feedback based on the user's emotions, learning effectiveness is improved.

[0304] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0305] Step 1:

[0306] Audio Acquisition

[0307] The terminal uses a headset to pick up the instruction voice of an experienced worker in real time.

[0308] How it works: The device (headset) captures the voice of an experienced worker at the work site giving instructions such as, "This part needs to be inspected once a week." The headset captures the voice data in real time and temporarily stores it locally.

[0309] Input: Voice guidance from an experienced worker

[0310] Output: Temporarily saved audio data

[0311] Step 2:

[0312] Sending audio data

[0313] The terminal transmits the voice data to the server.

[0314] What it does: Your device sends locally stored audio data to a server over Wi-Fi or a cellular connection, securely using the SSL protocol.

[0315] Input: Temporarily saved audio data

[0316] Output: Audio data sent to the server

[0317] Step 3:

[0318] Audio analysis

[0319] The server analyzes the received audio data.

[0320] Specific operation: The server uses a speech recognition API (e.g., Google Speech-to-Text) to convert the voice data into natural language text. The converted text data is in the form of "This part needs to be inspected once a week."

[0321] Input: Audio data sent to the server

[0322] Output: Parsed text data

[0323] Step 4:

[0324] Extraction of important information

[0325] The server extracts important information from the text data obtained by speech analysis.

[0326] Specific operation: The server uses a natural language processing library (e.g., NLTK) to extract keywords such as "parts inspection" and "once a week" from the text data. The extracted information is then stored in a database.

[0327] Input: Parsed text data

[0328] Output: Extracted important information

[0329] Step 5:

[0330] Manual generation and updating

[0331] The server generates and updates the work manual based on the extracted information.

[0332] Specific operation: Based on the extracted information such as "part inspection" and "once a week," the server uses the manual generation means to add a new procedure to the "part inspection manual" page. The new procedure will state "Part A should be inspected once a week."

[0333] Input: Extracted sensitive information

[0334] Output: Updated operating manual

[0335] Step 6:

[0336] User sentiment analysis

[0337] The server analyzes the user's emotions.

[0338] Specific operation: When a user browses the manual through the web interface, the server uses an emotion analysis API (e.g., Microsoft Azure Emotion API) to analyze the user's facial expressions and tone of voice. If an emotion such as "low comprehension" is detected, the system displays additional explanations in the manual.

[0339] Input: User's facial expression data, voice tone

[0340] Output: Sentiment analysis results and adjusted manual content

[0341] Step 7:

[0342] Manual confirmation and correction instructions

[0343] Users can check the manual and provide correction instructions through a web interface.

[0344] Specific operation: The user can view the created work manual and, if there are any errors or additional explanations needed, provide correction instructions through the web interface. For example, feedback can be provided for the part that says "Part A should be inspected once a week."

[0345] Input: User views and feedback

[0346] Output: Correction instruction data

[0347] Step 8:

[0348] Manual Translation

[0349] The server translates the work manual into multiple languages.

[0350] Specific operation: The server uses a translation API (e.g., Google Translate API) to translate the generated work manual into multiple languages. If a foreign worker wants to view the manual in Spanish, the server translates the manual into Spanish, and the user can view it from the same web interface.

[0351] Input: Operation manual and selected language

[0352] Output: Translated work manual

[0353] In this way, by appropriately processing and calculating data at each processing step, it is possible to efficiently digitize the knowledge of experienced workers and provide feedback based on the user's emotions.

[0354] (Application example 2)

[0355] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] At manufacturing and construction sites, there is a need to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees. Important information, such as work procedures and precautions, is often communicated verbally on site, which can lead to inaccurate communication and reduced work efficiency and safety. Furthermore, at sites with multinational workers, language barriers can hinder work comprehension. Furthermore, there are concerns that feedback that takes into account emotions during learning may be ineffective.

[0357] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0358] In this invention, the server includes a voice acquisition means, a voice analysis means, an information extraction means, a manual generation means, a translation means, and an emotion engine means, which makes it possible to acquire in real time the instruction voices given by experienced workers on-site, automatically generate the contents as an operation manual, translate the contents into multiple languages, and provide feedback based on the user's emotions.

[0359] The "voice acquisition means" refers to a device and software for acquiring in real time the instruction voice given by an experienced worker on-site.

[0360] The "voice analysis means" refers to a device and software that analyzes acquired voice data and converts it into natural language.

[0361] The "information extraction means" refers to a device and software that extracts instruction content and procedures from the text data converted by the voice analysis means.

[0362] The "manual generation means" refers to a device and software that automatically generates and updates a work manual based on the information extracted by the information extraction means.

[0363] The "interface means" refers to a device and software that allows a user to view the generated work manual and give correction instructions.

[0364] The "translation means" refers to a device and software that translates the generated operation manual into multiple languages.

[0365] The "emotion engine means" is a device and software that analyzes the user's emotions and adjusts the contents of the operation manual based on the results.

[0366] A "smartphone" is a type of portable information terminal, a communication device that can be used to capture, analyze, display, and interface with voice.

[0367] "Smart glasses" are glasses-type devices with a built-in display for displaying information.

[0368] A "head-mounted display" is a display device that a user wears on their head, and is a terminal for displaying work manuals and instruction information.

[0369] This invention provides a system for efficiently transferring the knowledge and skills of veteran workers to the next generation of employees at manufacturing and construction sites. This system acquires the voice of the veteran worker giving instructions on-site, analyzes the content, automatically generates a work manual, and further analyzes the user's emotions and provides feedback to improve learning effectiveness.

[0370] System configuration

[0371] The system consists of the following major components:

[0372] 1. Audio acquisition method

[0373] An experienced worker wears a headset and captures their voice in real time. For example, if a worker says, "This part needs to be inspected once a week," the headset captures this voice. An example of the hardware used is a typical smart headset.

[0374] 2. Audio analysis methods

[0375] Voice data is sent to a server, which then converts it into natural language using speech recognition technology (such as Google Cloud Speech API).

[0376] 3. Information extraction means

[0377] Using natural language processing technology (such as the NLTK library) on the server, instruction content and procedures are extracted from the text data. For example, important information such as "parts inspection" and "once a week" is extracted.

[0378] 4. Manual Generation Methods

[0379] Based on the extracted information, the server automatically generates and updates the work manual. This process uses software such as the Django framework. A new step in the work manual might be written as "Part A should be inspected once a week."

[0380] 5. Interface Methods

[0381] A web-based interface is provided that allows users (next-generation employees) to view the work manual and provide correction instructions as necessary. This interface uses React.js and other tools.

[0382] 6. Translation Methods

[0383] The server translates the generated work manual into multiple languages, making it available to foreign workers. The translation is performed using the Amazon Translate API. For example, if the manual is translated into Spanish, the user can view the manual in Spanish.

[0384] 7. Emotional Engine Means

[0385] As users browse the manual, their emotions are analyzed in real time using OpenCV and the Microsoft Azure Emotion API. For example, if the user is judged to have a "low level of understanding," more detailed explanations are added accordingly.

[0386] Examples of concrete examples and prompts

[0387] If the emotion engine determines that a worker who has looked at the contents of a work manual has a "low level of understanding," phrases that explain the contents in more detail are added.

[0388] Example prompt sentence:

[0389] "Please explain the contents of this manual in more detail. In particular, the following information is missing: Specific inspection procedures for parts."

[0390] Example result:

[0391] "Detailed text: It is important to inspect part A once a week. Specifically, 1. Remove cover, 2. Visually inspect, 3. Clean if necessary, 4. Reinstall part. Follow this procedure."

[0392] This invention not only digitizes the tacit knowledge of veteran workers and keeps them up-to-date as work manuals, but also provides feedback based on the user's emotions, which allows for the smooth transfer of know-how to the next generation of employees, improving work efficiency and safety on-site, and enhancing learning effectiveness.

[0393] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0394] Step 1:

[0395] Audio Acquisition

[0396] An experienced worker wearing a device (headset) receives real-time instruction voice on-site. The input voice might say, for example, "This part needs to be inspected once a week." The headset captures this voice data and temporarily stores it. At this time, the voice data is saved in local storage.

[0397] Step 2:

[0398] Sending audio data

[0399] The device sends the temporarily saved audio data to the server. The input is the audio data captured by the headset. A secure communication method such as HTTPS is used as the transmission protocol. The server stores the received audio data.

[0400] Step 3:

[0401] Audio analysis

[0402] The server receives the voice data and converts it into natural language text using a voice analysis tool. This process uses the Google Cloud Speech API. The input is the transmitted voice data, and the output is text data. For example, if the voice data is "This part needs to be inspected once a week," the output text data will be "This part needs to be inspected once a week."

[0403] Step 4:

[0404] Information Extraction

[0405] The server uses information extraction methods to extract instructional content and important procedures from the generated text data. This process uses natural language processing technology (such as the NLTK library). The input is the analyzed text data, and the output is the extracted information. For example, "parts inspection" and "once a week" are extracted.

[0406] Step 5:

[0407] Manual Generation

[0408] The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This process uses the Django framework, etc. The input is the extracted information, and the output is the generated work manual. For example, a new procedure in the manual might state, "Part A should be inspected once a week."

[0409] Step 6:

[0410] Interface provided

[0411] It provides a web-based interface for users to view the manual and provide correction instructions. This interface uses React.js etc. The input is the generated work manual, and the output is a web page that users can view.

[0412] Step 7:

[0413] Translation tools

[0414] The server translates the generated work manual into multiple languages. This process uses the Amazon Translate API. The input is the generated work manual, and the output is the translated manual. For example, the manual can be translated into Spanish, allowing users to view the manual in Spanish.

[0415] Step 8:

[0416] Emotion analysis

[0417] The system analyzes emotions in real time as users read manuals. This process uses OpenCV and the Microsoft Azure Emotion API. The input is the user's facial expressions and tone of voice, and the output is the analyzed emotional data. For example, if the user is determined to have a "low level of understanding," the system generates a prompt to add more detailed explanations based on the result.

[0418] Step 9:

[0419] Applying Feedback

[0420] The content of the work manual is adjusted based on the results of the analysis by the emotion engine means. The input is emotion data, and the output is a revised work manual. For example, if the level of understanding is determined to be "low," a prompt sentence is added, detailing the specific inspection procedures for the manual.

[0421] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0422] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0423] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0424] [Second embodiment]

[0425] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0426] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0427] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0428] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0429] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0430] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0431] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0432] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0433] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0434] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0435] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0436] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0437] This invention is a system for efficiently transferring the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and is configured as follows.

[0438] System configuration

[0439] The system consists of the following main components:

[0440] 1. Voice acquisition means: The terminal acquires the voice of the experienced worker giving instructions. Specifically, the experienced worker wearing a headset captures the voice data of the real-time work instruction and explanations given on-site.

[0441] 2. Voice analysis means: The device transmits the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology.

[0442] 3. Information extraction means: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the voice analysis means. This is done using natural language processing technology.

[0443] 4. Manual generation means: The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This ensures that the latest instruction information is always reflected.

[0444] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[0445] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API or similar.

[0446] Specific examples of program processing

[0447] Operation of the audio acquisition means

[0448] The device (headset) picks up the voice of an experienced worker instructing in real time. For example, an experienced worker may say, "This part needs to be inspected once a week." The device collects this voice and temporarily stores it locally.

[0449] Sending and analyzing voice data

[0450] The device sends voice data to the server, which receives the data and uses a voice analysis tool to convert it into natural language text, such as "This part needs to be inspected once a week."

[0451] Information Extraction and Manual Generation

[0452] The server uses the information extraction means to extract information such as "part inspection," "once a week," and "necessary" from the text data. Based on the extracted information, the server adds a new procedure to the "part inspection manual" page using the manual generation means. The new procedure states, "Inspect part A once a week."

[0453] Manual review and translation

[0454] A user checks a manual through a web interface. For example, a new employee opens the "Parts Inspection Manual" and views the section that says "Part A should be inspected once a week." If a foreign worker wants to view the manual in Spanish, the server uses a translation tool to translate the manual into Spanish. After translation, the user can view the manual in Spanish through the same web interface.

[0455] In this way, the present invention digitally and automatically visualizes the tacit knowledge of experienced workers, enabling them to be constantly updated as the latest work manuals. Furthermore, by translating the knowledge into multiple languages, the system can be used in international work sites. This allows for the smooth transfer of know-how to the next generation of employees, improving work efficiency and safety on-site.

[0456] The processing flow will be explained below.

[0457] Step 1:

[0458] The device receives audio

[0459] An experienced worker wears a headset and provides on-site work instructions. The device captures the worker's voice in real time through a microphone and temporarily stores it locally as audio data.

[0460] Step 2:

[0461] The device sends the voice data to the server

[0462] At regular intervals, the device encrypts the collected voice data and transmits it to the server through a secure channel.

[0463] Step 3:

[0464] The server analyzes the voice data

[0465] The server decompresses the received voice data and passes it to the voice analysis module, which uses natural language processing technology to convert the voice data into text format.

[0466] Step 4:

[0467] The server extracts the information

[0468] The server analyzes the text data and extracts specific keywords (e.g., work procedures, time, location) that correspond to the business context, using natural language processing technology.

[0469] Step 5:

[0470] The server generates the operation manual

[0471] Based on the information extracted by the server, the first edition of the work manual is automatically generated, including new work procedures and precautions. This manual summarizes specific procedures in an easy-to-understand manner using bullet points and diagrams.

[0472] Step 6:

[0473] Server updates manual

[0474] The server analyzes the new voice data, adds or modifies relevant information to the existing work manual, and updates the manual to reflect the latest information.

[0475] Step 7:

[0476] User checks the manual

[0477] Next-generation employees and foreign workers can access and review the generated work manuals through a web interface, and can easily find specific work procedures and precautions using the search function.

[0478] Step 8:

[0479] The server translates the manual

[0480] If the user requests the work manual in a different language, the server invokes the automatic translation module to convert the manual into the specified language, and the translated manual is displayed according to the user's language setting.

[0481] Example 1

[0482] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0483] On-site work instruction often relies on the experience and knowledge of veteran workers, and if this knowledge is not passed on properly, it hinders the improvement of the skills and knowledge of the next generation of workers. Furthermore, if the content of instruction is not standardized, there is a risk of work efficiency and safety declining. Furthermore, language barriers exist in international work sites, requiring multilingual support. To resolve these issues, an efficient and standardized knowledge transfer system is needed.

[0484] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0485] In this invention, the server includes a voice analysis means, an information extraction means, and a manual generation means, which automatically converts the knowledge and skills of experienced workers from voice to text, extracts important instructional content, and generates and updates a work manual based on the extracted information.

[0486] "Audio acquisition means" refers to equipment and software for collecting the instructional audio given by experienced workers on-site.

[0487] "Speech analysis means" refers to technology for analyzing captured speech data and converting it into natural language text.

[0488] "Information extraction means" refers to a technology for extracting important information such as instruction content and procedures from the text data converted by the voice analysis means.

[0489] "Manual generation means" refers to a system that automatically generates and updates a work manual based on the extracted information.

[0490] The "interface means" refers to an interface that allows a user to view the generated work manual and give correction instructions.

[0491] "Translation means" refers to the technology or system for translating the generated work manual into multiple languages.

[0492] This invention is a system for efficiently transferring the knowledge and skills of veteran workers to the next generation of employees at manufacturing and construction sites. The system consists of the following main components:

[0493] 1. Voice acquisition method: A method in which the terminal acquires the voice of an experienced worker giving instructions. Specifically, an experienced worker wearing a headset captures real-time work instruction and explanations on-site as voice data.

[0494] 2. Voice analysis: The device sends the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[0495] 3. Information extraction: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the speech analysis method. This is done using natural language processing technology (e.g., spaCy, NLTK).

[0496] 4. Manual generation means: The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This ensures that the latest instruction information is always reflected.

[0497] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[0498] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API (e.g., Google Translate API).

[0499] Specific examples of program processing

[0500] The device receives the voice instructions of the experienced worker through the headset and temporarily stores the voice data locally. The device then sends the voice data to a server. The server receives the voice data and converts it into natural language text using a voice analysis tool. For example, if an experienced worker instructs, "This part needs to be inspected once a week," the voice instructions are converted into text.

[0501] Next, the server uses information extraction means to extract important information from the text data, such as "part inspection," "once a week," and "necessary." Based on this information, the server automatically updates the work manual using manual generation means and adds new procedures. For example, a new procedure might read, "Inspect part A once a week."

[0502] Users can check the updated work manual through a web interface and make corrections or add comments as necessary. The server also uses a translation means to translate the generated manual into multiple languages. For example, if a foreign worker wants to view the manual in Spanish, the server translates it and the user can view the manual in Spanish from the same web interface.

[0503] Prompt Sentence Examples

[0504] ---

[0505] An experienced worker instructed, "This part needs to be inspected once a week." Add a new procedure to the work manual: "Inspect part A once a week."

[0506] ---

[0507] This system digitizes the tacit knowledge of veteran workers and makes it possible to efficiently pass it on to the next generation of employees. Multilingual support allows for smooth use even in international work sites. This improves work efficiency and ensures safety.

[0508] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0509] Step 1: Audio capture

[0510] Subject: Device

[0511] Specific operation: The terminal (experienced worker wearing a headset) receives real-time instruction audio on-site.

[0512] Input: Operator's voice instruction

[0513] Data processing: Using a microphone to capture audio signals and store them as digital audio data.

[0514] Output: Digital audio data

[0515] Step 2: Sending audio data

[0516] Subject: Device

[0517] Specific operation: The device temporarily stores the voice data locally and then transmits it to the server over the network.

[0518] Input: Digital audio data

[0519] Data processing: Converts voice data into a network format and transmits it

[0520] Output: Audio data sent to the server

[0521] Step 3: Audio analysis

[0522] Subject: Server

[0523] What it does: The server converts the received voice data into natural language text using speech analysis techniques, such as a speech recognition API (e.g., Google Cloud Speech-to-Text).

[0524] Input: Audio data sent to the server

[0525] Data processing: Analyzes voice data and converts it into natural language text

[0526] Output: Parsed text data

[0527] Step 4: Information Extraction

[0528] Subject: Server

[0529] Specific operation: The server processes the text data obtained from the speech analysis means using information extraction means to extract important instructional content and work procedures. Natural language processing technology (e.g., spaCy, NLTK) is used.

[0530] Input: Parsed text data

[0531] Data processing: Extracting important information from text

[0532] Output: Extracted information

[0533] Step 5: Manual generation

[0534] Subject: Server

[0535] Specific operation: The server automatically generates and updates the work manual based on the data obtained by the information extraction means.

[0536] Input: Extracted information

[0537] Data processing: Manual creation and updating

[0538] Output: Updated operating manual

[0539] Step 6: Manual review and correction

[0540] Subject: User

[0541] Specific operation: The user uses a web interface to view the generated work manual and add correction instructions or comments as necessary.

[0542] Input: Updated Operation Manual

[0543] Data processing: Adding corrections and comments

[0544] Output: Revised operating manual

[0545] Step 7: Manual Translation

[0546] Subject: Server

[0547] Specific operation: The server uses a translation tool to translate the work manual into multiple languages, using a translation API (e.g., Google Translate API).

[0548] Input: Corrected operation manual

[0549] Data processing: Manual translation

[0550] Output: Translated work manual

[0551] In this way, the system can digitize the knowledge of veteran workers and efficiently pass it on to the next generation. By detailing the processing content for each specific step, the overall picture of the operation can be clearly understood.

[0552] (Application example 1)

[0553] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0554] It is important to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees in on-site work. However, previous methods lacked consistency and accuracy in training content, and were not multilingual, which resulted in impacts on work efficiency and safety. Furthermore, manual recording and transmission methods made it difficult to automate and reference in real time.

[0555] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0556] In this invention, the server includes a voice acquisition means for acquiring voice guidance provided by an experienced worker on-site, a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting it into natural language, an information extraction means for extracting guidance content and procedures from the text data converted by the voice analysis means, a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means, an interface means for a user to view the generated work manual and give correction instructions, a translation means for translating the work manual into other languages, and a machine operation means for a machine to automatically refer to the work manual and perform work. This makes it possible to efficiently transfer the knowledge and skills of experienced workers, achieve consistency and accuracy in work, and support multiple languages, while enabling work automation and real-time reference.

[0557] "Veteran workers" are workers with long-term experience and deep knowledge in the field.

[0558] "Site" refers to the location where actual work, such as manufacturing or construction, takes place.

[0559] "Instruction voice" is a voice for instruction or explanation given by an experienced worker.

[0560] The "voice acquisition means" refers to a device or system that collects the instruction voice of an experienced worker.

[0561] "Audio data" refers to information in which instructional audio is recorded in digital format.

[0562] "Speech analysis means" refers to a device or software that analyzes acquired speech data and converts it into natural language text.

[0563] "Text data" is character string information converted by speech analysis.

[0564] "Information extraction means" refers to devices or software that extract important instructional content and procedures from text data.

[0565] "Instruction content" refers to specific instructions and explanations such as work procedures and precautions.

[0566] A "procedure" refers to the steps or method for performing a task.

[0567] The "manual generation means" refers to a device or software that automatically creates and updates a work manual based on the extracted information.

[0568] A "work manual" is a document that describes specific work procedures and instructions to be followed on-site.

[0569] "User" refers to the workers and administrators who use this system.

[0570] "Interface means" refers to devices and software that allow a user to view and correct the operation manual.

[0571] "Translation means" refers to devices or software that translates work manuals into multiple languages.

[0572] "Other languages" are languages ​​other than one's native language.

[0573] "Machine operating means" refers to devices and software that allow a machine to automatically perform work based on the created work manual.

[0574] "Machine" refers to an automated device that can perform simple tasks or complex processes.

[0575] The following describes an embodiment of the present invention. This system is designed to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees, and is composed of the following main components:

[0576] System Program

[0577] The audio capture means is used to capture the instruction voice given by experienced workers on-site. For this purpose, audio recording devices, smartphones, and smart headsets are used. The audio of the experienced worker giving instructions such as "This part needs to be inspected once a week" is recorded and saved locally.

[0578] The speech analysis means analyzes the acquired speech data and converts it into natural language text. This process uses the Google Speech Recognition API and Python libraries. The acquired speech is converted into text data, such as "This part needs to be inspected once a week."

[0579] The information extraction means extracts instruction content and procedures from the text data converted by the voice analysis means. This process uses natural language processing technology to extract key keywords (e.g., inspection, once a week, necessary, etc.).

[0580] The manual generation means automatically generates and updates the work manual based on the extracted information. This allows the work manual to always reflect the latest instruction information and be saved in JSON format.

[0581] The interface means allows the user to view the generated work manual and give correction instructions. This interface is web-based and designed to be easily accessible by the user.

[0582] The translation means translates the work manual into other languages ​​(e.g., Spanish). This process uses the Google Translate API. By translating the generated work manual, it can be used by foreign workers.

[0583] The machine operation means allows the robot to automatically refer to the created work manual and perform the work, which allows the robot to act according to the procedure, improving the accuracy and safety of on-site work.

[0584] Specific examples

[0585] For example, in a factory, a robot can capture the voice instructions of an experienced worker in real time and record the weekly part inspection procedure as text data. This data is automatically analyzed and generated as a work manual. The generated manual is translated into Spanish, making it suitable for foreign workers as well. The robot can refer to this manual and perform part inspection work with high precision.

[0586] Prompt Sentence Examples

[0587] "Take audio containing part inspection procedures and convert it to text."

[0588] Extract important keywords from the captured text.

[0589] "Generate a parts inspection manual based on the extracted information."

[0590] "Please translate the generated manual into Spanish."

[0591] In this way, the system of the present invention efficiently transfers the knowledge and skills of experienced workers, achieves consistency and accuracy in work, and supports multiple languages, while enabling automation of work and real-time reference.

[0592] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0593] Step 1:

[0594] Voice acquisition: The terminal acquires the voice of an experienced worker giving on-site instruction. This is done using a voice recorder or smart headset. The input is the voice of the experienced worker, and the voice data is saved locally as output. This voice data is used for subsequent analysis.

[0595] Step 2:

[0596] Speech analysis: The device sends the captured voice data to the server. The server uses the Google Speech Recognition API to convert the voice data into natural language text data. The input is voice data, and the output is text data, which is used for subsequent information extraction.

[0597] Step 3:

[0598] Information extraction: The server uses natural language processing technology to extract instruction content and procedures from the text data converted by the speech analysis means. The input is text data, and keywords for the instruction content and procedures are generated as output. Specifically, text analysis is performed, and keywords such as "inspection," "once a week," and "necessary" are extracted.

[0599] Step 4:

[0600] Manual generation: The server automatically generates and updates a work manual based on the information extracted by the information extraction means. The input is the extracted keywords, and the output is a structured work manual. This manual is saved in JSON format and used for subsequent translation and reference.

[0601] Step 5:

[0602] Translation: The server translates the generated work manual into another language. For example, it uses the Google Translation API to translate a Japanese manual into Spanish. The input is the Japanese work manual, and the output is a translated work manual. This makes it possible to accommodate foreign workers.

[0603] Step 6:

[0604] Manual Reference: Users can view and correct work manuals through a web-based interface. The input is the user's viewing instructions and correction instructions, and the output is the displayed or updated manual. This interface is designed to allow easy access for users.

[0605] Step 7:

[0606] Machine operation: The terminal operates the machine based on the operation manual obtained from the server. The input is the operation manual, and the output is the machine performing the operation. A specific operation would be, for example, a robot inspecting parts according to a procedure. This improves work accuracy and safety.

[0607] Based on these processing steps, the knowledge and skills of experienced workers can be efficiently transferred, ensuring consistency and accuracy of work, and multilingual support, while enabling automation of work and real-time reference.

[0608] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0609] This invention is a system that efficiently transfers the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and improves the effectiveness of learning by recognizing the user's emotions. It consists of the following components:

[0610] System configuration

[0611] The system consists of the following main components:

[0612] 1. Voice acquisition means: The terminal acquires the voice of an experienced worker in real time. Specifically, the experienced worker wearing a headset captures the voice data of real-time work instruction and explanations given on-site.

[0613] 2. Voice analysis means: The device transmits the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology.

[0614] 3. Information extraction: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the voice analysis method. This is done using natural language processing technology.

[0615] 4. Manual generation method: The server automatically generates and updates the work manual based on the extracted information, ensuring that the latest instruction information is always reflected.

[0616] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[0617] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API, etc.

[0618] 7. Emotion Engine: Recognizes the user's emotions and provides manual feedback based on the emotional data. In conjunction with voice analysis, it can analyze the user's emotions in real time.

[0619] Specific examples of program processing

[0620] Operation of the audio acquisition means

[0621] The terminal (headset) picks up the voice of the experienced worker instructing in real time. When the experienced worker instructs, "This part needs to be inspected once a week," the terminal collects this voice and temporarily stores it locally.

[0622] Sending and analyzing voice data

[0623] The device sends voice data to the server, which receives the data and converts it into natural language text using a speech analysis tool. The textual content will be in the form of "This part needs to be inspected once a week."

[0624] Information Extraction and Manual Generation

[0625] The server uses the information extraction means to extract information such as "parts inspection" and "once a week" from the text data. Based on the extracted information, the server uses the manual generation means to add a new procedure to the "parts inspection manual" page. The new procedure will state, "Inspect part A once a week."

[0626] Emotion Engine Operation

[0627] The server uses an emotion engine to analyze the user's emotions in real time as they view the manual. It analyzes the user's facial expressions and tone of voice to detect emotions such as "low level of understanding" or "dissatisfaction." Depending on the results, it adjusts the content of the work manual or displays additional explanations.

[0628] Manual review and translation

[0629] A user checks a manual through a web interface. For example, a new employee opens the "Parts Inspection Manual" and sees the section that says "Part A should be inspected once a week." If a foreign worker wants to view the manual in Spanish, the server uses a translation tool to translate the manual into Spanish. After translation, the user can view the manual in Spanish through the same web interface.

[0630] In this way, the present invention not only digitizes the tacit knowledge of veteran workers and keeps them updated as the latest work manual, but also provides feedback based on the user's emotions. This allows for the smooth transfer of know-how to the next generation of employees, improving not only work efficiency and safety on-site, but also the effectiveness of learning.

[0631] The processing flow will be explained below.

[0632] Step 1:

[0633] The device receives audio

[0634] An experienced worker wears a headset and instructs a new worker at the work site, saying, "This part needs to be inspected once a week." The device (headset) picks up this voice in real time and temporarily stores it locally as voice data.

[0635] Step 2:

[0636] The device sends the voice data to the server

[0637] The device periodically encrypts and transmits the audio data over a secure channel to the server, at intervals optimized to accommodate battery and network usage.

[0638] Step 3:

[0639] The server analyzes the voice data

[0640] The server decompresses the received voice data and passes it to the voice analysis module, which converts the data into text format, resulting in the text "This part needs to be inspected once a week."

[0641] Step 4:

[0642] The server extracts the information

[0643] The server analyzes the text data and uses natural language processing technology to extract important information, including keywords such as "parts," "inspection," and "once a week." This information is used in later steps to generate the manual.

[0644] Step 5:

[0645] The server generates the operation manual

[0646] Based on the data obtained by the information extraction means, the server adds new work procedures to the work manual, specifically including a new procedure such as "Part A should be inspected once a week."

[0647] Step 6:

[0648] Server updates manual

[0649] Each time additional voice data is sent, the server updates the existing work manual, ensuring that the manual always reflects the most up-to-date information.

[0650] Step 7:

[0651] The server analyzes the user's emotions

[0652] When a user browses the work manual through the web interface, an emotion engine installed on the server analyzes the user's voice and facial expressions. If the user is confused, it provides feedback such as displaying additional explanations.

[0653] Step 8:

[0654] User checks the manual

[0655] A user (e.g., a new worker) checks the work manual from the web interface. The user opens the "Parts Inspection Manual" and sees the statement that "Part A should be inspected once a week."

[0656] Step 9:

[0657] The server translates the manual

[0658] If a foreign worker wishes to read the manual in Spanish, the server uses a translation means to convert the manual into Spanish, and the translated manual is displayed on the user's interface.

[0659] Step 10:

[0660] The server provides feedback based on emotional data

[0661] The emotion engine analyzes the user's reactions, and if it determines that the user's level of understanding is low, the server automatically inserts additional explanations, videos, and illustrations into the manual to support the user's learning.

[0662] In this way, the present invention not only digitizes the knowledge of experienced workers and automatically generates and updates work manuals, but also provides feedback according to the user's emotional state, thereby facilitating the transfer of knowledge to the next generation of employees and improving learning effectiveness.

[0663] Example 2

[0664] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0665] Efficiently transferring the knowledge and skills of veteran workers to the next generation of employees and improving on-site work efficiency and safety are important issues at many manufacturing and construction sites. Furthermore, providing appropriate feedback is necessary to improve user learning, but it is essential to consider the user's emotions when providing such feedback. However, with previous systems, it was difficult to analyze the user's emotions in real time and reflect them in the feedback.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0667] In this invention, the server includes a means for acquiring the instruction voice of an experienced worker on-site, a means for analyzing the voice data acquired by the voice acquisition means and converting it into natural language, and a means for extracting instruction content and procedures from the text data converted by the voice analysis means. This new technology enables the digitization of the knowledge of experienced workers and smooth knowledge transfer to the next generation of employees. The server also includes an interface means for users to view the generated work manual and issue correction instructions, and an emotion engine that analyzes the user's emotions and provides feedback based on the emotion data, making it possible to provide flexible feedback based on the user's emotions.

[0668] "Audio acquisition means" refers to devices or technology that acquires in real time the instructional audio given by experienced workers on-site.

[0669] "Speech analysis means" refers to devices or technologies that analyze acquired voice data and convert it into natural language.

[0670] "Information extraction means" refers to a device or technology that extracts important information such as instruction content and procedures from the text data converted by the voice analysis means.

[0671] "Manual generation means" refers to a device or technology that automatically generates and updates a work manual based on the extracted information.

[0672] "Interface means" refers to an operation screen or system that allows a user to view the generated work manual and give instructions for correction.

[0673] An "emotion engine" refers to a device or technology that analyzes a user's emotions and provides feedback based on that emotional data.

[0674] "Translation means" refers to devices and technologies that translate work manuals into multiple languages.

[0675] A "headset" refers to a device worn by an experienced worker to capture audio.

[0676] This invention is a system that efficiently transfers the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and improves the effectiveness of learning by recognizing the user's emotions. The components of this system and their specific operation are described below.

[0677] System configuration

[0678] The system consists of the following main components:

[0679] 1. Audio acquisition method

[0680] The device uses a headset to capture the voice of an experienced worker in real time. For example, when an experienced worker wearing a headset provides instruction on-site and explains aloud, "This part needs to be inspected once a week," the device collects this voice and temporarily stores it locally.

[0681] 2. Audio analysis methods

[0682] The device sends the voice data to the server, which then uses voice recognition technology to convert the data into natural language text. Specific software used is a voice recognition API (e.g., Google Speech-to-Text). This process converts the voice data into text data, which is formatted as "This part needs to be inspected once a week."

[0683] 3. Information extraction means

[0684] The server uses natural language processing technology to extract important information from the text data, such as instruction content, work procedures, and points to note. Specifically, it uses a natural language processing library (e.g., NLTK) to extract keywords such as "parts inspection" and "once a week" from the text data.

[0685] 4. Manual Generation Methods

[0686] The server automatically generates and updates the work manual based on the extracted information. This ensures that the latest instruction information is always reflected. A manual page is generated that states, as a new procedure, that "Part A should be inspected once a week."

[0687] 5. Interface Methods

[0688] It provides an interface that allows users to view the generated work manual and provide correction instructions as necessary. The interface is web-based and designed for easy user access. For example, a new employee can open the "Parts Inspection Manual" and view the section that says "Part A should be inspected once a week."

[0689] 6. Emotion Engine

[0690] The server analyzes the user's emotions in real time as they view the manual and provides feedback. Specifically, it uses an emotion analysis API (e.g., Microsoft Azure Emotion API) that uses a webcam and microphone to analyze the user's facial expressions and tone of voice. For example, if the user's facial expression indicates a lack of understanding, the system will display additional explanations.

[0691] 7. Translation Methods

[0692] The server translates the generated work manual into multiple languages, making it available to foreign workers. The specific software used is a translation API (e.g., Google Translate API). For example, if a foreign worker wants to view the manual in Spanish, the server translates the manual into Spanish, and the user can view it from the same web interface.

[0693] Examples of concrete examples and prompts

[0694] For example, if an experienced worker instructs, "This part needs to be inspected once a week," the following sequence of actions will occur:

[0695] 1. The device captures audio through the headset and temporarily stores the audio data locally.

[0696] 2. The device sends the voice data to the server, which converts the voice data into text.

[0697] 3. The server extracts important information from the text data.

[0698] 4. The server adds the new information to the manual.

[0699] 5. The user checks the manual through the web interface and provides correction instructions.

[0700] 6. The server analyzes the user's sentiment and displays additional explanations if necessary.

[0701] 7. The server translates the manual so that users can view it in multiple languages.

[0702] Example prompt sentence:

[0703] "Please explain a system that captures in real time the voice of an experienced worker at a manufacturing site giving instructions on parts that require inspection once a week, converts that content into natural language, and reflects it in an operation manual. As a concrete example, please show instructions on part A."

[0704] In this way, the knowledge of veteran workers can be efficiently digitized and smoothly transferred to the next generation of employees. Also, by providing feedback based on the user's emotions, learning effectiveness is improved.

[0705] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0706] Step 1:

[0707] Audio Acquisition

[0708] The terminal uses a headset to pick up the instruction voice of an experienced worker in real time.

[0709] How it works: The device (headset) captures the voice of an experienced worker at the work site giving instructions such as, "This part needs to be inspected once a week." The headset captures the voice data in real time and temporarily stores it locally.

[0710] Input: Voice guidance from an experienced worker

[0711] Output: Temporarily saved audio data

[0712] Step 2:

[0713] Sending audio data

[0714] The terminal transmits the voice data to the server.

[0715] What it does: Your device sends locally stored audio data to a server over Wi-Fi or a cellular connection, securely using the SSL protocol.

[0716] Input: Temporarily saved audio data

[0717] Output: Audio data sent to the server

[0718] Step 3:

[0719] Audio analysis

[0720] The server analyzes the received audio data.

[0721] Specific operation: The server uses a speech recognition API (e.g., Google Speech-to-Text) to convert the voice data into natural language text. The converted text data is in the form of "This part needs to be inspected once a week."

[0722] Input: Audio data sent to the server

[0723] Output: Parsed text data

[0724] Step 4:

[0725] Extraction of important information

[0726] The server extracts important information from the text data obtained by speech analysis.

[0727] Specific operation: The server uses a natural language processing library (e.g., NLTK) to extract keywords such as "parts inspection" and "once a week" from the text data. The extracted information is then stored in a database.

[0728] Input: Parsed text data

[0729] Output: Extracted important information

[0730] Step 5:

[0731] Manual generation and updating

[0732] The server generates and updates the work manual based on the extracted information.

[0733] Specific operation: Based on the extracted information such as "part inspection" and "once a week," the server uses the manual generation means to add a new procedure to the "part inspection manual" page. The new procedure will state "Part A should be inspected once a week."

[0734] Input: Extracted sensitive information

[0735] Output: Updated operating manual

[0736] Step 6:

[0737] User sentiment analysis

[0738] The server analyzes the user's emotions.

[0739] Specific operation: When a user browses the manual through the web interface, the server uses an emotion analysis API (e.g., Microsoft Azure Emotion API) to analyze the user's facial expressions and tone of voice. If an emotion such as "low comprehension" is detected, the system displays additional explanations in the manual.

[0740] Input: User's facial expression data, voice tone

[0741] Output: Sentiment analysis results and adjusted manual content

[0742] Step 7:

[0743] Manual confirmation and correction instructions

[0744] Users can check the manual and provide correction instructions through a web interface.

[0745] Specific operation: The user can view the created work manual and, if there are any errors or additional explanations needed, provide correction instructions through the web interface. For example, feedback can be provided for the part that says "Part A should be inspected once a week."

[0746] Input: User views and feedback

[0747] Output: Correction instruction data

[0748] Step 8:

[0749] Manual Translation

[0750] The server translates the work manual into multiple languages.

[0751] Specific operation: The server uses a translation API (e.g., Google Translate API) to translate the generated work manual into multiple languages. If a foreign worker wants to view the manual in Spanish, the server translates the manual into Spanish, and the user can view it from the same web interface.

[0752] Input: Operation manual and selected language

[0753] Output: Translated work manual

[0754] In this way, by appropriately processing and calculating data at each processing step, it is possible to efficiently digitize the knowledge of experienced workers and provide feedback based on the user's emotions.

[0755] (Application example 2)

[0756] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0757] At manufacturing and construction sites, there is a need to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees. Important information, such as work procedures and precautions, is often communicated verbally on site, which can lead to inaccurate communication and reduced work efficiency and safety. Furthermore, at sites with multinational workers, language barriers can hinder work comprehension. Furthermore, there are concerns that feedback that takes into account emotions during learning may be ineffective.

[0758] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0759] In this invention, the server includes a voice acquisition means, a voice analysis means, an information extraction means, a manual generation means, a translation means, and an emotion engine means, which makes it possible to acquire in real time the instruction voices given by experienced workers on-site, automatically generate the contents as an operation manual, translate the contents into multiple languages, and provide feedback based on the user's emotions.

[0760] The "voice acquisition means" refers to a device and software for acquiring in real time the instruction voice given by an experienced worker on-site.

[0761] The "voice analysis means" refers to a device and software that analyzes acquired voice data and converts it into natural language.

[0762] The "information extraction means" refers to a device and software that extracts instruction content and procedures from the text data converted by the voice analysis means.

[0763] The "manual generation means" refers to a device and software that automatically generates and updates a work manual based on the information extracted by the information extraction means.

[0764] The "interface means" refers to a device and software that allows a user to view the generated work manual and give correction instructions.

[0765] The "translation means" refers to a device and software that translates the generated operation manual into multiple languages.

[0766] The "emotion engine means" is a device and software that analyzes the user's emotions and adjusts the contents of the operation manual based on the results.

[0767] A "smartphone" is a type of portable information terminal, a communication device that can be used to capture, analyze, display, and interface with voice.

[0768] "Smart glasses" are glasses-type devices with a built-in display for displaying information.

[0769] A "head-mounted display" is a display device that a user wears on their head, and is a terminal for displaying work manuals and instruction information.

[0770] This invention provides a system for efficiently transferring the knowledge and skills of veteran workers to the next generation of employees at manufacturing and construction sites. This system acquires the voice of the veteran worker giving instructions on-site, analyzes the content, automatically generates a work manual, and further analyzes the user's emotions and provides feedback to improve learning effectiveness.

[0771] System configuration

[0772] The system consists of the following major components:

[0773] 1. Audio acquisition method

[0774] An experienced worker wears a headset and captures their voice in real time. For example, if a worker says, "This part needs to be inspected once a week," the headset captures this voice. An example of the hardware used is a typical smart headset.

[0775] 2. Audio analysis methods

[0776] Voice data is sent to a server, which then converts it into natural language using speech recognition technology (such as Google Cloud Speech API).

[0777] 3. Information extraction means

[0778] Using natural language processing technology (such as the NLTK library) on the server, instruction content and procedures are extracted from the text data. For example, important information such as "parts inspection" and "once a week" is extracted.

[0779] 4. Manual Generation Methods

[0780] Based on the extracted information, the server automatically generates and updates the work manual. This process uses software such as the Django framework. A new step in the work manual might be written as "Part A should be inspected once a week."

[0781] 5. Interface Methods

[0782] A web-based interface is provided that allows users (next-generation employees) to view the work manual and provide correction instructions as necessary. This interface uses React.js and other tools.

[0783] 6. Translation Methods

[0784] The server translates the generated work manual into multiple languages, making it available to foreign workers. The translation is performed using the Amazon Translate API. For example, if the manual is translated into Spanish, the user can view the manual in Spanish.

[0785] 7. Emotional Engine Means

[0786] As users browse the manual, their emotions are analyzed in real time using OpenCV and the Microsoft Azure Emotion API. For example, if the user is judged to have a "low level of understanding," more detailed explanations are added accordingly.

[0787] Examples of concrete examples and prompts

[0788] If the emotion engine determines that a worker who has looked at the contents of a work manual has a "low level of understanding," phrases that explain the contents in more detail are added.

[0789] Example prompt sentence:

[0790] "Please explain the contents of this manual in more detail. In particular, the following information is missing: Specific inspection procedures for parts."

[0791] Example result:

[0792] "Detailed text: It is important to inspect part A once a week. Specifically, 1. Remove cover, 2. Visually inspect, 3. Clean if necessary, 4. Reinstall part. Follow this procedure."

[0793] This invention not only digitizes the tacit knowledge of veteran workers and keeps them up-to-date as work manuals, but also provides feedback based on the user's emotions, which allows for the smooth transfer of know-how to the next generation of employees, improving work efficiency and safety on-site, and enhancing learning effectiveness.

[0794] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0795] Step 1:

[0796] Audio Acquisition

[0797] An experienced worker wearing a device (headset) receives real-time instruction voice on-site. The input voice might say, for example, "This part needs to be inspected once a week." The headset captures this voice data and temporarily stores it. At this time, the voice data is saved in local storage.

[0798] Step 2:

[0799] Sending audio data

[0800] The device sends the temporarily saved audio data to the server. The input is the audio data captured by the headset. A secure communication method such as HTTPS is used as the transmission protocol. The server stores the received audio data.

[0801] Step 3:

[0802] Audio analysis

[0803] The server receives the voice data and converts it into natural language text using a voice analysis tool. This process uses the Google Cloud Speech API. The input is the transmitted voice data, and the output is text data. For example, if the voice data is "This part needs to be inspected once a week," the output text data will be "This part needs to be inspected once a week."

[0804] Step 4:

[0805] Information Extraction

[0806] The server uses information extraction methods to extract instructional content and important procedures from the generated text data. This process uses natural language processing technology (such as the NLTK library). The input is the analyzed text data, and the output is the extracted information. For example, "parts inspection" and "once a week" are extracted.

[0807] Step 5:

[0808] Manual Generation

[0809] The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This process uses the Django framework, etc. The input is the extracted information, and the output is the generated work manual. For example, a new procedure in the manual might state, "Part A should be inspected once a week."

[0810] Step 6:

[0811] Interface provided

[0812] It provides a web-based interface for users to view the manual and provide correction instructions. This interface uses React.js etc. The input is the generated work manual, and the output is a web page that users can view.

[0813] Step 7:

[0814] Translation tools

[0815] The server translates the generated work manual into multiple languages. This process uses the Amazon Translate API. The input is the generated work manual, and the output is the translated manual. For example, the manual can be translated into Spanish, allowing users to view the manual in Spanish.

[0816] Step 8:

[0817] Emotion analysis

[0818] The system analyzes emotions in real time as users read manuals. This process uses OpenCV and the Microsoft Azure Emotion API. The input is the user's facial expressions and tone of voice, and the output is the analyzed emotional data. For example, if the user is determined to have a "low level of understanding," the system generates a prompt to add more detailed explanations based on the result.

[0819] Step 9:

[0820] Applying Feedback

[0821] The content of the work manual is adjusted based on the results of the analysis by the emotion engine means. The input is emotion data, and the output is a revised work manual. For example, if the level of understanding is determined to be "low," a prompt sentence is added, detailing the specific inspection procedures for the manual.

[0822] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0823] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0824] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0825] [Third embodiment]

[0826] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0827] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0828] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0829] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0830] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0831] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0832] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0833] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0834] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0835] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0836] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0837] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0838] This invention is a system for efficiently transferring the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and is configured as follows.

[0839] System configuration

[0840] The system consists of the following main components:

[0841] 1. Voice acquisition means: The terminal acquires the voice of the experienced worker giving instructions. Specifically, the experienced worker wearing a headset captures the voice data of the real-time work instruction and explanations given on-site.

[0842] 2. Voice analysis means: The device transmits the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology.

[0843] 3. Information extraction means: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the voice analysis means. This is done using natural language processing technology.

[0844] 4. Manual generation means: The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This ensures that the latest instruction information is always reflected.

[0845] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[0846] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API or similar.

[0847] Specific examples of program processing

[0848] Operation of the audio acquisition means

[0849] The device (headset) picks up the voice of an experienced worker instructing in real time. For example, an experienced worker may say, "This part needs to be inspected once a week." The device collects this voice and temporarily stores it locally.

[0850] Sending and analyzing voice data

[0851] The device sends voice data to the server, which receives the data and uses a voice analysis tool to convert it into natural language text, such as "This part needs to be inspected once a week."

[0852] Information Extraction and Manual Generation

[0853] The server uses the information extraction means to extract information such as "part inspection," "once a week," and "necessary" from the text data. Based on the extracted information, the server adds a new procedure to the "part inspection manual" page using the manual generation means. The new procedure states, "Inspect part A once a week."

[0854] Manual review and translation

[0855] A user checks a manual through a web interface. For example, a new employee opens the "Parts Inspection Manual" and views the section that says "Part A should be inspected once a week." If a foreign worker wants to view the manual in Spanish, the server uses a translation tool to translate the manual into Spanish. After translation, the user can view the manual in Spanish through the same web interface.

[0856] In this way, the present invention digitally and automatically visualizes the tacit knowledge of experienced workers, enabling them to be constantly updated as the latest work manuals. Furthermore, by translating the knowledge into multiple languages, the system can be used in international work sites. This allows for the smooth transfer of know-how to the next generation of employees, improving work efficiency and safety on-site.

[0857] The processing flow will be explained below.

[0858] Step 1:

[0859] The device receives audio

[0860] An experienced worker wears a headset and provides on-site work instructions. The device captures the worker's voice in real time through a microphone and temporarily stores it locally as audio data.

[0861] Step 2:

[0862] The device sends the voice data to the server

[0863] At regular intervals, the device encrypts the collected voice data and transmits it to the server through a secure channel.

[0864] Step 3:

[0865] The server analyzes the voice data

[0866] The server decompresses the received voice data and passes it to the voice analysis module, which uses natural language processing technology to convert the voice data into text format.

[0867] Step 4:

[0868] The server extracts the information

[0869] The server analyzes the text data and extracts specific keywords (e.g., work procedures, time, location) that correspond to the business context, using natural language processing technology.

[0870] Step 5:

[0871] The server generates the operation manual

[0872] Based on the information extracted by the server, the first edition of the work manual is automatically generated, including new work procedures and precautions. This manual summarizes specific procedures in an easy-to-understand manner using bullet points and diagrams.

[0873] Step 6:

[0874] Server updates manual

[0875] The server analyzes the new voice data, adds or modifies relevant information to the existing work manual, and updates the manual to reflect the latest information.

[0876] Step 7:

[0877] User checks the manual

[0878] Next-generation employees and foreign workers can access and review the generated work manuals through a web interface, and can easily find specific work procedures and precautions using the search function.

[0879] Step 8:

[0880] The server translates the manual

[0881] If the user requests the work manual in a different language, the server invokes the automatic translation module to convert the manual into the specified language, and the translated manual is displayed according to the user's language setting.

[0882] Example 1

[0883] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0884] On-site work instruction often relies on the experience and knowledge of veteran workers, and if this knowledge is not passed on properly, it hinders the improvement of the skills and knowledge of the next generation of workers. Furthermore, if the content of instruction is not standardized, there is a risk of work efficiency and safety declining. Furthermore, language barriers exist in international work sites, requiring multilingual support. To resolve these issues, an efficient and standardized knowledge transfer system is needed.

[0885] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0886] In this invention, the server includes a voice analysis means, an information extraction means, and a manual generation means, which automatically converts the knowledge and skills of experienced workers from voice to text, extracts important instructional content, and generates and updates a work manual based on the extracted information.

[0887] "Audio acquisition means" refers to equipment and software for collecting the instructional audio given by experienced workers on-site.

[0888] "Speech analysis means" refers to technology for analyzing captured speech data and converting it into natural language text.

[0889] "Information extraction means" refers to a technology for extracting important information such as instruction content and procedures from the text data converted by the voice analysis means.

[0890] "Manual generation means" refers to a system that automatically generates and updates a work manual based on the extracted information.

[0891] The "interface means" refers to an interface that allows a user to view the generated work manual and give correction instructions.

[0892] "Translation means" refers to the technology or system for translating the generated work manual into multiple languages.

[0893] This invention is a system for efficiently transferring the knowledge and skills of veteran workers to the next generation of employees at manufacturing and construction sites. The system consists of the following main components:

[0894] 1. Voice acquisition method: A method in which the terminal acquires the voice of an experienced worker giving instructions. Specifically, an experienced worker wearing a headset captures real-time work instruction and explanations on-site as voice data.

[0895] 2. Voice analysis: The device sends the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[0896] 3. Information extraction: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the speech analysis method. This is done using natural language processing technology (e.g., spaCy, NLTK).

[0897] 4. Manual generation means: The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This ensures that the latest instruction information is always reflected.

[0898] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[0899] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API (e.g., Google Translate API).

[0900] Specific examples of program processing

[0901] The device receives the voice instructions of the experienced worker through the headset and temporarily stores the voice data locally. The device then sends the voice data to a server. The server receives the voice data and converts it into natural language text using a voice analysis tool. For example, if an experienced worker instructs, "This part needs to be inspected once a week," the voice instructions are converted into text.

[0902] Next, the server uses information extraction means to extract important information from the text data, such as "part inspection," "once a week," and "necessary." Based on this information, the server automatically updates the work manual using manual generation means and adds new procedures. For example, a new procedure might read, "Inspect part A once a week."

[0903] Users can check the updated work manual through a web interface and make corrections or add comments as necessary. The server also uses a translation means to translate the generated manual into multiple languages. For example, if a foreign worker wants to view the manual in Spanish, the server translates it and the user can view the manual in Spanish from the same web interface.

[0904] Prompt Sentence Examples

[0905] ---

[0906] An experienced worker instructed, "This part needs to be inspected once a week." Add a new procedure to the work manual: "Inspect part A once a week."

[0907] ---

[0908] This system digitizes the tacit knowledge of veteran workers and makes it possible to efficiently pass it on to the next generation of employees. Multilingual support allows for smooth use even in international work sites. This improves work efficiency and ensures safety.

[0909] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0910] Step 1: Audio capture

[0911] Subject: Device

[0912] Specific operation: The terminal (experienced worker wearing a headset) receives real-time instruction audio on-site.

[0913] Input: Operator's voice instruction

[0914] Data processing: Using a microphone to capture audio signals and store them as digital audio data.

[0915] Output: Digital audio data

[0916] Step 2: Sending audio data

[0917] Subject: Device

[0918] Specific operation: The device temporarily stores the voice data locally and then transmits it to the server over the network.

[0919] Input: Digital audio data

[0920] Data processing: Converts voice data into a network format and transmits it

[0921] Output: Audio data sent to the server

[0922] Step 3: Audio analysis

[0923] Subject: Server

[0924] What it does: The server converts the received voice data into natural language text using speech analysis techniques, such as a speech recognition API (e.g., Google Cloud Speech-to-Text).

[0925] Input: Audio data sent to the server

[0926] Data processing: Analyzes voice data and converts it into natural language text

[0927] Output: Parsed text data

[0928] Step 4: Information Extraction

[0929] Subject: Server

[0930] Specific operation: The server processes the text data obtained from the speech analysis means using information extraction means to extract important instructional content and work procedures. Natural language processing technology (e.g., spaCy, NLTK) is used.

[0931] Input: Parsed text data

[0932] Data processing: Extracting important information from text

[0933] Output: Extracted information

[0934] Step 5: Manual generation

[0935] Subject: Server

[0936] Specific operation: The server automatically generates and updates the work manual based on the data obtained by the information extraction means.

[0937] Input: Extracted information

[0938] Data processing: Manual creation and updating

[0939] Output: Updated operating manual

[0940] Step 6: Manual review and correction

[0941] Subject: User

[0942] Specific operation: The user uses a web interface to view the generated work manual and add correction instructions or comments as necessary.

[0943] Input: Updated Operation Manual

[0944] Data processing: Adding corrections and comments

[0945] Output: Revised operating manual

[0946] Step 7: Manual Translation

[0947] Subject: Server

[0948] Specific operation: The server uses a translation tool to translate the work manual into multiple languages, using a translation API (e.g., Google Translate API).

[0949] Input: Corrected operation manual

[0950] Data processing: Manual translation

[0951] Output: Translated work manual

[0952] In this way, the system can digitize the knowledge of veteran workers and efficiently pass it on to the next generation. By detailing the processing content for each specific step, the overall picture of the operation can be clearly understood.

[0953] (Application example 1)

[0954] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0955] It is important to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees in on-site work. However, previous methods lacked consistency and accuracy in training content, and were not multilingual, which resulted in impacts on work efficiency and safety. Furthermore, manual recording and transmission methods made it difficult to automate and reference in real time.

[0956] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0957] In this invention, the server includes a voice acquisition means for acquiring voice guidance provided by an experienced worker on-site, a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting it into natural language, an information extraction means for extracting guidance content and procedures from the text data converted by the voice analysis means, a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means, an interface means for a user to view the generated work manual and give correction instructions, a translation means for translating the work manual into other languages, and a machine operation means for a machine to automatically refer to the work manual and perform work. This makes it possible to efficiently transfer the knowledge and skills of experienced workers, achieve consistency and accuracy in work, and support multiple languages, while enabling work automation and real-time reference.

[0958] "Veteran workers" are workers with long-term experience and deep knowledge in the field.

[0959] "Site" refers to the location where actual work, such as manufacturing or construction, takes place.

[0960] "Instruction voice" is a voice for instruction or explanation given by an experienced worker.

[0961] The "voice acquisition means" refers to a device or system that collects the instruction voice of an experienced worker.

[0962] "Audio data" refers to information in which instructional audio is recorded in digital format.

[0963] "Speech analysis means" refers to a device or software that analyzes acquired speech data and converts it into natural language text.

[0964] "Text data" is character string information converted by speech analysis.

[0965] "Information extraction means" refers to devices or software that extract important instructional content and procedures from text data.

[0966] "Instruction content" refers to specific instructions and explanations such as work procedures and precautions.

[0967] A "procedure" refers to the steps or method for performing a task.

[0968] The "manual generation means" refers to a device or software that automatically creates and updates a work manual based on the extracted information.

[0969] A "work manual" is a document that describes specific work procedures and instructions to be followed on-site.

[0970] "User" refers to the workers and administrators who use this system.

[0971] "Interface means" refers to devices and software that allow a user to view and correct the operation manual.

[0972] "Translation means" refers to devices or software that translates work manuals into multiple languages.

[0973] "Other languages" are languages ​​other than one's native language.

[0974] "Machine operating means" refers to devices and software that allow a machine to automatically perform work based on the created work manual.

[0975] "Machine" refers to an automated device that can perform simple tasks or complex processes.

[0976] The following describes an embodiment of the present invention. This system is designed to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees, and is composed of the following main components:

[0977] System Program

[0978] The audio capture means is used to capture the instruction voice given by experienced workers on-site. For this purpose, audio recording devices, smartphones, and smart headsets are used. The audio of the experienced worker giving instructions such as "This part needs to be inspected once a week" is recorded and saved locally.

[0979] The speech analysis means analyzes the acquired speech data and converts it into natural language text. This process uses the Google Speech Recognition API and Python libraries. The acquired speech is converted into text data, such as "This part needs to be inspected once a week."

[0980] The information extraction means extracts instruction content and procedures from the text data converted by the voice analysis means. This process uses natural language processing technology to extract key keywords (e.g., inspection, once a week, necessary, etc.).

[0981] The manual generation means automatically generates and updates the work manual based on the extracted information. This allows the work manual to always reflect the latest instruction information and be saved in JSON format.

[0982] The interface means allows the user to view the generated work manual and give correction instructions. This interface is web-based and designed to be easily accessible by the user.

[0983] The translation means translates the work manual into other languages ​​(e.g., Spanish). This process uses the Google Translate API. By translating the generated work manual, it can be used by foreign workers.

[0984] The machine operation means allows the robot to automatically refer to the created work manual and perform the work, which allows the robot to act according to the procedure, improving the accuracy and safety of on-site work.

[0985] Specific examples

[0986] For example, in a factory, a robot can capture the voice instructions of an experienced worker in real time and record the weekly part inspection procedure as text data. This data is automatically analyzed and generated as a work manual. The generated manual is translated into Spanish, making it suitable for foreign workers as well. The robot can refer to this manual and perform part inspection work with high precision.

[0987] Prompt Sentence Examples

[0988] "Take audio containing part inspection procedures and convert it to text."

[0989] Extract important keywords from the captured text.

[0990] "Generate a parts inspection manual based on the extracted information."

[0991] "Please translate the generated manual into Spanish."

[0992] In this way, the system of the present invention efficiently transfers the knowledge and skills of experienced workers, achieves consistency and accuracy in work, and supports multiple languages, while enabling automation of work and real-time reference.

[0993] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0994] Step 1:

[0995] Voice acquisition: The terminal acquires the voice of an experienced worker giving on-site instruction. This is done using a voice recorder or smart headset. The input is the voice of the experienced worker, and the voice data is saved locally as output. This voice data is used for subsequent analysis.

[0996] Step 2:

[0997] Speech analysis: The device sends the captured voice data to the server. The server uses the Google Speech Recognition API to convert the voice data into natural language text data. The input is voice data, and the output is text data, which is used for subsequent information extraction.

[0998] Step 3:

[0999] Information extraction: The server uses natural language processing technology to extract instruction content and procedures from the text data converted by the speech analysis means. The input is text data, and keywords for the instruction content and procedures are generated as output. Specifically, text analysis is performed, and keywords such as "inspection," "once a week," and "necessary" are extracted.

[1000] Step 4:

[1001] Manual generation: The server automatically generates and updates a work manual based on the information extracted by the information extraction means. The input is the extracted keywords, and the output is a structured work manual. This manual is saved in JSON format and used for subsequent translation and reference.

[1002] Step 5:

[1003] Translation: The server translates the generated work manual into another language. For example, it uses the Google Translation API to translate a Japanese manual into Spanish. The input is the Japanese work manual, and the output is a translated work manual. This makes it possible to accommodate foreign workers.

[1004] Step 6:

[1005] Manual Reference: Users can view and correct work manuals through a web-based interface. The input is the user's viewing instructions and correction instructions, and the output is the displayed or updated manual. This interface is designed to allow easy access for users.

[1006] Step 7:

[1007] Machine operation: The terminal operates the machine based on the operation manual obtained from the server. The input is the operation manual, and the output is the machine performing the operation. A specific operation would be, for example, a robot inspecting parts according to a procedure. This improves work accuracy and safety.

[1008] Based on these processing steps, the knowledge and skills of experienced workers can be efficiently transferred, ensuring consistency and accuracy of work, and multilingual support, while enabling automation of work and real-time reference.

[1009] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1010] This invention is a system that efficiently transfers the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and improves the effectiveness of learning by recognizing the user's emotions. It consists of the following components:

[1011] System configuration

[1012] The system consists of the following main components:

[1013] 1. Voice acquisition means: The terminal acquires the voice of an experienced worker in real time. Specifically, the experienced worker wearing a headset captures the voice data of real-time work instruction and explanations given on-site.

[1014] 2. Voice analysis means: The device transmits the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology.

[1015] 3. Information extraction: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the voice analysis method. This is done using natural language processing technology.

[1016] 4. Manual generation method: The server automatically generates and updates the work manual based on the extracted information, ensuring that the latest instruction information is always reflected.

[1017] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[1018] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API, etc.

[1019] 7. Emotion Engine: Recognizes the user's emotions and provides manual feedback based on the emotional data. In conjunction with voice analysis, it can analyze the user's emotions in real time.

[1020] Specific examples of program processing

[1021] Operation of the audio acquisition means

[1022] The terminal (headset) picks up the voice of the experienced worker instructing in real time. When the experienced worker instructs, "This part needs to be inspected once a week," the terminal collects this voice and temporarily stores it locally.

[1023] Sending and analyzing voice data

[1024] The device sends voice data to the server, which receives the data and converts it into natural language text using a speech analysis tool. The textual content will be in the form of "This part needs to be inspected once a week."

[1025] Information Extraction and Manual Generation

[1026] The server uses the information extraction means to extract information such as "parts inspection" and "once a week" from the text data. Based on the extracted information, the server uses the manual generation means to add a new procedure to the "parts inspection manual" page. The new procedure will state, "Inspect part A once a week."

[1027] Emotion Engine Operation

[1028] The server uses an emotion engine to analyze the user's emotions in real time as they view the manual. It analyzes the user's facial expressions and tone of voice to detect emotions such as "low level of understanding" or "dissatisfaction." Depending on the results, it adjusts the content of the work manual or displays additional explanations.

[1029] Manual review and translation

[1030] A user checks a manual through a web interface. For example, a new employee opens the "Parts Inspection Manual" and sees the section that says "Part A should be inspected once a week." If a foreign worker wants to view the manual in Spanish, the server uses a translation tool to translate the manual into Spanish. After translation, the user can view the manual in Spanish through the same web interface.

[1031] In this way, the present invention not only digitizes the tacit knowledge of veteran workers and keeps them updated as the latest work manual, but also provides feedback based on the user's emotions. This allows for the smooth transfer of know-how to the next generation of employees, improving not only work efficiency and safety on-site, but also the effectiveness of learning.

[1032] The processing flow will be explained below.

[1033] Step 1:

[1034] The device receives audio

[1035] An experienced worker wears a headset and instructs a new worker at the work site, saying, "This part needs to be inspected once a week." The device (headset) picks up this voice in real time and temporarily stores it locally as voice data.

[1036] Step 2:

[1037] The device sends the voice data to the server

[1038] The device periodically encrypts and transmits the audio data over a secure channel to the server, at intervals optimized to accommodate battery and network usage.

[1039] Step 3:

[1040] The server analyzes the voice data

[1041] The server decompresses the received voice data and passes it to the voice analysis module, which converts the data into text format, resulting in the text "This part needs to be inspected once a week."

[1042] Step 4:

[1043] The server extracts the information

[1044] The server analyzes the text data and uses natural language processing technology to extract important information, including keywords such as "parts," "inspection," and "once a week." This information is used in later steps to generate the manual.

[1045] Step 5:

[1046] The server generates the operation manual

[1047] Based on the data obtained by the information extraction means, the server adds new work procedures to the work manual, specifically including a new procedure such as "Part A should be inspected once a week."

[1048] Step 6:

[1049] Server updates manual

[1050] Each time additional voice data is sent, the server updates the existing work manual, ensuring that the manual always reflects the most up-to-date information.

[1051] Step 7:

[1052] The server analyzes the user's emotions

[1053] When a user browses the work manual through the web interface, an emotion engine installed on the server analyzes the user's voice and facial expressions. If the user is confused, it provides feedback such as displaying additional explanations.

[1054] Step 8:

[1055] User checks the manual

[1056] A user (e.g., a new worker) checks the work manual from the web interface. The user opens the "Parts Inspection Manual" and sees the statement that "Part A should be inspected once a week."

[1057] Step 9:

[1058] The server translates the manual

[1059] If a foreign worker wishes to read the manual in Spanish, the server uses a translation means to convert the manual into Spanish, and the translated manual is displayed on the user's interface.

[1060] Step 10:

[1061] The server provides feedback based on emotional data

[1062] The emotion engine analyzes the user's reactions, and if it determines that the user's level of understanding is low, the server automatically inserts additional explanations, videos, and illustrations into the manual to support the user's learning.

[1063] In this way, the present invention not only digitizes the knowledge of experienced workers and automatically generates and updates work manuals, but also provides feedback according to the user's emotional state, thereby facilitating the transfer of knowledge to the next generation of employees and improving learning effectiveness.

[1064] Example 2

[1065] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1066] Efficiently transferring the knowledge and skills of veteran workers to the next generation of employees and improving on-site work efficiency and safety are important issues at many manufacturing and construction sites. Furthermore, providing appropriate feedback is necessary to improve user learning, but it is essential to consider the user's emotions when providing such feedback. However, with previous systems, it was difficult to analyze the user's emotions in real time and reflect them in the feedback.

[1067] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1068] In this invention, the server includes a means for acquiring the instruction voice of an experienced worker on-site, a means for analyzing the voice data acquired by the voice acquisition means and converting it into natural language, and a means for extracting instruction content and procedures from the text data converted by the voice analysis means. This new technology enables the digitization of the knowledge of experienced workers and smooth knowledge transfer to the next generation of employees. The server also includes an interface means for users to view the generated work manual and issue correction instructions, and an emotion engine that analyzes the user's emotions and provides feedback based on the emotion data, making it possible to provide flexible feedback based on the user's emotions.

[1069] "Audio acquisition means" refers to devices or technology that acquires in real time the instructional audio given by experienced workers on-site.

[1070] "Speech analysis means" refers to devices or technologies that analyze acquired voice data and convert it into natural language.

[1071] "Information extraction means" refers to a device or technology that extracts important information such as instruction content and procedures from the text data converted by the voice analysis means.

[1072] "Manual generation means" refers to a device or technology that automatically generates and updates a work manual based on the extracted information.

[1073] "Interface means" refers to an operation screen or system that allows a user to view the generated work manual and give instructions for correction.

[1074] An "emotion engine" refers to a device or technology that analyzes a user's emotions and provides feedback based on that emotional data.

[1075] "Translation means" refers to devices and technologies that translate work manuals into multiple languages.

[1076] A "headset" refers to a device worn by an experienced worker to capture audio.

[1077] This invention is a system that efficiently transfers the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and improves the effectiveness of learning by recognizing the user's emotions. The components of this system and their specific operation are described below.

[1078] System configuration

[1079] The system consists of the following main components:

[1080] 1. Audio acquisition method

[1081] The device uses a headset to capture the voice of an experienced worker in real time. For example, when an experienced worker wearing a headset provides instruction on-site and explains aloud, "This part needs to be inspected once a week," the device collects this voice and temporarily stores it locally.

[1082] 2. Audio analysis methods

[1083] The device sends the voice data to the server, which then uses voice recognition technology to convert the data into natural language text. Specific software used is a voice recognition API (e.g., Google Speech-to-Text). This process converts the voice data into text data, which is formatted as "This part needs to be inspected once a week."

[1084] 3. Information extraction means

[1085] The server uses natural language processing technology to extract important information from the text data, such as instruction content, work procedures, and points to note. Specifically, it uses a natural language processing library (e.g., NLTK) to extract keywords such as "parts inspection" and "once a week" from the text data.

[1086] 4. Manual Generation Methods

[1087] The server automatically generates and updates the work manual based on the extracted information. This ensures that the latest instruction information is always reflected. A manual page is generated that states, as a new procedure, that "Part A should be inspected once a week."

[1088] 5. Interface Methods

[1089] It provides an interface that allows users to view the generated work manual and provide correction instructions as necessary. The interface is web-based and designed for easy user access. For example, a new employee can open the "Parts Inspection Manual" and view the section that says "Part A should be inspected once a week."

[1090] 6. Emotion Engine

[1091] The server analyzes the user's emotions in real time as they view the manual and provides feedback. Specifically, it uses an emotion analysis API (e.g., Microsoft Azure Emotion API) that uses a webcam and microphone to analyze the user's facial expressions and tone of voice. For example, if the user's facial expression indicates a lack of understanding, the system will display additional explanations.

[1092] 7. Translation Methods

[1093] The server translates the generated work manual into multiple languages, making it available to foreign workers. The specific software used is a translation API (e.g., Google Translate API). For example, if a foreign worker wants to view the manual in Spanish, the server translates the manual into Spanish, and the user can view it from the same web interface.

[1094] Examples of concrete examples and prompts

[1095] For example, if an experienced worker instructs, "This part needs to be inspected once a week," the following sequence of actions will occur:

[1096] 1. The device captures audio through the headset and temporarily stores the audio data locally.

[1097] 2. The device sends the voice data to the server, which converts the voice data into text.

[1098] 3. The server extracts important information from the text data.

[1099] 4. The server adds the new information to the manual.

[1100] 5. The user checks the manual through the web interface and provides correction instructions.

[1101] 6. The server analyzes the user's sentiment and displays additional explanations if necessary.

[1102] 7. The server translates the manual so that users can view it in multiple languages.

[1103] Example prompt sentence:

[1104] "Please explain a system that captures in real time the voice of an experienced worker at a manufacturing site giving instructions on parts that require inspection once a week, converts that content into natural language, and reflects it in an operation manual. As a concrete example, please show instructions on part A."

[1105] In this way, the knowledge of veteran workers can be efficiently digitized and smoothly transferred to the next generation of employees. Also, by providing feedback based on the user's emotions, learning effectiveness is improved.

[1106] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1107] Step 1:

[1108] Audio Acquisition

[1109] The terminal uses a headset to pick up the instruction voice of an experienced worker in real time.

[1110] How it works: The device (headset) captures the voice of an experienced worker at the work site giving instructions such as, "This part needs to be inspected once a week." The headset captures the voice data in real time and temporarily stores it locally.

[1111] Input: Voice guidance from an experienced worker

[1112] Output: Temporarily saved audio data

[1113] Step 2:

[1114] Sending audio data

[1115] The terminal transmits the voice data to the server.

[1116] What it does: Your device sends locally stored audio data to a server over Wi-Fi or a cellular connection, securely using the SSL protocol.

[1117] Input: Temporarily saved audio data

[1118] Output: Audio data sent to the server

[1119] Step 3:

[1120] Audio analysis

[1121] The server analyzes the received audio data.

[1122] Specific operation: The server uses a speech recognition API (e.g., Google Speech-to-Text) to convert the voice data into natural language text. The converted text data is in the form of "This part needs to be inspected once a week."

[1123] Input: Audio data sent to the server

[1124] Output: Parsed text data

[1125] Step 4:

[1126] Extraction of important information

[1127] The server extracts important information from the text data obtained by speech analysis.

[1128] Specific operation: The server uses a natural language processing library (e.g., NLTK) to extract keywords such as "parts inspection" and "once a week" from the text data. The extracted information is then stored in a database.

[1129] Input: Parsed text data

[1130] Output: Extracted important information

[1131] Step 5:

[1132] Manual generation and updating

[1133] The server generates and updates the work manual based on the extracted information.

[1134] Specific operation: Based on the extracted information such as "part inspection" and "once a week," the server uses the manual generation means to add a new procedure to the "part inspection manual" page. The new procedure will state "Part A should be inspected once a week."

[1135] Input: Extracted sensitive information

[1136] Output: Updated operating manual

[1137] Step 6:

[1138] User sentiment analysis

[1139] The server analyzes the user's emotions.

[1140] Specific operation: When a user browses the manual through the web interface, the server uses an emotion analysis API (e.g., Microsoft Azure Emotion API) to analyze the user's facial expressions and tone of voice. If an emotion such as "low comprehension" is detected, the system displays additional explanations in the manual.

[1141] Input: User's facial expression data, voice tone

[1142] Output: Sentiment analysis results and adjusted manual content

[1143] Step 7:

[1144] Manual confirmation and correction instructions

[1145] Users can check the manual and provide correction instructions through a web interface.

[1146] Specific operation: The user can view the created work manual and, if there are any errors or additional explanations needed, provide correction instructions through the web interface. For example, feedback can be provided for the part that says "Part A should be inspected once a week."

[1147] Input: User views and feedback

[1148] Output: Correction instruction data

[1149] Step 8:

[1150] Manual Translation

[1151] The server translates the work manual into multiple languages.

[1152] Specific operation: The server uses a translation API (e.g., Google Translate API) to translate the generated work manual into multiple languages. If a foreign worker wants to view the manual in Spanish, the server translates the manual into Spanish, and the user can view it from the same web interface.

[1153] Input: Operation manual and selected language

[1154] Output: Translated work manual

[1155] In this way, by appropriately processing and calculating data at each processing step, it is possible to efficiently digitize the knowledge of experienced workers and provide feedback based on the user's emotions.

[1156] (Application example 2)

[1157] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1158] At manufacturing and construction sites, there is a need to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees. Important information, such as work procedures and precautions, is often communicated verbally on site, which can lead to inaccurate communication and reduced work efficiency and safety. Furthermore, at sites with multinational workers, language barriers can hinder work comprehension. Furthermore, there are concerns that feedback that takes into account emotions during learning may be ineffective.

[1159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1160] In this invention, the server includes a voice acquisition means, a voice analysis means, an information extraction means, a manual generation means, a translation means, and an emotion engine means, which makes it possible to acquire in real time the instruction voices given by experienced workers on-site, automatically generate the contents as an operation manual, translate the contents into multiple languages, and provide feedback based on the user's emotions.

[1161] The "voice acquisition means" refers to a device and software for acquiring in real time the instruction voice given by an experienced worker on-site.

[1162] The "voice analysis means" refers to a device and software that analyzes acquired voice data and converts it into natural language.

[1163] The "information extraction means" refers to a device and software that extracts instruction content and procedures from the text data converted by the voice analysis means.

[1164] The "manual generation means" refers to a device and software that automatically generates and updates a work manual based on the information extracted by the information extraction means.

[1165] The "interface means" refers to a device and software that allows a user to view the generated work manual and give correction instructions.

[1166] The "translation means" refers to a device and software that translates the generated operation manual into multiple languages.

[1167] The "emotion engine means" is a device and software that analyzes the user's emotions and adjusts the contents of the operation manual based on the results.

[1168] A "smartphone" is a type of portable information terminal, a communication device that can be used to capture, analyze, display, and interface with voice.

[1169] "Smart glasses" are glasses-type devices with a built-in display for displaying information.

[1170] A "head-mounted display" is a display device that a user wears on their head, and is a terminal for displaying work manuals and instruction information.

[1171] This invention provides a system for efficiently transferring the knowledge and skills of veteran workers to the next generation of employees at manufacturing and construction sites. This system acquires the voice of the veteran worker giving instructions on-site, analyzes the content, automatically generates a work manual, and further analyzes the user's emotions and provides feedback to improve learning effectiveness.

[1172] System configuration

[1173] The system consists of the following major components:

[1174] 1. Audio acquisition method

[1175] An experienced worker wears a headset and captures their voice in real time. For example, if a worker says, "This part needs to be inspected once a week," the headset captures this voice. An example of the hardware used is a typical smart headset.

[1176] 2. Audio analysis methods

[1177] Voice data is sent to a server, which then converts it into natural language using speech recognition technology (such as Google Cloud Speech API).

[1178] 3. Information extraction means

[1179] Using natural language processing technology (such as the NLTK library) on the server, instruction content and procedures are extracted from the text data. For example, important information such as "parts inspection" and "once a week" is extracted.

[1180] 4. Manual Generation Methods

[1181] Based on the extracted information, the server automatically generates and updates the work manual. This process uses software such as the Django framework. A new step in the work manual might be written as "Part A should be inspected once a week."

[1182] 5. Interface Methods

[1183] A web-based interface is provided that allows users (next-generation employees) to view the work manual and provide correction instructions as necessary. This interface uses React.js and other tools.

[1184] 6. Translation Methods

[1185] The server translates the generated work manual into multiple languages, making it available to foreign workers. The translation is performed using the Amazon Translate API. For example, if the manual is translated into Spanish, the user can view the manual in Spanish.

[1186] 7. Emotional Engine Means

[1187] As users browse the manual, their emotions are analyzed in real time using OpenCV and the Microsoft Azure Emotion API. For example, if the user is judged to have a "low level of understanding," more detailed explanations are added accordingly.

[1188] Examples of concrete examples and prompts

[1189] If the emotion engine determines that a worker who has looked at the contents of a work manual has a "low level of understanding," phrases that explain the contents in more detail are added.

[1190] Example prompt sentence:

[1191] "Please explain the contents of this manual in more detail. In particular, the following information is missing: Specific inspection procedures for parts."

[1192] Example result:

[1193] "Detailed text: It is important to inspect part A once a week. Specifically, 1. Remove cover, 2. Visually inspect, 3. Clean if necessary, 4. Reinstall part. Follow this procedure."

[1194] This invention not only digitizes the tacit knowledge of veteran workers and keeps them up-to-date as work manuals, but also provides feedback based on the user's emotions, which allows for the smooth transfer of know-how to the next generation of employees, improving work efficiency and safety on-site, and enhancing learning effectiveness.

[1195] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1196] Step 1:

[1197] Audio Acquisition

[1198] An experienced worker wearing a device (headset) receives real-time instruction voice on-site. The input voice might say, for example, "This part needs to be inspected once a week." The headset captures this voice data and temporarily stores it. At this time, the voice data is saved in local storage.

[1199] Step 2:

[1200] Sending audio data

[1201] The device sends the temporarily saved audio data to the server. The input is the audio data captured by the headset. A secure communication method such as HTTPS is used as the transmission protocol. The server stores the received audio data.

[1202] Step 3:

[1203] Audio analysis

[1204] The server receives the voice data and converts it into natural language text using a voice analysis tool. This process uses the Google Cloud Speech API. The input is the transmitted voice data, and the output is text data. For example, if the voice data is "This part needs to be inspected once a week," the output text data will be "This part needs to be inspected once a week."

[1205] Step 4:

[1206] Information Extraction

[1207] The server uses information extraction methods to extract instructional content and important procedures from the generated text data. This process uses natural language processing technology (such as the NLTK library). The input is the analyzed text data, and the output is the extracted information. For example, "parts inspection" and "once a week" are extracted.

[1208] Step 5:

[1209] Manual Generation

[1210] The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This process uses the Django framework, etc. The input is the extracted information, and the output is the generated work manual. For example, a new procedure in the manual might state, "Part A should be inspected once a week."

[1211] Step 6:

[1212] Interface provided

[1213] It provides a web-based interface for users to view the manual and provide correction instructions. This interface uses React.js etc. The input is the generated work manual, and the output is a web page that users can view.

[1214] Step 7:

[1215] Translation tools

[1216] The server translates the generated work manual into multiple languages. This process uses the Amazon Translate API. The input is the generated work manual, and the output is the translated manual. For example, the manual can be translated into Spanish, allowing users to view the manual in Spanish.

[1217] Step 8:

[1218] Emotion analysis

[1219] The system analyzes emotions in real time as users read manuals. This process uses OpenCV and the Microsoft Azure Emotion API. The input is the user's facial expressions and tone of voice, and the output is the analyzed emotional data. For example, if the user is determined to have a "low level of understanding," the system generates a prompt to add more detailed explanations based on the result.

[1220] Step 9:

[1221] Applying Feedback

[1222] The content of the work manual is adjusted based on the results of the analysis by the emotion engine means. The input is emotion data, and the output is a revised work manual. For example, if the level of understanding is determined to be "low," a prompt sentence is added, detailing the specific inspection procedures for the manual.

[1223] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1224] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1225] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1226] [Fourth embodiment]

[1227] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1228] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1229] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1230] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1231] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1232] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1233] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1234] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1235] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1236] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1237] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1238] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1239] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1240] This invention is a system for efficiently transferring the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and is configured as follows.

[1241] System configuration

[1242] The system consists of the following main components:

[1243] 1. Voice acquisition means: The terminal acquires the voice of the experienced worker giving instructions. Specifically, the experienced worker wearing a headset captures the voice data of the real-time work instruction and explanations given on-site.

[1244] 2. Voice analysis means: The device transmits the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology.

[1245] 3. Information extraction means: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the voice analysis means. This is done using natural language processing technology.

[1246] 4. Manual generation means: The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This ensures that the latest instruction information is always reflected.

[1247] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[1248] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API or similar.

[1249] Specific examples of program processing

[1250] Operation of the audio acquisition means

[1251] The device (headset) picks up the voice of an experienced worker instructing in real time. For example, an experienced worker may say, "This part needs to be inspected once a week." The device collects this voice and temporarily stores it locally.

[1252] Sending and analyzing voice data

[1253] The device sends voice data to the server, which receives the data and uses a voice analysis tool to convert it into natural language text, such as "This part needs to be inspected once a week."

[1254] Information Extraction and Manual Generation

[1255] The server uses the information extraction means to extract information such as "part inspection," "once a week," and "necessary" from the text data. Based on the extracted information, the server adds a new procedure to the "part inspection manual" page using the manual generation means. The new procedure states, "Inspect part A once a week."

[1256] Manual review and translation

[1257] A user checks a manual through a web interface. For example, a new employee opens the "Parts Inspection Manual" and views the section that says "Part A should be inspected once a week." If a foreign worker wants to view the manual in Spanish, the server uses a translation tool to translate the manual into Spanish. After translation, the user can view the manual in Spanish through the same web interface.

[1258] In this way, the present invention digitally and automatically visualizes the tacit knowledge of experienced workers, enabling them to be constantly updated as the latest work manuals. Furthermore, by translating the knowledge into multiple languages, the system can be used in international work sites. This allows for the smooth transfer of know-how to the next generation of employees, improving work efficiency and safety on-site.

[1259] The processing flow will be explained below.

[1260] Step 1:

[1261] The device receives audio

[1262] An experienced worker wears a headset and provides on-site work instructions. The device captures the worker's voice in real time through a microphone and temporarily stores it locally as audio data.

[1263] Step 2:

[1264] The device sends the voice data to the server

[1265] At regular intervals, the device encrypts the collected voice data and transmits it to the server through a secure channel.

[1266] Step 3:

[1267] The server analyzes the voice data

[1268] The server decompresses the received voice data and passes it to the voice analysis module, which uses natural language processing technology to convert the voice data into text format.

[1269] Step 4:

[1270] The server extracts the information

[1271] The server analyzes the text data and extracts specific keywords (e.g., work procedures, time, location) that correspond to the business context, using natural language processing technology.

[1272] Step 5:

[1273] The server generates the operation manual

[1274] Based on the information extracted by the server, the first edition of the work manual is automatically generated, including new work procedures and precautions. This manual summarizes specific procedures in an easy-to-understand manner using bullet points and diagrams.

[1275] Step 6:

[1276] Server updates manual

[1277] The server analyzes the new voice data, adds or modifies relevant information to the existing work manual, and updates the manual to reflect the latest information.

[1278] Step 7:

[1279] User checks the manual

[1280] Next-generation employees and foreign workers can access and review the generated work manuals through a web interface, and can easily find specific work procedures and precautions using the search function.

[1281] Step 8:

[1282] The server translates the manual

[1283] If the user requests the work manual in a different language, the server invokes the automatic translation module to convert the manual into the specified language, and the translated manual is displayed according to the user's language setting.

[1284] Example 1

[1285] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1286] On-site work instruction often relies on the experience and knowledge of veteran workers, and if this knowledge is not passed on properly, it hinders the improvement of the skills and knowledge of the next generation of workers. Furthermore, if the content of instruction is not standardized, there is a risk of work efficiency and safety declining. Furthermore, language barriers exist in international work sites, requiring multilingual support. To resolve these issues, an efficient and standardized knowledge transfer system is needed.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1288] In this invention, the server includes a voice analysis means, an information extraction means, and a manual generation means, which automatically converts the knowledge and skills of experienced workers from voice to text, extracts important instructional content, and generates and updates a work manual based on the extracted information.

[1289] "Audio acquisition means" refers to equipment and software for collecting the instructional audio given by experienced workers on-site.

[1290] "Speech analysis means" refers to technology for analyzing captured speech data and converting it into natural language text.

[1291] "Information extraction means" refers to a technology for extracting important information such as instruction content and procedures from the text data converted by the voice analysis means.

[1292] "Manual generation means" refers to a system that automatically generates and updates a work manual based on the extracted information.

[1293] The "interface means" refers to an interface that allows a user to view the generated work manual and give correction instructions.

[1294] "Translation means" refers to the technology or system for translating the generated work manual into multiple languages.

[1295] This invention is a system for efficiently transferring the knowledge and skills of veteran workers to the next generation of employees at manufacturing and construction sites. The system consists of the following main components:

[1296] 1. Voice acquisition method: A method in which the terminal acquires the voice of an experienced worker giving instructions. Specifically, an experienced worker wearing a headset captures real-time work instruction and explanations on-site as voice data.

[1297] 2. Voice analysis: The device sends the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[1298] 3. Information extraction: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the speech analysis method. This is done using natural language processing technology (e.g., spaCy, NLTK).

[1299] 4. Manual generation means: The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This ensures that the latest instruction information is always reflected.

[1300] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[1301] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API (e.g., Google Translate API).

[1302] Specific examples of program processing

[1303] The device receives the voice instructions of the experienced worker through the headset and temporarily stores the voice data locally. The device then sends the voice data to a server. The server receives the voice data and converts it into natural language text using a voice analysis tool. For example, if an experienced worker instructs, "This part needs to be inspected once a week," the voice instructions are converted into text.

[1304] Next, the server uses information extraction means to extract important information from the text data, such as "part inspection," "once a week," and "necessary." Based on this information, the server automatically updates the work manual using manual generation means and adds new procedures. For example, a new procedure might read, "Inspect part A once a week."

[1305] Users can check the updated work manual through a web interface and make corrections or add comments as necessary. The server also uses a translation means to translate the generated manual into multiple languages. For example, if a foreign worker wants to view the manual in Spanish, the server translates it and the user can view the manual in Spanish from the same web interface.

[1306] Prompt Sentence Examples

[1307] ---

[1308] An experienced worker instructed, "This part needs to be inspected once a week." Add a new procedure to the work manual: "Inspect part A once a week."

[1309] ---

[1310] This system digitizes the tacit knowledge of veteran workers and makes it possible to efficiently pass it on to the next generation of employees. Multilingual support allows for smooth use even in international work sites. This improves work efficiency and ensures safety.

[1311] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1312] Step 1: Audio capture

[1313] Subject: Device

[1314] Specific operation: The terminal (experienced worker wearing a headset) receives real-time instruction audio on-site.

[1315] Input: Operator's voice instruction

[1316] Data processing: Using a microphone to capture audio signals and store them as digital audio data.

[1317] Output: Digital audio data

[1318] Step 2: Sending audio data

[1319] Subject: Device

[1320] Specific operation: The device temporarily stores the voice data locally and then transmits it to the server over the network.

[1321] Input: Digital audio data

[1322] Data processing: Converts voice data into a network format and transmits it

[1323] Output: Audio data sent to the server

[1324] Step 3: Audio analysis

[1325] Subject: Server

[1326] What it does: The server converts the received voice data into natural language text using speech analysis techniques, such as a speech recognition API (e.g., Google Cloud Speech-to-Text).

[1327] Input: Audio data sent to the server

[1328] Data processing: Analyzes voice data and converts it into natural language text

[1329] Output: Parsed text data

[1330] Step 4: Information Extraction

[1331] Subject: Server

[1332] Specific operation: The server processes the text data obtained from the speech analysis means using information extraction means to extract important instructional content and work procedures. Natural language processing technology (e.g., spaCy, NLTK) is used.

[1333] Input: Parsed text data

[1334] Data processing: Extracting important information from text

[1335] Output: Extracted information

[1336] Step 5: Manual generation

[1337] Subject: Server

[1338] Specific operation: The server automatically generates and updates the work manual based on the data obtained by the information extraction means.

[1339] Input: Extracted information

[1340] Data processing: Manual creation and updating

[1341] Output: Updated operating manual

[1342] Step 6: Manual review and correction

[1343] Subject: User

[1344] Specific operation: The user uses a web interface to view the generated work manual and add correction instructions or comments as necessary.

[1345] Input: Updated Operation Manual

[1346] Data processing: Adding corrections and comments

[1347] Output: Revised operating manual

[1348] Step 7: Manual Translation

[1349] Subject: Server

[1350] Specific operation: The server uses a translation tool to translate the work manual into multiple languages, using a translation API (e.g., Google Translate API).

[1351] Input: Corrected operation manual

[1352] Data processing: Manual translation

[1353] Output: Translated work manual

[1354] In this way, the system can digitize the knowledge of veteran workers and efficiently pass it on to the next generation. By detailing the processing content for each specific step, the overall picture of the operation can be clearly understood.

[1355] (Application example 1)

[1356] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1357] It is important to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees in on-site work. However, previous methods lacked consistency and accuracy in training content, and were not multilingual, which resulted in impacts on work efficiency and safety. Furthermore, manual recording and transmission methods made it difficult to automate and reference in real time.

[1358] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1359] In this invention, the server includes a voice acquisition means for acquiring voice guidance provided by an experienced worker on-site, a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting it into natural language, an information extraction means for extracting guidance content and procedures from the text data converted by the voice analysis means, a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means, an interface means for a user to view the generated work manual and give correction instructions, a translation means for translating the work manual into other languages, and a machine operation means for a machine to automatically refer to the work manual and perform work. This makes it possible to efficiently transfer the knowledge and skills of experienced workers, achieve consistency and accuracy in work, and support multiple languages, while enabling work automation and real-time reference.

[1360] "Veteran workers" are workers with long-term experience and deep knowledge in the field.

[1361] "Site" refers to the location where actual work, such as manufacturing or construction, takes place.

[1362] "Instruction voice" is a voice for instruction or explanation given by an experienced worker.

[1363] The "voice acquisition means" refers to a device or system that collects the instruction voice of an experienced worker.

[1364] "Audio data" refers to information in which instructional audio is recorded in digital format.

[1365] "Speech analysis means" refers to a device or software that analyzes acquired speech data and converts it into natural language text.

[1366] "Text data" is character string information converted by speech analysis.

[1367] "Information extraction means" refers to devices or software that extract important instructional content and procedures from text data.

[1368] "Instruction content" refers to specific instructions and explanations such as work procedures and precautions.

[1369] A "procedure" refers to the steps or method for performing a task.

[1370] The "manual generation means" refers to a device or software that automatically creates and updates a work manual based on the extracted information.

[1371] A "work manual" is a document that describes specific work procedures and instructions to be followed on-site.

[1372] "User" refers to the workers and administrators who use this system.

[1373] "Interface means" refers to devices and software that allow a user to view and correct the operation manual.

[1374] "Translation means" refers to devices or software that translates work manuals into multiple languages.

[1375] "Other languages" are languages ​​other than one's native language.

[1376] "Machine operating means" refers to devices and software that allow a machine to automatically perform work based on the created work manual.

[1377] "Machine" refers to an automated device that can perform simple tasks or complex processes.

[1378] The following describes an embodiment of the present invention. This system is designed to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees, and is composed of the following main components:

[1379] System Program

[1380] The audio capture means is used to capture the instruction voice given by experienced workers on-site. For this purpose, audio recording devices, smartphones, and smart headsets are used. The audio of the experienced worker giving instructions such as "This part needs to be inspected once a week" is recorded and saved locally.

[1381] The speech analysis means analyzes the acquired speech data and converts it into natural language text. This process uses the Google Speech Recognition API and Python libraries. The acquired speech is converted into text data, such as "This part needs to be inspected once a week."

[1382] The information extraction means extracts instruction content and procedures from the text data converted by the voice analysis means. This process uses natural language processing technology to extract key keywords (e.g., inspection, once a week, necessary, etc.).

[1383] The manual generation means automatically generates and updates the work manual based on the extracted information. This allows the work manual to always reflect the latest instruction information and be saved in JSON format.

[1384] The interface means allows the user to view the generated work manual and give correction instructions. This interface is web-based and designed to be easily accessible by the user.

[1385] The translation means translates the work manual into other languages ​​(e.g., Spanish). This process uses the Google Translate API. By translating the generated work manual, it can be used by foreign workers.

[1386] The machine operation means allows the robot to automatically refer to the created work manual and perform the work, which allows the robot to act according to the procedure, improving the accuracy and safety of on-site work.

[1387] Specific examples

[1388] For example, in a factory, a robot can capture the voice instructions of an experienced worker in real time and record the weekly part inspection procedure as text data. This data is automatically analyzed and generated as a work manual. The generated manual is translated into Spanish, making it suitable for foreign workers as well. The robot can refer to this manual and perform part inspection work with high precision.

[1389] Prompt Sentence Examples

[1390] "Take audio containing part inspection procedures and convert it to text."

[1391] Extract important keywords from the captured text.

[1392] "Generate a parts inspection manual based on the extracted information."

[1393] "Please translate the generated manual into Spanish."

[1394] In this way, the system of the present invention efficiently transfers the knowledge and skills of experienced workers, achieves consistency and accuracy in work, and supports multiple languages, while enabling automation of work and real-time reference.

[1395] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1396] Step 1:

[1397] Voice acquisition: The terminal acquires the voice of an experienced worker giving on-site instruction. This is done using a voice recorder or smart headset. The input is the voice of the experienced worker, and the voice data is saved locally as output. This voice data is used for subsequent analysis.

[1398] Step 2:

[1399] Speech analysis: The device sends the captured voice data to the server. The server uses the Google Speech Recognition API to convert the voice data into natural language text data. The input is voice data, and the output is text data, which is used for subsequent information extraction.

[1400] Step 3:

[1401] Information extraction: The server uses natural language processing technology to extract instruction content and procedures from the text data converted by the speech analysis means. The input is text data, and keywords for the instruction content and procedures are generated as output. Specifically, text analysis is performed, and keywords such as "inspection," "once a week," and "necessary" are extracted.

[1402] Step 4:

[1403] Manual generation: The server automatically generates and updates a work manual based on the information extracted by the information extraction means. The input is the extracted keywords, and the output is a structured work manual. This manual is saved in JSON format and used for subsequent translation and reference.

[1404] Step 5:

[1405] Translation: The server translates the generated work manual into another language. For example, it uses the Google Translation API to translate a Japanese manual into Spanish. The input is the Japanese work manual, and the output is a translated work manual. This makes it possible to accommodate foreign workers.

[1406] Step 6:

[1407] Manual Reference: Users can view and correct work manuals through a web-based interface. The input is the user's viewing instructions and correction instructions, and the output is the displayed or updated manual. This interface is designed to allow easy access for users.

[1408] Step 7:

[1409] Machine operation: The terminal operates the machine based on the operation manual obtained from the server. The input is the operation manual, and the output is the machine performing the operation. A specific operation would be, for example, a robot inspecting parts according to a procedure. This improves work accuracy and safety.

[1410] Based on these processing steps, the knowledge and skills of experienced workers can be efficiently transferred, ensuring consistency and accuracy of work, and multilingual support, while enabling automation of work and real-time reference.

[1411] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1412] This invention is a system that efficiently transfers the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and improves the effectiveness of learning by recognizing the user's emotions. It consists of the following components:

[1413] System configuration

[1414] The system consists of the following main components:

[1415] 1. Voice acquisition means: The terminal acquires the voice of an experienced worker in real time. Specifically, the experienced worker wearing a headset captures the voice data of real-time work instruction and explanations given on-site.

[1416] 2. Voice analysis means: The device transmits the acquired voice data to the server, which then converts the voice data into natural language using voice recognition technology.

[1417] 3. Information extraction: The server extracts important information such as instruction content, work procedures, and important points from the text data generated by the voice analysis method. This is done using natural language processing technology.

[1418] 4. Manual generation method: The server automatically generates and updates the work manual based on the extracted information, ensuring that the latest instruction information is always reflected.

[1419] 5. Interface: Provides an interface for users to view the generated work manual and provide correction instructions as necessary. This interface is web-based and designed to be easily accessible by users.

[1420] 6. Translation method: The server translates the generated work manual into multiple languages ​​so that it can be used by foreign workers. This is done using a translation API, etc.

[1421] 7. Emotion Engine: Recognizes the user's emotions and provides manual feedback based on the emotional data. In conjunction with voice analysis, it can analyze the user's emotions in real time.

[1422] Specific examples of program processing

[1423] Operation of the audio acquisition means

[1424] The terminal (headset) picks up the voice of the experienced worker instructing in real time. When the experienced worker instructs, "This part needs to be inspected once a week," the terminal collects this voice and temporarily stores it locally.

[1425] Sending and analyzing voice data

[1426] The device sends voice data to the server, which receives the data and converts it into natural language text using a speech analysis tool. The textual content will be in the form of "This part needs to be inspected once a week."

[1427] Information Extraction and Manual Generation

[1428] The server uses the information extraction means to extract information such as "parts inspection" and "once a week" from the text data. Based on the extracted information, the server uses the manual generation means to add a new procedure to the "parts inspection manual" page. The new procedure will state, "Inspect part A once a week."

[1429] Emotion Engine Operation

[1430] The server uses an emotion engine to analyze the user's emotions in real time as they view the manual. It analyzes the user's facial expressions and tone of voice to detect emotions such as "low level of understanding" or "dissatisfaction." Depending on the results, it adjusts the content of the work manual or displays additional explanations.

[1431] Manual review and translation

[1432] A user checks a manual through a web interface. For example, a new employee opens the "Parts Inspection Manual" and sees the section that says "Part A should be inspected once a week." If a foreign worker wants to view the manual in Spanish, the server uses a translation tool to translate the manual into Spanish. After translation, the user can view the manual in Spanish through the same web interface.

[1433] In this way, the present invention not only digitizes the tacit knowledge of veteran workers and keeps them updated as the latest work manual, but also provides feedback based on the user's emotions. This allows for the smooth transfer of know-how to the next generation of employees, improving not only work efficiency and safety on-site, but also the effectiveness of learning.

[1434] The processing flow will be explained below.

[1435] Step 1:

[1436] The device receives audio

[1437] An experienced worker wears a headset and instructs a new worker at the work site, saying, "This part needs to be inspected once a week." The device (headset) picks up this voice in real time and temporarily stores it locally as voice data.

[1438] Step 2:

[1439] The device sends the voice data to the server

[1440] The device periodically encrypts and transmits the audio data over a secure channel to the server, at intervals optimized to accommodate battery and network usage.

[1441] Step 3:

[1442] The server analyzes the voice data

[1443] The server decompresses the received voice data and passes it to the voice analysis module, which converts the data into text format, resulting in the text "This part needs to be inspected once a week."

[1444] Step 4:

[1445] The server extracts the information

[1446] The server analyzes the text data and uses natural language processing technology to extract important information, including keywords such as "parts," "inspection," and "once a week." This information is used in later steps to generate the manual.

[1447] Step 5:

[1448] The server generates the operation manual

[1449] Based on the data obtained by the information extraction means, the server adds new work procedures to the work manual, specifically including a new procedure such as "Part A should be inspected once a week."

[1450] Step 6:

[1451] Server updates manual

[1452] Each time additional voice data is sent, the server updates the existing work manual, ensuring that the manual always reflects the most up-to-date information.

[1453] Step 7:

[1454] The server analyzes the user's emotions

[1455] When a user browses the work manual through the web interface, an emotion engine installed on the server analyzes the user's voice and facial expressions. If the user is confused, it provides feedback such as displaying additional explanations.

[1456] Step 8:

[1457] User checks the manual

[1458] A user (e.g., a new worker) checks the work manual from the web interface. The user opens the "Parts Inspection Manual" and sees the statement that "Part A should be inspected once a week."

[1459] Step 9:

[1460] The server translates the manual

[1461] If a foreign worker wishes to read the manual in Spanish, the server uses a translation means to convert the manual into Spanish, and the translated manual is displayed on the user's interface.

[1462] Step 10:

[1463] The server provides feedback based on emotional data

[1464] The emotion engine analyzes the user's reactions, and if it determines that the user's level of understanding is low, the server automatically inserts additional explanations, videos, and illustrations into the manual to support the user's learning.

[1465] In this way, the present invention not only digitizes the knowledge of experienced workers and automatically generates and updates work manuals, but also provides feedback according to the user's emotional state, thereby facilitating the transfer of knowledge to the next generation of employees and improving learning effectiveness.

[1466] Example 2

[1467] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1468] Efficiently transferring the knowledge and skills of veteran workers to the next generation of employees and improving on-site work efficiency and safety are important issues at many manufacturing and construction sites. Furthermore, providing appropriate feedback is necessary to improve user learning, but it is essential to consider the user's emotions when providing such feedback. However, with previous systems, it was difficult to analyze the user's emotions in real time and reflect them in the feedback.

[1469] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1470] In this invention, the server includes a means for acquiring the instruction voice of an experienced worker on-site, a means for analyzing the voice data acquired by the voice acquisition means and converting it into natural language, and a means for extracting instruction content and procedures from the text data converted by the voice analysis means. This new technology enables the digitization of the knowledge of experienced workers and smooth knowledge transfer to the next generation of employees. The server also includes an interface means for users to view the generated work manual and issue correction instructions, and an emotion engine that analyzes the user's emotions and provides feedback based on the emotion data, making it possible to provide flexible feedback based on the user's emotions.

[1471] "Audio acquisition means" refers to devices or technology that acquires in real time the instructional audio given by experienced workers on-site.

[1472] "Speech analysis means" refers to devices or technologies that analyze acquired voice data and convert it into natural language.

[1473] "Information extraction means" refers to a device or technology that extracts important information such as instruction content and procedures from the text data converted by the voice analysis means.

[1474] "Manual generation means" refers to a device or technology that automatically generates and updates a work manual based on the extracted information.

[1475] "Interface means" refers to an operation screen or system that allows a user to view the generated work manual and give instructions for correction.

[1476] An "emotion engine" refers to a device or technology that analyzes a user's emotions and provides feedback based on that emotional data.

[1477] "Translation means" refers to devices and technologies that translate work manuals into multiple languages.

[1478] A "headset" refers to a device worn by an experienced worker to capture audio.

[1479] This invention is a system that efficiently transfers the knowledge and skills of veteran workers at manufacturing and construction sites to the next generation of employees, and improves the effectiveness of learning by recognizing the user's emotions. The components of this system and their specific operation are described below.

[1480] System configuration

[1481] The system consists of the following main components:

[1482] 1. Audio acquisition method

[1483] The device uses a headset to capture the voice of an experienced worker in real time. For example, when an experienced worker wearing a headset provides instruction on-site and explains aloud, "This part needs to be inspected once a week," the device collects this voice and temporarily stores it locally.

[1484] 2. Audio analysis methods

[1485] The device sends the voice data to the server, which then uses voice recognition technology to convert the data into natural language text. Specific software used is a voice recognition API (e.g., Google Speech-to-Text). This process converts the voice data into text data, which is formatted as "This part needs to be inspected once a week."

[1486] 3. Information extraction means

[1487] The server uses natural language processing technology to extract important information from the text data, such as instruction content, work procedures, and points to note. Specifically, it uses a natural language processing library (e.g., NLTK) to extract keywords such as "parts inspection" and "once a week" from the text data.

[1488] 4. Manual Generation Methods

[1489] The server automatically generates and updates the work manual based on the extracted information. This ensures that the latest instruction information is always reflected. A manual page is generated that states, as a new procedure, that "Part A should be inspected once a week."

[1490] 5. Interface Methods

[1491] It provides an interface that allows users to view the generated work manual and provide correction instructions as necessary. The interface is web-based and designed for easy user access. For example, a new employee can open the "Parts Inspection Manual" and view the section that says "Part A should be inspected once a week."

[1492] 6. Emotion Engine

[1493] The server analyzes the user's emotions in real time as they view the manual and provides feedback. Specifically, it uses an emotion analysis API (e.g., Microsoft Azure Emotion API) that uses a webcam and microphone to analyze the user's facial expressions and tone of voice. For example, if the user's facial expression indicates a lack of understanding, the system will display additional explanations.

[1494] 7. Translation Methods

[1495] The server translates the generated work manual into multiple languages, making it available to foreign workers. The specific software used is a translation API (e.g., Google Translate API). For example, if a foreign worker wants to view the manual in Spanish, the server translates the manual into Spanish, and the user can view it from the same web interface.

[1496] Examples of concrete examples and prompts

[1497] For example, if an experienced worker instructs, "This part needs to be inspected once a week," the following sequence of actions will occur:

[1498] 1. The device captures audio through the headset and temporarily stores the audio data locally.

[1499] 2. The device sends the voice data to the server, which converts the voice data into text.

[1500] 3. The server extracts important information from the text data.

[1501] 4. The server adds the new information to the manual.

[1502] 5. The user checks the manual through the web interface and provides correction instructions.

[1503] 6. The server analyzes the user's sentiment and displays additional explanations if necessary.

[1504] 7. The server translates the manual so that users can view it in multiple languages.

[1505] Example prompt sentence:

[1506] "Please explain a system that captures in real time the voice of an experienced worker at a manufacturing site giving instructions on parts that require inspection once a week, converts that content into natural language, and reflects it in an operation manual. As a concrete example, please show instructions on part A."

[1507] In this way, the knowledge of veteran workers can be efficiently digitized and smoothly transferred to the next generation of employees. Also, by providing feedback based on the user's emotions, learning effectiveness is improved.

[1508] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1509] Step 1:

[1510] Audio Acquisition

[1511] The terminal uses a headset to pick up the instruction voice of an experienced worker in real time.

[1512] How it works: The device (headset) captures the voice of an experienced worker at the work site giving instructions such as, "This part needs to be inspected once a week." The headset captures the voice data in real time and temporarily stores it locally.

[1513] Input: Voice guidance from an experienced worker

[1514] Output: Temporarily saved audio data

[1515] Step 2:

[1516] Sending audio data

[1517] The terminal transmits the voice data to the server.

[1518] What it does: Your device sends locally stored audio data to a server over Wi-Fi or a cellular connection, securely using the SSL protocol.

[1519] Input: Temporarily saved audio data

[1520] Output: Audio data sent to the server

[1521] Step 3:

[1522] Audio analysis

[1523] The server analyzes the received audio data.

[1524] Specific operation: The server uses a speech recognition API (e.g., Google Speech-to-Text) to convert the voice data into natural language text. The converted text data is in the form of "This part needs to be inspected once a week."

[1525] Input: Audio data sent to the server

[1526] Output: Parsed text data

[1527] Step 4:

[1528] Extraction of important information

[1529] The server extracts important information from the text data obtained by speech analysis.

[1530] Specific operation: The server uses a natural language processing library (e.g., NLTK) to extract keywords such as "parts inspection" and "once a week" from the text data. The extracted information is then stored in a database.

[1531] Input: Parsed text data

[1532] Output: Extracted important information

[1533] Step 5:

[1534] Manual generation and updating

[1535] The server generates and updates the work manual based on the extracted information.

[1536] Specific operation: Based on the extracted information such as "part inspection" and "once a week," the server uses the manual generation means to add a new procedure to the "part inspection manual" page. The new procedure will state "Part A should be inspected once a week."

[1537] Input: Extracted sensitive information

[1538] Output: Updated operating manual

[1539] Step 6:

[1540] User sentiment analysis

[1541] The server analyzes the user's emotions.

[1542] Specific operation: When a user browses the manual through the web interface, the server uses an emotion analysis API (e.g., Microsoft Azure Emotion API) to analyze the user's facial expressions and tone of voice. If an emotion such as "low comprehension" is detected, the system displays additional explanations in the manual.

[1543] Input: User's facial expression data, voice tone

[1544] Output: Sentiment analysis results and adjusted manual content

[1545] Step 7:

[1546] Manual confirmation and correction instructions

[1547] Users can check the manual and provide correction instructions through a web interface.

[1548] Specific operation: The user can view the created work manual and, if there are any errors or additional explanations needed, provide correction instructions through the web interface. For example, feedback can be provided for the part that says "Part A should be inspected once a week."

[1549] Input: User views and feedback

[1550] Output: Correction instruction data

[1551] Step 8:

[1552] Manual Translation

[1553] The server translates the work manual into multiple languages.

[1554] Specific operation: The server uses a translation API (e.g., Google Translate API) to translate the generated work manual into multiple languages. If a foreign worker wants to view the manual in Spanish, the server translates the manual into Spanish, and the user can view it from the same web interface.

[1555] Input: Operation manual and selected language

[1556] Output: Translated work manual

[1557] In this way, by appropriately processing and calculating data at each processing step, it is possible to efficiently digitize the knowledge of experienced workers and provide feedback based on the user's emotions.

[1558] (Application example 2)

[1559] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1560] At manufacturing and construction sites, there is a need to efficiently transfer the knowledge and skills of veteran workers to the next generation of employees. Important information, such as work procedures and precautions, is often communicated verbally on site, which can lead to inaccurate communication and reduced work efficiency and safety. Furthermore, at sites with multinational workers, language barriers can hinder work comprehension. Furthermore, there are concerns that feedback that takes into account emotions during learning may be ineffective.

[1561] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1562] In this invention, the server includes a voice acquisition means, a voice analysis means, an information extraction means, a manual generation means, a translation means, and an emotion engine means, which makes it possible to acquire in real time the instruction voices given by experienced workers on-site, automatically generate the contents as an operation manual, translate the contents into multiple languages, and provide feedback based on the user's emotions.

[1563] The "voice acquisition means" refers to a device and software for acquiring in real time the instruction voice given by an experienced worker on-site.

[1564] The "voice analysis means" refers to a device and software that analyzes acquired voice data and converts it into natural language.

[1565] The "information extraction means" refers to a device and software that extracts instruction content and procedures from the text data converted by the voice analysis means.

[1566] The "manual generation means" refers to a device and software that automatically generates and updates a work manual based on the information extracted by the information extraction means.

[1567] The "interface means" refers to a device and software that allows a user to view the generated work manual and give correction instructions.

[1568] The "translation means" refers to a device and software that translates the generated operation manual into multiple languages.

[1569] The "emotion engine means" is a device and software that analyzes the user's emotions and adjusts the contents of the operation manual based on the results.

[1570] A "smartphone" is a type of portable information terminal, a communication device that can be used to capture, analyze, display, and interface with voice.

[1571] "Smart glasses" are glasses-type devices with a built-in display for displaying information.

[1572] A "head-mounted display" is a display device that a user wears on their head, and is a terminal for displaying work manuals and instruction information.

[1573] This invention provides a system for efficiently transferring the knowledge and skills of veteran workers to the next generation of employees at manufacturing and construction sites. This system acquires the voice of the veteran worker giving instructions on-site, analyzes the content, automatically generates a work manual, and further analyzes the user's emotions and provides feedback to improve learning effectiveness.

[1574] System configuration

[1575] The system consists of the following major components:

[1576] 1. Audio acquisition method

[1577] An experienced worker wears a headset and captures their voice in real time. For example, if a worker says, "This part needs to be inspected once a week," the headset captures this voice. An example of the hardware used is a typical smart headset.

[1578] 2. Audio analysis methods

[1579] Voice data is sent to a server, which then converts it into natural language using speech recognition technology (such as Google Cloud Speech API).

[1580] 3. Information extraction means

[1581] Using natural language processing technology (such as the NLTK library) on the server, instruction content and procedures are extracted from the text data. For example, important information such as "parts inspection" and "once a week" is extracted.

[1582] 4. Manual Generation Methods

[1583] Based on the extracted information, the server automatically generates and updates the work manual. This process uses software such as the Django framework. A new step in the work manual might be written as "Part A should be inspected once a week."

[1584] 5. Interface Methods

[1585] A web-based interface is provided that allows users (next-generation employees) to view the work manual and provide correction instructions as necessary. This interface uses React.js and other tools.

[1586] 6. Translation Methods

[1587] The server translates the generated work manual into multiple languages, making it available to foreign workers. The translation is performed using the Amazon Translate API. For example, if the manual is translated into Spanish, the user can view the manual in Spanish.

[1588] 7. Emotional Engine Means

[1589] As users browse the manual, their emotions are analyzed in real time using OpenCV and the Microsoft Azure Emotion API. For example, if the user is judged to have a "low level of understanding," more detailed explanations are added accordingly.

[1590] Examples of concrete examples and prompts

[1591] If the emotion engine determines that a worker who has looked at the contents of a work manual has a "low level of understanding," phrases that explain the contents in more detail are added.

[1592] Example prompt sentence:

[1593] "Please explain the contents of this manual in more detail. In particular, the following information is missing: Specific inspection procedures for parts."

[1594] Example result:

[1595] "Detailed text: It is important to inspect part A once a week. Specifically, 1. Remove cover, 2. Visually inspect, 3. Clean if necessary, 4. Reinstall part. Follow this procedure."

[1596] This invention not only digitizes the tacit knowledge of veteran workers and keeps them up-to-date as work manuals, but also provides feedback based on the user's emotions, which allows for the smooth transfer of know-how to the next generation of employees, improving work efficiency and safety on-site, and enhancing learning effectiveness.

[1597] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1598] Step 1:

[1599] Audio Acquisition

[1600] An experienced worker wearing a device (headset) receives real-time instruction voice on-site. The input voice might say, for example, "This part needs to be inspected once a week." The headset captures this voice data and temporarily stores it. At this time, the voice data is saved in local storage.

[1601] Step 2:

[1602] Sending audio data

[1603] The device sends the temporarily saved audio data to the server. The input is the audio data captured by the headset. A secure communication method such as HTTPS is used as the transmission protocol. The server stores the received audio data.

[1604] Step 3:

[1605] Audio analysis

[1606] The server receives the voice data and converts it into natural language text using a voice analysis tool. This process uses the Google Cloud Speech API. The input is the transmitted voice data, and the output is text data. For example, if the voice data is "This part needs to be inspected once a week," the output text data will be "This part needs to be inspected once a week."

[1607] Step 4:

[1608] Information Extraction

[1609] The server uses information extraction methods to extract instructional content and important procedures from the generated text data. This process uses natural language processing technology (such as the NLTK library). The input is the analyzed text data, and the output is the extracted information. For example, "parts inspection" and "once a week" are extracted.

[1610] Step 5:

[1611] Manual Generation

[1612] The server automatically generates and updates the work manual based on the information extracted by the information extraction means. This process uses the Django framework, etc. The input is the extracted information, and the output is the generated work manual. For example, a new procedure in the manual might state, "Part A should be inspected once a week."

[1613] Step 6:

[1614] Interface provided

[1615] It provides a web-based interface for users to view the manual and provide correction instructions. This interface uses React.js etc. The input is the generated work manual, and the output is a web page that users can view.

[1616] Step 7:

[1617] Translation tools

[1618] The server translates the generated work manual into multiple languages. This process uses the Amazon Translate API. The input is the generated work manual, and the output is the translated manual. For example, the manual can be translated into Spanish, allowing users to view the manual in Spanish.

[1619] Step 8:

[1620] Emotion analysis

[1621] The system analyzes emotions in real time as users read manuals. This process uses OpenCV and the Microsoft Azure Emotion API. The input is the user's facial expressions and tone of voice, and the output is the analyzed emotional data. For example, if the user is determined to have a "low level of understanding," the system generates a prompt to add more detailed explanations based on the result.

[1622] Step 9:

[1623] Applying Feedback

[1624] The content of the work manual is adjusted based on the results of the analysis by the emotion engine means. The input is emotion data, and the output is a revised work manual. For example, if the level of understanding is determined to be "low," a prompt sentence is added, detailing the specific inspection procedures for the manual.

[1625] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1626] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1627] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1628] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1629] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1630] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1631] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1632] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1633] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1634] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1635] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1636] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1637] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1638] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1639] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1640] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1641] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1642] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1643] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1644] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1645] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1646] The following is further disclosed regarding the above embodiment.

[1647] (Claim 1)

[1648] a voice acquisition means for acquiring voice instructions given by an experienced worker on-site;

[1649] a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting the voice data into a natural language;

[1650] an information extraction means for extracting instruction content and procedures from the text data converted by the voice analysis means;

[1651] a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means;

[1652] an interface means for a user to view the generated operation manual and give instructions for correction;

[1653] A system including:

[1654] (Claim 2)

[1655] 10. The system of claim 1, further comprising a translation means for translating the operation manual into multiple languages.

[1656] (Claim 3)

[1657] 2. The system according to claim 1, wherein the voice acquisition means is a headset that is worn by an experienced worker.

[1658] "Example 1"

[1659] (Claim 1)

[1660] a voice acquisition means for acquiring voice instructions given by an experienced worker on-site;

[1661] a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting the voice data into a natural language;

[1662] an information extraction means for extracting instruction content and procedures from the text data converted by the voice analysis means;

[1663] a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means;

[1664] an interface means for a user to view the generated operation manual and give instructions for correction;

[1665] a translation means for translating the operation manual into multiple languages;

[1666] A system including:

[1667] (Claim 2)

[1668] 2. The system of claim 1, wherein the voice capture means is a headset worn by an experienced worker.

[1669] (Claim 3)

[1670] 10. The system of claim 1, further comprising: means for transmitting the audio data to a server.

[1671] "Application Example 1"

[1672] (Claim 1)

[1673] a voice acquisition means for acquiring voice instructions given by an experienced worker on-site;

[1674] a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting the voice data into a natural language;

[1675] an information extraction means for extracting instruction content and procedures from the text data converted by the voice analysis means;

[1676] a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means;

[1677] an interface means for a user to view the generated operation manual and give instructions for correction;

[1678] a translation means for translating the operation manual into another language;

[1679] a machine operating means for automatically referring to the operation manual and performing the operation;

[1680] A system including:

[1681] (Claim 2)

[1682] 10. The system of claim 1, wherein the work manual can be updated and reviewed in real time through a user interface.

[1683] (Claim 3)

[1684] 2. The system according to claim 1, wherein the voice acquisition means is a voice recording device that is worn by an experienced worker.

[1685] "Example 2: Combining Emotion Engines"

[1686] (Claim 1)

[1687] a voice acquisition means for acquiring voice instructions given by an experienced worker on-site;

[1688] a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting the voice data into a natural language;

[1689] an information extraction means for extracting instruction content and procedures from the text data converted by the voice analysis means;

[1690] a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means;

[1691] an interface means for a user to view the generated operation manual and give instructions for correction;

[1692] an emotion engine that analyzes the emotion of the user and provides feedback based on the emotion data;

[1693] A system including:

[1694] (Claim 2)

[1695] 10. The system of claim 1, further comprising a translation means for translating the operation manual into multiple languages.

[1696] (Claim 3)

[1697] 2. The system according to claim 1, wherein the voice acquisition means is a headset that is worn by an experienced worker.

[1698] "Application example 2 when combining emotion engines"

[1699] (Claim 1)

[1700] a voice acquisition means for acquiring voice instructions given by an experienced worker on-site;

[1701] a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting the voice data into a natural language;

[1702] an information extraction means for extracting instruction content and procedures from the text data converted by the voice analysis means;

[1703] a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means;

[1704] an interface means for a user to view the generated operation manual and give instructions for correction;

[1705] a translation means for translating the generated operation manual into multiple languages;

[1706] emotion engine means for analyzing the emotion of a user and adjusting the content of the operation manual based on the result of the analysis;

[1707] A system including:

[1708] (Claim 2)

[1709] 10. The system of claim 1, further comprising means for displaying the generated work manual on a smartphone, smart glasses, or a head-mounted display.

[1710] (Claim 3)

[1711] 2. The system according to claim 1, wherein the voice acquisition means is a headset that is worn by an experienced worker. [Explanation of symbols]

[1712] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a voice acquisition means for acquiring voice instructions given by an experienced worker on-site; a voice analysis means for analyzing the voice data acquired by the voice acquisition means and converting the voice data into a natural language; an information extraction means for extracting instruction content and procedures from the text data converted by the voice analysis means; a manual generation means for automatically generating and updating a work manual based on the information extracted by the information extraction means; an interface means for a user to view the generated operation manual and give instructions for correction; A system including:

2. 2. The system according to claim 1, further comprising a translation means for translating said operation manual into multiple languages.

3. 2. The system according to claim 1, wherein the voice acquisition means is a headset that is worn by an experienced worker.

Images