system

Abstract

Provide a system. A video capture device, A means of transcribing the audio in the video; a means of summarizing the transcribed text; A means of extracting important images from within a video; a means for generating a manual by combining the summarized text and extracted images; a means for providing the generated manual to a user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Video manuals are commonplace in many modern workplaces, but they present the problem of making it difficult to pinpoint the information needed. Particularly when it comes to practical procedures for on-site work or household tasks, there is no time to rewatch the entire video. For this reason, traditional manuals with still images and text remain highly valued, but there has been a demand for a way to create them efficiently. [Means for solving the problem]

[0005] The present invention relates to a system that records a video while explaining its contents and transcribes and summarizes the audio using AI technology. The system includes a means for recording a video, a means for transcribing the audio from the video, a means for summarizing the transcribed text, a means for extracting important images from the video, a means for generating a manual by combining the summarized text and the extracted images, and a means for providing the generated manual to the user. This system enables users to easily and quickly create manuals containing pinpoint information, promoting widespread use in businesses and homes.

[0006] "Means for taking videos" refers to devices or methods used by users to record the state of their work, and includes camera functions installed on smartphones, smart glasses, etc.

[0007] A "means for transcribing audio from a filmed video" is a device or method that analyzes the audio data in the video and converts it into text data using voice recognition technology.

[0008] A "means for summarizing transcribed text" is a device or method that extracts necessary information from the obtained text data and generates a concise, to-the-point text.

[0009] A "means for extracting important images from a video" is a device or method that analyzes video frames to identify specific actions or important moments and extracts them as still images.

[0010] The "means for generating a manual by combining summarized text and extracted images" refers to a device or method for arranging summarized text and still images based on a template to create a manual in document format.

[0011] The "means for providing the generated manual to the user" refers to a device or method for storing the generated manual document in cloud storage or the like and generating a link that allows the user to access the manual. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] The system based on this invention allows users to film their work and then uses AI technology to efficiently generate manuals from the video. The basic configuration and operation of this system are described below.

[0034] Video recording and audio description

[0035] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, the user explains each step while assembling furniture.

[0036] Video and audio uploads

[0037] After the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted to ensure security.

[0038] Audio transcription

[0039] The server transcribes the received audio data using automatic speech recognition (ASR) technology, converting the audio data into text and recording any specific instructions or instructions contained within.

[0040] Text summary

[0041] The server's natural language processing (NLP) engine analyzes and summarizes the transcribed text, eliminating redundant information and leaving only the key points. This summary text focuses on the key steps or actions the user needs to take.

[0042] Image extraction

[0043] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes, such as when a user removes a part or screws it together, and then cuts out those scenes as still images.

[0044] Automatic manual generation

[0045] Using the extracted still images and summarized text, the server uses a manual generation engine to create a manual, which is generated in PDF or HTML format and provided in a visually easy-to-understand format.

[0046] Manual distribution

[0047] The generated manual is stored in cloud storage by the server, and the device provides the manual as a downloadable link, which users can use to obtain and share the manual.

[0048] Specific examples

[0049] For example, if a user is assembling new furniture at home, they can use their smartphone to film the process. They can explain the steps as they go along, and verbally mention important points about screws and parts. After filming is complete, the video and audio data are uploaded to a server, where ASR technology transcribes the audio. An NLP engine then summarizes the text, and CV technology extracts key scenes. Finally, the summarized text and still images are combined to generate a concise visual manual, which can be accessed by the user via cloud storage.

[0050] Such a system will eliminate the problems associated with conventional video manuals and allow users to efficiently obtain the information they need.

[0051] The processing flow will be explained below.

[0052] Step 1:

[0053] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, when removing a part, the user may say, "Remove part A."

[0054] Step 2:

[0055] Once the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted before being sent.

[0056] Step 3:

[0057] The server receives the uploaded video and audio data, temporarily stores the received data, and prepares it for transfer to the next process.

[0058] Step 4:

[0059] The server extracts the audio data from the video file, which is then passed to an automatic speech recognition (ASR) engine.

[0060] Step 5:

[0061] The ASR engine analyzes the voice data and transcribes it. For example, "Remove part A" is generated as text.

[0062] Step 6:

[0063] The server sends the transcribed text to a natural language processing (NLP) engine for analysis and summarization, eliminating redundancies and extracting only the important information.

[0064] Step 7:

[0065] The server passes the video data to a computer vision (CV) engine, which identifies important scenes. For example, it extracts frames of a scene where part A is being picked up and held up to the camera.

[0066] Step 8:

[0067] The CV engine generates still images from the identified scenes. The generated still images (e.g., an image of the moment part A is removed) are saved as material for creating manuals.

[0068] Step 9:

[0069] The server combines the summarized text and still images based on a template, and the manual generation engine uses the template to automatically generate a manual in PDF or HTML format.

[0070] Step 10:

[0071] The server saves the generated manual in cloud storage. Once the upload to the storage is complete, a download link is generated.

[0072] Step 11:

[0073] The terminal notifies the user of the download link for the generated manual, which the user can use to download, view, or share the manual.

[0074] In this way, the user can efficiently and quickly create a manual that combines still images and summary text from the video that he or she has taken.

[0075] Example 1

[0076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0077] Conventional manual generation methods have the problem of requiring a great deal of time and effort to describe work procedures in detail using text and images. Furthermore, methods that utilize video require users to spend a long time watching the video, making it difficult to search and organize information. This reduces the user's work efficiency and makes it difficult for them to confirm accurate procedures.

[0078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0079] In this invention, the server includes means for a user to record a video of the work, means for uploading the recorded video and audio data to the server, means for the server to transcribe the audio data using automatic speech recognition technology, means for summarizing the transcribed text using natural language processing technology, means for extracting important scenes from the video data using computer vision technology, means for combining the summarized text and extracted images to generate a manual, and means for storing the generated manual in cloud storage and providing it to the user, thereby enabling the user to efficiently obtain the necessary information and accurately check the work procedures.

[0080] "User" refers to a person who uses the system to record video of their work and obtain or refer to the manual.

[0081] "Device" refers to the hardware device a user uses to capture video and upload the data to a server. Examples include smartphones and smart glasses.

[0082] The "server" refers to a system that receives data uploaded by users, transcribes audio, summarizes text, extracts images, and ultimately generates and provides manuals.

[0083] "Audio data" refers to digital information of the voice recorded when the user explains the task.

[0084] "Automatic speech recognition technology (ASR)" refers to technology for converting voice data into text data.

[0085] "Transcription" refers to the process of converting audio data into text form.

[0086] "Natural language processing technology (NLP)" refers to the technology of analyzing transcribed text and extracting summaries and meanings.

[0087] A "summary" is a short text that removes redundant parts from transcribed text and summarizes only the important points.

[0088] "Computer vision technology (CV)" refers to technology that automatically extracts and analyzes specific information from images and videos.

[0089] An "important scene" refers to a particularly important part or moment in a video, and refers to a scene that contains information that a user needs.

[0090] A "still image" refers to an individual frame extracted from a video, meaning an image that captures a specific moment in time.

[0091] "Manual generation" refers to the process of combining summarized text with extracted still images to create a document containing instructions and instructions.

[0092] "Cloud storage" refers to remote servers that store and make data accessible over the internet.

[0093] The system based on this invention allows users to take videos of their work and then uses AI technology to efficiently generate manuals from those videos. The basic configuration and operation of this system are described below.

[0094] Video recording and audio description

[0095] The user uses a device such as a smartphone or smart glasses to record a video of the work being done. While recording, the user verbally explains the work. For example, while assembling furniture, the user might explain the detailed steps, such as "Install this screw in this position. Next, insert this part into this hole."

[0096] Video and audio uploads

[0097] After the recording is complete, the device uploads the video and audio data to the server. The data is encrypted before transmission to ensure security. Specifically, the data is encrypted using the SSL / TLS protocol.

[0098] Audio transcription

[0099] The server transcribes the received voice data using automatic speech recognition (ASR). For example, it uses the Google (registered trademark) Cloud Speech-to-Text API to convert the voice data into text. The text data includes steps such as "1. Take out the screw. 2. Insert the screw into the screw hole."

[0100] Text summary

[0101] The server analyzes and summarizes the transcribed text using natural language processing (NLP) techniques such as OpenAI's GPT model. This removes redundant information and leaves only the key points. For example, the summary text might include concise instructions such as "1. Remove the screw. 2. Insert the screw into the screw hole."

[0102] Image extraction

[0103] The server analyzes the video data using computer vision (CV) technologies such as OpenCV and automatically extracts important scenes, such as when a user removes a part or screws it together, and cuts out the scene as a still image.

[0104] Automatic manual generation

[0105] The server generates a manual using the summarized text and extracted still images. Specifically, it uses a manual generation engine such as LaTeX or Pandoc. The generated manual is provided in PDF or HTML format. This results in a visually easy-to-understand manual.

[0106] Manual distribution

[0107] The generated manual is stored in cloud storage (e.g., Amazon S3, Google Drive, etc.) by the server, and a link to the cloud storage is provided to the user, who can download or view the manual through the link.

[0108] Specific examples

[0109] As a concrete example, consider a user assembling new furniture at home. The user uses their smartphone to record a video of the work process and verbally explain the steps, such as "Use screw A here to attach board B to board C." After completing the recording, the user uploads the video and audio data to a server. The server transcribes the audio data using the Google Cloud Speech-to-Text API and summarizes it using OpenAI's GPT model. It also uses OpenCV to extract key scenes. Finally, the server combines the summarized text with the extracted still images to generate a concise visual manual. This manual is stored in cloud storage and a link is provided to the user.

[0110] Prompt Sentence Examples

[0111] "This video contains instructions for assembling furniture. Please follow the instructions verbally as you go through the process. Upload the video and audio data and generate a manual."

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1:

[0114] The user takes a video of the work they are doing. The work is recorded using a smartphone or smart glasses. For example, if the user is assembling furniture, they will explain each step as they go along. This video will record all of the steps.

[0115] Input: Video and audio of your work.

[0116] Output: Video file (including video and audio).

[0117] Step 2:

[0118] The device uploads the captured video and audio data to a server. Using a dedicated application, the data is encrypted using SSL / TLS and sent to the server.

[0119] Input: Video file.

[0120] Output: Video file saved on the server.

[0121] Step 3:

[0122] The server transcribes the received audio data using automatic speech recognition (ASR) technology and converts the audio into text using the Google Cloud Speech-to-Text API.

[0123] Input: Audio data from the video.

[0124] Output: Transcribed text data.

[0125] Step 4:

[0126] The server summarizes the transcribed text using natural language processing (NLP) techniques, using OpenAI's GPT model to extract important information from the text and remove redundancies.

[0127] Input: Transcribed text data.

[0128] Output: Summarized text data.

[0129] Step 5:

[0130] The server uses computer vision (CV) technology to extract important scenes from the video data. Using OpenCV, it identifies important scenes in the video and extracts them as still images. For example, this includes the moment when a user removes a part or tightens a screw.

[0131] Input: Video file.

[0132] Output: Still image files of important scenes.

[0133] Step 6:

[0134] The server generates a manual by combining the summarized text and extracted still images. Using LaTeX and Pandoc, the manual is created in PDF or HTML format for easy visual understanding.

[0135] Input: Abstracted text data, still image files.

[0136] Output: The generated manual (PDF or HTML format).

[0137] Step 7:

[0138] The server saves the generated manuals in cloud storage and provides them to users. It saves them using Amazon S3 or Google Drive and creates a download link.

[0139] Input: Generated manual.

[0140] Output: A download link that users can access.

[0141] (Application example 1)

[0142] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0143] In modern factories, it is important to efficiently record robot maintenance procedures and create maintenance manuals that can be reproduced by other technicians. However, traditional methods require manual recording and manual creation, which is time-consuming and labor-intensive. It is also difficult to record all detailed procedures and precautions, which can lead to inconsistent quality.

[0144] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0145] In this invention, the server includes means for shooting video, means for transcribing audio from the shot video, means for summarizing the transcribed text, means for extracting important images from the video, means for generating a manual by combining the summarized text and the extracted images, means for providing the generated manual to a user, and means for recording robot maintenance procedures as video and automatically generating a maintenance manual. This makes it possible to automatically and efficiently generate high-quality, reproducible maintenance manuals from videos recorded by technicians during work.

[0146] "Means for capturing video" refers to a device and method that allows a user to record the state of work using a terminal such as a smartphone or smart glasses.

[0147] A "means for transcribing audio from a video" is a device and method that uses automatic speech recognition (ASR) technology to convert audio data from a video into text data.

[0148] The "means for summarizing transcribed text" refers to an apparatus and method that uses a natural language processing (NLP) engine to remove redundant parts of transcribed text and extract key points.

[0149] The "means for extracting important images from video" refers to an apparatus and method that uses computer vision (CV) technology to identify important scenes from video data and extract them as still images.

[0150] The "means for generating a manual by combining summarized text and extracted images" refers to an apparatus and method for combining summarized text and extracted images to generate a visually easy-to-understand manual.

[0151] The "means for providing the generated manual to the user" refers to an apparatus and method for providing the generated manual to the user in a form that is easily accessible to the user via cloud storage or a download link.

[0152] The "means for recording robot maintenance procedures on video and automatically generating a maintenance manual" refers to a series of devices and methods for recording factory robot maintenance work on video and automatically creating a maintenance manual from the video.

[0153] To implement this invention, a user uses a device such as a smartphone or smart glasses to record video of the work they are doing. The user then verbally explains the necessary steps and key points during the work, and simultaneously records this as audio data. After the recording is complete, the device uploads the video and audio data to a server. The data is encrypted to ensure security.

[0154] The server converts the voice data into text using automatic speech recognition (ASR) technology, specifically software such as Google Cloud Speech-to-Text. The converted text is then summarized using a natural language processing (NLP) engine, such as the generative AI model GPT-3®. The summarized text focuses on important steps and explanations and eliminates redundant information.

[0155] Furthermore, the server uses computer vision (CV) technology to extract important scenes from the video data. The specific software used for this is OpenCV. The extracted important scenes are cut out as still images, and it is made clear which steps each scene corresponds to.

[0156] The server uses a manual generation engine to generate a maintenance manual by combining the summarized text and extracted images. The manual is generated in PDF or HTML format and provided in an easy-to-understand format. The generated manual is stored in cloud storage (e.g., AWS (registered trademark) S3), and users can obtain the manual as a downloadable link.

[0157] As a concrete example, consider a scenario in which a technician performing maintenance work on a robot in a factory records a video of the maintenance procedure on a smartphone. The technician explains each step as he or she works. The video and audio data are then uploaded to a server, where the audio is transcribed using ASR technology. An NLP engine summarizes the text, and CV technology extracts important scenes. The generated manual can then be accessed by the technician via cloud storage, and can also be shared with other technicians.

[0158] Examples of prompts that can be used include:

[0159] Summarize the robot maintenance procedures below:

[0160] 1. Remove the screws

[0161] 2. Remove the parts

[0162] 3. Install the new parts

[0163] 4. Retighten the screws

[0164] 5. Check the operation

[0165] summary:

[0166] The present invention allows users to easily generate efficient and high-quality maintenance manuals.

[0167] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0168] Step 1:

[0169] The user uses a device such as a smartphone or smart glasses to record a video of the work being done, and at the same time verbally explains the work content. The input is the work the user is doing and an explanation of it, and the output is video data and audio data. The specific operations are to launch the camera app, start recording, and verbally explain the work procedures and important points.

[0170] Step 2:

[0171] The device uploads the captured video and audio data to the server. At this time, the data is encrypted to ensure security. The input is the video and audio data stored on the device, and the output is the encrypted data stored on the server. Specifically, the upload application is used to transfer the data to the server.

[0172] Step 3:

[0173] The server converts the voice data into text using automatic speech recognition (ASR) technology. The software used is Google Cloud Speech-to-Text. The input is encrypted voice data, and the output is transcribed text data. Specifically, the server decrypts the voice data and uses ASR technology to convert the voice into text format.

[0174] Step 4:

[0175] The server uses a natural language processing (NLP) engine to summarize the transcribed text. The software used is a generative AI model such as GPT-3. The input is the transcribed text data, and the output is the summarized text data. Specifically, the text data is input into the NLP engine, and a summary is generated that focuses on important steps and explanations.

[0176] Step 5:

[0177] The server uses computer vision (CV) technology to extract important scenes from video data. The software used is OpenCV. The input is video data, and the output is extracted still image data. Specifically, it analyzes the video data and extracts important scenes (such as the moment a part is removed or a screw is fastened) as still images.

[0178] Step 6:

[0179] The server generates a manual by combining the summarized text and extracted images. The input is the summarized text data and extracted still image data, and the output is the manual data (PDF or HTML format). Specifically, it combines the summarized text and still images to determine the layout and generate a visually easy-to-understand manual.

[0180] Step 7:

[0181] The generated manual is stored in cloud storage and a link is provided for users to download it. The input is the manual data, and the output is the manual stored on the cloud and its download link. The specific operation is to upload the manual to a cloud storage service (e.g., AWS S3) and provide the link to the user.

[0182] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0183] The system based on this invention is designed to efficiently generate manuals from videos taken by users, and by combining it with an emotion engine that analyzes the user's emotions, it provides more appropriate manuals. The basic configuration and operation of this system are described below.

[0184] Video recording and audio description

[0185] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, the user explains each step while assembling furniture.

[0186] Video and audio uploads

[0187] After the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted to ensure security.

[0188] Audio transcription

[0189] The server transcribes the received audio data using automatic speech recognition (ASR) technology, converting the audio data into text and recording any specific instructions or instructions contained within.

[0190] Recognizing user emotions

[0191] The server uses an emotion engine to analyze the user's emotions from their voice and facial expressions, for example, determining whether they are confused or progressing smoothly based on their tone and intonation.

[0192] Text summary

[0193] The server's natural language processing (NLP) engine analyzes and summarizes the transcribed text, eliminating redundant information and leaving only the key points. This summary text focuses on the key steps or actions the user needs to take.

[0194] Image extraction

[0195] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes, such as when a user removes a part or screws it together, and then cuts out those scenes as still images.

[0196] Identifying important scenes based on emotions

[0197] Based on the user's emotional information recognized by the emotion engine, scenes that the user found particularly interesting or difficult are preferentially extracted, thereby improving the accuracy of extracting important scenes.

[0198] Automatic manual generation

[0199] Using the extracted still images and summarized text, the server uses a manual generation engine to create a manual, which is generated in PDF or HTML format and provided in a visually easy-to-understand format.

[0200] Manual distribution

[0201] The generated manual is stored in cloud storage by the server, and the device provides the manual as a downloadable link, which users can use to obtain and share the manual.

[0202] Specific examples

[0203] For example, if a user is assembling new furniture at home, they can use their smartphone to record a video of the work process. The steps are explained as they go along, with important points about screws and parts being mentioned verbally. After filming is complete, the video and audio data are uploaded to a server, where the audio is transcribed using ASR technology. An NLP engine then summarizes the text, and CV technology extracts key scenes. An emotion engine identifies situations where the user is confused and prioritizes the extraction of still images from those scenes. Finally, the summarized text and still images are combined to generate a concise visual manual, which can be accessed by the user via cloud storage. In this way, users can efficiently obtain the information they need and receive focused support on areas they are struggling with.

[0204] The processing flow will be explained below.

[0205] Step 1:

[0206] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, while assembling furniture, the user might explain, "First, remove part A."

[0207] Step 2:

[0208] Once the recording is complete, the device uploads the video and audio data to a server, where it is encrypted and securely transmitted.

[0209] Step 3:

[0210] The server receives the uploaded video and audio data, verifies the integrity of the files, and then begins the next process.

[0211] Step 4:

[0212] The server extracts the audio data from the video file, which is then passed to an automatic speech recognition (ASR) engine.

[0213] Step 5:

[0214] The ASR engine analyzes the voice data and transcribes it. The resulting transcription is "Remove part A."

[0215] Step 6:

[0216] The server sends the transcribed text to a natural language processing (NLP) engine for analysis and summarization. For example, a concise summary such as "Extract part A" is generated.

[0217] Step 7:

[0218] The server uses an emotion engine to analyze the user's voice and facial expressions, for example, to determine whether the user is confused or confused based on changes in voice tone or facial expression.

[0219] Step 8:

[0220] The server passes the video data to a computer vision (CV) engine to identify important scenes, such as when the user is confused or when important operations are performed.

[0221] Step 9:

[0222] Still images are generated from the scenes identified by the CV engine. Important work scenes and scenes that the user was interested in are saved as still images (e.g., the moment part A is removed).

[0223] Step 10:

[0224] The server combines the summarized text and extracted still images based on a template, and the manual generation engine uses the template to automatically generate a manual in PDF or HTML format.

[0225] Step 11:

[0226] The server saves the generated manual to cloud storage. Once the save is complete, a download link is generated and sent to the device.

[0227] Step 12:

[0228] The terminal notifies the user of the download link for the generated manual, which the user can use to download, view, and share the manual.

[0229] In this way, users can efficiently and quickly create manuals that combine still images and summary text from captured videos. Furthermore, the emotion engine prioritizes support for scenes where users find it difficult.

[0230] Example 2

[0231] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] Conventional manual generation systems often fail to meet actual needs because they do not consider the user's emotions or level of difficulty, even when using video or audio data. Furthermore, they sometimes fail to properly identify important scenes or summarize text, resulting in manuals that are difficult for users to understand. Furthermore, there are issues with the method of distributing the generated manuals, making them difficult for users to access.

[0233] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for shooting a video, a means for transcribing audio from the shot video, a means for summarizing the transcribed text, a means for extracting important images from the video, a means for analyzing a user's emotions, a means for identifying important scenes based on the analyzed emotional information, a means for generating a manual by combining the summarized text and the extracted images, and a means for providing the generated manual to the user. This makes it possible to take the user's emotions into consideration, appropriately extract important scenes, and combine them with the summarized text, thereby providing a manual that is easy for the user to understand and meets their needs. The generated manual is also stored in cloud storage, and a downloadable link is provided for the user to easily access.

[0234] "Means for shooting video" refers to the functions or devices that allow users to record video using digital devices.

[0235] "Means for transcribing audio from a video" refers to a function or device that uses voice recognition technology to convert audio from a video into text data.

[0236] A "means for summarizing transcribed text" is a function or device that uses natural language processing technology to extract important information from long pieces of text and summarize it concisely.

[0237] "Means for extracting important images from a video" refers to a function or device that uses computer vision technology to select specific scenes or images from a video.

[0238] A "means for analyzing user emotions" is a function or device that uses machine learning models or algorithms based on audio and video data to determine a user's emotional state.

[0239] The "means for identifying important scenes based on analyzed emotional information" refers to a function or device that, based on the results of the user's emotional analysis, prioritizes extracting scenes that the user finds particularly interesting or difficult.

[0240] The "means for generating a manual by combining summarized text and extracted images" refers to a function or device that integrates summarized text with important images to create a manual in a visually easy-to-understand format.

[0241] The "means for providing the generated manual to the user" refers to a function or device that provides the user with an access link to the manual stored in cloud storage, allowing the user to download or refer to the manual.

[0242] The system based on this invention is designed to efficiently generate manuals from videos taken by users, and by combining it with an emotion engine that analyzes the user's emotions, it provides more appropriate manuals. The basic configuration and operation of this system are described below.

[0243] Video recording and audio description

[0244] The user uses a device such as a smartphone or smart glasses to record a video of the work being done. While recording, the user verbally explains the work. For example, while assembling furniture, the user explains each step, giving specific instructions such as "Use these screws to connect the boards."

[0245] Video and audio uploads

[0246] After the recording is complete, the device uploads the video and audio data to the server. The data is encrypted to ensure security. The device transmits the video and audio data to the server using the SSL / TLS protocol.

[0247] Audio transcription

[0248] The server transcribes the received voice data using automatic speech recognition (ASR) technology, such as Google Cloud Speech-to-Text or IBM Watson® Speech to Text. The voice data is converted into text, and each step and explanation is recorded.

[0249] Recognizing user emotions

[0250] The server uses an emotion engine to analyze emotions from the user's voice and facial expressions. For example, it uses the Microsoft® Azure® Emotion API to determine the user's emotions from the tone and intonation of the voice and facial expressions. The server then sends the analyzed voice data and video data to the emotion engine for emotion analysis.

[0251] Text summary

[0252] The transcribed text is analyzed and summarized by a natural language processing (NLP) engine on the server. Redundant information is removed, leaving only the key points. This process uses techniques such as OpenAI GPT-3 and BERT. The server then stores the generated summary text in a database.

[0253] Image extraction

[0254] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes. Using OpenCV, it identifies the moment when the user removes a part or connects it with a screw, and cuts them out as still images.

[0255] Identifying important scenes based on emotions

[0256] Based on the user's emotional information recognized by the emotion engine, scenes that the user found particularly interesting or difficult are preferentially extracted. This improves the accuracy of extracting important scenes. Based on the emotion analysis results, the server preferentially selects still images that match scenes that the user found difficult.

[0257] Automatic manual generation

[0258] The server uses the extracted still images and summarized text to create a manual using a manual generation engine. It uses a combination of LaTeX, Markdown, and Pandoc to generate the manual in PDF or HTML format. The generated document is saved in the specified format.

[0259] Manual distribution

[0260] The generated manual is stored by the server in cloud storage (e.g., Amazon S3 or Google Drive). The user is provided with a link to download the manual from their device. The server uploads the generated manual to cloud storage and generates a link. The user can download the manual from the link in the app.

[0261] Specific examples

[0262] For example, this system could be used when a user is assembling new furniture at home. The user uses their smartphone to record video of the work process while verbally explaining the steps and important points. The video and audio data are uploaded to a server, and the audio is transcribed using ASR technology. The text is then summarized using an NLP engine, and important scenes are extracted using CV technology. An emotion engine identifies confusing moments and prioritizes the extraction of still images from those moments. Finally, the summarized text and still images are combined to generate a concise manual. The user can retrieve this manual from cloud storage and efficiently obtain the information they need.

[0263] Prompt Sentence Examples

[0264] Here are some examples of prompts to input to a generative AI model:

[0265] Design a program that automatically generates a manual for assembling furniture based on a video taken by a user, following the steps below:

[0266] 1. Uploading video and audio data

[0267] 2. Audio transcription (using ASR technology)

[0268] 3. User sentiment analysis (using emotion engine)

[0269] 4. Text Summarization (using NLP techniques)

[0270] 5. Extraction of important scenes from videos (using CV technology)

[0271] 6. Generating a manual based on the extracted scenes and summary text (using a manual generation engine)

[0272] 7. Save the manual to the cloud and provide a link to the user

[0273] Specifically, this involves using technologies such as Google Cloud Speech-to-Text, Microsoft Azure Emotion API, OpenAI GPT-3, OpenCV, LaTeX, and Amazon S3.

[0274] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0275] Step 1:

[0276] The user records a video of their work using a device such as a smartphone or smart glasses. The input is the user's work and an explanation of it, and the output is video data. The user opens a camera app and presses the record button to start recording. As the user works, they provide appropriate explanations.

[0277] Step 2:

[0278] After the recording is complete, the device uploads the video and audio data to the server. The input is the recorded video data, and the output is the video and audio data sent to the server. The device encrypts the data using the SSL / TLS protocol and sends it securely to the server.

[0279] Step 3:

[0280] The server transcribes the received voice data using automatic speech recognition (ASR) technology. The input is voice data and the output is text data. The server sends the voice data to an ASR engine (e.g., Google Cloud Speech-to-Text) and converts the voice into text format.

[0281] Step 4:

[0282] The server uses an emotion engine to analyze emotions from the user's voice and facial expressions. The input is audio and video data, and the output is the emotion analysis results. The server sends the audio and video to the emotion engine (for example, a general emotion analysis API) for emotion analysis.

[0283] Step 5:

[0284] The transcribed text is analyzed and summarized by the server's natural language processing (NLP) engine. The input is the transcribed text, and the output is the summary text. The server inputs the text data into an NLP engine (e.g., OpenAI GPT-3 or BERT) to generate a summary.

[0285] Step 6:

[0286] The server analyzes video data using computer vision (CV) technology and automatically extracts important scenes. The input is video data and the output is important still images. The server analyzes video frames every second and uses CV technology (e.g., OpenCV) to detect specific actions and extract them as still images.

[0287] Step 7:

[0288] Based on the user's emotional information recognized by the emotion engine, scenes that are of particular interest or difficulty are extracted with priority. The input is the emotion analysis results and still images, and the output is a prioritized list of important still images. The server compares the emotion analysis results with the still images and prioritizes saving scenes that the user found confusing.

[0289] Step 8:

[0290] The server uses the extracted still images and summarized text to create a manual using a manual generation engine. The input is the summary text and still images, and the output is a manual in PDF or HTML format. The server uses a combination of LaTeX, Markdown, and Pandoc to generate a visually easy-to-understand manual.

[0291] Step 9:

[0292] The generated manual is stored in cloud storage by the server. The input is the generated manual, and the output is an access link to the manual on the cloud. The server uploads the manual to cloud storage (e.g., Amazon S3 or Google Drive) and provides the user with a downloadable link.

[0293] (Application example 2)

[0294] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0295] In modern factories, creating work procedure manuals is time-consuming and labor-intensive, and relies on monotonous, impersonal information, making it difficult for workers to understand and put into practice. Furthermore, because the feelings and difficulties of individual workers are not taken into consideration, efficiency is often low. Therefore, there is a demand for shortening work times and reducing errors, but current manual generation methods have limitations.

[0296] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0297] In this invention, the server includes: [means for shooting video;] [means for transcribing audio from the shot video;] [means for summarizing the transcribed text;] [means for extracting important images from the video;] [means for generating a manual by combining the summarized text and extracted images;] [means for providing the generated manual to the user;] [means for analyzing the user's emotions from their voice and facial expressions;] [means for identifying important scenes based on the user's emotional information; and [means for recording and analyzing work performed by the robot.] This enables the user to generate more understandable and effective manuals that take into account the emotions and difficulties of each work procedure.

[0298] "Means for taking videos" refers to a function that allows a user to use a device such as a smartphone or robot to record the work situation and procedures as video.

[0299] "Means for transcribing audio from video footage" refers to a function that converts audio data from video footage into text data using automatic speech recognition technology.

[0300] "Means for summarizing transcribed text" refers to the function of extracting important points from transcribed text data and summarizing them in a concise form. This function uses natural language processing technology.

[0301] "Means for extracting important images from video" refers to a function that automatically selects specific scenes or important moments from captured video data and extracts them as still images. This uses computer vision technology.

[0302] The "means for generating a manual by combining summarized text and extracted images" is a function that automatically creates a visually easy-to-understand manual by combining summarized text data with extracted still images.

[0303] The "means for providing the generated manual to the user" is a function for storing the generated manual in cloud storage and providing a link so that the user can download or access the manual.

[0304] "Means for analyzing user emotions from voice and facial expressions" refers to a function that identifies and analyzes the user's emotional state from the tone of their voice and facial expressions in the video. This is done using an emotion engine.

[0305] The "means for identifying important scenes based on the user's emotional information" is a function that prioritizes extracting scenes that the user is particularly interested in or finds difficult, based on data obtained through emotional analysis.

[0306] "Means for recording and analyzing work performed by robots" refers to a function that records and records the various tasks performed by factory robots, analyzes the data, and generates manuals.

[0307] To implement this invention, it is necessary to configure a system that includes the following main means: The system has means for shooting video, transcribing audio, summarizing text, extracting important images, generating and providing manuals, analyzing user emotions, and recording and analyzing work performed by a robot.

[0308] The server provides a means for users to record video of their work through their devices, such as smartphones or factory robots. The user can record video and provide verbal instructions while working, and the video and audio data are stored on the device. This data is then encrypted and uploaded to a cloud-based server.

[0309] The server automatically transcribes the uploaded audio data using the Google Cloud Speech API, converting it into text so that its content can be understood. The text data is then summarized using natural language processing techniques from Hugging Face's Transformers library. The summarized text is stripped of redundant information and focuses on the key points.

[0310] The server then analyzes the video data using OpenCV to extract specific scenes and important moments. This allows scenes and steps that the user considered particularly important during the task to be extracted as images. The server then analyzes the user's emotions using the TENSORFLOW® model. The server determines the user's emotional state based on their voice tone and facial expressions, and based on this information, prioritizes the extraction of scenes in which the user found the task particularly challenging or interesting.

[0311] By combining these summary texts with extracted images of important scenes, the manual generation engine automatically creates a visually concise manual. The generated manual is saved in cloud storage in PDF or HTML format, and users can obtain and share the manual via a downloadable link.

[0312] (Example)

[0313] For example, when a factory worker records a robotic maintenance procedure for a new machine, they film the robot performing the task. This video and the audio data within it are uploaded to a server, where they are transcribed and summarized using automatic speech recognition and natural language processing technologies. Furthermore, computer vision technology extracts key scenes, and emotion analysis identifies parts where the user had difficulty. Finally, a manual summarizing the key points is generated.

[0314] (Example of a prompt)

[0315] "Explain the machine maintenance procedure, record each step on video, highlight the key points and generate a manual."

[0316] "Record a video of the assembly process and generate a summary manual based on the user's voice description and sentiment analysis."

[0317] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0318] Step 1:

[0319] Users use devices such as smartphones or factory robots to record video of work situations and procedures. While recording the video, the user verbally explains each step, and video data (video and audio) is input into the device.

[0320] Step 2:

[0321] The device uploads the recorded video data to a cloud-based server. During the upload process, the video data is encrypted and processed to ensure security. The encrypted video data is input to the server as an output.

[0322] Step 3:

[0323] The server automatically transcribes the audio data in the video using the Google Cloud Speech API. This process converts the audio data into text data. The input is audio data, and the output is transcribed text data.

[0324] Step 4:

[0325] The server uses Hugging Face's Transformers library to summarize the transcribed text data. Summarization extracts key points from the text and appropriately compresses the amount of information. The input is the transcribed text data, and the output is the summarized text data.

[0326] Step 5:

[0327] The server uses OpenCV to extract important image scenes from video data. It identifies specific frames and scenes and extracts them as still images. The input is video data, and the output is a set of extracted still images.

[0328] Step 6:

[0329] The server uses a TensorFlow model to analyze emotions from the user's voice tone and facial expressions in the video. Emotion analysis identifies the emotional state, such as difficulty or joy, felt by the user at each step. The input is audio and video data, and the output is the emotion analysis results.

[0330] Step 7:

[0331] Based on the emotion analysis results, the server prioritizes and extracts scenes that the user finds particularly difficult or that the user deems important. This selection process identifies still images with high importance. The input is the emotion analysis results and the extracted still images, and the output is the prioritized still images.

[0332] Step 8:

[0333] The server combines the summarized text data with a set of selected still images to automatically generate a manual. The generated manual is visually easy to understand, emphasizing important procedures and key points. The input is the summarized text data and a set of selected still images, and the output is the generated manual (in PDF or HTML format).

[0334] Step 9:

[0335] The server stores the generated manual in cloud storage and provides a download link to the user, through which the user can access the generated manual and share it as needed. The input is the generated manual, and the output is the download link.

[0336] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0337] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0338] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0339] [Second embodiment]

[0340] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0341] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0342] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0343] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0344] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0345] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0346] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0347] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0348] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0349] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0350] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0351] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0352] The system based on this invention allows users to film their work and then uses AI technology to efficiently generate manuals from the video. The basic configuration and operation of this system are described below.

[0353] Video recording and audio description

[0354] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, the user explains each step while assembling furniture.

[0355] Video and audio uploads

[0356] After the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted to ensure security.

[0357] Audio transcription

[0358] The server transcribes the received audio data using automatic speech recognition (ASR) technology, converting the audio data into text and recording any specific instructions or instructions contained within.

[0359] Text summary

[0360] The server's natural language processing (NLP) engine analyzes and summarizes the transcribed text, eliminating redundant information and leaving only the key points. This summary text focuses on the key steps or actions the user needs to take.

[0361] Image extraction

[0362] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes, such as when a user removes a part or screws it together, and then cuts out those scenes as still images.

[0363] Automatic manual generation

[0364] Using the extracted still images and summarized text, the server uses a manual generation engine to create a manual, which is generated in PDF or HTML format and provided in a visually easy-to-understand format.

[0365] Manual distribution

[0366] The generated manual is stored in cloud storage by the server, and the device provides the manual as a downloadable link, which users can use to obtain and share the manual.

[0367] Specific examples

[0368] For example, if a user is assembling new furniture at home, they can use their smartphone to film the process. They can explain the steps as they go along, and verbally mention important points about screws and parts. After filming is complete, the video and audio data are uploaded to a server, where ASR technology transcribes the audio. An NLP engine then summarizes the text, and CV technology extracts key scenes. Finally, the summarized text and still images are combined to generate a concise visual manual, which can be accessed by the user via cloud storage.

[0369] Such a system will eliminate the problems associated with conventional video manuals and allow users to efficiently obtain the information they need.

[0370] The processing flow will be explained below.

[0371] Step 1:

[0372] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, when removing a part, the user may say, "Remove part A."

[0373] Step 2:

[0374] Once the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted before being sent.

[0375] Step 3:

[0376] The server receives the uploaded video and audio data, temporarily stores the received data, and prepares it for transfer to the next process.

[0377] Step 4:

[0378] The server extracts the audio data from the video file, which is then passed to an automatic speech recognition (ASR) engine.

[0379] Step 5:

[0380] The ASR engine analyzes the voice data and transcribes it. For example, "Remove part A" is generated as text.

[0381] Step 6:

[0382] The server sends the transcribed text to a natural language processing (NLP) engine for analysis and summarization, eliminating redundancies and extracting only the important information.

[0383] Step 7:

[0384] The server passes the video data to a computer vision (CV) engine, which identifies important scenes. For example, it extracts frames of a scene where part A is being picked up and held up to the camera.

[0385] Step 8:

[0386] The CV engine generates still images from the identified scenes. The generated still images (e.g., an image of the moment part A is removed) are saved as material for creating manuals.

[0387] Step 9:

[0388] The server combines the summarized text and still images based on a template, and the manual generation engine uses the template to automatically generate a manual in PDF or HTML format.

[0389] Step 10:

[0390] The server saves the generated manual in cloud storage. Once the upload to the storage is complete, a download link is generated.

[0391] Step 11:

[0392] The terminal notifies the user of the download link for the generated manual, which the user can use to download, view, or share the manual.

[0393] In this way, the user can efficiently and quickly create a manual that combines still images and summary text from the video that he or she has taken.

[0394] Example 1

[0395] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0396] Conventional manual generation methods have the problem of requiring a great deal of time and effort to describe work procedures in detail using text and images. Furthermore, methods that utilize video require users to spend a long time watching the video, making it difficult to search and organize information. This reduces the user's work efficiency and makes it difficult for them to confirm accurate procedures.

[0397] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0398] In this invention, the server includes means for a user to record a video of the work, means for uploading the recorded video and audio data to the server, means for the server to transcribe the audio data using automatic speech recognition technology, means for summarizing the transcribed text using natural language processing technology, means for extracting important scenes from the video data using computer vision technology, means for combining the summarized text and extracted images to generate a manual, and means for storing the generated manual in cloud storage and providing it to the user, thereby enabling the user to efficiently obtain the necessary information and accurately check the work procedures.

[0399] "User" refers to a person who uses the system to record video of their work and obtain or refer to the manual.

[0400] "Device" refers to the hardware device a user uses to capture video and upload the data to a server. Examples include smartphones and smart glasses.

[0401] The "server" refers to a system that receives data uploaded by users, transcribes audio, summarizes text, extracts images, and ultimately generates and provides manuals.

[0402] "Audio data" refers to digital information of the voice recorded when the user explains the task.

[0403] "Automatic speech recognition technology (ASR)" refers to technology for converting voice data into text data.

[0404] "Transcription" refers to the process of converting audio data into text form.

[0405] "Natural language processing technology (NLP)" refers to the technology of analyzing transcribed text and extracting summaries and meanings.

[0406] A "summary" is a short text that removes redundant parts from transcribed text and summarizes only the important points.

[0407] "Computer vision technology (CV)" refers to technology that automatically extracts and analyzes specific information from images and videos.

[0408] An "important scene" refers to a particularly important part or moment in a video, and refers to a scene that contains information that a user needs.

[0409] A "still image" refers to an individual frame extracted from a video, meaning an image that captures a specific moment in time.

[0410] "Manual generation" refers to the process of combining summarized text with extracted still images to create a document containing instructions and instructions.

[0411] "Cloud storage" refers to remote servers that store and make data accessible over the internet.

[0412] The system based on this invention allows users to take videos of their work and then uses AI technology to efficiently generate manuals from those videos. The basic configuration and operation of this system are described below.

[0413] Video recording and audio description

[0414] The user uses a device such as a smartphone or smart glasses to record a video of the work being done. While recording, the user verbally explains the work. For example, while assembling furniture, the user might explain the detailed steps, such as "Install this screw in this position. Next, insert this part into this hole."

[0415] Video and audio uploads

[0416] After the recording is complete, the device uploads the video and audio data to the server. The data is encrypted before transmission to ensure security. Specifically, the data is encrypted using the SSL / TLS protocol.

[0417] Audio transcription

[0418] The server transcribes the received voice data using automatic speech recognition (ASR). For example, it uses the Google Cloud Speech-to-Text API to convert the voice data into text. The text data includes steps such as "1. Take out the screw. 2. Insert the screw into the screw hole."

[0419] Text summary

[0420] The server analyzes and summarizes the transcribed text using natural language processing (NLP) techniques such as OpenAI's GPT model. This removes redundant information and leaves only the key points. For example, the summary text might include concise steps such as "1. Remove the screw. 2. Insert the screw into the screw hole."

[0421] Image extraction

[0422] The server analyzes the video data using computer vision (CV) technologies such as OpenCV and automatically extracts important scenes, such as when a user removes a part or screws it together, and cuts out the scene as a still image.

[0423] Automatic manual generation

[0424] The server generates a manual using the summarized text and extracted still images. Specifically, it uses a manual generation engine such as LaTeX or Pandoc. The generated manual is provided in PDF or HTML format. This results in a visually easy-to-understand manual.

[0425] Manual distribution

[0426] The generated manual is stored in cloud storage (e.g., Amazon S3, Google Drive, etc.) by the server, and a link to the cloud storage is provided to the user, who can download or view the manual through the link.

[0427] Specific examples

[0428] As a concrete example, consider a user assembling new furniture at home. The user uses their smartphone to record a video of the work process and verbally explain the steps, such as "Use screw A here to attach board B to board C." After completing the recording, the user uploads the video and audio data to a server. The server transcribes the audio data using the Google Cloud Speech-to-Text API and summarizes it using OpenAI's GPT model. It also uses OpenCV to extract key scenes. Finally, the server combines the summarized text with the extracted still images to generate a concise visual manual. This manual is stored in cloud storage and a link is provided to the user.

[0429] Prompt Sentence Examples

[0430] "This video contains instructions for assembling furniture. Please follow the instructions verbally as you go through the process. Upload the video and audio data and generate a manual."

[0431] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0432] Step 1:

[0433] The user takes a video of the work they are doing. The work is recorded using a smartphone or smart glasses. For example, if the user is assembling furniture, they will explain each step as they go along. This video will record all of the steps.

[0434] Input: Video and audio of your work.

[0435] Output: Video file (including video and audio).

[0436] Step 2:

[0437] The device uploads the captured video and audio data to a server. Using a dedicated application, the data is encrypted using SSL / TLS and sent to the server.

[0438] Input: Video file.

[0439] Output: Video file saved on the server.

[0440] Step 3:

[0441] The server transcribes the received audio data using automatic speech recognition (ASR) technology and converts the audio into text using the Google Cloud Speech-to-Text API.

[0442] Input: Audio data from the video.

[0443] Output: Transcribed text data.

[0444] Step 4:

[0445] The server summarizes the transcribed text using natural language processing (NLP) techniques, using OpenAI's GPT model to extract important information from the text and remove redundancies.

[0446] Input: Transcribed text data.

[0447] Output: Summarized text data.

[0448] Step 5:

[0449] The server uses computer vision (CV) technology to extract important scenes from the video data. Using OpenCV, it identifies important scenes in the video and extracts them as still images. For example, this includes the moment when a user removes a part or tightens a screw.

[0450] Input: Video file.

[0451] Output: Still image files of important scenes.

[0452] Step 6:

[0453] The server generates a manual by combining the summarized text and extracted still images. Using LaTeX and Pandoc, the manual is created in PDF or HTML format for easy visual understanding.

[0454] Input: Abstracted text data, still image files.

[0455] Output: The generated manual (PDF or HTML format).

[0456] Step 7:

[0457] The server saves the generated manuals in cloud storage and provides them to users. It saves them using Amazon S3 or Google Drive and creates a download link.

[0458] Input: Generated manual.

[0459] Output: A download link that users can access.

[0460] (Application example 1)

[0461] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0462] In modern factories, it is important to efficiently record robot maintenance procedures and create maintenance manuals that can be reproduced by other technicians. However, traditional methods require manual recording and manual creation, which is time-consuming and labor-intensive. It is also difficult to record all detailed procedures and precautions, which can lead to inconsistent quality.

[0463] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0464] In this invention, the server includes means for shooting video, means for transcribing audio from the shot video, means for summarizing the transcribed text, means for extracting important images from the video, means for generating a manual by combining the summarized text and the extracted images, means for providing the generated manual to a user, and means for recording robot maintenance procedures as video and automatically generating a maintenance manual. This makes it possible to automatically and efficiently generate high-quality, reproducible maintenance manuals from videos recorded by technicians during work.

[0465] "Means for capturing video" refers to a device and method that allows a user to record the state of work using a terminal such as a smartphone or smart glasses.

[0466] A "means for transcribing audio from a video" is a device and method that uses automatic speech recognition (ASR) technology to convert audio data from a video into text data.

[0467] The "means for summarizing transcribed text" refers to an apparatus and method that uses a natural language processing (NLP) engine to remove redundant parts of transcribed text and extract key points.

[0468] The "means for extracting important images from video" refers to an apparatus and method that uses computer vision (CV) technology to identify important scenes from video data and extract them as still images.

[0469] The "means for generating a manual by combining summarized text and extracted images" refers to an apparatus and method for combining summarized text and extracted images to generate a visually easy-to-understand manual.

[0470] The "means for providing the generated manual to the user" refers to an apparatus and method for providing the generated manual to the user in a form that is easily accessible to the user via cloud storage or a download link.

[0471] The "means for recording robot maintenance procedures on video and automatically generating a maintenance manual" refers to a series of devices and methods for recording factory robot maintenance work on video and automatically creating a maintenance manual from the video.

[0472] To implement this invention, a user uses a device such as a smartphone or smart glasses to record video of the work they are doing. The user then verbally explains the necessary steps and key points during the work, and simultaneously records this as audio data. After the recording is complete, the device uploads the video and audio data to a server. The data is encrypted to ensure security.

[0473] The server converts the audio data into text using automatic speech recognition (ASR) technology, specifically Google Cloud Speech-to-Text. The converted text is then summarized using a natural language processing (NLP) engine, such as the generative AI model GPT-3. The summarized text focuses on important steps and explanations and eliminates redundant information.

[0474] Furthermore, the server uses computer vision (CV) technology to extract important scenes from the video data. The specific software used for this is OpenCV. The extracted important scenes are cut out as still images, and it is made clear which steps each scene corresponds to.

[0475] The server uses a manual generation engine to generate a maintenance manual by combining the summarized text and extracted images. The manual is generated in PDF or HTML format and provided in an easy-to-understand format. The generated manual is stored in cloud storage (e.g., AWS S3), and users can obtain the manual as a downloadable link.

[0476] As a concrete example, consider a scenario in which a technician performing maintenance work on a robot in a factory records a video of the maintenance procedure on a smartphone. The technician explains each step as he or she works. The video and audio data are then uploaded to a server, where the audio is transcribed using ASR technology. An NLP engine summarizes the text, and CV technology extracts important scenes. The generated manual can then be accessed by the technician via cloud storage, and can also be shared with other technicians.

[0477] Examples of prompts that can be used include:

[0478] Summarize the robot maintenance procedures below:

[0479] 1. Remove the screws

[0480] 2. Remove the parts

[0481] 3. Install the new parts

[0482] 4. Retighten the screws

[0483] 5. Check the operation

[0484] summary:

[0485] The present invention allows users to easily generate efficient and high-quality maintenance manuals.

[0486] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0487] Step 1:

[0488] The user uses a device such as a smartphone or smart glasses to record a video of the work being done, and at the same time verbally explains the work content. The input is the work the user is doing and an explanation of it, and the output is video data and audio data. The specific operations are to launch the camera app, start recording, and verbally explain the work procedures and important points.

[0489] Step 2:

[0490] The device uploads the captured video and audio data to the server. At this time, the data is encrypted to ensure security. The input is the video and audio data stored on the device, and the output is the encrypted data stored on the server. Specifically, the upload application is used to transfer the data to the server.

[0491] Step 3:

[0492] The server converts the voice data into text using automatic speech recognition (ASR) technology. The software used is Google Cloud Speech-to-Text. The input is encrypted voice data, and the output is transcribed text data. Specifically, the server decrypts the voice data and uses ASR technology to convert the voice into text format.

[0493] Step 4:

[0494] The server uses a natural language processing (NLP) engine to summarize the transcribed text. The software used is a generative AI model such as GPT-3. The input is the transcribed text data, and the output is the summarized text data. Specifically, the text data is input into the NLP engine, and a summary is generated that focuses on important steps and explanations.

[0495] Step 5:

[0496] The server uses computer vision (CV) technology to extract important scenes from video data. The software used is OpenCV. The input is video data, and the output is extracted still image data. Specifically, it analyzes the video data and extracts important scenes (such as the moment a part is removed or a screw is fastened) as still images.

[0497] Step 6:

[0498] The server generates a manual by combining the summarized text and extracted images. The input is the summarized text data and extracted still image data, and the output is the manual data (PDF or HTML format). Specifically, it combines the summarized text and still images to determine the layout and generate a visually easy-to-understand manual.

[0499] Step 7:

[0500] The generated manual is stored in cloud storage and a link is provided for users to download it. The input is the manual data, and the output is the manual stored on the cloud and its download link. The specific operation is to upload the manual to a cloud storage service (e.g., AWS S3) and provide the link to the user.

[0501] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0502] The system based on this invention is designed to efficiently generate manuals from videos taken by users, and by combining it with an emotion engine that analyzes the user's emotions, it provides more appropriate manuals. The basic configuration and operation of this system are described below.

[0503] Video recording and audio description

[0504] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, the user explains each step while assembling furniture.

[0505] Video and audio uploads

[0506] After the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted to ensure security.

[0507] Audio transcription

[0508] The server transcribes the received audio data using automatic speech recognition (ASR) technology, converting the audio data into text and recording any specific instructions or instructions contained within.

[0509] Recognizing user emotions

[0510] The server uses an emotion engine to analyze the user's emotions from their voice and facial expressions, for example, determining whether they are confused or progressing smoothly based on their tone and intonation.

[0511] Text summary

[0512] The server's natural language processing (NLP) engine analyzes and summarizes the transcribed text, eliminating redundant information and leaving only the key points. This summary text focuses on the key steps or actions the user needs to take.

[0513] Image extraction

[0514] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes, such as when a user removes a part or screws it together, and then cuts out those scenes as still images.

[0515] Identifying important scenes based on emotions

[0516] Based on the user's emotional information recognized by the emotion engine, scenes that the user found particularly interesting or difficult are preferentially extracted, thereby improving the accuracy of extracting important scenes.

[0517] Automatic manual generation

[0518] Using the extracted still images and summarized text, the server uses a manual generation engine to create a manual, which is generated in PDF or HTML format and provided in a visually easy-to-understand format.

[0519] Manual distribution

[0520] The generated manual is stored in cloud storage by the server, and the device provides the manual as a downloadable link, which users can use to obtain and share the manual.

[0521] Specific examples

[0522] For example, if a user is assembling new furniture at home, they can use their smartphone to record a video of the work process. The steps are explained as they go along, with important points about screws and parts being mentioned verbally. After filming is complete, the video and audio data are uploaded to a server, where the audio is transcribed using ASR technology. An NLP engine then summarizes the text, and CV technology extracts key scenes. An emotion engine identifies situations where the user is confused and prioritizes the extraction of still images from those scenes. Finally, the summarized text and still images are combined to generate a concise visual manual, which can be accessed by the user via cloud storage. In this way, users can efficiently obtain the information they need and receive focused support on areas they are struggling with.

[0523] The processing flow will be explained below.

[0524] Step 1:

[0525] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, while assembling furniture, the user might explain, "First, remove part A."

[0526] Step 2:

[0527] Once the recording is complete, the device uploads the video and audio data to a server, where it is encrypted and securely transmitted.

[0528] Step 3:

[0529] The server receives the uploaded video and audio data, verifies the integrity of the files, and then begins the next process.

[0530] Step 4:

[0531] The server extracts the audio data from the video file, which is then passed to an automatic speech recognition (ASR) engine.

[0532] Step 5:

[0533] The ASR engine analyzes the voice data and transcribes it. The resulting transcription is "Remove part A."

[0534] Step 6:

[0535] The server sends the transcribed text to a natural language processing (NLP) engine for analysis and summarization. For example, a concise summary such as "Extract part A" is generated.

[0536] Step 7:

[0537] The server uses an emotion engine to analyze the user's voice and facial expressions, for example, to determine whether the user is confused or confused based on changes in voice tone or facial expression.

[0538] Step 8:

[0539] The server passes the video data to a computer vision (CV) engine to identify important scenes, such as when the user is confused or when important operations are performed.

[0540] Step 9:

[0541] Still images are generated from the scenes identified by the CV engine. Important work scenes and scenes that the user was interested in are saved as still images (e.g., the moment part A is removed).

[0542] Step 10:

[0543] The server combines the summarized text and extracted still images based on a template, and the manual generation engine uses the template to automatically generate a manual in PDF or HTML format.

[0544] Step 11:

[0545] The server saves the generated manual to cloud storage. Once the save is complete, a download link is generated and sent to the device.

[0546] Step 12:

[0547] The terminal notifies the user of the download link for the generated manual, which the user can use to download, view, and share the manual.

[0548] In this way, users can efficiently and quickly create manuals that combine still images and summary text from captured videos. Furthermore, the emotion engine prioritizes support for scenes where users find it difficult.

[0549] Example 2

[0550] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0551] Conventional manual generation systems often fail to meet actual needs because they do not consider the user's emotions or level of difficulty, even when using video or audio data. Furthermore, they sometimes fail to properly identify important scenes or summarize text, resulting in manuals that are difficult for users to understand. Furthermore, there are issues with the method of distributing the generated manuals, making them difficult for users to access.

[0552] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for shooting a video, a means for transcribing audio from the shot video, a means for summarizing the transcribed text, a means for extracting important images from the video, a means for analyzing a user's emotions, a means for identifying important scenes based on the analyzed emotional information, a means for generating a manual by combining the summarized text and the extracted images, and a means for providing the generated manual to the user. This makes it possible to take the user's emotions into consideration, appropriately extract important scenes, and combine them with the summarized text, thereby providing a manual that is easy for the user to understand and meets their needs. The generated manual is also stored in cloud storage, and a downloadable link is provided for the user to easily access.

[0553] "Means for shooting video" refers to the functions or devices that allow users to record video using digital devices.

[0554] "Means for transcribing audio from a video" refers to a function or device that uses voice recognition technology to convert audio from a video into text data.

[0555] A "means for summarizing transcribed text" is a function or device that uses natural language processing technology to extract important information from long pieces of text and summarize it concisely.

[0556] "Means for extracting important images from a video" refers to a function or device that uses computer vision technology to select specific scenes or images from a video.

[0557] A "means for analyzing user emotions" is a function or device that uses machine learning models or algorithms based on audio and video data to determine a user's emotional state.

[0558] The "means for identifying important scenes based on analyzed emotional information" refers to a function or device that, based on the results of the user's emotional analysis, prioritizes extracting scenes that the user finds particularly interesting or difficult.

[0559] The "means for generating a manual by combining summarized text and extracted images" refers to a function or device that integrates summarized text with important images to create a manual in a visually easy-to-understand format.

[0560] The "means for providing the generated manual to the user" refers to a function or device that provides the user with an access link to the manual stored in cloud storage, allowing the user to download or refer to the manual.

[0561] The system based on this invention is designed to efficiently generate manuals from videos taken by users, and by combining it with an emotion engine that analyzes the user's emotions, it provides more appropriate manuals. The basic configuration and operation of this system are described below.

[0562] Video recording and audio description

[0563] The user uses a device such as a smartphone or smart glasses to record a video of the work being done. While recording, the user verbally explains the work. For example, while assembling furniture, the user explains each step, giving specific instructions such as "Use these screws to connect the boards."

[0564] Video and audio uploads

[0565] After the recording is complete, the device uploads the video and audio data to the server. The data is encrypted to ensure security. The device transmits the video and audio data to the server using the SSL / TLS protocol.

[0566] Audio transcription

[0567] The server transcribes the received voice data using automatic speech recognition (ASR) technology, such as Google Cloud Speech-to-Text or IBM Watson Speech to Text, and converts the voice data into text, recording each step and explanation.

[0568] Recognizing user emotions

[0569] The server uses an emotion engine to analyze emotions from the user's voice and facial expressions. For example, it uses the Microsoft Azure Emotion API to determine the user's emotions from the tone, intonation, and facial expressions of the voice. The server then sends the analyzed voice data and video data to the emotion engine for emotion analysis.

[0570] Text summary

[0571] The transcribed text is analyzed and summarized by a natural language processing (NLP) engine on the server. Redundant information is removed, leaving only the key points. This process uses techniques such as OpenAI GPT-3 and BERT. The server then stores the generated summary text in a database.

[0572] Image extraction

[0573] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes. Using OpenCV, it identifies the moment when the user removes a part or connects it with a screw, and cuts them out as still images.

[0574] Identifying important scenes based on emotions

[0575] Based on the user's emotional information recognized by the emotion engine, scenes that the user found particularly interesting or difficult are preferentially extracted. This improves the accuracy of extracting important scenes. Based on the emotion analysis results, the server preferentially selects still images that match scenes that the user found difficult.

[0576] Automatic manual generation

[0577] The server uses the extracted still images and summarized text to create a manual using a manual generation engine. It uses a combination of LaTeX, Markdown, and Pandoc to generate the manual in PDF or HTML format. The generated document is saved in the specified format.

[0578] Manual distribution

[0579] The generated manual is stored by the server in cloud storage (e.g., Amazon S3 or Google Drive). The user is provided with a link to download the manual from their device. The server uploads the generated manual to cloud storage and generates a link. The user can download the manual from the link in the app.

[0580] Specific examples

[0581] For example, this system could be used when a user is assembling new furniture at home. The user uses their smartphone to record video of the work process while verbally explaining the steps and important points. The video and audio data are uploaded to a server, and the audio is transcribed using ASR technology. The text is then summarized using an NLP engine, and important scenes are extracted using CV technology. An emotion engine identifies confusing moments and prioritizes the extraction of still images from those moments. Finally, the summarized text and still images are combined to generate a concise manual. The user can retrieve this manual from cloud storage and efficiently obtain the information they need.

[0582] Prompt Sentence Examples

[0583] Here are some examples of prompts to input to a generative AI model:

[0584] Design a program that automatically generates a manual for assembling furniture based on a video taken by a user, following the steps below:

[0585] 1. Uploading video and audio data

[0586] 2. Audio transcription (using ASR technology)

[0587] 3. User sentiment analysis (using emotion engine)

[0588] 4. Text Summarization (using NLP techniques)

[0589] 5. Extraction of important scenes from videos (using CV technology)

[0590] 6. Generating a manual based on the extracted scenes and summary text (using a manual generation engine)

[0591] 7. Save the manual to the cloud and provide a link to the user

[0592] Specifically, this involves using technologies such as Google Cloud Speech-to-Text, Microsoft Azure Emotion API, OpenAI GPT-3, OpenCV, LaTeX, and Amazon S3.

[0593] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0594] Step 1:

[0595] The user records a video of their work using a device such as a smartphone or smart glasses. The input is the user's work and an explanation of it, and the output is video data. The user opens a camera app and presses the record button to start recording. As the user works, they provide appropriate explanations.

[0596] Step 2:

[0597] After the recording is complete, the device uploads the video and audio data to the server. The input is the recorded video data, and the output is the video and audio data sent to the server. The device encrypts the data using the SSL / TLS protocol and sends it securely to the server.

[0598] Step 3:

[0599] The server transcribes the received voice data using automatic speech recognition (ASR) technology. The input is voice data and the output is text data. The server sends the voice data to an ASR engine (e.g., Google Cloud Speech-to-Text) and converts the voice into text format.

[0600] Step 4:

[0601] The server uses an emotion engine to analyze emotions from the user's voice and facial expressions. The input is audio and video data, and the output is the emotion analysis results. The server sends the audio and video to the emotion engine (for example, a general emotion analysis API) for emotion analysis.

[0602] Step 5:

[0603] The transcribed text is analyzed and summarized by the server's natural language processing (NLP) engine. The input is the transcribed text, and the output is the summary text. The server inputs the text data into an NLP engine (e.g., OpenAI GPT-3 or BERT) to generate a summary.

[0604] Step 6:

[0605] The server analyzes video data using computer vision (CV) technology and automatically extracts important scenes. The input is video data and the output is important still images. The server analyzes video frames every second and uses CV technology (e.g., OpenCV) to detect specific actions and extract them as still images.

[0606] Step 7:

[0607] Based on the user's emotional information recognized by the emotion engine, scenes that are of particular interest or difficulty are extracted with priority. The input is the emotion analysis results and still images, and the output is a prioritized list of important still images. The server compares the emotion analysis results with the still images and prioritizes saving scenes that the user found confusing.

[0608] Step 8:

[0609] The server uses the extracted still images and summarized text to create a manual using a manual generation engine. The input is the summary text and still images, and the output is a manual in PDF or HTML format. The server uses a combination of LaTeX, Markdown, and Pandoc to generate a visually easy-to-understand manual.

[0610] Step 9:

[0611] The generated manual is stored in cloud storage by the server. The input is the generated manual, and the output is an access link to the manual on the cloud. The server uploads the manual to cloud storage (e.g., Amazon S3 or Google Drive) and provides the user with a downloadable link.

[0612] (Application example 2)

[0613] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] In modern factories, creating work procedure manuals is time-consuming and labor-intensive, and relies on monotonous, impersonal information, making it difficult for workers to understand and put into practice. Furthermore, because the feelings and difficulties of individual workers are not taken into consideration, efficiency is often low. Therefore, there is a demand for shortening work times and reducing errors, but current manual generation methods have limitations.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0616] In this invention, the server includes: [means for shooting video;] [means for transcribing audio from the shot video;] [means for summarizing the transcribed text;] [means for extracting important images from the video;] [means for generating a manual by combining the summarized text and extracted images;] [means for providing the generated manual to the user;] [means for analyzing the user's emotions from their voice and facial expressions;] [means for identifying important scenes based on the user's emotional information; and [means for recording and analyzing work performed by the robot.] This enables the user to generate more understandable and effective manuals that take into account the emotions and difficulties of each work procedure.

[0617] "Means for taking videos" refers to a function that allows a user to use a device such as a smartphone or robot to record the work situation and procedures as video.

[0618] "Means for transcribing audio from video footage" refers to a function that converts audio data from video footage into text data using automatic speech recognition technology.

[0619] "Means for summarizing transcribed text" refers to the function of extracting important points from transcribed text data and summarizing them in a concise form. This function uses natural language processing technology.

[0620] "Means for extracting important images from video" refers to a function that automatically selects specific scenes or important moments from captured video data and extracts them as still images. This uses computer vision technology.

[0621] The "means for generating a manual by combining summarized text and extracted images" is a function that automatically creates a visually easy-to-understand manual by combining summarized text data with extracted still images.

[0622] The "means for providing the generated manual to the user" is a function for storing the generated manual in cloud storage and providing a link so that the user can download or access the manual.

[0623] "Means for analyzing user emotions from voice and facial expressions" refers to a function that identifies and analyzes the user's emotional state from the tone of their voice and facial expressions in the video. This is done using an emotion engine.

[0624] The "means for identifying important scenes based on the user's emotional information" is a function that prioritizes extracting scenes that the user is particularly interested in or finds difficult, based on data obtained through emotional analysis.

[0625] "Means for recording and analyzing work performed by robots" refers to a function that records and records the various tasks performed by factory robots, analyzes the data, and generates manuals.

[0626] To implement this invention, it is necessary to configure a system that includes the following main means: The system has means for shooting video, transcribing audio, summarizing text, extracting important images, generating and providing manuals, analyzing user emotions, and recording and analyzing work performed by a robot.

[0627] The server provides a means for users to record video of their work through their devices, such as smartphones or factory robots. The user can record video and provide verbal instructions while working, and the video and audio data are stored on the device. This data is then encrypted and uploaded to a cloud-based server.

[0628] The server automatically transcribes the uploaded audio data using the Google Cloud Speech API, converting it into text so that its content can be understood. The text data is then summarized using natural language processing techniques from Hugging Face's Transformers library. The summarized text is stripped of redundant information and focuses on the key points.

[0629] The server then analyzes the video data using OpenCV to extract specific scenes and important moments. This allows the scenes and steps the user considered particularly important during the task to be extracted as images. The server then analyzes the user's emotions using a TensorFlow model. The server determines the user's emotional state based on their voice tone and facial expressions, and based on this information, prioritizes the extraction of scenes in which the user found the task particularly challenging or interesting.

[0630] By combining these summary texts with extracted images of important scenes, the manual generation engine automatically creates a visually concise manual. The generated manual is saved in cloud storage in PDF or HTML format, and users can obtain and share the manual via a downloadable link.

[0631] (Example)

[0632] For example, when a factory worker records a robotic maintenance procedure for a new machine, they film the robot performing the task. This video and the audio data within it are uploaded to a server, where they are transcribed and summarized using automatic speech recognition and natural language processing technologies. Furthermore, computer vision technology extracts key scenes, and emotion analysis identifies parts where the user had difficulty. Finally, a manual summarizing the key points is generated.

[0633] (Example of a prompt)

[0634] "Explain the machine maintenance procedure, record each step on video, highlight the key points and generate a manual."

[0635] "Record a video of the assembly process and generate a summary manual based on the user's voice description and sentiment analysis."

[0636] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0637] Step 1:

[0638] Users use devices such as smartphones or factory robots to record video of work situations and procedures. While recording the video, the user verbally explains each step, and video data (video and audio) is input into the device.

[0639] Step 2:

[0640] The device uploads the recorded video data to a cloud-based server. During the upload process, the video data is encrypted and processed to ensure security. The encrypted video data is input to the server as an output.

[0641] Step 3:

[0642] The server automatically transcribes the audio data in the video using the Google Cloud Speech API. This process converts the audio data into text data. The input is audio data, and the output is transcribed text data.

[0643] Step 4:

[0644] The server uses Hugging Face's Transformers library to summarize the transcribed text data. Summarization extracts key points from the text and appropriately compresses the amount of information. The input is the transcribed text data, and the output is the summarized text data.

[0645] Step 5:

[0646] The server uses OpenCV to extract important image scenes from video data. It identifies specific frames and scenes and extracts them as still images. The input is video data, and the output is a set of extracted still images.

[0647] Step 6:

[0648] The server uses a TensorFlow model to analyze emotions from the user's voice tone and facial expressions in the video. Emotion analysis identifies the emotional state, such as difficulty or joy, felt by the user at each step. The input is audio and video data, and the output is the emotion analysis results.

[0649] Step 7:

[0650] Based on the emotion analysis results, the server prioritizes and extracts scenes that the user finds particularly difficult or that the user deems important. This selection process identifies still images with high importance. The input is the emotion analysis results and the extracted still images, and the output is the prioritized still images.

[0651] Step 8:

[0652] The server combines the summarized text data with a set of selected still images to automatically generate a manual. The generated manual is visually easy to understand, emphasizing important procedures and key points. The input is the summarized text data and a set of selected still images, and the output is the generated manual (in PDF or HTML format).

[0653] Step 9:

[0654] The server stores the generated manual in cloud storage and provides a download link to the user, through which the user can access the generated manual and share it as needed. The input is the generated manual, and the output is the download link.

[0655] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0656] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0657] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0658] [Third embodiment]

[0659] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0660] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0661] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0662] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0663] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0664] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0665] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0666] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0667] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0668] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0669] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0670] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0671] The system based on this invention allows users to film their work and then uses AI technology to efficiently generate manuals from the video. The basic configuration and operation of this system are described below.

[0672] Video recording and audio description

[0673] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, the user explains each step while assembling furniture.

[0674] Video and audio uploads

[0675] After the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted to ensure security.

[0676] Audio transcription

[0677] The server transcribes the received audio data using automatic speech recognition (ASR) technology, converting the audio data into text and recording any specific instructions or instructions contained within.

[0678] Text summary

[0679] The server's natural language processing (NLP) engine analyzes and summarizes the transcribed text, eliminating redundant information and leaving only the key points. This summary text focuses on the key steps or actions the user needs to take.

[0680] Image extraction

[0681] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes, such as when a user removes a part or screws it together, and then cuts out those scenes as still images.

[0682] Automatic manual generation

[0683] Using the extracted still images and summarized text, the server uses a manual generation engine to create a manual, which is generated in PDF or HTML format and provided in a visually easy-to-understand format.

[0684] Manual distribution

[0685] The generated manual is stored in cloud storage by the server, and the device provides the manual as a downloadable link, which users can use to obtain and share the manual.

[0686] Specific examples

[0687] For example, if a user is assembling new furniture at home, they can use their smartphone to film the process. They can explain the steps as they go along, and verbally mention important points about screws and parts. After filming is complete, the video and audio data are uploaded to a server, where ASR technology transcribes the audio. An NLP engine then summarizes the text, and CV technology extracts key scenes. Finally, the summarized text and still images are combined to generate a concise visual manual, which can be accessed by the user via cloud storage.

[0688] Such a system will eliminate the problems associated with conventional video manuals and allow users to efficiently obtain the information they need.

[0689] The processing flow will be explained below.

[0690] Step 1:

[0691] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, when removing a part, the user may say, "Remove part A."

[0692] Step 2:

[0693] Once the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted before being sent.

[0694] Step 3:

[0695] The server receives the uploaded video and audio data, temporarily stores the received data, and prepares it for transfer to the next process.

[0696] Step 4:

[0697] The server extracts the audio data from the video file, which is then passed to an automatic speech recognition (ASR) engine.

[0698] Step 5:

[0699] The ASR engine analyzes the voice data and transcribes it. For example, "Remove part A" is generated as text.

[0700] Step 6:

[0701] The server sends the transcribed text to a natural language processing (NLP) engine for analysis and summarization, eliminating redundancies and extracting only the important information.

[0702] Step 7:

[0703] The server passes the video data to a computer vision (CV) engine, which identifies important scenes. For example, it extracts frames of a scene where part A is being picked up and held up to the camera.

[0704] Step 8:

[0705] The CV engine generates still images from the identified scenes. The generated still images (e.g., an image of the moment part A is removed) are saved as material for creating manuals.

[0706] Step 9:

[0707] The server combines the summarized text and still images based on a template, and the manual generation engine uses the template to automatically generate a manual in PDF or HTML format.

[0708] Step 10:

[0709] The server saves the generated manual in cloud storage. Once the upload to the storage is complete, a download link is generated.

[0710] Step 11:

[0711] The terminal notifies the user of the download link for the generated manual, which the user can use to download, view, or share the manual.

[0712] In this way, the user can efficiently and quickly create a manual that combines still images and summary text from the video that he or she has taken.

[0713] Example 1

[0714] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0715] Conventional manual generation methods have the problem of requiring a great deal of time and effort to describe work procedures in detail using text and images. Furthermore, methods that utilize video require users to spend a long time watching the video, making it difficult to search and organize information. This reduces the user's work efficiency and makes it difficult for them to confirm accurate procedures.

[0716] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0717] In this invention, the server includes means for a user to record a video of the work, means for uploading the recorded video and audio data to the server, means for the server to transcribe the audio data using automatic speech recognition technology, means for summarizing the transcribed text using natural language processing technology, means for extracting important scenes from the video data using computer vision technology, means for combining the summarized text and extracted images to generate a manual, and means for storing the generated manual in cloud storage and providing it to the user, thereby enabling the user to efficiently obtain the necessary information and accurately check the work procedures.

[0718] "User" refers to a person who uses the system to record video of their work and obtain or refer to the manual.

[0719] "Device" refers to the hardware device a user uses to capture video and upload the data to a server. Examples include smartphones and smart glasses.

[0720] The "server" refers to a system that receives data uploaded by users, transcribes audio, summarizes text, extracts images, and ultimately generates and provides manuals.

[0721] "Audio data" refers to digital information of the voice recorded when the user explains the task.

[0722] "Automatic speech recognition technology (ASR)" refers to technology for converting voice data into text data.

[0723] "Transcription" refers to the process of converting audio data into text form.

[0724] "Natural language processing technology (NLP)" refers to the technology of analyzing transcribed text and extracting summaries and meanings.

[0725] A "summary" is a short text that removes redundant parts from transcribed text and summarizes only the important points.

[0726] "Computer vision technology (CV)" refers to technology that automatically extracts and analyzes specific information from images and videos.

[0727] An "important scene" refers to a particularly important part or moment in a video, and refers to a scene that contains information that a user needs.

[0728] A "still image" refers to an individual frame extracted from a video, meaning an image that captures a specific moment in time.

[0729] "Manual generation" refers to the process of combining summarized text with extracted still images to create a document containing instructions and instructions.

[0730] "Cloud storage" refers to remote servers that store and make data accessible over the internet.

[0731] The system based on this invention allows users to take videos of their work and then uses AI technology to efficiently generate manuals from those videos. The basic configuration and operation of this system are described below.

[0732] Video recording and audio description

[0733] The user uses a device such as a smartphone or smart glasses to record a video of the work being done. While recording, the user verbally explains the work. For example, while assembling furniture, the user might explain the detailed steps, such as "Install this screw in this position. Next, insert this part into this hole."

[0734] Video and audio uploads

[0735] After the recording is complete, the device uploads the video and audio data to the server. The data is encrypted before transmission to ensure security. Specifically, the data is encrypted using the SSL / TLS protocol.

[0736] Audio transcription

[0737] The server transcribes the received voice data using automatic speech recognition (ASR). For example, it uses the Google Cloud Speech-to-Text API to convert the voice data into text. The text data includes steps such as "1. Take out the screw. 2. Insert the screw into the screw hole."

[0738] Text summary

[0739] The server analyzes and summarizes the transcribed text using natural language processing (NLP) techniques such as OpenAI's GPT model. This removes redundant information and leaves only the key points. For example, the summary text might include concise steps such as "1. Remove the screw. 2. Insert the screw into the screw hole."

[0740] Image extraction

[0741] The server analyzes the video data using computer vision (CV) technologies such as OpenCV and automatically extracts important scenes, such as when a user removes a part or screws it together, and cuts out the scene as a still image.

[0742] Automatic manual generation

[0743] The server generates a manual using the summarized text and extracted still images. Specifically, it uses a manual generation engine such as LaTeX or Pandoc. The generated manual is provided in PDF or HTML format. This results in a visually easy-to-understand manual.

[0744] Manual distribution

[0745] The generated manual is stored in cloud storage (e.g., Amazon S3, Google Drive, etc.) by the server, and a link to the cloud storage is provided to the user, who can download or view the manual through the link.

[0746] Specific examples

[0747] As a concrete example, consider a user assembling new furniture at home. The user uses their smartphone to record a video of the work process and verbally explain the steps, such as "Use screw A here to attach board B to board C." After completing the recording, the user uploads the video and audio data to a server. The server transcribes the audio data using the Google Cloud Speech-to-Text API and summarizes it using OpenAI's GPT model. It also uses OpenCV to extract key scenes. Finally, the server combines the summarized text with the extracted still images to generate a concise visual manual. This manual is stored in cloud storage and a link is provided to the user.

[0748] Prompt Sentence Examples

[0749] "This video contains instructions for assembling furniture. Please follow the instructions verbally as you go through the process. Upload the video and audio data and generate a manual."

[0750] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0751] Step 1:

[0752] The user takes a video of the work they are doing. The work is recorded using a smartphone or smart glasses. For example, if the user is assembling furniture, they will explain each step as they go along. This video will record all of the steps.

[0753] Input: Video and audio of your work.

[0754] Output: Video file (including video and audio).

[0755] Step 2:

[0756] The device uploads the captured video and audio data to a server. Using a dedicated application, the data is encrypted using SSL / TLS and sent to the server.

[0757] Input: Video file.

[0758] Output: Video file saved on the server.

[0759] Step 3:

[0760] The server transcribes the received audio data using automatic speech recognition (ASR) technology and converts the audio into text using the Google Cloud Speech-to-Text API.

[0761] Input: Audio data from the video.

[0762] Output: Transcribed text data.

[0763] Step 4:

[0764] The server summarizes the transcribed text using natural language processing (NLP) techniques, using OpenAI's GPT model to extract important information from the text and remove redundancies.

[0765] Input: Transcribed text data.

[0766] Output: Summarized text data.

[0767] Step 5:

[0768] The server uses computer vision (CV) technology to extract important scenes from the video data. Using OpenCV, it identifies important scenes in the video and extracts them as still images. For example, this includes the moment when a user removes a part or tightens a screw.

[0769] Input: Video file.

[0770] Output: Still image files of important scenes.

[0771] Step 6:

[0772] The server generates a manual by combining the summarized text and extracted still images. Using LaTeX and Pandoc, the manual is created in PDF or HTML format for easy visual understanding.

[0773] Input: Abstracted text data, still image files.

[0774] Output: The generated manual (PDF or HTML format).

[0775] Step 7:

[0776] The server saves the generated manuals in cloud storage and provides them to users. It saves them using Amazon S3 or Google Drive and creates a download link.

[0777] Input: Generated manual.

[0778] Output: A download link that users can access.

[0779] (Application example 1)

[0780] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0781] In modern factories, it is important to efficiently record robot maintenance procedures and create maintenance manuals that can be reproduced by other technicians. However, traditional methods require manual recording and manual creation, which is time-consuming and labor-intensive. It is also difficult to record all detailed procedures and precautions, which can lead to inconsistent quality.

[0782] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0783] In this invention, the server includes means for shooting video, means for transcribing audio from the shot video, means for summarizing the transcribed text, means for extracting important images from the video, means for generating a manual by combining the summarized text and the extracted images, means for providing the generated manual to a user, and means for recording robot maintenance procedures as video and automatically generating a maintenance manual. This makes it possible to automatically and efficiently generate high-quality, reproducible maintenance manuals from videos recorded by technicians during work.

[0784] "Means for capturing video" refers to a device and method that allows a user to record the state of work using a terminal such as a smartphone or smart glasses.

[0785] A "means for transcribing audio from a video" is a device and method that uses automatic speech recognition (ASR) technology to convert audio data from a video into text data.

[0786] The "means for summarizing transcribed text" refers to an apparatus and method that uses a natural language processing (NLP) engine to remove redundant parts of transcribed text and extract key points.

[0787] The "means for extracting important images from video" refers to an apparatus and method that uses computer vision (CV) technology to identify important scenes from video data and extract them as still images.

[0788] The "means for generating a manual by combining summarized text and extracted images" refers to an apparatus and method for combining summarized text and extracted images to generate a visually easy-to-understand manual.

[0789] The "means for providing the generated manual to the user" refers to an apparatus and method for providing the generated manual to the user in a form that is easily accessible to the user via cloud storage or a download link.

[0790] The "means for recording robot maintenance procedures on video and automatically generating a maintenance manual" refers to a series of devices and methods for recording factory robot maintenance work on video and automatically creating a maintenance manual from the video.

[0791] To implement this invention, a user uses a device such as a smartphone or smart glasses to record video of the work they are doing. The user then verbally explains the necessary steps and key points during the work, and simultaneously records this as audio data. After the recording is complete, the device uploads the video and audio data to a server. The data is encrypted to ensure security.

[0792] The server converts the audio data into text using automatic speech recognition (ASR) technology, specifically Google Cloud Speech-to-Text. The converted text is then summarized using a natural language processing (NLP) engine, such as the generative AI model GPT-3. The summarized text focuses on important steps and explanations and eliminates redundant information.

[0793] Furthermore, the server uses computer vision (CV) technology to extract important scenes from the video data. The specific software used for this is OpenCV. The extracted important scenes are cut out as still images, and it is made clear which steps each scene corresponds to.

[0794] The server uses a manual generation engine to generate a maintenance manual by combining the summarized text and extracted images. The manual is generated in PDF or HTML format and provided in an easy-to-understand format. The generated manual is stored in cloud storage (e.g., AWS S3), and users can obtain the manual as a downloadable link.

[0795] As a concrete example, consider a scenario in which a technician performing maintenance work on a robot in a factory records a video of the maintenance procedure on a smartphone. The technician explains each step as he or she works. The video and audio data are then uploaded to a server, where the audio is transcribed using ASR technology. An NLP engine summarizes the text, and CV technology extracts important scenes. The generated manual can then be accessed by the technician via cloud storage, and can also be shared with other technicians.

[0796] Examples of prompts that can be used include:

[0797] Summarize the robot maintenance procedures below:

[0798] 1. Remove the screws

[0799] 2. Remove the parts

[0800] 3. Install the new parts

[0801] 4. Retighten the screws

[0802] 5. Check the operation

[0803] summary:

[0804] The present invention allows users to easily generate efficient and high-quality maintenance manuals.

[0805] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0806] Step 1:

[0807] The user uses a device such as a smartphone or smart glasses to record a video of the work being done, and at the same time verbally explains the work content. The input is the work the user is doing and an explanation of it, and the output is video data and audio data. The specific operations are to launch the camera app, start recording, and verbally explain the work procedures and important points.

[0808] Step 2:

[0809] The device uploads the captured video and audio data to the server. At this time, the data is encrypted to ensure security. The input is the video and audio data stored on the device, and the output is the encrypted data stored on the server. Specifically, the upload application is used to transfer the data to the server.

[0810] Step 3:

[0811] The server converts the voice data into text using automatic speech recognition (ASR) technology. The software used is Google Cloud Speech-to-Text. The input is encrypted voice data, and the output is transcribed text data. Specifically, the server decrypts the voice data and uses ASR technology to convert the voice into text format.

[0812] Step 4:

[0813] The server uses a natural language processing (NLP) engine to summarize the transcribed text. The software used is a generative AI model such as GPT-3. The input is the transcribed text data, and the output is the summarized text data. Specifically, the text data is input into the NLP engine, and a summary is generated that focuses on important steps and explanations.

[0814] Step 5:

[0815] The server uses computer vision (CV) technology to extract important scenes from video data. The software used is OpenCV. The input is video data, and the output is extracted still image data. Specifically, it analyzes the video data and extracts important scenes (such as the moment a part is removed or a screw is fastened) as still images.

[0816] Step 6:

[0817] The server generates a manual by combining the summarized text and extracted images. The input is the summarized text data and extracted still image data, and the output is the manual data (PDF or HTML format). Specifically, it combines the summarized text and still images to determine the layout and generate a visually easy-to-understand manual.

[0818] Step 7:

[0819] The generated manual is stored in cloud storage and a link is provided for users to download it. The input is the manual data, and the output is the manual stored on the cloud and its download link. The specific operation is to upload the manual to a cloud storage service (e.g., AWS S3) and provide the link to the user.

[0820] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0821] The system based on this invention is designed to efficiently generate manuals from videos taken by users, and by combining it with an emotion engine that analyzes the user's emotions, it provides more appropriate manuals. The basic configuration and operation of this system are described below.

[0822] Video recording and audio description

[0823] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, the user explains each step while assembling furniture.

[0824] Video and audio uploads

[0825] After the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted to ensure security.

[0826] Audio transcription

[0827] The server transcribes the received audio data using automatic speech recognition (ASR) technology, converting the audio data into text and recording any specific instructions or instructions contained within.

[0828] Recognizing user emotions

[0829] The server uses an emotion engine to analyze the user's emotions from their voice and facial expressions, for example, determining whether they are confused or progressing smoothly based on their tone and intonation.

[0830] Text summary

[0831] The server's natural language processing (NLP) engine analyzes and summarizes the transcribed text, eliminating redundant information and leaving only the key points. This summary text focuses on the key steps or actions the user needs to take.

[0832] Image extraction

[0833] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes, such as when a user removes a part or screws it together, and then cuts out those scenes as still images.

[0834] Identifying important scenes based on emotions

[0835] Based on the user's emotional information recognized by the emotion engine, scenes that the user found particularly interesting or difficult are preferentially extracted, thereby improving the accuracy of extracting important scenes.

[0836] Automatic manual generation

[0837] Using the extracted still images and summarized text, the server uses a manual generation engine to create a manual, which is generated in PDF or HTML format and provided in a visually easy-to-understand format.

[0838] Manual distribution

[0839] The generated manual is stored in cloud storage by the server, and the device provides the manual as a downloadable link, which users can use to obtain and share the manual.

[0840] Specific examples

[0841] For example, if a user is assembling new furniture at home, they can use their smartphone to record a video of the work process. The steps are explained as they go along, with important points about screws and parts being mentioned verbally. After filming is complete, the video and audio data are uploaded to a server, where the audio is transcribed using ASR technology. An NLP engine then summarizes the text, and CV technology extracts key scenes. An emotion engine identifies situations where the user is confused and prioritizes the extraction of still images from those scenes. Finally, the summarized text and still images are combined to generate a concise visual manual, which can be accessed by the user via cloud storage. In this way, users can efficiently obtain the information they need and receive focused support on areas they are struggling with.

[0842] The processing flow will be explained below.

[0843] Step 1:

[0844] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, while assembling furniture, the user might explain, "First, remove part A."

[0845] Step 2:

[0846] Once the recording is complete, the device uploads the video and audio data to a server, where it is encrypted and securely transmitted.

[0847] Step 3:

[0848] The server receives the uploaded video and audio data, verifies the integrity of the files, and then begins the next process.

[0849] Step 4:

[0850] The server extracts the audio data from the video file, which is then passed to an automatic speech recognition (ASR) engine.

[0851] Step 5:

[0852] The ASR engine analyzes the voice data and transcribes it. The resulting transcription is "Remove part A."

[0853] Step 6:

[0854] The server sends the transcribed text to a natural language processing (NLP) engine for analysis and summarization. For example, a concise summary such as "Extract part A" is generated.

[0855] Step 7:

[0856] The server uses an emotion engine to analyze the user's voice and facial expressions, for example, to determine whether the user is confused or confused based on changes in voice tone or facial expression.

[0857] Step 8:

[0858] The server passes the video data to a computer vision (CV) engine to identify important scenes, such as when the user is confused or when important operations are performed.

[0859] Step 9:

[0860] Still images are generated from the scenes identified by the CV engine. Important work scenes and scenes that the user was interested in are saved as still images (e.g., the moment part A is removed).

[0861] Step 10:

[0862] The server combines the summarized text and extracted still images based on a template, and the manual generation engine uses the template to automatically generate a manual in PDF or HTML format.

[0863] Step 11:

[0864] The server saves the generated manual to cloud storage. Once the save is complete, a download link is generated and sent to the device.

[0865] Step 12:

[0866] The terminal notifies the user of the download link for the generated manual, which the user can use to download, view, and share the manual.

[0867] In this way, users can efficiently and quickly create manuals that combine still images and summary text from captured videos. Furthermore, the emotion engine prioritizes support for scenes where users find it difficult.

[0868] Example 2

[0869] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0870] Conventional manual generation systems often fail to meet actual needs because they do not consider the user's emotions or level of difficulty, even when using video or audio data. Furthermore, they sometimes fail to properly identify important scenes or summarize text, resulting in manuals that are difficult for users to understand. Furthermore, there are issues with the method of distributing the generated manuals, making them difficult for users to access.

[0871] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for shooting a video, a means for transcribing audio from the shot video, a means for summarizing the transcribed text, a means for extracting important images from the video, a means for analyzing a user's emotions, a means for identifying important scenes based on the analyzed emotional information, a means for generating a manual by combining the summarized text and the extracted images, and a means for providing the generated manual to the user. This makes it possible to take the user's emotions into consideration, appropriately extract important scenes, and combine them with the summarized text, thereby providing a manual that is easy for the user to understand and meets their needs. The generated manual is also stored in cloud storage, and a downloadable link is provided for the user to easily access.

[0872] "Means for shooting video" refers to the functions or devices that allow users to record video using digital devices.

[0873] "Means for transcribing audio from a video" refers to a function or device that uses voice recognition technology to convert audio from a video into text data.

[0874] A "means for summarizing transcribed text" is a function or device that uses natural language processing technology to extract important information from long pieces of text and summarize it concisely.

[0875] "Means for extracting important images from a video" refers to a function or device that uses computer vision technology to select specific scenes or images from a video.

[0876] A "means for analyzing user emotions" is a function or device that uses machine learning models or algorithms based on audio and video data to determine a user's emotional state.

[0877] The "means for identifying important scenes based on analyzed emotional information" refers to a function or device that, based on the results of the user's emotional analysis, prioritizes extracting scenes that the user finds particularly interesting or difficult.

[0878] The "means for generating a manual by combining summarized text and extracted images" refers to a function or device that integrates summarized text with important images to create a manual in a visually easy-to-understand format.

[0879] The "means for providing the generated manual to the user" refers to a function or device that provides the user with an access link to the manual stored in cloud storage, allowing the user to download or refer to the manual.

[0880] The system based on this invention is designed to efficiently generate manuals from videos taken by users, and by combining it with an emotion engine that analyzes the user's emotions, it provides more appropriate manuals. The basic configuration and operation of this system are described below.

[0881] Video recording and audio description

[0882] The user uses a device such as a smartphone or smart glasses to record a video of the work being done. While recording, the user verbally explains the work. For example, while assembling furniture, the user explains each step, giving specific instructions such as "Use these screws to connect the boards."

[0883] Video and audio uploads

[0884] After the recording is complete, the device uploads the video and audio data to the server. The data is encrypted to ensure security. The device transmits the video and audio data to the server using the SSL / TLS protocol.

[0885] Audio transcription

[0886] The server transcribes the received voice data using automatic speech recognition (ASR) technology, such as Google Cloud Speech-to-Text or IBM Watson Speech to Text, and converts the voice data into text, recording each step and explanation.

[0887] Recognizing user emotions

[0888] The server uses an emotion engine to analyze emotions from the user's voice and facial expressions. For example, it uses the Microsoft Azure Emotion API to determine the user's emotions from the tone, intonation, and facial expressions of the voice. The server then sends the analyzed voice data and video data to the emotion engine for emotion analysis.

[0889] Text summary

[0890] The transcribed text is analyzed and summarized by a natural language processing (NLP) engine on the server. Redundant information is removed, leaving only the key points. This process uses techniques such as OpenAI GPT-3 and BERT. The server then stores the generated summary text in a database.

[0891] Image extraction

[0892] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes. Using OpenCV, it identifies the moment when the user removes a part or connects it with a screw, and cuts them out as still images.

[0893] Identifying important scenes based on emotions

[0894] Based on the user's emotional information recognized by the emotion engine, scenes that the user found particularly interesting or difficult are preferentially extracted. This improves the accuracy of extracting important scenes. Based on the emotion analysis results, the server preferentially selects still images that match scenes that the user found difficult.

[0895] Automatic manual generation

[0896] The server uses the extracted still images and summarized text to create a manual using a manual generation engine. It uses a combination of LaTeX, Markdown, and Pandoc to generate the manual in PDF or HTML format. The generated document is saved in the specified format.

[0897] Manual distribution

[0898] The generated manual is stored by the server in cloud storage (e.g., Amazon S3 or Google Drive). The user is provided with a link to download the manual from their device. The server uploads the generated manual to cloud storage and generates a link. The user can download the manual from the link in the app.

[0899] Specific examples

[0900] For example, this system could be used when a user is assembling new furniture at home. The user uses their smartphone to record video of the work process while verbally explaining the steps and important points. The video and audio data are uploaded to a server, and the audio is transcribed using ASR technology. The text is then summarized using an NLP engine, and important scenes are extracted using CV technology. An emotion engine identifies confusing moments and prioritizes the extraction of still images from those moments. Finally, the summarized text and still images are combined to generate a concise manual. The user can retrieve this manual from cloud storage and efficiently obtain the information they need.

[0901] Prompt Sentence Examples

[0902] Here are some examples of prompts to input to a generative AI model:

[0903] Design a program that automatically generates a manual for assembling furniture based on a video taken by a user, following the steps below:

[0904] 1. Uploading video and audio data

[0905] 2. Audio transcription (using ASR technology)

[0906] 3. User sentiment analysis (using emotion engine)

[0907] 4. Text Summarization (using NLP techniques)

[0908] 5. Extraction of important scenes from videos (using CV technology)

[0909] 6. Generating a manual based on the extracted scenes and summary text (using a manual generation engine)

[0910] 7. Save the manual to the cloud and provide a link to the user

[0911] Specifically, this involves using technologies such as Google Cloud Speech-to-Text, Microsoft Azure Emotion API, OpenAI GPT-3, OpenCV, LaTeX, and Amazon S3.

[0912] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0913] Step 1:

[0914] The user records a video of their work using a device such as a smartphone or smart glasses. The input is the user's work and an explanation of it, and the output is video data. The user opens a camera app and presses the record button to start recording. As the user works, they provide appropriate explanations.

[0915] Step 2:

[0916] After the recording is complete, the device uploads the video and audio data to the server. The input is the recorded video data, and the output is the video and audio data sent to the server. The device encrypts the data using the SSL / TLS protocol and sends it securely to the server.

[0917] Step 3:

[0918] The server transcribes the received voice data using automatic speech recognition (ASR) technology. The input is voice data and the output is text data. The server sends the voice data to an ASR engine (e.g., Google Cloud Speech-to-Text) and converts the voice into text format.

[0919] Step 4:

[0920] The server uses an emotion engine to analyze emotions from the user's voice and facial expressions. The input is audio and video data, and the output is the emotion analysis results. The server sends the audio and video to the emotion engine (for example, a general emotion analysis API) for emotion analysis.

[0921] Step 5:

[0922] The transcribed text is analyzed and summarized by the server's natural language processing (NLP) engine. The input is the transcribed text, and the output is the summary text. The server inputs the text data into an NLP engine (e.g., OpenAI GPT-3 or BERT) to generate a summary.

[0923] Step 6:

[0924] The server analyzes video data using computer vision (CV) technology and automatically extracts important scenes. The input is video data and the output is important still images. The server analyzes video frames every second and uses CV technology (e.g., OpenCV) to detect specific actions and extract them as still images.

[0925] Step 7:

[0926] Based on the user's emotional information recognized by the emotion engine, scenes that are of particular interest or difficulty are extracted with priority. The input is the emotion analysis results and still images, and the output is a prioritized list of important still images. The server compares the emotion analysis results with the still images and prioritizes saving scenes that the user found confusing.

[0927] Step 8:

[0928] The server uses the extracted still images and summarized text to create a manual using a manual generation engine. The input is the summary text and still images, and the output is a manual in PDF or HTML format. The server uses a combination of LaTeX, Markdown, and Pandoc to generate a visually easy-to-understand manual.

[0929] Step 9:

[0930] The generated manual is stored in cloud storage by the server. The input is the generated manual, and the output is an access link to the manual on the cloud. The server uploads the manual to cloud storage (e.g., Amazon S3 or Google Drive) and provides the user with a downloadable link.

[0931] (Application example 2)

[0932] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0933] In modern factories, creating work procedure manuals is time-consuming and labor-intensive, and relies on monotonous, impersonal information, making it difficult for workers to understand and put into practice. Furthermore, because the feelings and difficulties of individual workers are not taken into consideration, efficiency is often low. Therefore, there is a demand for shortening work times and reducing errors, but current manual generation methods have limitations.

[0934] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0935] In this invention, the server includes: [means for shooting video;] [means for transcribing audio from the shot video;] [means for summarizing the transcribed text;] [means for extracting important images from the video;] [means for generating a manual by combining the summarized text and extracted images;] [means for providing the generated manual to the user;] [means for analyzing the user's emotions from their voice and facial expressions;] [means for identifying important scenes based on the user's emotional information; and [means for recording and analyzing work performed by the robot.] This enables the user to generate more understandable and effective manuals that take into account the emotions and difficulties of each work procedure.

[0936] "Means for taking videos" refers to a function that allows a user to use a device such as a smartphone or robot to record the work situation and procedures as video.

[0937] "Means for transcribing audio from video footage" refers to a function that converts audio data from video footage into text data using automatic speech recognition technology.

[0938] "Means for summarizing transcribed text" refers to the function of extracting important points from transcribed text data and summarizing them in a concise form. This function uses natural language processing technology.

[0939] "Means for extracting important images from video" refers to a function that automatically selects specific scenes or important moments from captured video data and extracts them as still images. This uses computer vision technology.

[0940] The "means for generating a manual by combining summarized text and extracted images" is a function that automatically creates a visually easy-to-understand manual by combining summarized text data with extracted still images.

[0941] The "means for providing the generated manual to the user" is a function for storing the generated manual in cloud storage and providing a link so that the user can download or access the manual.

[0942] "Means for analyzing user emotions from voice and facial expressions" refers to a function that identifies and analyzes the user's emotional state from the tone of their voice and facial expressions in the video. This is done using an emotion engine.

[0943] The "means for identifying important scenes based on the user's emotional information" is a function that prioritizes extracting scenes that the user is particularly interested in or finds difficult, based on data obtained through emotional analysis.

[0944] "Means for recording and analyzing work performed by robots" refers to a function that records and records the various tasks performed by factory robots, analyzes the data, and generates manuals.

[0945] To implement this invention, it is necessary to configure a system that includes the following main means: The system has means for shooting video, transcribing audio, summarizing text, extracting important images, generating and providing manuals, analyzing user emotions, and recording and analyzing work performed by a robot.

[0946] The server provides a means for users to record video of their work through their devices, such as smartphones or factory robots. The user can record video and provide verbal instructions while working, and the video and audio data are stored on the device. This data is then encrypted and uploaded to a cloud-based server.

[0947] The server automatically transcribes the uploaded audio data using the Google Cloud Speech API, converting it into text so that its content can be understood. The text data is then summarized using natural language processing techniques from Hugging Face's Transformers library. The summarized text is stripped of redundant information and focuses on the key points.

[0948] The server then analyzes the video data using OpenCV to extract specific scenes and important moments. This allows the scenes and steps the user considered particularly important during the task to be extracted as images. The server then analyzes the user's emotions using a TensorFlow model. The server determines the user's emotional state based on their voice tone and facial expressions, and based on this information, prioritizes the extraction of scenes in which the user found the task particularly challenging or interesting.

[0949] By combining these summary texts with extracted images of important scenes, the manual generation engine automatically creates a visually concise manual. The generated manual is saved in cloud storage in PDF or HTML format, and users can obtain and share the manual via a downloadable link.

[0950] (Example)

[0951] For example, when a factory worker records a robotic maintenance procedure for a new machine, they film the robot performing the task. This video and the audio data within it are uploaded to a server, where they are transcribed and summarized using automatic speech recognition and natural language processing technologies. Furthermore, computer vision technology extracts key scenes, and emotion analysis identifies parts where the user had difficulty. Finally, a manual summarizing the key points is generated.

[0952] (Example of a prompt)

[0953] "Explain the machine maintenance procedure, record each step on video, highlight the key points and generate a manual."

[0954] "Record a video of the assembly process and generate a summary manual based on the user's voice description and sentiment analysis."

[0955] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0956] Step 1:

[0957] Users use devices such as smartphones or factory robots to record video of work situations and procedures. While recording the video, the user verbally explains each step, and video data (video and audio) is input into the device.

[0958] Step 2:

[0959] The device uploads the recorded video data to a cloud-based server. During the upload process, the video data is encrypted and processed to ensure security. The encrypted video data is input to the server as an output.

[0960] Step 3:

[0961] The server automatically transcribes the audio data in the video using the Google Cloud Speech API. This process converts the audio data into text data. The input is audio data, and the output is transcribed text data.

[0962] Step 4:

[0963] The server uses Hugging Face's Transformers library to summarize the transcribed text data. Summarization extracts key points from the text and appropriately compresses the amount of information. The input is the transcribed text data, and the output is the summarized text data.

[0964] Step 5:

[0965] The server uses OpenCV to extract important image scenes from video data. It identifies specific frames and scenes and extracts them as still images. The input is video data, and the output is a set of extracted still images.

[0966] Step 6:

[0967] The server uses a TensorFlow model to analyze emotions from the user's voice tone and facial expressions in the video. Emotion analysis identifies the emotional state, such as difficulty or joy, felt by the user at each step. The input is audio and video data, and the output is the emotion analysis results.

[0968] Step 7:

[0969] Based on the emotion analysis results, the server prioritizes and extracts scenes that the user finds particularly difficult or that the user deems important. This selection process identifies still images with high importance. The input is the emotion analysis results and the extracted still images, and the output is the prioritized still images.

[0970] Step 8:

[0971] The server combines the summarized text data with a set of selected still images to automatically generate a manual. The generated manual is visually easy to understand, emphasizing important procedures and key points. The input is the summarized text data and a set of selected still images, and the output is the generated manual (in PDF or HTML format).

[0972] Step 9:

[0973] The server stores the generated manual in cloud storage and provides a download link to the user, through which the user can access the generated manual and share it as needed. The input is the generated manual, and the output is the download link.

[0974] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0975] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0976] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0977] [Fourth embodiment]

[0978] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0979] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0980] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0981] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0982] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0983] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0984] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0985] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0986] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0987] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0988] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0989] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0990] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0991] The system based on this invention allows users to film their work and then uses AI technology to efficiently generate manuals from the video. The basic configuration and operation of this system are described below.

[0992] Video recording and audio description

[0993] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, the user explains each step while assembling furniture.

[0994] Video and audio uploads

[0995] After the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted to ensure security.

[0996] Audio transcription

[0997] The server transcribes the received audio data using automatic speech recognition (ASR) technology, converting the audio data into text and recording any specific instructions or instructions contained within.

[0998] Text summary

[0999] The server's natural language processing (NLP) engine analyzes and summarizes the transcribed text, eliminating redundant information and leaving only the key points. This summary text focuses on the key steps or actions the user needs to take.

[1000] Image extraction

[1001] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes, such as when a user removes a part or screws it together, and then cuts out those scenes as still images.

[1002] Automatic manual generation

[1003] Using the extracted still images and summarized text, the server uses a manual generation engine to create a manual, which is generated in PDF or HTML format and provided in a visually easy-to-understand format.

[1004] Manual distribution

[1005] The generated manual is stored in cloud storage by the server, and the device provides the manual as a downloadable link, which users can use to obtain and share the manual.

[1006] Specific examples

[1007] For example, if a user is assembling new furniture at home, they can use their smartphone to film the process. They can explain the steps as they go along, and verbally mention important points about screws and parts. After filming is complete, the video and audio data are uploaded to a server, where ASR technology transcribes the audio. An NLP engine then summarizes the text, and CV technology extracts key scenes. Finally, the summarized text and still images are combined to generate a concise visual manual, which can be accessed by the user via cloud storage.

[1008] Such a system will eliminate the problems associated with conventional video manuals and allow users to efficiently obtain the information they need.

[1009] The processing flow will be explained below.

[1010] Step 1:

[1011] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, when removing a part, the user may say, "Remove part A."

[1012] Step 2:

[1013] Once the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted before being sent.

[1014] Step 3:

[1015] The server receives the uploaded video and audio data, temporarily stores the received data, and prepares it for transfer to the next process.

[1016] Step 4:

[1017] The server extracts the audio data from the video file, which is then passed to an automatic speech recognition (ASR) engine.

[1018] Step 5:

[1019] The ASR engine analyzes the voice data and transcribes it. For example, "Remove part A" is generated as text.

[1020] Step 6:

[1021] The server sends the transcribed text to a natural language processing (NLP) engine for analysis and summarization, eliminating redundancies and extracting only the important information.

[1022] Step 7:

[1023] The server passes the video data to a computer vision (CV) engine, which identifies important scenes. For example, it extracts frames of a scene where part A is being picked up and held up to the camera.

[1024] Step 8:

[1025] The CV engine generates still images from the identified scenes. The generated still images (e.g., an image of the moment part A is removed) are saved as material for creating manuals.

[1026] Step 9:

[1027] The server combines the summarized text and still images based on a template, and the manual generation engine uses the template to automatically generate a manual in PDF or HTML format.

[1028] Step 10:

[1029] The server saves the generated manual in cloud storage. Once the upload to the storage is complete, a download link is generated.

[1030] Step 11:

[1031] The terminal notifies the user of the download link for the generated manual, which the user can use to download, view, or share the manual.

[1032] In this way, the user can efficiently and quickly create a manual that combines still images and summary text from the video that he or she has taken.

[1033] Example 1

[1034] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1035] Conventional manual generation methods have the problem of requiring a great deal of time and effort to describe work procedures in detail using text and images. Furthermore, methods that utilize video require users to spend a long time watching the video, making it difficult to search and organize information. This reduces the user's work efficiency and makes it difficult for them to confirm accurate procedures.

[1036] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1037] In this invention, the server includes means for a user to record a video of the work, means for uploading the recorded video and audio data to the server, means for the server to transcribe the audio data using automatic speech recognition technology, means for summarizing the transcribed text using natural language processing technology, means for extracting important scenes from the video data using computer vision technology, means for combining the summarized text and extracted images to generate a manual, and means for storing the generated manual in cloud storage and providing it to the user, thereby enabling the user to efficiently obtain the necessary information and accurately check the work procedures.

[1038] "User" refers to a person who uses the system to record video of their work and obtain or refer to the manual.

[1039] "Device" refers to the hardware device a user uses to capture video and upload the data to a server. Examples include smartphones and smart glasses.

[1040] The "server" refers to a system that receives data uploaded by users, transcribes audio, summarizes text, extracts images, and ultimately generates and provides manuals.

[1041] "Audio data" refers to digital information of the voice recorded when the user explains the task.

[1042] "Automatic speech recognition technology (ASR)" refers to technology for converting voice data into text data.

[1043] "Transcription" refers to the process of converting audio data into text form.

[1044] "Natural language processing technology (NLP)" refers to the technology of analyzing transcribed text and extracting summaries and meanings.

[1045] A "summary" is a short text that removes redundant parts from transcribed text and summarizes only the important points.

[1046] "Computer vision technology (CV)" refers to technology that automatically extracts and analyzes specific information from images and videos.

[1047] An "important scene" refers to a particularly important part or moment in a video, and refers to a scene that contains information that a user needs.

[1048] A "still image" refers to an individual frame extracted from a video, meaning an image that captures a specific moment in time.

[1049] "Manual generation" refers to the process of combining summarized text with extracted still images to create a document containing instructions and instructions.

[1050] "Cloud storage" refers to remote servers that store and make data accessible over the internet.

[1051] The system based on this invention allows users to take videos of their work and then uses AI technology to efficiently generate manuals from those videos. The basic configuration and operation of this system are described below.

[1052] Video recording and audio description

[1053] The user uses a device such as a smartphone or smart glasses to record a video of the work being done. While recording, the user verbally explains the work. For example, while assembling furniture, the user might explain the detailed steps, such as "Install this screw in this position. Next, insert this part into this hole."

[1054] Video and audio uploads

[1055] After the recording is complete, the device uploads the video and audio data to the server. The data is encrypted before transmission to ensure security. Specifically, the data is encrypted using the SSL / TLS protocol.

[1056] Audio transcription

[1057] The server transcribes the received voice data using automatic speech recognition (ASR). For example, it uses the Google Cloud Speech-to-Text API to convert the voice data into text. The text data includes steps such as "1. Take out the screw. 2. Insert the screw into the screw hole."

[1058] Text summary

[1059] The server analyzes and summarizes the transcribed text using natural language processing (NLP) techniques such as OpenAI's GPT model. This removes redundant information and leaves only the key points. For example, the summary text might include concise steps such as "1. Remove the screw. 2. Insert the screw into the screw hole."

[1060] Image extraction

[1061] The server analyzes the video data using computer vision (CV) technologies such as OpenCV and automatically extracts important scenes, such as when a user removes a part or screws it together, and cuts out the scene as a still image.

[1062] Automatic manual generation

[1063] The server generates a manual using the summarized text and extracted still images. Specifically, it uses a manual generation engine such as LaTeX or Pandoc. The generated manual is provided in PDF or HTML format. This results in a visually easy-to-understand manual.

[1064] Manual distribution

[1065] The generated manual is stored in cloud storage (e.g., Amazon S3, Google Drive, etc.) by the server, and a link to the cloud storage is provided to the user, who can download or view the manual through the link.

[1066] Specific examples

[1067] As a concrete example, consider a user assembling new furniture at home. The user uses their smartphone to record a video of the work process and verbally explain the steps, such as "Use screw A here to attach board B to board C." After completing the recording, the user uploads the video and audio data to a server. The server transcribes the audio data using the Google Cloud Speech-to-Text API and summarizes it using OpenAI's GPT model. It also uses OpenCV to extract key scenes. Finally, the server combines the summarized text with the extracted still images to generate a concise visual manual. This manual is stored in cloud storage and a link is provided to the user.

[1068] Prompt Sentence Examples

[1069] "This video contains instructions for assembling furniture. Please follow the instructions verbally as you go through the process. Upload the video and audio data and generate a manual."

[1070] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1071] Step 1:

[1072] The user takes a video of the work they are doing. The work is recorded using a smartphone or smart glasses. For example, if the user is assembling furniture, they will explain each step as they go along. This video will record all of the steps.

[1073] Input: Video and audio of your work.

[1074] Output: Video file (including video and audio).

[1075] Step 2:

[1076] The device uploads the captured video and audio data to a server. Using a dedicated application, the data is encrypted using SSL / TLS and sent to the server.

[1077] Input: Video file.

[1078] Output: Video file saved on the server.

[1079] Step 3:

[1080] The server transcribes the received audio data using automatic speech recognition (ASR) technology and converts the audio into text using the Google Cloud Speech-to-Text API.

[1081] Input: Audio data from the video.

[1082] Output: Transcribed text data.

[1083] Step 4:

[1084] The server summarizes the transcribed text using natural language processing (NLP) techniques, using OpenAI's GPT model to extract important information from the text and remove redundancies.

[1085] Input: Transcribed text data.

[1086] Output: Summarized text data.

[1087] Step 5:

[1088] The server uses computer vision (CV) technology to extract important scenes from the video data. Using OpenCV, it identifies important scenes in the video and extracts them as still images. For example, this includes the moment when a user removes a part or tightens a screw.

[1089] Input: Video file.

[1090] Output: Still image files of important scenes.

[1091] Step 6:

[1092] The server generates a manual by combining the summarized text and extracted still images. Using LaTeX and Pandoc, the manual is created in PDF or HTML format for easy visual understanding.

[1093] Input: Abstracted text data, still image files.

[1094] Output: The generated manual (PDF or HTML format).

[1095] Step 7:

[1096] The server saves the generated manuals in cloud storage and provides them to users. It saves them using Amazon S3 or Google Drive and creates a download link.

[1097] Input: Generated manual.

[1098] Output: A download link that users can access.

[1099] (Application example 1)

[1100] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1101] In modern factories, it is important to efficiently record robot maintenance procedures and create maintenance manuals that can be reproduced by other technicians. However, traditional methods require manual recording and manual creation, which is time-consuming and labor-intensive. It is also difficult to record all detailed procedures and precautions, which can lead to inconsistent quality.

[1102] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1103] In this invention, the server includes means for shooting video, means for transcribing audio from the shot video, means for summarizing the transcribed text, means for extracting important images from the video, means for generating a manual by combining the summarized text and the extracted images, means for providing the generated manual to a user, and means for recording robot maintenance procedures as video and automatically generating a maintenance manual. This makes it possible to automatically and efficiently generate high-quality, reproducible maintenance manuals from videos recorded by technicians during work.

[1104] "Means for capturing video" refers to a device and method that allows a user to record the state of work using a terminal such as a smartphone or smart glasses.

[1105] A "means for transcribing audio from a video" is a device and method that uses automatic speech recognition (ASR) technology to convert audio data from a video into text data.

[1106] The "means for summarizing transcribed text" refers to an apparatus and method that uses a natural language processing (NLP) engine to remove redundant parts of transcribed text and extract key points.

[1107] The "means for extracting important images from video" refers to an apparatus and method that uses computer vision (CV) technology to identify important scenes from video data and extract them as still images.

[1108] The "means for generating a manual by combining summarized text and extracted images" refers to an apparatus and method for combining summarized text and extracted images to generate a visually easy-to-understand manual.

[1109] The "means for providing the generated manual to the user" refers to an apparatus and method for providing the generated manual to the user in a form that is easily accessible to the user via cloud storage or a download link.

[1110] The "means for recording robot maintenance procedures on video and automatically generating a maintenance manual" refers to a series of devices and methods for recording factory robot maintenance work on video and automatically creating a maintenance manual from the video.

[1111] To implement this invention, a user uses a device such as a smartphone or smart glasses to record video of the work they are doing. The user then verbally explains the necessary steps and key points during the work, and simultaneously records this as audio data. After the recording is complete, the device uploads the video and audio data to a server. The data is encrypted to ensure security.

[1112] The server converts the audio data into text using automatic speech recognition (ASR) technology, specifically Google Cloud Speech-to-Text. The converted text is then summarized using a natural language processing (NLP) engine, such as the generative AI model GPT-3. The summarized text focuses on important steps and explanations and eliminates redundant information.

[1113] Furthermore, the server uses computer vision (CV) technology to extract important scenes from the video data. The specific software used for this is OpenCV. The extracted important scenes are cut out as still images, and it is made clear which steps each scene corresponds to.

[1114] The server uses a manual generation engine to generate a maintenance manual by combining the summarized text and extracted images. The manual is generated in PDF or HTML format and provided in an easy-to-understand format. The generated manual is stored in cloud storage (e.g., AWS S3), and users can obtain the manual as a downloadable link.

[1115] As a concrete example, consider a scenario in which a technician performing maintenance work on a robot in a factory records a video of the maintenance procedure on a smartphone. The technician explains each step as he or she works. The video and audio data are then uploaded to a server, where the audio is transcribed using ASR technology. An NLP engine summarizes the text, and CV technology extracts important scenes. The generated manual can then be accessed by the technician via cloud storage, and can also be shared with other technicians.

[1116] Examples of prompts that can be used include:

[1117] Summarize the robot maintenance procedures below:

[1118] 1. Remove the screws

[1119] 2. Remove the parts

[1120] 3. Install the new parts

[1121] 4. Retighten the screws

[1122] 5. Check the operation

[1123] summary:

[1124] The present invention allows users to easily generate efficient and high-quality maintenance manuals.

[1125] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1126] Step 1:

[1127] The user uses a device such as a smartphone or smart glasses to record a video of the work being done, and at the same time verbally explains the work content. The input is the work the user is doing and an explanation of it, and the output is video data and audio data. The specific operations are to launch the camera app, start recording, and verbally explain the work procedures and important points.

[1128] Step 2:

[1129] The device uploads the captured video and audio data to the server. At this time, the data is encrypted to ensure security. The input is the video and audio data stored on the device, and the output is the encrypted data stored on the server. Specifically, the upload application is used to transfer the data to the server.

[1130] Step 3:

[1131] The server converts the voice data into text using automatic speech recognition (ASR) technology. The software used is Google Cloud Speech-to-Text. The input is encrypted voice data, and the output is transcribed text data. Specifically, the server decrypts the voice data and uses ASR technology to convert the voice into text format.

[1132] Step 4:

[1133] The server uses a natural language processing (NLP) engine to summarize the transcribed text. The software used is a generative AI model such as GPT-3. The input is the transcribed text data, and the output is the summarized text data. Specifically, the text data is input into the NLP engine, and a summary is generated that focuses on important steps and explanations.

[1134] Step 5:

[1135] The server uses computer vision (CV) technology to extract important scenes from video data. The software used is OpenCV. The input is video data, and the output is extracted still image data. Specifically, it analyzes the video data and extracts important scenes (such as the moment a part is removed or a screw is fastened) as still images.

[1136] Step 6:

[1137] The server generates a manual by combining the summarized text and extracted images. The input is the summarized text data and extracted still image data, and the output is the manual data (PDF or HTML format). Specifically, it combines the summarized text and still images to determine the layout and generate a visually easy-to-understand manual.

[1138] Step 7:

[1139] The generated manual is stored in cloud storage and a link is provided for users to download it. The input is the manual data, and the output is the manual stored on the cloud and its download link. The specific operation is to upload the manual to a cloud storage service (e.g., AWS S3) and provide the link to the user.

[1140] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1141] The system based on this invention is designed to efficiently generate manuals from videos taken by users, and by combining it with an emotion engine that analyzes the user's emotions, it provides more appropriate manuals. The basic configuration and operation of this system are described below.

[1142] Video recording and audio description

[1143] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, the user explains each step while assembling furniture.

[1144] Video and audio uploads

[1145] After the recording is complete, the device uploads the video and audio data to a server, where the data is encrypted to ensure security.

[1146] Audio transcription

[1147] The server transcribes the received audio data using automatic speech recognition (ASR) technology, converting the audio data into text and recording any specific instructions or instructions contained within.

[1148] Recognizing user emotions

[1149] The server uses an emotion engine to analyze the user's emotions from their voice and facial expressions, for example, determining whether they are confused or progressing smoothly based on their tone and intonation.

[1150] Text summary

[1151] The server's natural language processing (NLP) engine analyzes and summarizes the transcribed text, eliminating redundant information and leaving only the key points. This summary text focuses on the key steps or actions the user needs to take.

[1152] Image extraction

[1153] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes, such as when a user removes a part or screws it together, and then cuts out those scenes as still images.

[1154] Identifying important scenes based on emotions

[1155] Based on the user's emotional information recognized by the emotion engine, scenes that the user found particularly interesting or difficult are preferentially extracted, thereby improving the accuracy of extracting important scenes.

[1156] Automatic manual generation

[1157] Using the extracted still images and summarized text, the server uses a manual generation engine to create a manual, which is generated in PDF or HTML format and provided in a visually easy-to-understand format.

[1158] Manual distribution

[1159] The generated manual is stored in cloud storage by the server, and the device provides the manual as a downloadable link, which users can use to obtain and share the manual.

[1160] Specific examples

[1161] For example, if a user is assembling new furniture at home, they can use their smartphone to record a video of the work process. The steps are explained as they go along, with important points about screws and parts being mentioned verbally. After filming is complete, the video and audio data are uploaded to a server, where the audio is transcribed using ASR technology. An NLP engine then summarizes the text, and CV technology extracts key scenes. An emotion engine identifies situations where the user is confused and prioritizes the extraction of still images from those scenes. Finally, the summarized text and still images are combined to generate a concise visual manual, which can be accessed by the user via cloud storage. In this way, users can efficiently obtain the information they need and receive focused support on areas they are struggling with.

[1162] The processing flow will be explained below.

[1163] Step 1:

[1164] The user records a video of the work using a device such as a smartphone or smart glasses. While recording, the user verbally explains the work. For example, while assembling furniture, the user might explain, "First, remove part A."

[1165] Step 2:

[1166] Once the recording is complete, the device uploads the video and audio data to a server, where it is encrypted and securely transmitted.

[1167] Step 3:

[1168] The server receives the uploaded video and audio data, verifies the integrity of the files, and then begins the next process.

[1169] Step 4:

[1170] The server extracts the audio data from the video file, which is then passed to an automatic speech recognition (ASR) engine.

[1171] Step 5:

[1172] The ASR engine analyzes the voice data and transcribes it. The resulting transcription is "Remove part A."

[1173] Step 6:

[1174] The server sends the transcribed text to a natural language processing (NLP) engine for analysis and summarization. For example, a concise summary such as "Extract part A" is generated.

[1175] Step 7:

[1176] The server uses an emotion engine to analyze the user's voice and facial expressions, for example, to determine whether the user is confused or confused based on changes in voice tone or facial expression.

[1177] Step 8:

[1178] The server passes the video data to a computer vision (CV) engine to identify important scenes, such as when the user is confused or when important operations are performed.

[1179] Step 9:

[1180] Still images are generated from the scenes identified by the CV engine. Important work scenes and scenes that the user was interested in are saved as still images (e.g., the moment part A is removed).

[1181] Step 10:

[1182] The server combines the summarized text and extracted still images based on a template, and the manual generation engine uses the template to automatically generate a manual in PDF or HTML format.

[1183] Step 11:

[1184] The server saves the generated manual to cloud storage. Once the save is complete, a download link is generated and sent to the device.

[1185] Step 12:

[1186] The terminal notifies the user of the download link for the generated manual, which the user can use to download, view, and share the manual.

[1187] In this way, users can efficiently and quickly create manuals that combine still images and summary text from captured videos. Furthermore, the emotion engine prioritizes support for scenes where users find it difficult.

[1188] Example 2

[1189] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1190] Conventional manual generation systems often fail to meet actual needs because they do not consider the user's emotions or level of difficulty, even when using video or audio data. Furthermore, they sometimes fail to properly identify important scenes or summarize text, resulting in manuals that are difficult for users to understand. Furthermore, there are issues with the method of distributing the generated manuals, making them difficult for users to access.

[1191] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for shooting a video, a means for transcribing audio from the shot video, a means for summarizing the transcribed text, a means for extracting important images from the video, a means for analyzing a user's emotions, a means for identifying important scenes based on the analyzed emotional information, a means for generating a manual by combining the summarized text and the extracted images, and a means for providing the generated manual to the user. This makes it possible to take the user's emotions into consideration, appropriately extract important scenes, and combine them with the summarized text, thereby providing a manual that is easy for the user to understand and meets their needs. The generated manual is also stored in cloud storage, and a downloadable link is provided for the user to easily access.

[1192] "Means for shooting video" refers to the functions or devices that allow users to record video using digital devices.

[1193] "Means for transcribing audio from a video" refers to a function or device that uses voice recognition technology to convert audio from a video into text data.

[1194] A "means for summarizing transcribed text" is a function or device that uses natural language processing technology to extract important information from long pieces of text and summarize it concisely.

[1195] "Means for extracting important images from a video" refers to a function or device that uses computer vision technology to select specific scenes or images from a video.

[1196] A "means for analyzing user emotions" is a function or device that uses machine learning models or algorithms based on audio and video data to determine a user's emotional state.

[1197] The "means for identifying important scenes based on analyzed emotional information" refers to a function or device that, based on the results of the user's emotional analysis, prioritizes extracting scenes that the user finds particularly interesting or difficult.

[1198] The "means for generating a manual by combining summarized text and extracted images" refers to a function or device that integrates summarized text with important images to create a manual in a visually easy-to-understand format.

[1199] The "means for providing the generated manual to the user" refers to a function or device that provides the user with an access link to the manual stored in cloud storage, allowing the user to download or refer to the manual.

[1200] The system based on this invention is designed to efficiently generate manuals from videos taken by users, and by combining it with an emotion engine that analyzes the user's emotions, it provides more appropriate manuals. The basic configuration and operation of this system are described below.

[1201] Video recording and audio description

[1202] The user uses a device such as a smartphone or smart glasses to record a video of the work being done. While recording, the user verbally explains the work. For example, while assembling furniture, the user explains each step, giving specific instructions such as "Use these screws to connect the boards."

[1203] Video and audio uploads

[1204] After the recording is complete, the device uploads the video and audio data to the server. The data is encrypted to ensure security. The device transmits the video and audio data to the server using the SSL / TLS protocol.

[1205] Audio transcription

[1206] The server transcribes the received voice data using automatic speech recognition (ASR) technology, such as Google Cloud Speech-to-Text or IBM Watson Speech to Text, and converts the voice data into text, recording each step and explanation.

[1207] Recognizing user emotions

[1208] The server uses an emotion engine to analyze emotions from the user's voice and facial expressions. For example, it uses the Microsoft Azure Emotion API to determine the user's emotions from the tone, intonation, and facial expressions of the voice. The server then sends the analyzed voice data and video data to the emotion engine for emotion analysis.

[1209] Text summary

[1210] The transcribed text is analyzed and summarized by a natural language processing (NLP) engine on the server. Redundant information is removed, leaving only the key points. This process uses techniques such as OpenAI GPT-3 and BERT. The server then stores the generated summary text in a database.

[1211] Image extraction

[1212] The server analyzes the video data using computer vision (CV) technology and automatically extracts important scenes. Using OpenCV, it identifies the moment when the user removes a part or connects it with a screw, and cuts them out as still images.

[1213] Identifying important scenes based on emotions

[1214] Based on the user's emotional information recognized by the emotion engine, scenes that the user found particularly interesting or difficult are preferentially extracted. This improves the accuracy of extracting important scenes. Based on the emotion analysis results, the server preferentially selects still images that match scenes that the user found difficult.

[1215] Automatic manual generation

[1216] The server uses the extracted still images and summarized text to create a manual using a manual generation engine. It uses a combination of LaTeX, Markdown, and Pandoc to generate the manual in PDF or HTML format. The generated document is saved in the specified format.

[1217] Manual distribution

[1218] The generated manual is stored by the server in cloud storage (e.g., Amazon S3 or Google Drive). The user is provided with a link to download the manual from their device. The server uploads the generated manual to cloud storage and generates a link. The user can download the manual from the link in the app.

[1219] Specific examples

[1220] For example, this system could be used when a user is assembling new furniture at home. The user uses their smartphone to record video of the work process while verbally explaining the steps and important points. The video and audio data are uploaded to a server, and the audio is transcribed using ASR technology. The text is then summarized using an NLP engine, and important scenes are extracted using CV technology. An emotion engine identifies confusing moments and prioritizes the extraction of still images from those moments. Finally, the summarized text and still images are combined to generate a concise manual. The user can retrieve this manual from cloud storage and efficiently obtain the information they need.

[1221] Prompt Sentence Examples

[1222] Here are some examples of prompts to input to a generative AI model:

[1223] Design a program that automatically generates a manual for assembling furniture based on a video taken by a user, following the steps below:

[1224] 1. Uploading video and audio data

[1225] 2. Audio transcription (using ASR technology)

[1226] 3. User sentiment analysis (using emotion engine)

[1227] 4. Text Summarization (using NLP techniques)

[1228] 5. Extraction of important scenes from videos (using CV technology)

[1229] 6. Generating a manual based on the extracted scenes and summary text (using a manual generation engine)

[1230] 7. Save the manual to the cloud and provide a link to the user

[1231] Specifically, this involves using technologies such as Google Cloud Speech-to-Text, Microsoft Azure Emotion API, OpenAI GPT-3, OpenCV, LaTeX, and Amazon S3.

[1232] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1233] Step 1:

[1234] The user records a video of their work using a device such as a smartphone or smart glasses. The input is the user's work and an explanation of it, and the output is video data. The user opens a camera app and presses the record button to start recording. As the user works, they provide appropriate explanations.

[1235] Step 2:

[1236] After the recording is complete, the device uploads the video and audio data to the server. The input is the recorded video data, and the output is the video and audio data sent to the server. The device encrypts the data using the SSL / TLS protocol and sends it securely to the server.

[1237] Step 3:

[1238] The server transcribes the received voice data using automatic speech recognition (ASR) technology. The input is voice data and the output is text data. The server sends the voice data to an ASR engine (e.g., Google Cloud Speech-to-Text) and converts the voice into text format.

[1239] Step 4:

[1240] The server uses an emotion engine to analyze emotions from the user's voice and facial expressions. The input is audio and video data, and the output is the emotion analysis results. The server sends the audio and video to the emotion engine (for example, a general emotion analysis API) for emotion analysis.

[1241] Step 5:

[1242] The transcribed text is analyzed and summarized by the server's natural language processing (NLP) engine. The input is the transcribed text, and the output is the summary text. The server inputs the text data into an NLP engine (e.g., OpenAI GPT-3 or BERT) to generate a summary.

[1243] Step 6:

[1244] The server analyzes video data using computer vision (CV) technology and automatically extracts important scenes. The input is video data and the output is important still images. The server analyzes video frames every second and uses CV technology (e.g., OpenCV) to detect specific actions and extract them as still images.

[1245] Step 7:

[1246] Based on the user's emotional information recognized by the emotion engine, scenes that are of particular interest or difficulty are extracted with priority. The input is the emotion analysis results and still images, and the output is a prioritized list of important still images. The server compares the emotion analysis results with the still images and prioritizes saving scenes that the user found confusing.

[1247] Step 8:

[1248] The server uses the extracted still images and summarized text to create a manual using a manual generation engine. The input is the summary text and still images, and the output is a manual in PDF or HTML format. The server uses a combination of LaTeX, Markdown, and Pandoc to generate a visually easy-to-understand manual.

[1249] Step 9:

[1250] The generated manual is stored in cloud storage by the server. The input is the generated manual, and the output is an access link to the manual on the cloud. The server uploads the manual to cloud storage (e.g., Amazon S3 or Google Drive) and provides the user with a downloadable link.

[1251] (Application example 2)

[1252] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1253] In modern factories, creating work procedure manuals is time-consuming and labor-intensive, and relies on monotonous, impersonal information, making it difficult for workers to understand and put into practice. Furthermore, because the feelings and difficulties of individual workers are not taken into consideration, efficiency is often low. Therefore, there is a demand for shortening work times and reducing errors, but current manual generation methods have limitations.

[1254] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1255] In this invention, the server includes: [means for shooting video;] [means for transcribing audio from the shot video;] [means for summarizing the transcribed text;] [means for extracting important images from the video;] [means for generating a manual by combining the summarized text and extracted images;] [means for providing the generated manual to the user;] [means for analyzing the user's emotions from their voice and facial expressions;] [means for identifying important scenes based on the user's emotional information; and [means for recording and analyzing work performed by the robot.] This enables the user to generate more understandable and effective manuals that take into account the emotions and difficulties of each work procedure.

[1256] "Means for taking videos" refers to a function that allows a user to use a device such as a smartphone or robot to record the work situation and procedures as video.

[1257] "Means for transcribing audio from video footage" refers to a function that converts audio data from video footage into text data using automatic speech recognition technology.

[1258] "Means for summarizing transcribed text" refers to the function of extracting important points from transcribed text data and summarizing them in a concise form. This function uses natural language processing technology.

[1259] "Means for extracting important images from video" refers to a function that automatically selects specific scenes or important moments from captured video data and extracts them as still images. This uses computer vision technology.

[1260] The "means for generating a manual by combining summarized text and extracted images" is a function that automatically creates a visually easy-to-understand manual by combining summarized text data with extracted still images.

[1261] The "means for providing the generated manual to the user" is a function for storing the generated manual in cloud storage and providing a link so that the user can download or access the manual.

[1262] "Means for analyzing user emotions from voice and facial expressions" refers to a function that identifies and analyzes the user's emotional state from the tone of their voice and facial expressions in the video. This is done using an emotion engine.

[1263] The "means for identifying important scenes based on the user's emotional information" is a function that prioritizes extracting scenes that the user is particularly interested in or finds difficult, based on data obtained through emotional analysis.

[1264] "Means for recording and analyzing work performed by robots" refers to a function that records and records the various tasks performed by factory robots, analyzes the data, and generates manuals.

[1265] To implement this invention, it is necessary to configure a system that includes the following main means: The system has means for shooting video, transcribing audio, summarizing text, extracting important images, generating and providing manuals, analyzing user emotions, and recording and analyzing work performed by a robot.

[1266] The server provides a means for users to record video of their work through their devices, such as smartphones or factory robots. The user can record video and provide verbal instructions while working, and the video and audio data are stored on the device. This data is then encrypted and uploaded to a cloud-based server.

[1267] The server automatically transcribes the uploaded audio data using the Google Cloud Speech API, converting it into text so that its content can be understood. The text data is then summarized using natural language processing techniques from Hugging Face's Transformers library. The summarized text is stripped of redundant information and focuses on the key points.

[1268] The server then analyzes the video data using OpenCV to extract specific scenes and important moments. This allows the scenes and steps the user considered particularly important during the task to be extracted as images. The server then analyzes the user's emotions using a TensorFlow model. The server determines the user's emotional state based on their voice tone and facial expressions, and based on this information, prioritizes the extraction of scenes in which the user found the task particularly challenging or interesting.

[1269] By combining these summary texts with extracted images of important scenes, the manual generation engine automatically creates a visually concise manual. The generated manual is saved in cloud storage in PDF or HTML format, and users can obtain and share the manual via a downloadable link.

[1270] (Example)

[1271] For example, when a factory worker records a robotic maintenance procedure for a new machine, they film the robot performing the task. This video and the audio data within it are uploaded to a server, where they are transcribed and summarized using automatic speech recognition and natural language processing technologies. Furthermore, computer vision technology extracts key scenes, and emotion analysis identifies parts where the user had difficulty. Finally, a manual summarizing the key points is generated.

[1272] (Example of a prompt)

[1273] "Explain the machine maintenance procedure, record each step on video, highlight the key points and generate a manual."

[1274] "Record a video of the assembly process and generate a summary manual based on the user's voice description and sentiment analysis."

[1275] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1276] Step 1:

[1277] Users use devices such as smartphones or factory robots to record video of work situations and procedures. While recording the video, the user verbally explains each step, and video data (video and audio) is input into the device.

[1278] Step 2:

[1279] The device uploads the recorded video data to a cloud-based server. During the upload process, the video data is encrypted and processed to ensure security. The encrypted video data is input to the server as an output.

[1280] Step 3:

[1281] The server automatically transcribes the audio data in the video using the Google Cloud Speech API. This process converts the audio data into text data. The input is audio data, and the output is transcribed text data.

[1282] Step 4:

[1283] The server uses Hugging Face's Transformers library to summarize the transcribed text data. Summarization extracts key points from the text and appropriately compresses the amount of information. The input is the transcribed text data, and the output is the summarized text data.

[1284] Step 5:

[1285] The server uses OpenCV to extract important image scenes from video data. It identifies specific frames and scenes and extracts them as still images. The input is video data, and the output is a set of extracted still images.

[1286] Step 6:

[1287] The server uses a TensorFlow model to analyze emotions from the user's voice tone and facial expressions in the video. Emotion analysis identifies the emotional state, such as difficulty or joy, felt by the user at each step. The input is audio and video data, and the output is the emotion analysis results.

[1288] Step 7:

[1289] Based on the emotion analysis results, the server prioritizes and extracts scenes that the user finds particularly difficult or that the user deems important. This selection process identifies still images with high importance. The input is the emotion analysis results and the extracted still images, and the output is the prioritized still images.

[1290] Step 8:

[1291] The server combines the summarized text data with a set of selected still images to automatically generate a manual. The generated manual is visually easy to understand, emphasizing important procedures and key points. The input is the summarized text data and a set of selected still images, and the output is the generated manual (in PDF or HTML format).

[1292] Step 9:

[1293] The server stores the generated manual in cloud storage and provides a download link to the user, through which the user can access the generated manual and share it as needed. The input is the generated manual, and the output is the download link.

[1294] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1295] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1296] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1297] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1298] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1299] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1300] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1301] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1302] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1303] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1304] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1305] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1306] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1307] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1308] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1309] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1310] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1311] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1312] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1313] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1314] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1315] The following is further disclosed regarding the above embodiment.

[1316] (Claim 1)

[1317] [Means for taking videos and

[1318] [Means to transcribe the audio in the video,

[1319] [Means of summarizing the transcribed text;

[1320] [Methods for extracting important images from videos,

[1321] [Means for generating a manual by combining the summarized text and extracted images;

[1322] [Means for providing the generated manual to the user;

[1323] A system including:

[1324] (Claim 2)

[1325] [The system of claim 1, which uses automatic speech recognition technology when transcribing the speech.

[1326] (Claim 3)

[1327] [The system of claim 1, wherein the extraction of significant images uses computer vision techniques.

[1328] "Example 1"

[1329] (Claim 1)

[1330] [Means for users to record videos of their work;

[1331] [Means for uploading the captured video and audio data to a server;

[1332] [Means for the server to transcribe the voice data using automatic speech recognition technology;

[1333] [Means for summarizing the transcribed text using natural language processing technology;

[1334] [Means for extracting important scenes from video data using computer vision technology;

[1335] [Means for generating a manual by combining the summarized text and extracted images;

[1336] [Means for storing the generated manual in cloud storage and providing it to users;

[1337] A system including:

[1338] (Claim 2)

[1339] [The system of claim 1, wherein the user records a video while verbally explaining the details of the work.

[1340] (Claim 3)

[1341] The system according to claim 1, wherein the summarized text and extracted images are laid out to generate a visually easy-to-understand manual.

[1342] "Application Example 1"

[1343] (Claim 1)

[1344] [Means for taking videos and

[1345] [Means to transcribe the audio in the video,

[1346] [Means of summarizing the transcribed text;

[1347] [Methods for extracting important images from videos,

[1348] [Means for generating a manual by combining the summarized text and extracted images;

[1349] [Means for providing the generated manual to the user;

[1350] [Means for recording robot maintenance procedures as videos and automatically generating maintenance manuals;

[1351] A system including:

[1352] (Claim 2)

[1353] [The system of claim 1, which uses automatic speech recognition technology when transcribing the speech.

[1354] (Claim 3)

[1355] [The system of claim 1, wherein the extraction of significant images uses computer vision techniques.

[1356] "Example 2: Combining Emotion Engines"

[1357] (Claim 1)

[1358] [Means for taking videos and

[1359] [Means to transcribe the audio in the video,

[1360] [Means of summarizing the transcribed text;

[1361] [Methods for extracting important images from videos,

[1362] [Means for analyzing user emotions;

[1363] [Means for identifying important scenes based on analyzed emotional information;

[1364] [Means for generating a manual by combining the summarized text and extracted images;

[1365] [Means for providing the generated manual to the user;

[1366] A system including:

[1367] (Claim 2)

[1368] [The system of claim 1, which uses automatic speech recognition technology when transcribing the speech.

[1369] (Claim 3)

[1370] [The system of claim 1, wherein the extraction of significant images uses computer vision techniques.

[1371] (Claim 4)

[1372] [The system of claim 1, wherein a machine learning model is used to analyze user emotions.

[1373] (Claim 5)

[1374] [The system of claim 1, wherein voice tone analysis and facial expression recognition techniques are used to identify important scenes using the results of user emotion analysis.

[1375] (Claim 6)

[1376] The system of claim 1 stores the generated manual in cloud storage and provides a link for the user to download it.

[1377] "Application example 2 when combining emotion engines"

[1378] (Claim 1)

[1379] [Means for taking videos and

[1380] [Means to transcribe the audio in the video,

[1381] [Means of summarizing the transcribed text;

[1382] [Methods for extracting important images from videos,

[1383] [Means for generating a manual by combining the summarized text and extracted images;

[1384] [Means for providing the generated manual to the user;

[1385] [Means for analyzing user emotions from voice and facial expressions,

[1386] [Means for identifying important scenes based on user emotion information;

[1387] [Means for recording and analyzing work performed by the robot;

[1388] A system including:

[1389] (Claim 2)

[1390] [The system of claim 1, which uses automatic speech recognition technology when transcribing the speech.

[1391] (Claim 3)

[1392] [The system of claim 1, wherein the extraction of significant images uses computer vision techniques. [Explanation of symbols]

[1393] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of capturing video; A means of transcribing the audio in the video; a means of summarizing the transcribed text; A means of extracting important images from within a video; a means for combining the summarized text and extracted images to generate a manual; a means for providing the generated manual to a user; A system including:

2. The system of claim 1 , wherein automatic speech recognition technology is used to transcribe the speech.

3. 10. The system of claim 1, wherein the extraction of significant images uses computer vision techniques.

Images