System

Abstract

A system is provided.SOLUTION: The system includes a means for inputting an instruction, a means for extracting text data and image data from the input instruction, a means for analyzing the extracted text data and image data to generate assembly procedure information, a means for generating an assembly moving image based on the generated assembly procedure information, and a means for distributing the generated assembly moving image to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When assembling products such as furniture or plastic models, if the accompanying instructions are difficult to read or the step-by-step images are only one-axis perspective and difficult to grasp in three dimensions, users often struggle to assemble and give up midway. Furthermore, the number of official and volunteer assembly videos is limited and expensive, making them difficult to implement. Given this background, there is a demand for a system that allows users to easily understand assembly steps and assemble accurately. [Means for solving the problem]

[0005] In order to solve the above problems, the following means is provided: A system including a means for inputting instructions, a means for extracting text data and image data from the input instructions, a means for analyzing the extracted text data and image data and generating assembly procedure information, a means for generating an assembly animation based on the generated assembly procedure information, and a means for delivering the generated assembly animation to the user provides an environment in which the user can easily understand the assembly procedure and assemble accurately.

[0006] Specifically, the system extracts text data from the instructions using OCR (Optical Character Recognition) technology, and then analyzes the extracted text and image data using a semantic analysis engine to convert each assembly step into an easy-to-understand format. This allows users to visually confirm the assembly steps, significantly improving assembly efficiency and success rates.

[0007] "Instructions" include documents and diagrams that accompany a product and explain assembly procedures, usage, etc.

[0008] "Means for inputting" refers to the technical means for receiving instructions from the user and inputting that data into the system.

[0009] "Text data" refers to the text information contained in the instructions, which has been converted into digital form using OCR technology.

[0010] "Image data" refers to the diagrams and illustrations contained in the instructions, stored in digital format.

[0011] "Means of extraction" refers to the technical means of obtaining text data or image data from the instructions and converting them into another format or file.

[0012] "Means for analysis" refers to technical means for analyzing extracted text data and image data and deciphering meanings and relationships.

[0013] "Assembly procedure information" is information that includes specific assembly methods and steps derived from the analyzed data.

[0014] "Means for generating" refers to the technical means for creating assembly videos and 3D models based on the analyzed assembly procedure information.

[0015] "Means for distribution" refers to the technical means for providing the generated assembled video to users, and includes methods such as streaming and downloading.

[0016] "OCR (Optical Character Recognition)" is a technology that reads character information from images and documents and converts it into electronic text.

[0017] A "semantic analysis engine" is software or algorithms that interpret input text or image data and analyze its meaning and relevance. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention relates to a system that automatically generates assembly animations based on instructions. The operation and configuration of the entire system will be explained below with specific procedures and examples.

[0040] Overall system configuration

[0041] 1. User Device:

[0042] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[0043] 2. Server:

[0044] The server is a central device with key functions such as instruction processing, video generation, analysis, etc. Multiple modules and engines work together.

[0045] Invention Implementation Process

[0046] 1. Upload the instructions

[0047] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[0048] Terminal: Receives the uploaded instruction file and sends it to the server.

[0049] 2. Analysis of Instructions

[0050] Server: The server analyzes the received instruction file.

[0051] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[0052] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[0053] Example: When a PDF of furniture instructions uploaded by a user arrives at the server, the OCR function is activated and extracts assembly steps such as "fix part A to part B." At the same time, image recognition technology identifies the diagrams of parts A and B and stores them in a database.

[0054] 3. Interpretation of instructional information

[0055] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[0056] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[0057] Example: The server uses a semantic analysis engine to interpret the instruction "Secure part A to part B with screws" and clarifies the specific alignment of the parts, the type of screw, and how to use it.

[0058] 4. Generating assembly videos

[0059] Server: Generates assembly animation based on assembly procedure information.

[0060] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[0061] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, screw rotation motion, etc.

[0062] Example: The server simulates the steps for assembling furniture using 3D modeling software and animates the process. The generated video reproduces the steps, such as "Place part A here, then place part B on top of it and fasten it with screws."

[0063] 5. Video distribution

[0064] Server: Stores the completed assembly video and generates streaming and download links.

[0065] Terminal: Receives the video link sent from the server and displays it to the user.

[0066] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[0067] summary

[0068] This system allows users to simply upload instructions and easily understand the visual assembly steps, allowing them to assemble products accurately, significantly reducing the time and effort required for assembly and improving the user experience.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] User: Logs in to the system and uploads instructions for the product to be assembled in image or PDF format.

[0072] Step 2:

[0073] Terminal: Receives the uploaded instruction file and sends it to the server.

[0074] Step 3:

[0075] Server: Receives the instruction manual file. After receiving it, it starts the OCR engine and converts the text information in the manual into digital text data.

[0076] Step 4:

[0077] Server: Extracts the text data after OCR processing, and simultaneously uses an image recognition algorithm to identify part information from diagrams and illustrations in the manual.

[0078] Step 5:

[0079] Server: The extracted text data and image data are input into a semantic analysis engine to analyze the relationships and meaning between procedures.

[0080] Step 6:

[0081] Server: Generates detailed assembly procedure information from the analysis results using a semantic analysis engine.

[0082] Step 7:

[0083] Server: Based on the assembly procedure information, a 3D model of the part is generated using 3D modeling software.

[0084] Step 8:

[0085] Server: The generated 3D model is input into a video generation engine, which records the assembly procedure as a video while simulating it.

[0086] Step 9:

[0087] Server: Renders the final assembled video, saves it, and generates a URL for the video file.

[0088] Step 10:

[0089] Device: Displays the video URL received from the server in the user interface, allowing the user to stream or download.

[0090] Step 11:

[0091] Users: Click on the video link to watch the video and then follow the instructions in the video to assemble the product. During playback, users can pause, rewind, fast forward, and more.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Traditional assembly instructions consist of only text and drawings, making them difficult for users to understand and often resulting in time-consuming assembly tasks. Misunderstandings and mistakes at certain steps can easily occur, leading to assembly failures and frustration. Furthermore, manually creating assembly videos requires specialized knowledge and time, resulting in significant effort.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animations based on the generated assembly procedure information, means for delivering the generated assembly animations to users, and means for identifying specific software modules (OCR technology, semantic analysis engine, 3D modeling software, etc.) used in each processing step, thereby enabling users to easily understand the visual assembly procedures and accurately assemble the product.

[0097] "Means for inputting instructions" refers to a function that allows users to upload image files or PDF format instructions to the system via their terminals.

[0098] "Means for extracting text data and image data" is a processing function for extracting text information and image information from the uploaded instructions.

[0099] The "means for analyzing the extracted text data and image data and generating assembly procedure information" is an analysis function for understanding the assembly procedure based on the extracted data and generating specific assembly instructions.

[0100] The "means for generating assembly animation" is a function for creating animation that visually represents assembly work using 3D models of parts based on the generated assembly procedure information.

[0101] The "means for distributing assembly animation to users" is a function for saving the generated assembly animation and providing it so that users can view or download it.

[0102] "OCR technology" is an abbreviation for Optical Character Recognition, and is a technology that converts image-formatted characters into digital text data.

[0103] A "semantic analysis engine" is an analysis engine that analyzes text and image data and understands their meaning.

[0104] "3D modeling software" is software for creating and editing three-dimensional objects and scenes on a computer.

[0105] This invention relates to a system that automatically generates assembly animations based on instructions. The operation and configuration of the entire system will be explained below with specific procedures and examples.

[0106] System configuration

[0107] User terminal

[0108] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[0109] server

[0110] The server is a central device with key functions such as instruction processing, video generation, analysis, etc. Multiple modules and engines work together.

[0111] Feature details

[0112] Upload instructions

[0113] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[0114] Terminal: Receives the uploaded instruction file and sends it to the server.

[0115] Analysis of the instructions

[0116] Server: The server analyzes the received instruction file.

[0117] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[0118] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[0119] Interpreting the instruction manual information

[0120] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[0121] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[0122] Assembly video generation

[0123] Server: Generates assembly animation based on assembly procedure information.

[0124] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[0125] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, and screw rotation motion.

[0126] Video distribution

[0127] Server: Stores the completed assembly video and generates streaming and download links.

[0128] Terminal: Receives the video link sent from the server and displays it to the user.

[0129] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[0130] Specific examples

[0131] When using furniture assembly instructions

[0132] The user uploads a PDF file of furniture instructions from their device. The server uses OCR to extract the text information, such as "Secure part A to part B with screws," and uses image recognition to identify the diagrams of parts A and B. The semantic analysis engine analyzes the text and interprets the specific steps (such as aligning the parts and the type of screws to use). The steps are animated using 3D modeling software, and the server sends the generated video link to the user's device. The user clicks the link and watches the video to accurately assemble the furniture.

[0133] Example prompts for generative AI models

[0134] I have uploaded assembly instructions for a piece of furniture. These instructions include the step of fastening part A to part B with screws. Please generate a 3D model and an assembly video based on these instructions.

[0135] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0136] Step 1: Upload your instructions

[0137] Input: Instructions (image file or PDF format) that come with items that users assemble, such as furniture or plastic models

[0138] Specific operation: The user clicks the "Upload instructions" button on the device, a local file selection dialog appears, the user selects the assembly instructions file and presses the "Open" button, and the file is uploaded to the system.

[0139] Output: The device displays a notification that the upload is complete and sends the instruction file to the server.

[0140] Step 2: Parse the instructions

[0141] Input: Uploaded instruction file

[0142] Specific operation: The server inputs the received instruction manual file into the analysis module. Using OCR, the text in the manual is converted into digital text data. At the same time, image recognition is performed to extract part information from diagrams and illustrations.

[0143] Output: Extracted text and image data

[0144] Step 3: Interpret the information in the instructions

[0145] Input: Extracted text and image data

[0146] Specific operation: The server inputs the data obtained through OCR and image recognition into a semantic analysis engine to generate assembly procedure information. For example, a command such as "Secure part A to part B with screws" is analyzed and the specific steps, tools required, and part alignment methods are interpreted.

[0147] Output: Assembly procedure information

[0148] Step 4: Generate assembly videos

[0149] Input: Assembly procedure information

[0150] Specific operation: The server uses 3D modeling software to generate a 3D model of the specified part. It then uses the generated 3D model to simulate the assembly procedure, and animates the simulation results using a video generation engine. The video includes the part selection procedure, precise alignment, screw rotation, and more.

[0151] Output: Assembly video file

[0152] Step 5: Publish your video

[0153] Input: Assembly video file

[0154] Specific operation: The server saves the assembled video and generates a URL or download link for streaming the video file. The server then sends the URL or download link to the device and notifies the user.

[0155] Output: User-accessible video URL and download link

[0156] Step 6: Watch the video and assemble

[0157] Input: Video URL or download link

[0158] Specific operation: The user clicks on the provided URL on their device to watch the assembly video. The video is streamed and the user can pause, rewind, fast forward, etc. The user then performs the actual assembly work by following the video.

[0159] Output: Understanding the assembled product and the exact assembly steps along the way

[0160] (Application example 1)

[0161] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] In modern manufacturing factories, efficient execution of complex assembly tasks is important, but they often require a great deal of effort and time. In particular, assembling new products requires workers to understand the manual and accurately grasp the procedures. However, differences in the ability to read and understand instructions and the experience of actually assembling by hand vary from person to person, leading to variations in quality and errors. Furthermore, as automation using robots advances, it is necessary for the robots to properly understand and reliably execute assembly procedures. To solve these problems and achieve efficient and uniform assembly work, a system is needed that clarifies assembly procedures and enables robots to accurately execute them.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0164] In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animation based on the generated assembly procedure information, means for distributing the generated assembly animation to a user, and means for generating and executing operation instructions for a robot based on the assembly procedure information. This not only provides assembly procedures that are easy to understand visually, but also enables the robot to perform assembly work accurately.

[0165] "Instructions" are documents that show how to assemble or operate a product.

[0166] "Text data" is text information extracted from the instructions.

[0167] "Image data" refers to visual information such as diagrams and illustrations extracted from the manual.

[0168] "Assembly procedure information" is information about a specific method for assembling a product, generated by analyzing text data and image data.

[0169] "OCR (Optical Character Recognition) technology" is a technology that converts characters in analog format, such as on paper or in images, into digital format.

[0170] A "semantic analysis engine" is software that analyzes the meaning of text and image data and understands their relationships.

[0171] "Video generation" is the process of creating a visually easy-to-view video based on assembly procedure information.

[0172] "Distribution" is the process of sending the generated video to users.

[0173] A "robot" is a machine that operates automatically according to a program and can perform assembly tasks.

[0174] The "operation instruction" is a specific instruction for the robot to perform an assembly task.

[0175] This invention relates to a system that automatically generates assembly animations based on instructions and provides operational instructions to a robot. The operation and configuration of the entire system will be explained below, along with specific procedures and examples.

[0176] Overall system configuration

[0177] 1. User Device

[0178] The user terminal is a device used to upload instructions and view the generated videos. Users access the system using a smartphone or tablet and operate it.

[0179] 2. Server

[0180] The server is a central device with key functions such as processing instructions, generating videos, analyzing them, and generating instructions for the robot. Multiple modules and engines work together.

[0181] Invention Implementation Process

[0182] Upload instructions

[0183] Users upload instructions for products such as furniture and plastic models to the system as image files or PDFs from their devices. The devices receive the uploaded instruction files and send them to the server.

[0184] Analysis of the instructions

[0185] The server uses OCR and image recognition technology to analyze the received instruction manual file. OCR technology (e.g., Tesseract) is used to convert the text into digital text data, and image recognition technology (e.g., OpenCV) is used to extract part information from diagrams and illustrations.

[0186] Interpreting the instruction manual information

[0187] The server analyzes the extracted text and image data to generate assembly procedure information. This analysis is performed using a semantic analysis engine, which analyzes the meaning of the text and image data and understands their relationships.

[0188] Assembly video generation

[0189] The server generates an assembly video based on the assembly procedure information. It uses 3D modeling software (e.g., Blender) to generate a 3D model of the part and performs a simulation. It uses a video generation engine (e.g., FFmpeg) to create an assembly video based on the generated 3D model and simulation results.

[0190] Video distribution

[0191] The server stores the completed assembly video and generates a streaming distribution or download link. The device receives the video link sent from the server and displays it to the user. The user can watch the video and actually perform the assembly work by following the displayed steps.

[0192] Instructions for the robot

[0193] Based on the assembly procedure information, the server generates and executes operation instructions for the robot, allowing the robot to accurately assemble the parts according to the generated instructions.

[0194] Specific examples

[0195] Let's say a user takes a photo of a piece of furniture instruction manual with their smartphone and uploads it to the application. The uploaded manual is converted into text using an OCR engine, and diagrams and illustrations are analyzed using an image recognition engine. Based on the analysis results, the server generates assembly procedures and simulates them using 3D modeling software. The generated video is sent to the user's device, and the user can proceed with the assembly work while watching it. Furthermore, a robot accurately performs the assembly work based on the operating instructions generated by the server.

[0196] Prompt Sentence Examples

[0197] Based on the furniture instructions below, please automatically analyze the assembly steps and generate the final assembly video.

[0198] "Secure part A to part B, then attach part C. Then place part D according to the instructions in step 3, and finally secure all parts together with screws."

[0199] Following this prompt, specific steps and component instructions are provided to ensure the model can be analyzed accurately.

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1: Upload your instructions

[0202] The user takes a photo of the instruction manual that comes with the product to be assembled using a smartphone or tablet, or saves it as a PDF file on the device. The device receives the photo or saved instruction file and sends it to the server. The input is an image file or PDF file of the instruction manual, and the output is the instruction data that is sent to the server. The specific operation is for the user to operate the application to select the file and press the upload button.

[0203] Step 2: Parse the instructions

[0204] The server analyzes the received instruction manual file using OCR and image recognition technology. Specifically, it uses an OCR engine (e.g., Tesseract) to convert the text into digital text data, and an image recognition engine (e.g., OpenCV) to extract part information from diagrams and illustrations. The input is the instruction manual data, and the output is text data and image data. Specifically, the server starts the OCR engine, scans the characters in the image, and converts them into text.

[0205] Step 3: Interpret the information in the instructions

[0206] The server analyzes the extracted text data and image data to generate assembly procedure information. A semantic analysis engine is used for this analysis. The input is text data and image data, and the output is assembly procedure information. Specifically, the semantic analysis engine analyzes the meaning of the text and understands the relationships between parts. For example, it interprets in detail the instruction "Secure part A to part B with a screw."

[0207] Step 4: Generate assembly videos

[0208] The server generates an assembly video based on the assembly procedure information. It uses 3D modeling software (e.g., Blender) to generate a 3D model of the part and performs a simulation. It uses a video generation engine (e.g., FFmpeg) to create an assembly video based on the generated 3D model and simulation results. The input is the assembly procedure information and the output is the assembly video. Specifically, the server starts the 3D modeling software, simulates the operation of the part, and encodes it as a video.

[0209] Step 5: Publish your video

[0210] The server stores the completed assembly video and generates a streaming distribution or download link. The terminal receives the video link sent from the server and displays it to the user. The input is the assembly video, and the output is the video link to the user's terminal. Specifically, the server uploads the video file to cloud storage and sends the link to the terminal.

[0211] Step 6: Instruct the robot to move

[0212] The server generates and executes operational instructions for the robot based on the generated assembly procedure information. Specifically, the instructions include specific operational instructions for the robot to accurately position and fix parts. The input is the assembly procedure information, and the output is the robot's operational instructions. The specific operation is that the server sends the operational instructions to the robot's control program, and the robot starts operating in accordance with the instructions.

[0213] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0214] This invention relates to a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations based on those emotions. The operation and configuration of the entire system are explained below, along with specific procedures and examples.

[0215] Overall system configuration

[0216] 1. User Device:

[0217] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[0218] 2. Server:

[0219] The server is a central device with key functions such as instruction processing, video generation, analysis, emotion recognition, etc. Multiple modules and engines work together.

[0220] 3. Emotion Engine:

[0221] This engine analyzes the user's facial expressions and voice to recognize their emotional state, and adapts the system's behavior according to the user's emotions.

[0222] Invention Implementation Process

[0223] 1. Upload the instructions

[0224] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[0225] Terminal: Receives the uploaded instruction file and sends it to the server.

[0226] 2. Analysis of Instructions

[0227] Server: The server analyzes the received instruction file.

[0228] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[0229] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[0230] Example: When a PDF of furniture instructions uploaded by a user arrives at the server, the OCR function is activated and extracts assembly steps such as "fix part A to part B." At the same time, image recognition technology identifies the diagrams of parts A and B and stores them in a database.

[0231] 3. Interpretation of instructional information

[0232] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[0233] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[0234] Example: The server uses a semantic analysis engine to interpret the instruction "Secure part A to part B with screws" and clarifies the specific alignment of the parts, the type of screw, and how to use it.

[0235] 4. Generating assembly videos

[0236] Server: Generates assembly animation based on assembly procedure information.

[0237] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[0238] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, screw rotation motion, etc.

[0239] Example: The server simulates the steps for assembling furniture using 3D modeling software and animates the process. The generated video reproduces the steps, such as "Place part A here, then place part B on top of it and fasten it with screws."

[0240] 5. Operational adaptation using emotion engine

[0241] Server: The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotional state.

[0242] Example: If the server recognizes the user's facial expression and determines that the user is confused, it will take appropriate action, such as slowing down the video playback speed.

[0243] 6. Video distribution

[0244] Server: Stores the completed assembly video and generates streaming and download links.

[0245] Device: Displays the video URL sent from the server in the user interface, allowing the user to stream or download.

[0246] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[0247] Example: As a user plays a video, an emotion engine monitors the user's emotional state and takes appropriate action, such as adjusting the playback speed or displaying additional information.

[0248] summary

[0249] This system allows users to easily understand the visual assembly procedures and assemble products accurately by simply uploading instructions. Furthermore, by using an emotion engine, it can provide optimal operations according to the user's emotions, improving the efficiency and satisfaction of assembly work.

[0250] The processing flow will be explained below.

[0251] Step 1:

[0252] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDF format.

[0253] Step 2:

[0254] Terminal: Receives the uploaded instruction file and sends it to the server.

[0255] Step 3:

[0256] Server: The server analyzes the received instruction file.

[0257] Step 4:

[0258] Server: Starts the OCR engine and converts the text information in the manual into digital text data.

[0259] Step 5:

[0260] Server: Extracts the text data after OCR processing, and simultaneously uses an image recognition algorithm to identify part information from diagrams and illustrations in the manual.

[0261] Step 6:

[0262] Server: The extracted text data and image data are input into a semantic analysis engine to analyze the relationships and meaning between procedures.

[0263] Step 7:

[0264] Server: Generates detailed assembly procedure information from the analysis results using a semantic analysis engine.

[0265] Step 8:

[0266] Server: Based on the assembly procedure information, a 3D model of the part is generated using 3D modeling software.

[0267] Step 9:

[0268] Server: The generated 3D model is input into a video generation engine, which records the assembly procedure as a video while simulating it.

[0269] Step 10:

[0270] Server: Renders the final assembled video, saves it, and generates a URL for the video file.

[0271] Step 11:

[0272] Server: Activates the emotion engine, analyzes the user's facial expressions and voice in real time, and recognizes their emotional state.

[0273] Step 12:

[0274] Server: Adapts the pace and content of video playback based on emotion recognition results from the emotion engine. For example, if a confused expression is detected, commentary may be added or playback speed may be slowed down.

[0275] Step 13:

[0276] Device: Displays the video URL received from the server in the user interface, allowing the user to stream or download.

[0277] Step 14:

[0278] Users: Click on the video link to watch the video and then assemble the parts by following the instructions in the video. During playback, users can pause, rewind, fast forward, and more.

[0279] Step 15:

[0280] Server: The emotion engine continuously monitors the user's emotions while the video is playing and adapts in real time if the user becomes confused.

[0281] Specific examples

[0282] When a user uploads a PDF instruction manual for the furniture they purchased to the system, the server receives the PDF and extracts the text data using OCR technology. At the same time, an image recognition algorithm identifies the parts diagrams in the manual. A semantic analysis engine analyzes the text and image data to generate specific assembly instructions. 3D modeling software then creates 3D models of each part, and uses them to generate assembly instructions as videos.

[0283] The generated video is then delivered to the user, who then watches the video while performing the assembly task. While the video is playing, the emotion engine monitors the user's facial expressions, and if it detects that the user is confused, it slows down the video playback speed or displays additional explanations to help the user understand. This approach allows the user to understand the instructions more intuitively and efficiently, and perform the assembly task accurately.

[0284] Example 2

[0285] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0286] Conventional assembly manuals require a lot of time and effort for users to understand the procedures, placing a significant burden on beginners and those who have difficulty reading manuals. Furthermore, because they do not adapt to the user's emotional state, they lack support for difficult-to-understand parts. This results in problems such as incorrect assembly methods and increased stress during the process.

[0287] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animation based on the generated assembly procedure information, means for recognizing the user's emotional state and adapting operations based on the recognized emotional state, and means for delivering the generated assembly animation to the user. This allows the user to easily understand the visual assembly procedures simply by uploading the instructions, and makes it possible to provide optimal support according to the user's emotional state.

[0288] "Instructions" are documents that explain the assembly procedures and usage of a product or part.

[0289] "Input means" refers to the device or software used to load instructions into the system.

[0290] "Text data" is a digital representation of the textual information contained in the instructions.

[0291] "Image data" means a digital representation of the diagrams and illustrations contained in the instructions.

[0292] The "extracting means" refers to a method or device for extracting text data and image data from the instructions.

[0293] "Means for analyzing" refers to methods or devices for processing extracted digital data to make sense of it and derive information.

[0294] "Assembly procedure information" is digital information that indicates the specific steps and methods required to assemble a product or part.

[0295] "Generating means" refers to a method or device for creating new data or information based on the analysis results.

[0296] "Assembly animation" is content in the form of animation that visually shows assembly procedures.

[0297] A "user" is a person who uses the system to read the instructions and watch the assembly animations.

[0298] "Emotional state" refers to a user's current mental and emotional state.

[0299] The "recognition means" is a method or device for identifying the emotional state of a user from their facial expression or voice.

[0300] "Means for adapting operations" are methods or devices for modifying system behavior or settings based on a recognized emotional state.

[0301] The "distribution means" refers to a method or device for providing the generated assembled animation to a user.

[0302] This invention relates to a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations based on those emotions.

[0303] Overall system configuration

[0304] User terminal

[0305] The user terminal is a device that users use to upload instructions and view generated assembly videos. Examples include personal computers (PCs) and smartphones. Users access the system from their terminals and perform various operations (e.g., uploading files, viewing videos).

[0306] server

[0307] The server is the hardware that manages the core functions of the system, such as data processing of instructions, video generation, analysis, and emotion recognition. The following software and engines run on the server:

[0308] Optical Character Recognition (OCR) software (e.g., Tesseract): converts the text in the instructions into digital data.

[0309] Image recognition engine (e.g., OpenCV, YOLO): Extracts component information from diagrams and illustrations in the manual.

[0310] Natural language processing engine (e.g., SpaCy, BERT): Analyzes the extracted data and generates assembly instructions.

[0311] 3D modeling software (e.g. Blender): Create 3D models of parts and perform simulations.

[0312] Video generation engine: Creates assembly videos based on the generated 3D models.

[0313] Emotion recognition software (e.g., facial expression analysis, voice analysis): Recognizes the user's emotional state and performs corresponding adaptive processing.

[0314] Emotion Engine

[0315] The emotion engine is software that analyzes the user's facial expressions and voice to recognize the user's emotional state and adapt the system's behavior accordingly. For example, if the user is confused, it can adjust the video playback speed or display additional explanations.

[0316] Invention Implementation Process

[0317] Specific example explanation

[0318] Let's say a user uploads a PDF of a piece of furniture's instructions from their computer. Here's what happens:

[0319] 1. Upload the instructions

[0320] The user selects the assembly manual PDF from the file manager on their PC and clicks the "Upload" button. The device then sends this file to the server as an HTTP POST request.

[0321] 2. Analysis of Instructions

[0322] The server processes the received instruction manual file. First, the OCR engine converts the text into digital data, and then the image recognition engine extracts part information from diagrams and illustrations. For example, it extracts the text "Fix part A to part B" and recognizes the diagrams of parts A and B.

[0323] 3. Interpretation of instructional information

[0324] The server uses a natural language processing engine to analyze text and image data and generate assembly instructions. It then interprets the extracted instruction, "Secure part A to part B with screws," in detail to identify the specific positions of the parts and the types of screws.

[0325] 4. Generating assembly videos

[0326] The server uses 3D modeling software to generate a 3D model of the part based on the assembly procedure information, and then simulates it.The animation generation engine uses the simulation results to create an assembly animation that shows the specific steps.

[0327] 5. Operational adaptation using emotion engine

[0328] While the user is watching the video, the server analyzes their facial expressions and voice to recognize their emotional state. If it determines that the user is confused, it will take appropriate action, such as slowing down the video playback speed or displaying supplementary explanations.

[0329] 6. Video distribution

[0330] The generated assembly video is saved on the server and a streaming URL is generated. The URL is sent to the device, and the user can click the "Play Video" button to watch the video. The video can also be downloaded if necessary.

[0331] Prompt Sentence Examples

[0332] Here are some examples of prompts for generative AI models:

[0333] A user uploaded a PDF instruction manual for assembling furniture. The server then used OCR technology to convert the text in the manual into digital text data and image recognition technology to extract component information. It then used a natural language processing engine to interpret the relationships between steps and generated an assembly video using 3D modeling software. It also used emotion recognition software to analyze the user's emotions and adapt the playback speed and supplementary explanations accordingly. If the user showed a confused expression, the server slowed down the video playback speed and displayed supplementary explanations at key points. Please provide an overview of this system and its steps, with examples.

[0334] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0335] Step 1:

[0336] Upload instructions

[0337] (User)

[0338] The user selects the assembly manual as an image file (JPEG, PNG) or PDF file on their device and presses the upload button. This operation selects the manual file.

[0339] (Terminal)

[0340] The terminal sends the instruction file selected by the user to the server as an HTTP POST request.

[0341] The input is a user-selected description file (JPEG, PNG or PDF) and the output is that file is sent to the server.

[0342] Specific behavior:

[0343] The user selects the instruction manual PDF from the file manager on their PC and clicks the upload button on the web application. The device sends the file to the server.

[0344] Step 2:

[0345] Analysis of the instructions

[0346] (server)

[0347] The server analyzes the received instruction manual file. First, it starts an OCR engine (e.g., Tesseract) to digitize the text in the manual. Next, it uses an image recognition engine (e.g., OpenCV or YOLO) to extract part information from the diagrams and illustrations in the manual.

[0348] The input is a manual file, and the output is text data extracted by OCR and part information identified by an image recognition engine.

[0349] Specific behavior:

[0350] The server uses OCR to extract the text from the instructions, "Fix part A to part B," and at the same time, the image recognition engine identifies the diagrams of parts A and B.

[0351] Step 3:

[0352] Interpreting the instruction manual information

[0353] (server)

[0354] Based on the extracted text data and image data, the server uses a natural language processing engine (e.g., SpaCy or BERT) to perform a detailed analysis of the text data and diagram data and generate assembly instructions.

[0355] The inputs include text data and image data from the OCR and image recognition engines, and the output is the generated assembly procedure information.

[0356] Specific behavior:

[0357] The server uses a natural language processing engine to interpret the instruction "fix part A to part B with a screw" in detail, identifying the exact part placement and type of screw, and generating digital assembly instructions.

[0358] Step 4:

[0359] Assembly video generation

[0360] (server)

[0361] Based on the generated assembly procedure information, the server uses 3D modeling software (such as Blender) to create a 3D model of the part. Based on this 3D model, the animation generation engine simulates the assembly procedure and generates an assembly animation that shows the specific movements.

[0362] The input is assembly procedure information from a natural language processing engine, and the output is a generated assembly video.

[0363] Specific behavior:

[0364] The 3D modeling software creates detailed 3D models of parts A and B, and then the video generation engine simulates these to generate a video of specific steps, such as "Place part A here, place part B on top of it, and secure it with screws."

[0365] Step 5:

[0366] Emotional engine for adaptive operation

[0367] (server)

[0368] As users watch videos, the server uses an emotion engine to analyze their facial expressions and voice to recognize their emotional state. If the user is confused, the server will adjust the video playback speed or display additional explanations.

[0369] Inputs include facial expression data and voice data of the user, and outputs include adjusting the playback speed and displaying supplementary explanations.

[0370] Specific behavior:

[0371] If a user shows a confused expression while watching a video, the server slows the video playback speed by 1.5 times and displays explanatory text on the screen at key points.

[0372] Step 6:

[0373] Video distribution

[0374] (server)

[0375] The completed assembly video is saved, and the server generates a URL for streaming or a download link.

[0376] (Terminal)

[0377] The terminal displays the video URL sent from the server on the user interface, allowing the user to stream or download the video.

[0378] (User)

[0379] Users can click the "Play Video" button on their device to watch the streamed video, pausing, rewinding, fast-forwarding, and other operations as needed.

[0380] The input is the generated assembled video, and the output is a streaming URL or download link provided to the user.

[0381] Specific behavior:

[0382] The user clicks the "Play Video" button on the device to play the generated video. If the user shows a confused expression during playback, the emotion engine adjusts the playback speed.

[0383] (Application example 2)

[0384] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0385] Understanding assembly procedures based on instructions is often difficult, especially for beginners or those working for the first time. Furthermore, the confusion and stress felt by users during the process can reduce work efficiency and accuracy. Conventional systems are unable to provide real-time assistance that responds to the user's emotions, making it difficult to improve work efficiency.

[0386] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animations based on the generated assembly procedure information, means for recognizing the user's emotions and adapting operations based on the emotions, and means for delivering the generated assembly animations to the user. This not only makes it easier for the user to understand the instructions, but also makes it possible to improve work efficiency and accuracy by providing support according to the user's emotions.

[0387] "Instructions" are documents that describe how to assemble or use a product or part.

[0388] "Input means" refers to a device or function that allows a user to load instructions into the system.

[0389] "Extraction means" refers to the technology or device used to extract text data or image data from the instructions.

[0390] The "means for analyzing and generating assembly procedure information" refers to a technique or device for understanding the extracted data and determining specific assembly procedures based on that data.

[0391] The "means for generating assembly animation" refers to a technique or device for creating a visual animation based on assembly procedure information.

[0392] "Distribution means" refers to the technology and devices used to deliver the generated assembly animation to the user's terminal.

[0393] "Means for recognizing user emotions" refers to technology or devices that analyze the user's facial expressions and voice and determine their emotional state.

[0394] "Operational adaptation" refers to techniques and devices for modifying system behavior based on perceived user emotions.

[0395] Overall system configuration

[0396] This invention is a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations accordingly. Specifically, it includes the following components:

[0397] 1. User Device:

[0398] A device used by users to upload instructions to the system and view the generated videos. Typically, a PC, smartphone, tablet, or other device is used.

[0399] 2. Server:

[0400] It is a central device with key functions such as instruction processing, video generation, analysis, and emotion recognition. Specifically, it includes OCR technology, image recognition technology, semantic analysis engine, 3D modeling software, and emotion recognition engine.

[0401] 3. Emotion Recognition Engine:

[0402] This technology analyzes a user's facial expressions and voice to recognize their emotional state. For example, it can use deep learning models such as Keras.

[0403] Operation explanation

[0404] 1. Upload the instructions

[0405] Users upload instructions for products such as furniture and plastic models to the system as image files or PDFs from their devices. This operation is performed through the user interface.

[0406] 2. Analysis of Instructions

[0407] The server analyzes the received instruction manual file, converting the text in the manual into digital data using OCR technology, and extracting part information from diagrams and illustrations using image recognition technology.

[0408] 3. Generating assembly procedure information

[0409] The extracted text data and image data are subjected to semantic analysis to generate assembly procedure information. The software used for this process includes a semantic analysis engine and a natural language processing library.

[0410] 4. Generating assembly videos

[0411] The server creates a 3D model based on the generated assembly procedure information, performs simulations, and finally generates an assembly video. This process is supported by a video generation engine.

[0412] 5. Emotion Recognition and Adaptive Manipulation

[0413] The server analyzes the user's facial expressions and voice to recognize their emotional state. For example, if it determines that the user is confused, it will take adaptive action, such as slowing down the video playback speed.

[0414] Specific examples

[0415] For example, consider the case of generating a video showing assembly procedures for a robot arm in a factory. When a worker uploads instructions, OCR and image recognition technology analyzes the instruction data and generates a video of the assembly procedures based on that. As the worker watches the video and performs the work, an emotion recognition engine monitors their emotions and adjusts the video playback speed or supplementary explanations as needed.

[0416] Prompt Sentence Examples

[0417] Upload a PDF of your furniture assembly instructions and we'll generate a video of the assembly process using adaptive control based on emotion recognition.

[0418] This system makes it easier for users to understand instructions and provides appropriate support according to their emotions, improving work efficiency and accuracy.

[0419] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0420] Step 1:

[0421] The user uploads the assembly instruction manual as a PDF or image file from the terminal to the system. The terminal sends the uploaded file to the server. The input is the assembly instruction file, and the output is the file sent to the server.

[0422] Step 2:

[0423] The server analyzes the received instruction manual file using OCR technology (e.g., Tesseract OCR) and extracts text data. It also uses image recognition technology (e.g., OpenCV or TENSORFLOW (registered trademark)) to extract component information from diagrams and illustrations in the manual. The input is the instruction manual file, and the output is text data and image data.

[0424] Step 3:

[0425] The server runs the extracted text data and image data through a semantic analysis engine to generate assembly procedure information. This uses a natural language processing library (e.g., NLTK or spaCy). The input is text data and image data, and the output is assembly procedure information.

[0426] Step 4:

[0427] The server uses 3D modeling software (e.g., Blender) based on the generated assembly procedure information to simulate each assembly step, and generates an assembly video using a video generation engine. The input is the assembly procedure information, and the output is the assembly video.

[0428] Step 5:

[0429] The server uses an emotion recognition engine (e.g., a deep learning model using Keras) to analyze the user's real-time facial expressions and voice. The server recognizes the user's emotional state and adjusts the video playback speed or displays supplementary explanations as needed. The input is the user's facial expression data and voice data, and the output is adjusted video playback and supplementary explanations.

[0430] Step 6:

[0431] The server delivers the generated assembled video to the user terminal. The terminal displays the video URL sent from the server on the user interface, allowing the user to stream or download. The input is the assembled video file, and the output is the generation and delivery of the video URL.

[0432] Step 7:

[0433] The user watches the assembly video on a device and actually performs the assembly work according to the displayed steps. While the video is playing, an emotion recognition engine monitors the user's emotional state and adjusts the playback speed or displays supplementary explanations as necessary. The input is emotion data, and the output is the adjusted assembly video and supplementary explanations.

[0434] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0435] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0436] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0437] [Second embodiment]

[0438] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0439] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0440] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0441] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0442] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0443] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0444] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0445] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0446] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0447] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0448] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0449] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0450] This invention relates to a system that automatically generates assembly animations based on instructions. The operation and configuration of the entire system will be explained below with specific procedures and examples.

[0451] Overall system configuration

[0452] 1. User Device:

[0453] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[0454] 2. Server:

[0455] The server is a central device with key functions such as instruction processing, video generation, analysis, etc. Multiple modules and engines work together.

[0456] Invention Implementation Process

[0457] 1. Upload the instructions

[0458] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[0459] Terminal: Receives the uploaded instruction file and sends it to the server.

[0460] 2. Analysis of Instructions

[0461] Server: The server analyzes the received instruction file.

[0462] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[0463] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[0464] Example: When a PDF of furniture instructions uploaded by a user arrives at the server, the OCR function is activated and extracts assembly steps such as "fix part A to part B." At the same time, image recognition technology identifies the diagrams of parts A and B and stores them in a database.

[0465] 3. Interpretation of instructional information

[0466] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[0467] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[0468] Example: The server uses a semantic analysis engine to interpret the instruction "Secure part A to part B with screws" and clarifies the specific alignment of the parts, the type of screw, and how to use it.

[0469] 4. Generating assembly videos

[0470] Server: Generates assembly animation based on assembly procedure information.

[0471] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[0472] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, screw rotation motion, etc.

[0473] Example: The server simulates the steps for assembling furniture using 3D modeling software and animates the process. The generated video reproduces the steps, such as "Place part A here, then place part B on top of it and fasten it with screws."

[0474] 5. Video distribution

[0475] Server: Stores the completed assembly video and generates streaming and download links.

[0476] Terminal: Receives the video link sent from the server and displays it to the user.

[0477] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[0478] summary

[0479] This system allows users to simply upload instructions and easily understand the visual assembly steps, allowing them to assemble products accurately, significantly reducing the time and effort required for assembly and improving the user experience.

[0480] The processing flow will be explained below.

[0481] Step 1:

[0482] User: Logs in to the system and uploads instructions for the product to be assembled in image or PDF format.

[0483] Step 2:

[0484] Terminal: Receives the uploaded instruction file and sends it to the server.

[0485] Step 3:

[0486] Server: Receives the instruction manual file. After receiving it, it starts the OCR engine and converts the text information in the manual into digital text data.

[0487] Step 4:

[0488] Server: Extracts the text data after OCR processing, and simultaneously uses an image recognition algorithm to identify part information from diagrams and illustrations in the manual.

[0489] Step 5:

[0490] Server: The extracted text data and image data are input into a semantic analysis engine to analyze the relationships and meaning between procedures.

[0491] Step 6:

[0492] Server: Generates detailed assembly procedure information from the analysis results using a semantic analysis engine.

[0493] Step 7:

[0494] Server: Based on the assembly procedure information, a 3D model of the part is generated using 3D modeling software.

[0495] Step 8:

[0496] Server: The generated 3D model is input into a video generation engine, which records the assembly procedure as a video while simulating it.

[0497] Step 9:

[0498] Server: Renders the final assembled video, saves it, and generates a URL for the video file.

[0499] Step 10:

[0500] Device: Displays the video URL received from the server in the user interface, allowing the user to stream or download.

[0501] Step 11:

[0502] Users: Click on the video link to watch the video and then follow the instructions in the video to assemble the product. During playback, users can pause, rewind, fast forward, and more.

[0503] Example 1

[0504] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0505] Traditional assembly instructions consist of only text and drawings, making them difficult for users to understand and often resulting in time-consuming assembly tasks. Misunderstandings and mistakes at certain steps can easily occur, leading to assembly failures and frustration. Furthermore, manually creating assembly videos requires specialized knowledge and time, resulting in significant effort.

[0506] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0507] In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animations based on the generated assembly procedure information, means for delivering the generated assembly animations to users, and means for identifying specific software modules (OCR technology, semantic analysis engine, 3D modeling software, etc.) used in each processing step, thereby enabling users to easily understand the visual assembly procedures and accurately assemble the product.

[0508] "Means for inputting instructions" refers to a function that allows users to upload image files or PDF format instructions to the system via their terminals.

[0509] "Means for extracting text data and image data" is a processing function for extracting text information and image information from the uploaded instructions.

[0510] The "means for analyzing the extracted text data and image data and generating assembly procedure information" is an analysis function for understanding the assembly procedure based on the extracted data and generating specific assembly instructions.

[0511] The "means for generating assembly animation" is a function for creating animation that visually represents assembly work using 3D models of parts based on the generated assembly procedure information.

[0512] The "means for distributing assembly animation to users" is a function for saving the generated assembly animation and providing it so that users can view or download it.

[0513] "OCR technology" is an abbreviation for Optical Character Recognition, and is a technology that converts image-formatted characters into digital text data.

[0514] A "semantic analysis engine" is an analysis engine that analyzes text and image data and understands their meaning.

[0515] "3D modeling software" is software for creating and editing three-dimensional objects and scenes on a computer.

[0516] This invention relates to a system that automatically generates assembly animations based on instructions. The operation and configuration of the entire system will be explained below with specific procedures and examples.

[0517] System configuration

[0518] User terminal

[0519] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[0520] server

[0521] The server is a central device with key functions such as instruction processing, video generation, analysis, etc. Multiple modules and engines work together.

[0522] Feature details

[0523] Upload instructions

[0524] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[0525] Terminal: Receives the uploaded instruction file and sends it to the server.

[0526] Analysis of the instructions

[0527] Server: The server analyzes the received instruction file.

[0528] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[0529] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[0530] Interpreting the instruction manual information

[0531] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[0532] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[0533] Assembly video generation

[0534] Server: Generates assembly animation based on assembly procedure information.

[0535] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[0536] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, and screw rotation motion.

[0537] Video distribution

[0538] Server: Stores the completed assembly video and generates streaming and download links.

[0539] Terminal: Receives the video link sent from the server and displays it to the user.

[0540] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[0541] Specific examples

[0542] When using furniture assembly instructions

[0543] The user uploads a PDF file of furniture instructions from their device. The server uses OCR to extract the text information, such as "Secure part A to part B with screws," and uses image recognition to identify the diagrams of parts A and B. The semantic analysis engine analyzes the text and interprets the specific steps (such as aligning the parts and the type of screws to use). The steps are animated using 3D modeling software, and the server sends the generated video link to the user's device. The user clicks the link and watches the video to accurately assemble the furniture.

[0544] Example prompts for generative AI models

[0545] I have uploaded assembly instructions for a piece of furniture. These instructions include the step of fastening part A to part B with screws. Please generate a 3D model and an assembly video based on these instructions.

[0546] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0547] Step 1: Upload your instructions

[0548] Input: Instructions (image file or PDF format) that come with items that users assemble, such as furniture or plastic models

[0549] Specific operation: The user clicks the "Upload instructions" button on the device, a local file selection dialog appears, the user selects the assembly instructions file and presses the "Open" button, and the file is uploaded to the system.

[0550] Output: The device displays a notification that the upload is complete and sends the instruction file to the server.

[0551] Step 2: Parse the instructions

[0552] Input: Uploaded instruction file

[0553] Specific operation: The server inputs the received instruction manual file into the analysis module. Using OCR, the text in the manual is converted into digital text data. At the same time, image recognition is performed to extract part information from diagrams and illustrations.

[0554] Output: Extracted text and image data

[0555] Step 3: Interpret the information in the instructions

[0556] Input: Extracted text and image data

[0557] Specific operation: The server inputs the data obtained through OCR and image recognition into a semantic analysis engine to generate assembly procedure information. For example, a command such as "Secure part A to part B with screws" is analyzed and the specific steps, tools required, and part alignment methods are interpreted.

[0558] Output: Assembly procedure information

[0559] Step 4: Generate assembly videos

[0560] Input: Assembly procedure information

[0561] Specific operation: The server uses 3D modeling software to generate a 3D model of the specified part. It then uses the generated 3D model to simulate the assembly procedure, and animates the simulation results using a video generation engine. The video includes the part selection procedure, precise alignment, screw rotation, and more.

[0562] Output: Assembly video file

[0563] Step 5: Publish your video

[0564] Input: Assembly video file

[0565] Specific operation: The server saves the assembled video and generates a URL or download link for streaming the video file. The server then sends the URL or download link to the device and notifies the user.

[0566] Output: User-accessible video URL and download link

[0567] Step 6: Watch the video and assemble

[0568] Input: Video URL or download link

[0569] Specific operation: The user clicks on the provided URL on their device to watch the assembly video. The video is streamed and the user can pause, rewind, fast forward, etc. The user then performs the actual assembly work by following the video.

[0570] Output: Understanding the assembled product and the exact assembly steps along the way

[0571] (Application example 1)

[0572] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0573] In modern manufacturing factories, efficient execution of complex assembly tasks is important, but they often require a great deal of effort and time. In particular, assembling new products requires workers to understand the manual and accurately grasp the procedures. However, differences in the ability to read and understand instructions and the experience of actually assembling by hand vary from person to person, leading to variations in quality and errors. Furthermore, as automation using robots advances, it is necessary for the robots to properly understand and reliably execute assembly procedures. To solve these problems and achieve efficient and uniform assembly work, a system is needed that clarifies assembly procedures and enables robots to accurately execute them.

[0574] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0575] In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animation based on the generated assembly procedure information, means for distributing the generated assembly animation to a user, and means for generating and executing operation instructions for a robot based on the assembly procedure information. This not only provides assembly procedures that are easy to understand visually, but also enables the robot to perform assembly work accurately.

[0576] "Instructions" are documents that show how to assemble or operate a product.

[0577] "Text data" is text information extracted from the instructions.

[0578] "Image data" refers to visual information such as diagrams and illustrations extracted from the manual.

[0579] "Assembly procedure information" is information about a specific method for assembling a product, generated by analyzing text data and image data.

[0580] "OCR (Optical Character Recognition) technology" is a technology that converts characters in analog format, such as on paper or in images, into digital format.

[0581] A "semantic analysis engine" is software that analyzes the meaning of text and image data and understands their relationships.

[0582] "Video generation" is the process of creating a visually easy-to-view video based on assembly procedure information.

[0583] "Distribution" is the process of sending the generated video to users.

[0584] A "robot" is a machine that operates automatically according to a program and can perform assembly tasks.

[0585] The "operation instruction" is a specific instruction for the robot to perform an assembly task.

[0586] This invention relates to a system that automatically generates assembly animations based on instructions and provides operational instructions to a robot. The operation and configuration of the entire system will be explained below, along with specific procedures and examples.

[0587] Overall system configuration

[0588] 1. User Device

[0589] The user terminal is a device used to upload instructions and view the generated videos. Users access the system using a smartphone or tablet and operate it.

[0590] 2. Server

[0591] The server is a central device with key functions such as processing instructions, generating videos, analyzing them, and generating instructions for the robot. Multiple modules and engines work together.

[0592] Invention Implementation Process

[0593] Upload instructions

[0594] Users upload instructions for products such as furniture and plastic models to the system as image files or PDFs from their devices. The devices receive the uploaded instruction files and send them to the server.

[0595] Analysis of the instructions

[0596] The server uses OCR and image recognition technology to analyze the received instruction manual file. OCR technology (e.g., Tesseract) is used to convert the text into digital text data, and image recognition technology (e.g., OpenCV) is used to extract part information from diagrams and illustrations.

[0597] Interpreting the instruction manual information

[0598] The server analyzes the extracted text and image data to generate assembly procedure information. This analysis is performed using a semantic analysis engine, which analyzes the meaning of the text and image data and understands their relationships.

[0599] Assembly video generation

[0600] The server generates an assembly video based on the assembly procedure information. It uses 3D modeling software (e.g., Blender) to generate a 3D model of the part and performs a simulation. It uses a video generation engine (e.g., FFmpeg) to create an assembly video based on the generated 3D model and simulation results.

[0601] Video distribution

[0602] The server stores the completed assembly video and generates a streaming distribution or download link. The device receives the video link sent from the server and displays it to the user. The user can watch the video and actually perform the assembly work by following the displayed steps.

[0603] Instructions for the robot

[0604] Based on the assembly procedure information, the server generates and executes operation instructions for the robot, allowing the robot to accurately assemble the parts according to the generated instructions.

[0605] Specific examples

[0606] Let's say a user takes a photo of a piece of furniture instruction manual with their smartphone and uploads it to the application. The uploaded manual is converted into text using an OCR engine, and diagrams and illustrations are analyzed using an image recognition engine. Based on the analysis results, the server generates assembly procedures and simulates them using 3D modeling software. The generated video is sent to the user's device, and the user can proceed with the assembly work while watching it. Furthermore, a robot accurately performs the assembly work based on the operating instructions generated by the server.

[0607] Prompt Sentence Examples

[0608] Based on the furniture instructions below, please automatically analyze the assembly steps and generate the final assembly video.

[0609] "Secure part A to part B, then attach part C. Then place part D according to the instructions in step 3, and finally secure all parts together with screws."

[0610] Following this prompt, specific steps and component instructions are provided to ensure the model can be analyzed accurately.

[0611] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0612] Step 1: Upload your instructions

[0613] The user takes a photo of the instruction manual that comes with the product to be assembled using a smartphone or tablet, or saves it as a PDF file on the device. The device receives the photo or saved instruction file and sends it to the server. The input is an image file or PDF file of the instruction manual, and the output is the instruction data that is sent to the server. The specific operation is for the user to operate the application to select the file and press the upload button.

[0614] Step 2: Parse the instructions

[0615] The server analyzes the received instruction manual file using OCR and image recognition technology. Specifically, it uses an OCR engine (e.g., Tesseract) to convert the text into digital text data, and an image recognition engine (e.g., OpenCV) to extract part information from diagrams and illustrations. The input is the instruction manual data, and the output is text data and image data. Specifically, the server starts the OCR engine, scans the characters in the image, and converts them into text.

[0616] Step 3: Interpret the information in the instructions

[0617] The server analyzes the extracted text data and image data to generate assembly procedure information. A semantic analysis engine is used for this analysis. The input is text data and image data, and the output is assembly procedure information. Specifically, the semantic analysis engine analyzes the meaning of the text and understands the relationships between parts. For example, it interprets in detail the instruction "Secure part A to part B with a screw."

[0618] Step 4: Generate assembly videos

[0619] The server generates an assembly video based on the assembly procedure information. It uses 3D modeling software (e.g., Blender) to generate a 3D model of the part and performs a simulation. It uses a video generation engine (e.g., FFmpeg) to create an assembly video based on the generated 3D model and simulation results. The input is the assembly procedure information and the output is the assembly video. Specifically, the server starts the 3D modeling software, simulates the operation of the part, and encodes it as a video.

[0620] Step 5: Publish your video

[0621] The server stores the completed assembly video and generates a streaming distribution or download link. The terminal receives the video link sent from the server and displays it to the user. The input is the assembly video, and the output is the video link to the user's terminal. Specifically, the server uploads the video file to cloud storage and sends the link to the terminal.

[0622] Step 6: Instruct the robot to move

[0623] The server generates and executes operational instructions for the robot based on the generated assembly procedure information. Specifically, the instructions include specific operational instructions for the robot to accurately position and fix parts. The input is the assembly procedure information, and the output is the robot's operational instructions. The specific operation is that the server sends the operational instructions to the robot's control program, and the robot starts operating in accordance with the instructions.

[0624] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0625] This invention relates to a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations based on those emotions. The operation and configuration of the entire system are explained below, along with specific procedures and examples.

[0626] Overall system configuration

[0627] 1. User Device:

[0628] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[0629] 2. Server:

[0630] The server is a central device with key functions such as instruction processing, video generation, analysis, emotion recognition, etc. Multiple modules and engines work together.

[0631] 3. Emotion Engine:

[0632] This engine analyzes the user's facial expressions and voice to recognize their emotional state, and adapts the system's behavior according to the user's emotions.

[0633] Invention Implementation Process

[0634] 1. Upload the instructions

[0635] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[0636] Terminal: Receives the uploaded instruction file and sends it to the server.

[0637] 2. Analysis of Instructions

[0638] Server: The server analyzes the received instruction file.

[0639] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[0640] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[0641] Example: When a PDF of furniture instructions uploaded by a user arrives at the server, the OCR function is activated and extracts assembly steps such as "fix part A to part B." At the same time, image recognition technology identifies the diagrams of parts A and B and stores them in a database.

[0642] 3. Interpretation of instructional information

[0643] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[0644] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[0645] Example: The server uses a semantic analysis engine to interpret the instruction "Secure part A to part B with screws" and clarifies the specific alignment of the parts, the type of screw, and how to use it.

[0646] 4. Generating assembly videos

[0647] Server: Generates assembly animation based on assembly procedure information.

[0648] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[0649] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, screw rotation motion, etc.

[0650] Example: The server simulates the steps for assembling furniture using 3D modeling software and animates the process. The generated video reproduces the steps, such as "Place part A here, then place part B on top of it and fasten it with screws."

[0651] 5. Operational adaptation using emotion engine

[0652] Server: The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotional state.

[0653] Example: If the server recognizes the user's facial expression and determines that the user is confused, it will take appropriate action, such as slowing down the video playback speed.

[0654] 6. Video distribution

[0655] Server: Stores the completed assembly video and generates streaming and download links.

[0656] Device: Displays the video URL sent from the server in the user interface, allowing the user to stream or download.

[0657] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[0658] Example: As a user plays a video, an emotion engine monitors the user's emotional state and takes appropriate action, such as adjusting the playback speed or displaying additional information.

[0659] summary

[0660] This system allows users to easily understand the visual assembly procedures and assemble products accurately by simply uploading instructions. Furthermore, by using an emotion engine, it can provide optimal operations according to the user's emotions, improving the efficiency and satisfaction of assembly work.

[0661] The processing flow will be explained below.

[0662] Step 1:

[0663] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDF format.

[0664] Step 2:

[0665] Terminal: Receives the uploaded instruction file and sends it to the server.

[0666] Step 3:

[0667] Server: The server analyzes the received instruction file.

[0668] Step 4:

[0669] Server: Starts the OCR engine and converts the text information in the manual into digital text data.

[0670] Step 5:

[0671] Server: Extracts the text data after OCR processing, and simultaneously uses an image recognition algorithm to identify part information from diagrams and illustrations in the manual.

[0672] Step 6:

[0673] Server: The extracted text data and image data are input into a semantic analysis engine to analyze the relationships and meaning between procedures.

[0674] Step 7:

[0675] Server: Generates detailed assembly procedure information from the analysis results using a semantic analysis engine.

[0676] Step 8:

[0677] Server: Based on the assembly procedure information, a 3D model of the part is generated using 3D modeling software.

[0678] Step 9:

[0679] Server: The generated 3D model is input into a video generation engine, which records the assembly procedure as a video while simulating it.

[0680] Step 10:

[0681] Server: Renders the final assembled video, saves it, and generates a URL for the video file.

[0682] Step 11:

[0683] Server: Activates the emotion engine, analyzes the user's facial expressions and voice in real time, and recognizes their emotional state.

[0684] Step 12:

[0685] Server: Adapts the pace and content of video playback based on emotion recognition results from the emotion engine. For example, if a confused expression is detected, commentary may be added or playback speed may be slowed down.

[0686] Step 13:

[0687] Device: Displays the video URL received from the server in the user interface, allowing the user to stream or download.

[0688] Step 14:

[0689] Users: Click on the video link to watch the video and then assemble the parts by following the instructions in the video. During playback, users can pause, rewind, fast forward, and more.

[0690] Step 15:

[0691] Server: The emotion engine continuously monitors the user's emotions while the video is playing and adapts in real time if the user becomes confused.

[0692] Specific examples

[0693] When a user uploads a PDF instruction manual for the furniture they purchased to the system, the server receives the PDF and extracts the text data using OCR technology. At the same time, an image recognition algorithm identifies the parts diagrams in the manual. A semantic analysis engine analyzes the text and image data to generate specific assembly instructions. 3D modeling software then creates 3D models of each part, and uses them to generate assembly instructions as videos.

[0694] The generated video is then delivered to the user, who then watches the video while performing the assembly task. While the video is playing, the emotion engine monitors the user's facial expressions, and if it detects that the user is confused, it slows down the video playback speed or displays additional explanations to help the user understand. This approach allows the user to understand the instructions more intuitively and efficiently, and perform the assembly task accurately.

[0695] Example 2

[0696] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0697] Conventional assembly manuals require a lot of time and effort for users to understand the procedures, placing a significant burden on beginners and those who have difficulty reading manuals. Furthermore, because they do not adapt to the user's emotional state, they lack support for difficult-to-understand parts. This results in problems such as incorrect assembly methods and increased stress during the process.

[0698] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animation based on the generated assembly procedure information, means for recognizing the user's emotional state and adapting operations based on the recognized emotional state, and means for delivering the generated assembly animation to the user. This allows the user to easily understand the visual assembly procedures simply by uploading the instructions, and makes it possible to provide optimal support according to the user's emotional state.

[0699] "Instructions" are documents that explain the assembly procedures and usage of a product or part.

[0700] "Input means" refers to the device or software used to load instructions into the system.

[0701] "Text data" is a digital representation of the textual information contained in the instructions.

[0702] "Image data" means a digital representation of the diagrams and illustrations contained in the instructions.

[0703] The "extracting means" refers to a method or device for extracting text data and image data from the instructions.

[0704] "Means for analyzing" refers to methods or devices for processing extracted digital data to make sense of it and derive information.

[0705] "Assembly procedure information" is digital information that indicates the specific steps and methods required to assemble a product or part.

[0706] "Generating means" refers to a method or device for creating new data or information based on the analysis results.

[0707] "Assembly animation" is content in the form of animation that visually shows assembly procedures.

[0708] A "user" is a person who uses the system to read the instructions and watch the assembly animations.

[0709] "Emotional state" refers to a user's current mental and emotional state.

[0710] The "recognition means" is a method or device for identifying the emotional state of a user from their facial expression or voice.

[0711] "Means for adapting operations" are methods or devices for modifying system behavior or settings based on a recognized emotional state.

[0712] The "distribution means" refers to a method or device for providing the generated assembled animation to a user.

[0713] This invention relates to a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations based on those emotions.

[0714] Overall system configuration

[0715] User terminal

[0716] The user terminal is a device that users use to upload instructions and view generated assembly videos. Examples include personal computers (PCs) and smartphones. Users access the system from their terminals and perform various operations (e.g., uploading files, viewing videos).

[0717] server

[0718] The server is the hardware that manages the core functions of the system, such as data processing of instructions, video generation, analysis, and emotion recognition. The following software and engines run on the server:

[0719] Optical Character Recognition (OCR) software (e.g., Tesseract): converts the text in the instructions into digital data.

[0720] Image recognition engine (e.g., OpenCV, YOLO): Extracts component information from diagrams and illustrations in the manual.

[0721] Natural language processing engine (e.g., SpaCy, BERT): Analyzes the extracted data and generates assembly instructions.

[0722] 3D modeling software (e.g. Blender): Create 3D models of parts and perform simulations.

[0723] Video generation engine: Creates assembly videos based on the generated 3D models.

[0724] Emotion recognition software (e.g., facial expression analysis, voice analysis): Recognizes the user's emotional state and performs corresponding adaptive processing.

[0725] Emotion Engine

[0726] The emotion engine is software that analyzes the user's facial expressions and voice to recognize the user's emotional state and adapt the system's behavior accordingly. For example, if the user is confused, it can adjust the video playback speed or display additional explanations.

[0727] Invention Implementation Process

[0728] Specific example explanation

[0729] Let's say a user uploads a PDF of a piece of furniture's instructions from their computer. Here's what happens:

[0730] 1. Upload the instructions

[0731] The user selects the assembly manual PDF from the file manager on their PC and clicks the "Upload" button. The device then sends this file to the server as an HTTP POST request.

[0732] 2. Analysis of Instructions

[0733] The server processes the received instruction manual file. First, the OCR engine converts the text into digital data, and then the image recognition engine extracts part information from diagrams and illustrations. For example, it extracts the text "Fix part A to part B" and recognizes the diagrams of parts A and B.

[0734] 3. Interpretation of instructional information

[0735] The server uses a natural language processing engine to analyze text and image data and generate assembly instructions. It then interprets the extracted instruction, "Secure part A to part B with screws," in detail to identify the specific positions of the parts and the types of screws.

[0736] 4. Generating assembly videos

[0737] The server uses 3D modeling software to generate a 3D model of the part based on the assembly procedure information, and then simulates it.The animation generation engine uses the simulation results to create an assembly animation that shows the specific steps.

[0738] 5. Operational adaptation using emotion engine

[0739] While the user is watching the video, the server analyzes their facial expressions and voice to recognize their emotional state. If it determines that the user is confused, it will take appropriate action, such as slowing down the video playback speed or displaying supplementary explanations.

[0740] 6. Video distribution

[0741] The generated assembly video is saved on the server and a streaming URL is generated. The URL is sent to the device, and the user can click the "Play Video" button to watch the video. The video can also be downloaded if necessary.

[0742] Prompt Sentence Examples

[0743] Here are some examples of prompts for generative AI models:

[0744] A user uploaded a PDF instruction manual for assembling furniture. The server then used OCR technology to convert the text in the manual into digital text data and image recognition technology to extract component information. It then used a natural language processing engine to interpret the relationships between steps and generated an assembly video using 3D modeling software. It also used emotion recognition software to analyze the user's emotions and adapt the playback speed and supplementary explanations accordingly. If the user showed a confused expression, the server slowed down the video playback speed and displayed supplementary explanations at key points. Please provide an overview of this system and its steps, with examples.

[0745] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0746] Step 1:

[0747] Upload instructions

[0748] (User)

[0749] The user selects the assembly manual as an image file (JPEG, PNG) or PDF file on their device and presses the upload button. This operation selects the manual file.

[0750] (Terminal)

[0751] The terminal sends the instruction file selected by the user to the server as an HTTP POST request.

[0752] The input is a user-selected description file (JPEG, PNG or PDF) and the output is that file is sent to the server.

[0753] Specific behavior:

[0754] The user selects the instruction manual PDF from the file manager on their PC and clicks the upload button on the web application. The device sends the file to the server.

[0755] Step 2:

[0756] Analysis of the instructions

[0757] (server)

[0758] The server analyzes the received instruction manual file. First, it starts an OCR engine (e.g., Tesseract) to digitize the text in the manual. Next, it uses an image recognition engine (e.g., OpenCV or YOLO) to extract part information from the diagrams and illustrations in the manual.

[0759] The input is a manual file, and the output is text data extracted by OCR and part information identified by an image recognition engine.

[0760] Specific behavior:

[0761] The server uses OCR to extract the text from the instructions, "Fix part A to part B," and at the same time, the image recognition engine identifies the diagrams of parts A and B.

[0762] Step 3:

[0763] Interpreting the instruction manual information

[0764] (server)

[0765] Based on the extracted text data and image data, the server uses a natural language processing engine (e.g., SpaCy or BERT) to perform a detailed analysis of the text data and diagram data and generate assembly instructions.

[0766] The inputs include text data and image data from the OCR and image recognition engines, and the output is the generated assembly procedure information.

[0767] Specific behavior:

[0768] The server uses a natural language processing engine to interpret the instruction "fix part A to part B with a screw" in detail, identifying the exact part placement and type of screw, and generating digital assembly instructions.

[0769] Step 4:

[0770] Assembly video generation

[0771] (server)

[0772] Based on the generated assembly procedure information, the server uses 3D modeling software (such as Blender) to create a 3D model of the part. Based on this 3D model, the animation generation engine simulates the assembly procedure and generates an assembly animation that shows the specific movements.

[0773] The input is assembly procedure information from a natural language processing engine, and the output is a generated assembly video.

[0774] Specific behavior:

[0775] The 3D modeling software creates detailed 3D models of parts A and B, and then the video generation engine simulates these to generate a video of specific steps, such as "Place part A here, place part B on top of it, and secure it with screws."

[0776] Step 5:

[0777] Emotional engine for adaptive operation

[0778] (server)

[0779] As users watch videos, the server uses an emotion engine to analyze their facial expressions and voice to recognize their emotional state. If the user is confused, the server will adjust the video playback speed or display additional explanations.

[0780] Inputs include facial expression data and voice data of the user, and outputs include adjusting the playback speed and displaying supplementary explanations.

[0781] Specific behavior:

[0782] If a user shows a confused expression while watching a video, the server slows the video playback speed by 1.5 times and displays explanatory text on the screen at key points.

[0783] Step 6:

[0784] Video distribution

[0785] (server)

[0786] The completed assembly video is saved, and the server generates a URL for streaming or a download link.

[0787] (Terminal)

[0788] The terminal displays the video URL sent from the server on the user interface, allowing the user to stream or download the video.

[0789] (User)

[0790] Users can click the "Play Video" button on their device to watch the streamed video, pausing, rewinding, fast-forwarding, and other operations as needed.

[0791] The input is the generated assembled video, and the output is a streaming URL or download link provided to the user.

[0792] Specific behavior:

[0793] The user clicks the "Play Video" button on the device to play the generated video. If the user shows a confused expression during playback, the emotion engine adjusts the playback speed.

[0794] (Application example 2)

[0795] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0796] Understanding assembly procedures based on instructions is often difficult, especially for beginners or those working for the first time. Furthermore, the confusion and stress felt by users during the process can reduce work efficiency and accuracy. Conventional systems are unable to provide real-time assistance that responds to the user's emotions, making it difficult to improve work efficiency.

[0797] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animations based on the generated assembly procedure information, means for recognizing the user's emotions and adapting operations based on the emotions, and means for delivering the generated assembly animations to the user. This not only makes it easier for the user to understand the instructions, but also makes it possible to improve work efficiency and accuracy by providing support according to the user's emotions.

[0798] "Instructions" are documents that describe how to assemble or use a product or part.

[0799] "Input means" refers to a device or function that allows a user to load instructions into the system.

[0800] "Extraction means" refers to the technology or device used to extract text data or image data from the instructions.

[0801] The "means for analyzing and generating assembly procedure information" refers to a technique or device for understanding the extracted data and determining specific assembly procedures based on that data.

[0802] The "means for generating assembly animation" refers to a technique or device for creating a visual animation based on assembly procedure information.

[0803] "Distribution means" refers to the technology and devices used to deliver the generated assembly animation to the user's terminal.

[0804] "Means for recognizing user emotions" refers to technology or devices that analyze the user's facial expressions and voice and determine their emotional state.

[0805] "Operational adaptation" refers to techniques and devices for modifying system behavior based on perceived user emotions.

[0806] Overall system configuration

[0807] This invention is a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations accordingly. Specifically, it includes the following components:

[0808] 1. User Device:

[0809] A device used by users to upload instructions to the system and view the generated videos. Typically, a PC, smartphone, tablet, or other device is used.

[0810] 2. Server:

[0811] It is a central device with key functions such as instruction processing, video generation, analysis, and emotion recognition. Specifically, it includes OCR technology, image recognition technology, semantic analysis engine, 3D modeling software, and emotion recognition engine.

[0812] 3. Emotion Recognition Engine:

[0813] This technology analyzes a user's facial expressions and voice to recognize their emotional state. For example, it can use deep learning models such as Keras.

[0814] Operation explanation

[0815] 1. Upload the instructions

[0816] Users upload instructions for products such as furniture and plastic models to the system as image files or PDFs from their devices. This operation is performed through the user interface.

[0817] 2. Analysis of Instructions

[0818] The server analyzes the received instruction manual file, converting the text in the manual into digital data using OCR technology, and extracting part information from diagrams and illustrations using image recognition technology.

[0819] 3. Generating assembly procedure information

[0820] The extracted text data and image data are subjected to semantic analysis to generate assembly procedure information. The software used for this process includes a semantic analysis engine and a natural language processing library.

[0821] 4. Generating assembly videos

[0822] The server creates a 3D model based on the generated assembly procedure information, performs simulations, and finally generates an assembly video. This process is supported by a video generation engine.

[0823] 5. Emotion Recognition and Adaptive Manipulation

[0824] The server analyzes the user's facial expressions and voice to recognize their emotional state. For example, if it determines that the user is confused, it will take adaptive action, such as slowing down the video playback speed.

[0825] Specific examples

[0826] For example, consider the case of generating a video showing assembly procedures for a robot arm in a factory. When a worker uploads instructions, OCR and image recognition technology analyzes the instruction data and generates a video of the assembly procedures based on that. As the worker watches the video and performs the work, an emotion recognition engine monitors their emotions and adjusts the video playback speed or supplementary explanations as needed.

[0827] Prompt Sentence Examples

[0828] Upload a PDF of your furniture assembly instructions and we'll generate a video of the assembly process using adaptive control based on emotion recognition.

[0829] This system makes it easier for users to understand instructions and provides appropriate support according to their emotions, improving work efficiency and accuracy.

[0830] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0831] Step 1:

[0832] The user uploads the assembly instruction manual as a PDF or image file from the terminal to the system. The terminal sends the uploaded file to the server. The input is the assembly instruction file, and the output is the file sent to the server.

[0833] Step 2:

[0834] The server analyzes the received instruction manual file using OCR technology (e.g., Tesseract OCR) and extracts text data. It also uses image recognition technology (e.g., OpenCV or TensorFlow) to extract part information from diagrams and illustrations in the manual. The input is the instruction manual file, and the output is text data and image data.

[0835] Step 3:

[0836] The server runs the extracted text data and image data through a semantic analysis engine to generate assembly procedure information. This uses a natural language processing library (e.g., NLTK or spaCy). The input is text data and image data, and the output is assembly procedure information.

[0837] Step 4:

[0838] The server uses 3D modeling software (e.g., Blender) based on the generated assembly procedure information to simulate each assembly step, and generates an assembly video using a video generation engine. The input is the assembly procedure information, and the output is the assembly video.

[0839] Step 5:

[0840] The server uses an emotion recognition engine (e.g., a deep learning model using Keras) to analyze the user's real-time facial expressions and voice. The server recognizes the user's emotional state and adjusts the video playback speed or displays supplementary explanations as needed. The input is the user's facial expression data and voice data, and the output is adjusted video playback and supplementary explanations.

[0841] Step 6:

[0842] The server delivers the generated assembled video to the user terminal. The terminal displays the video URL sent from the server on the user interface, allowing the user to stream or download. The input is the assembled video file, and the output is the generation and delivery of the video URL.

[0843] Step 7:

[0844] The user watches the assembly video on a device and actually performs the assembly work according to the displayed steps. While the video is playing, an emotion recognition engine monitors the user's emotional state and adjusts the playback speed or displays supplementary explanations as necessary. The input is emotion data, and the output is the adjusted assembly video and supplementary explanations.

[0845] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0846] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0847] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0848] [Third embodiment]

[0849] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0850] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0851] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0852] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0853] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0854] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0855] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0856] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0857] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0858] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0859] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0860] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0861] This invention relates to a system that automatically generates assembly animations based on instructions. The operation and configuration of the entire system will be explained below with specific procedures and examples.

[0862] Overall system configuration

[0863] 1. User Device:

[0864] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[0865] 2. Server:

[0866] The server is a central device with key functions such as instruction processing, video generation, analysis, etc. Multiple modules and engines work together.

[0867] Invention Implementation Process

[0868] 1. Upload the instructions

[0869] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[0870] Terminal: Receives the uploaded instruction file and sends it to the server.

[0871] 2. Analysis of Instructions

[0872] Server: The server analyzes the received instruction file.

[0873] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[0874] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[0875] Example: When a PDF of furniture instructions uploaded by a user arrives at the server, the OCR function is activated and extracts assembly steps such as "fix part A to part B." At the same time, image recognition technology identifies the diagrams of parts A and B and stores them in a database.

[0876] 3. Interpretation of instructional information

[0877] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[0878] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[0879] Example: The server uses a semantic analysis engine to interpret the instruction "Secure part A to part B with screws" and clarifies the specific alignment of the parts, the type of screw, and how to use it.

[0880] 4. Generating assembly videos

[0881] Server: Generates assembly animation based on assembly procedure information.

[0882] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[0883] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, screw rotation motion, etc.

[0884] Example: The server simulates the steps for assembling furniture using 3D modeling software and animates the process. The generated video reproduces the steps, such as "Place part A here, then place part B on top of it and fasten it with screws."

[0885] 5. Video distribution

[0886] Server: Stores the completed assembly video and generates streaming and download links.

[0887] Terminal: Receives the video link sent from the server and displays it to the user.

[0888] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[0889] summary

[0890] This system allows users to simply upload instructions and easily understand the visual assembly steps, allowing them to assemble products accurately, significantly reducing the time and effort required for assembly and improving the user experience.

[0891] The processing flow will be explained below.

[0892] Step 1:

[0893] User: Logs in to the system and uploads instructions for the product to be assembled in image or PDF format.

[0894] Step 2:

[0895] Terminal: Receives the uploaded instruction file and sends it to the server.

[0896] Step 3:

[0897] Server: Receives the instruction manual file. After receiving it, it starts the OCR engine and converts the text information in the manual into digital text data.

[0898] Step 4:

[0899] Server: Extracts the text data after OCR processing, and simultaneously uses an image recognition algorithm to identify part information from diagrams and illustrations in the manual.

[0900] Step 5:

[0901] Server: The extracted text data and image data are input into a semantic analysis engine to analyze the relationships and meaning between procedures.

[0902] Step 6:

[0903] Server: Generates detailed assembly procedure information from the analysis results using a semantic analysis engine.

[0904] Step 7:

[0905] Server: Based on the assembly procedure information, a 3D model of the part is generated using 3D modeling software.

[0906] Step 8:

[0907] Server: The generated 3D model is input into a video generation engine, which records the assembly procedure as a video while simulating it.

[0908] Step 9:

[0909] Server: Renders the final assembled video, saves it, and generates a URL for the video file.

[0910] Step 10:

[0911] Device: Displays the video URL received from the server in the user interface, allowing the user to stream or download.

[0912] Step 11:

[0913] Users: Click on the video link to watch the video and then follow the instructions in the video to assemble the product. During playback, users can pause, rewind, fast forward, and more.

[0914] Example 1

[0915] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0916] Traditional assembly instructions consist of only text and drawings, making them difficult for users to understand and often resulting in time-consuming assembly tasks. Misunderstandings and mistakes at certain steps can easily occur, leading to assembly failures and frustration. Furthermore, manually creating assembly videos requires specialized knowledge and time, resulting in significant effort.

[0917] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0918] In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animations based on the generated assembly procedure information, means for delivering the generated assembly animations to users, and means for identifying specific software modules (OCR technology, semantic analysis engine, 3D modeling software, etc.) used in each processing step, thereby enabling users to easily understand the visual assembly procedures and accurately assemble the product.

[0919] "Means for inputting instructions" refers to a function that allows users to upload image files or PDF format instructions to the system via their terminals.

[0920] "Means for extracting text data and image data" is a processing function for extracting text information and image information from the uploaded instructions.

[0921] The "means for analyzing the extracted text data and image data and generating assembly procedure information" is an analysis function for understanding the assembly procedure based on the extracted data and generating specific assembly instructions.

[0922] The "means for generating assembly animation" is a function for creating animation that visually represents assembly work using 3D models of parts based on the generated assembly procedure information.

[0923] The "means for distributing assembly animation to users" is a function for saving the generated assembly animation and providing it so that users can view or download it.

[0924] "OCR technology" is an abbreviation for Optical Character Recognition, and is a technology that converts image-formatted characters into digital text data.

[0925] A "semantic analysis engine" is an analysis engine that analyzes text and image data and understands their meaning.

[0926] "3D modeling software" is software for creating and editing three-dimensional objects and scenes on a computer.

[0927] This invention relates to a system that automatically generates assembly animations based on instructions. The operation and configuration of the entire system will be explained below with specific procedures and examples.

[0928] System configuration

[0929] User terminal

[0930] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[0931] server

[0932] The server is a central device with key functions such as instruction processing, video generation, analysis, etc. Multiple modules and engines work together.

[0933] Feature details

[0934] Upload instructions

[0935] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[0936] Terminal: Receives the uploaded instruction file and sends it to the server.

[0937] Analysis of the instructions

[0938] Server: The server analyzes the received instruction file.

[0939] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[0940] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[0941] Interpreting the instruction manual information

[0942] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[0943] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[0944] Assembly video generation

[0945] Server: Generates assembly animation based on assembly procedure information.

[0946] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[0947] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, and screw rotation motion.

[0948] Video distribution

[0949] Server: Stores the completed assembly video and generates streaming and download links.

[0950] Terminal: Receives the video link sent from the server and displays it to the user.

[0951] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[0952] Specific examples

[0953] When using furniture assembly instructions

[0954] The user uploads a PDF file of furniture instructions from their device. The server uses OCR to extract the text information, such as "Secure part A to part B with screws," and uses image recognition to identify the diagrams of parts A and B. The semantic analysis engine analyzes the text and interprets the specific steps (such as aligning the parts and the type of screws to use). The steps are animated using 3D modeling software, and the server sends the generated video link to the user's device. The user clicks the link and watches the video to accurately assemble the furniture.

[0955] Example prompts for generative AI models

[0956] I have uploaded assembly instructions for a piece of furniture. These instructions include the step of fastening part A to part B with screws. Please generate a 3D model and an assembly video based on these instructions.

[0957] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0958] Step 1: Upload your instructions

[0959] Input: Instructions (image file or PDF format) that come with items that users assemble, such as furniture or plastic models

[0960] Specific operation: The user clicks the "Upload instructions" button on the device, a local file selection dialog appears, the user selects the assembly instructions file and presses the "Open" button, and the file is uploaded to the system.

[0961] Output: The device displays a notification that the upload is complete and sends the instruction file to the server.

[0962] Step 2: Parse the instructions

[0963] Input: Uploaded instruction file

[0964] Specific operation: The server inputs the received instruction manual file into the analysis module. Using OCR, the text in the manual is converted into digital text data. At the same time, image recognition is performed to extract part information from diagrams and illustrations.

[0965] Output: Extracted text and image data

[0966] Step 3: Interpret the information in the instructions

[0967] Input: Extracted text and image data

[0968] Specific operation: The server inputs the data obtained through OCR and image recognition into a semantic analysis engine to generate assembly procedure information. For example, a command such as "Secure part A to part B with screws" is analyzed and the specific steps, tools required, and part alignment methods are interpreted.

[0969] Output: Assembly procedure information

[0970] Step 4: Generate assembly videos

[0971] Input: Assembly procedure information

[0972] Specific operation: The server uses 3D modeling software to generate a 3D model of the specified part. It then uses the generated 3D model to simulate the assembly procedure, and animates the simulation results using a video generation engine. The video includes the part selection procedure, precise alignment, screw rotation, and more.

[0973] Output: Assembly video file

[0974] Step 5: Publish your video

[0975] Input: Assembly video file

[0976] Specific operation: The server saves the assembled video and generates a URL or download link for streaming the video file. The server then sends the URL or download link to the device and notifies the user.

[0977] Output: User-accessible video URL and download link

[0978] Step 6: Watch the video and assemble

[0979] Input: Video URL or download link

[0980] Specific operation: The user clicks on the provided URL on their device to watch the assembly video. The video is streamed and the user can pause, rewind, fast forward, etc. The user then performs the actual assembly work by following the video.

[0981] Output: Understanding the assembled product and the exact assembly steps along the way

[0982] (Application example 1)

[0983] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0984] In modern manufacturing factories, efficient execution of complex assembly tasks is important, but they often require a great deal of effort and time. In particular, assembling new products requires workers to understand the manual and accurately grasp the procedures. However, differences in the ability to read and understand instructions and the experience of actually assembling by hand vary from person to person, leading to variations in quality and errors. Furthermore, as automation using robots advances, it is necessary for the robots to properly understand and reliably execute assembly procedures. To solve these problems and achieve efficient and uniform assembly work, a system is needed that clarifies assembly procedures and enables robots to accurately execute them.

[0985] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0986] In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animation based on the generated assembly procedure information, means for distributing the generated assembly animation to a user, and means for generating and executing operation instructions for a robot based on the assembly procedure information. This not only provides assembly procedures that are easy to understand visually, but also enables the robot to perform assembly work accurately.

[0987] "Instructions" are documents that show how to assemble or operate a product.

[0988] "Text data" is text information extracted from the instructions.

[0989] "Image data" refers to visual information such as diagrams and illustrations extracted from the manual.

[0990] "Assembly procedure information" is information about a specific method for assembling a product, generated by analyzing text data and image data.

[0991] "OCR (Optical Character Recognition) technology" is a technology that converts characters in analog format, such as on paper or in images, into digital format.

[0992] A "semantic analysis engine" is software that analyzes the meaning of text and image data and understands their relationships.

[0993] "Video generation" is the process of creating a visually easy-to-view video based on assembly procedure information.

[0994] "Distribution" is the process of sending the generated video to users.

[0995] A "robot" is a machine that operates automatically according to a program and can perform assembly tasks.

[0996] The "operation instruction" is a specific instruction for the robot to perform an assembly task.

[0997] This invention relates to a system that automatically generates assembly animations based on instructions and provides operational instructions to a robot. The operation and configuration of the entire system will be explained below, along with specific procedures and examples.

[0998] Overall system configuration

[0999] 1. User Device

[1000] The user terminal is a device used to upload instructions and view the generated videos. Users access the system using a smartphone or tablet and operate it.

[1001] 2. Server

[1002] The server is a central device with key functions such as processing instructions, generating videos, analyzing them, and generating instructions for the robot. Multiple modules and engines work together.

[1003] Invention Implementation Process

[1004] Upload instructions

[1005] Users upload instructions for products such as furniture and plastic models to the system as image files or PDFs from their devices. The devices receive the uploaded instruction files and send them to the server.

[1006] Analysis of the instructions

[1007] The server uses OCR and image recognition technology to analyze the received instruction manual file. OCR technology (e.g., Tesseract) is used to convert the text into digital text data, and image recognition technology (e.g., OpenCV) is used to extract part information from diagrams and illustrations.

[1008] Interpreting the instruction manual information

[1009] The server analyzes the extracted text and image data to generate assembly procedure information. This analysis is performed using a semantic analysis engine, which analyzes the meaning of the text and image data and understands their relationships.

[1010] Assembly video generation

[1011] The server generates an assembly video based on the assembly procedure information. It uses 3D modeling software (e.g., Blender) to generate a 3D model of the part and performs a simulation. It uses a video generation engine (e.g., FFmpeg) to create an assembly video based on the generated 3D model and simulation results.

[1012] Video distribution

[1013] The server stores the completed assembly video and generates a streaming distribution or download link. The device receives the video link sent from the server and displays it to the user. The user can watch the video and actually perform the assembly work by following the displayed steps.

[1014] Instructions for the robot

[1015] Based on the assembly procedure information, the server generates and executes operation instructions for the robot, allowing the robot to accurately assemble the parts according to the generated instructions.

[1016] Specific examples

[1017] Let's say a user takes a photo of a piece of furniture instruction manual with their smartphone and uploads it to the application. The uploaded manual is converted into text using an OCR engine, and diagrams and illustrations are analyzed using an image recognition engine. Based on the analysis results, the server generates assembly procedures and simulates them using 3D modeling software. The generated video is sent to the user's device, and the user can proceed with the assembly work while watching it. Furthermore, a robot accurately performs the assembly work based on the operating instructions generated by the server.

[1018] Prompt Sentence Examples

[1019] Based on the furniture instructions below, please automatically analyze the assembly steps and generate the final assembly video.

[1020] "Secure part A to part B, then attach part C. Then place part D according to the instructions in step 3, and finally secure all parts together with screws."

[1021] Following this prompt, specific steps and component instructions are provided to ensure the model can be analyzed accurately.

[1022] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1023] Step 1: Upload your instructions

[1024] The user takes a photo of the instruction manual that comes with the product to be assembled using a smartphone or tablet, or saves it as a PDF file on the device. The device receives the photo or saved instruction file and sends it to the server. The input is an image file or PDF file of the instruction manual, and the output is the instruction data that is sent to the server. The specific operation is for the user to operate the application to select the file and press the upload button.

[1025] Step 2: Parse the instructions

[1026] The server analyzes the received instruction manual file using OCR and image recognition technology. Specifically, it uses an OCR engine (e.g., Tesseract) to convert the text into digital text data, and an image recognition engine (e.g., OpenCV) to extract part information from diagrams and illustrations. The input is the instruction manual data, and the output is text data and image data. Specifically, the server starts the OCR engine, scans the characters in the image, and converts them into text.

[1027] Step 3: Interpret the information in the instructions

[1028] The server analyzes the extracted text data and image data to generate assembly procedure information. A semantic analysis engine is used for this analysis. The input is text data and image data, and the output is assembly procedure information. Specifically, the semantic analysis engine analyzes the meaning of the text and understands the relationships between parts. For example, it interprets in detail the instruction "Secure part A to part B with a screw."

[1029] Step 4: Generate assembly videos

[1030] The server generates an assembly video based on the assembly procedure information. It uses 3D modeling software (e.g., Blender) to generate a 3D model of the part and performs a simulation. It uses a video generation engine (e.g., FFmpeg) to create an assembly video based on the generated 3D model and simulation results. The input is the assembly procedure information and the output is the assembly video. Specifically, the server starts the 3D modeling software, simulates the operation of the part, and encodes it as a video.

[1031] Step 5: Publish your video

[1032] The server stores the completed assembly video and generates a streaming distribution or download link. The terminal receives the video link sent from the server and displays it to the user. The input is the assembly video, and the output is the video link to the user's terminal. Specifically, the server uploads the video file to cloud storage and sends the link to the terminal.

[1033] Step 6: Instruct the robot to move

[1034] The server generates and executes operational instructions for the robot based on the generated assembly procedure information. Specifically, the instructions include specific operational instructions for the robot to accurately position and fix parts. The input is the assembly procedure information, and the output is the robot's operational instructions. The specific operation is that the server sends the operational instructions to the robot's control program, and the robot starts operating in accordance with the instructions.

[1035] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1036] This invention relates to a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations based on those emotions. The operation and configuration of the entire system are explained below, along with specific procedures and examples.

[1037] Overall system configuration

[1038] 1. User Device:

[1039] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[1040] 2. Server:

[1041] The server is a central device with key functions such as instruction processing, video generation, analysis, emotion recognition, etc. Multiple modules and engines work together.

[1042] 3. Emotion Engine:

[1043] This engine analyzes the user's facial expressions and voice to recognize their emotional state, and adapts the system's behavior according to the user's emotions.

[1044] Invention Implementation Process

[1045] 1. Upload the instructions

[1046] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[1047] Terminal: Receives the uploaded instruction file and sends it to the server.

[1048] 2. Analysis of Instructions

[1049] Server: The server analyzes the received instruction file.

[1050] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[1051] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[1052] Example: When a PDF of furniture instructions uploaded by a user arrives at the server, the OCR function is activated and extracts assembly steps such as "fix part A to part B." At the same time, image recognition technology identifies the diagrams of parts A and B and stores them in a database.

[1053] 3. Interpretation of instructional information

[1054] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[1055] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[1056] Example: The server uses a semantic analysis engine to interpret the instruction "Secure part A to part B with screws" and clarifies the specific alignment of the parts, the type of screw, and how to use it.

[1057] 4. Generating assembly videos

[1058] Server: Generates assembly animation based on assembly procedure information.

[1059] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[1060] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, screw rotation motion, etc.

[1061] Example: The server simulates the steps for assembling furniture using 3D modeling software and animates the process. The generated video reproduces the steps, such as "Place part A here, then place part B on top of it and fasten it with screws."

[1062] 5. Operational adaptation using emotion engine

[1063] Server: The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotional state.

[1064] Example: If the server recognizes the user's facial expression and determines that the user is confused, it will take appropriate action, such as slowing down the video playback speed.

[1065] 6. Video distribution

[1066] Server: Stores the completed assembly video and generates streaming and download links.

[1067] Device: Displays the video URL sent from the server in the user interface, allowing the user to stream or download.

[1068] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[1069] Example: As a user plays a video, an emotion engine monitors the user's emotional state and takes appropriate action, such as adjusting the playback speed or displaying additional information.

[1070] summary

[1071] This system allows users to easily understand the visual assembly procedures and assemble products accurately by simply uploading instructions. Furthermore, by using an emotion engine, it can provide optimal operations according to the user's emotions, improving the efficiency and satisfaction of assembly work.

[1072] The processing flow will be explained below.

[1073] Step 1:

[1074] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDF format.

[1075] Step 2:

[1076] Terminal: Receives the uploaded instruction file and sends it to the server.

[1077] Step 3:

[1078] Server: The server analyzes the received instruction file.

[1079] Step 4:

[1080] Server: Starts the OCR engine and converts the text information in the manual into digital text data.

[1081] Step 5:

[1082] Server: Extracts the text data after OCR processing, and simultaneously uses an image recognition algorithm to identify part information from diagrams and illustrations in the manual.

[1083] Step 6:

[1084] Server: The extracted text data and image data are input into a semantic analysis engine to analyze the relationships and meaning between procedures.

[1085] Step 7:

[1086] Server: Generates detailed assembly procedure information from the analysis results using a semantic analysis engine.

[1087] Step 8:

[1088] Server: Based on the assembly procedure information, a 3D model of the part is generated using 3D modeling software.

[1089] Step 9:

[1090] Server: The generated 3D model is input into a video generation engine, which records the assembly procedure as a video while simulating it.

[1091] Step 10:

[1092] Server: Renders the final assembled video, saves it, and generates a URL for the video file.

[1093] Step 11:

[1094] Server: Activates the emotion engine, analyzes the user's facial expressions and voice in real time, and recognizes their emotional state.

[1095] Step 12:

[1096] Server: Adapts the pace and content of video playback based on emotion recognition results from the emotion engine. For example, if a confused expression is detected, commentary may be added or playback speed may be slowed down.

[1097] Step 13:

[1098] Device: Displays the video URL received from the server in the user interface, allowing the user to stream or download.

[1099] Step 14:

[1100] Users: Click on the video link to watch the video and then assemble the parts by following the instructions in the video. During playback, users can pause, rewind, fast forward, and more.

[1101] Step 15:

[1102] Server: The emotion engine continuously monitors the user's emotions while the video is playing and adapts in real time if the user becomes confused.

[1103] Specific examples

[1104] When a user uploads a PDF instruction manual for the furniture they purchased to the system, the server receives the PDF and extracts the text data using OCR technology. At the same time, an image recognition algorithm identifies the parts diagrams in the manual. A semantic analysis engine analyzes the text and image data to generate specific assembly instructions. 3D modeling software then creates 3D models of each part, and uses them to generate assembly instructions as videos.

[1105] The generated video is then delivered to the user, who then watches the video while performing the assembly task. While the video is playing, the emotion engine monitors the user's facial expressions, and if it detects that the user is confused, it slows down the video playback speed or displays additional explanations to help the user understand. This approach allows the user to understand the instructions more intuitively and efficiently, and perform the assembly task accurately.

[1106] Example 2

[1107] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1108] Conventional assembly manuals require a lot of time and effort for users to understand the procedures, placing a significant burden on beginners and those who have difficulty reading manuals. Furthermore, because they do not adapt to the user's emotional state, they lack support for difficult-to-understand parts. This results in problems such as incorrect assembly methods and increased stress during the process.

[1109] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animation based on the generated assembly procedure information, means for recognizing the user's emotional state and adapting operations based on the recognized emotional state, and means for delivering the generated assembly animation to the user. This allows the user to easily understand the visual assembly procedures simply by uploading the instructions, and makes it possible to provide optimal support according to the user's emotional state.

[1110] "Instructions" are documents that explain the assembly procedures and usage of a product or part.

[1111] "Input means" refers to the device or software used to load instructions into the system.

[1112] "Text data" is a digital representation of the textual information contained in the instructions.

[1113] "Image data" means a digital representation of the diagrams and illustrations contained in the instructions.

[1114] The "extracting means" refers to a method or device for extracting text data and image data from the instructions.

[1115] "Means for analyzing" refers to methods or devices for processing extracted digital data to make sense of it and derive information.

[1116] "Assembly procedure information" is digital information that indicates the specific steps and methods required to assemble a product or part.

[1117] "Generating means" refers to a method or device for creating new data or information based on the analysis results.

[1118] "Assembly animation" is content in the form of animation that visually shows assembly procedures.

[1119] A "user" is a person who uses the system to read the instructions and watch the assembly animations.

[1120] "Emotional state" refers to a user's current mental and emotional state.

[1121] The "recognition means" is a method or device for identifying the emotional state of a user from their facial expression or voice.

[1122] "Means for adapting operations" are methods or devices for modifying system behavior or settings based on a recognized emotional state.

[1123] The "distribution means" refers to a method or device for providing the generated assembled animation to a user.

[1124] This invention relates to a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations based on those emotions.

[1125] Overall system configuration

[1126] User terminal

[1127] The user terminal is a device that users use to upload instructions and view generated assembly videos. Examples include personal computers (PCs) and smartphones. Users access the system from their terminals and perform various operations (e.g., uploading files, viewing videos).

[1128] server

[1129] The server is the hardware that manages the core functions of the system, such as data processing of instructions, video generation, analysis, and emotion recognition. The following software and engines run on the server:

[1130] Optical Character Recognition (OCR) software (e.g., Tesseract): converts the text in the instructions into digital data.

[1131] Image recognition engine (e.g., OpenCV, YOLO): Extracts component information from diagrams and illustrations in the manual.

[1132] Natural language processing engine (e.g., SpaCy, BERT): Analyzes the extracted data and generates assembly instructions.

[1133] 3D modeling software (e.g. Blender): Create 3D models of parts and perform simulations.

[1134] Video generation engine: Creates assembly videos based on the generated 3D models.

[1135] Emotion recognition software (e.g., facial expression analysis, voice analysis): Recognizes the user's emotional state and performs corresponding adaptive processing.

[1136] Emotion Engine

[1137] The emotion engine is software that analyzes the user's facial expressions and voice to recognize the user's emotional state and adapt the system's behavior accordingly. For example, if the user is confused, it can adjust the video playback speed or display additional explanations.

[1138] Invention Implementation Process

[1139] Specific example explanation

[1140] Let's say a user uploads a PDF of a piece of furniture's instructions from their computer. Here's what happens:

[1141] 1. Upload the instructions

[1142] The user selects the assembly manual PDF from the file manager on their PC and clicks the "Upload" button. The device then sends this file to the server as an HTTP POST request.

[1143] 2. Analysis of Instructions

[1144] The server processes the received instruction manual file. First, the OCR engine converts the text into digital data, and then the image recognition engine extracts part information from diagrams and illustrations. For example, it extracts the text "Fix part A to part B" and recognizes the diagrams of parts A and B.

[1145] 3. Interpretation of instructional information

[1146] The server uses a natural language processing engine to analyze text and image data and generate assembly instructions. It then interprets the extracted instruction, "Secure part A to part B with screws," in detail to identify the specific positions of the parts and the types of screws.

[1147] 4. Generating assembly videos

[1148] The server uses 3D modeling software to generate a 3D model of the part based on the assembly procedure information, and then simulates it.The animation generation engine uses the simulation results to create an assembly animation that shows the specific steps.

[1149] 5. Operational adaptation using emotion engine

[1150] While the user is watching the video, the server analyzes their facial expressions and voice to recognize their emotional state. If it determines that the user is confused, it will take appropriate action, such as slowing down the video playback speed or displaying supplementary explanations.

[1151] 6. Video distribution

[1152] The generated assembly video is saved on the server and a streaming URL is generated. The URL is sent to the device, and the user can click the "Play Video" button to watch the video. The video can also be downloaded if necessary.

[1153] Prompt Sentence Examples

[1154] Here are some examples of prompts for generative AI models:

[1155] A user uploaded a PDF instruction manual for assembling furniture. The server then used OCR technology to convert the text in the manual into digital text data and image recognition technology to extract component information. It then used a natural language processing engine to interpret the relationships between steps and generated an assembly video using 3D modeling software. It also used emotion recognition software to analyze the user's emotions and adapt the playback speed and supplementary explanations accordingly. If the user showed a confused expression, the server slowed down the video playback speed and displayed supplementary explanations at key points. Please provide an overview of this system and its steps, with examples.

[1156] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1157] Step 1:

[1158] Upload instructions

[1159] (User)

[1160] The user selects the assembly manual as an image file (JPEG, PNG) or PDF file on their device and presses the upload button. This operation selects the manual file.

[1161] (Terminal)

[1162] The terminal sends the instruction file selected by the user to the server as an HTTP POST request.

[1163] The input is a user-selected description file (JPEG, PNG or PDF) and the output is that file is sent to the server.

[1164] Specific behavior:

[1165] The user selects the instruction manual PDF from the file manager on their PC and clicks the upload button on the web application. The device sends the file to the server.

[1166] Step 2:

[1167] Analysis of the instructions

[1168] (server)

[1169] The server analyzes the received instruction manual file. First, it starts an OCR engine (e.g., Tesseract) to digitize the text in the manual. Next, it uses an image recognition engine (e.g., OpenCV or YOLO) to extract part information from the diagrams and illustrations in the manual.

[1170] The input is a manual file, and the output is text data extracted by OCR and part information identified by an image recognition engine.

[1171] Specific behavior:

[1172] The server uses OCR to extract the text from the instructions, "Fix part A to part B," and at the same time, the image recognition engine identifies the diagrams of parts A and B.

[1173] Step 3:

[1174] Interpreting the instruction manual information

[1175] (server)

[1176] Based on the extracted text data and image data, the server uses a natural language processing engine (e.g., SpaCy or BERT) to perform a detailed analysis of the text data and diagram data and generate assembly instructions.

[1177] The inputs include text data and image data from the OCR and image recognition engines, and the output is the generated assembly procedure information.

[1178] Specific behavior:

[1179] The server uses a natural language processing engine to interpret the instruction "fix part A to part B with a screw" in detail, identifying the exact part placement and type of screw, and generating digital assembly instructions.

[1180] Step 4:

[1181] Assembly video generation

[1182] (server)

[1183] Based on the generated assembly procedure information, the server uses 3D modeling software (such as Blender) to create a 3D model of the part. Based on this 3D model, the animation generation engine simulates the assembly procedure and generates an assembly animation that shows the specific movements.

[1184] The input is assembly procedure information from a natural language processing engine, and the output is a generated assembly video.

[1185] Specific behavior:

[1186] The 3D modeling software creates detailed 3D models of parts A and B, and then the video generation engine simulates these to generate a video of specific steps, such as "Place part A here, place part B on top of it, and secure it with screws."

[1187] Step 5:

[1188] Emotional engine for adaptive operation

[1189] (server)

[1190] As users watch videos, the server uses an emotion engine to analyze their facial expressions and voice to recognize their emotional state. If the user is confused, the server will adjust the video playback speed or display additional explanations.

[1191] Inputs include facial expression data and voice data of the user, and outputs include adjusting the playback speed and displaying supplementary explanations.

[1192] Specific behavior:

[1193] If a user shows a confused expression while watching a video, the server slows the video playback speed by 1.5 times and displays explanatory text on the screen at key points.

[1194] Step 6:

[1195] Video distribution

[1196] (server)

[1197] The completed assembly video is saved, and the server generates a URL for streaming or a download link.

[1198] (Terminal)

[1199] The terminal displays the video URL sent from the server on the user interface, allowing the user to stream or download the video.

[1200] (User)

[1201] Users can click the "Play Video" button on their device to watch the streamed video, pausing, rewinding, fast-forwarding, and other operations as needed.

[1202] The input is the generated assembled video, and the output is a streaming URL or download link provided to the user.

[1203] Specific behavior:

[1204] The user clicks the "Play Video" button on the device to play the generated video. If the user shows a confused expression during playback, the emotion engine adjusts the playback speed.

[1205] (Application example 2)

[1206] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1207] Understanding assembly procedures based on instructions is often difficult, especially for beginners or those working for the first time. Furthermore, the confusion and stress felt by users during the process can reduce work efficiency and accuracy. Conventional systems are unable to provide real-time assistance that responds to the user's emotions, making it difficult to improve work efficiency.

[1208] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animations based on the generated assembly procedure information, means for recognizing the user's emotions and adapting operations based on the emotions, and means for delivering the generated assembly animations to the user. This not only makes it easier for the user to understand the instructions, but also makes it possible to improve work efficiency and accuracy by providing support according to the user's emotions.

[1209] "Instructions" are documents that describe how to assemble or use a product or part.

[1210] "Input means" refers to a device or function that allows a user to load instructions into the system.

[1211] "Extraction means" refers to the technology or device used to extract text data or image data from the instructions.

[1212] The "means for analyzing and generating assembly procedure information" refers to a technique or device for understanding the extracted data and determining specific assembly procedures based on that data.

[1213] The "means for generating assembly animation" refers to a technique or device for creating a visual animation based on assembly procedure information.

[1214] "Distribution means" refers to the technology and devices used to deliver the generated assembly animation to the user's terminal.

[1215] "Means for recognizing user emotions" refers to technology or devices that analyze the user's facial expressions and voice and determine their emotional state.

[1216] "Operational adaptation" refers to techniques and devices for modifying system behavior based on perceived user emotions.

[1217] Overall system configuration

[1218] This invention is a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations accordingly. Specifically, it includes the following components:

[1219] 1. User Device:

[1220] A device used by users to upload instructions to the system and view the generated videos. Typically, a PC, smartphone, tablet, or other device is used.

[1221] 2. Server:

[1222] It is a central device with key functions such as instruction processing, video generation, analysis, and emotion recognition. Specifically, it includes OCR technology, image recognition technology, semantic analysis engine, 3D modeling software, and emotion recognition engine.

[1223] 3. Emotion Recognition Engine:

[1224] This technology analyzes a user's facial expressions and voice to recognize their emotional state. For example, it can use deep learning models such as Keras.

[1225] Operation explanation

[1226] 1. Upload the instructions

[1227] Users upload instructions for products such as furniture and plastic models to the system as image files or PDFs from their devices. This operation is performed through the user interface.

[1228] 2. Analysis of Instructions

[1229] The server analyzes the received instruction manual file, converting the text in the manual into digital data using OCR technology, and extracting part information from diagrams and illustrations using image recognition technology.

[1230] 3. Generating assembly procedure information

[1231] The extracted text data and image data are subjected to semantic analysis to generate assembly procedure information. The software used for this process includes a semantic analysis engine and a natural language processing library.

[1232] 4. Generating assembly videos

[1233] The server creates a 3D model based on the generated assembly procedure information, performs simulations, and finally generates an assembly video. This process is supported by a video generation engine.

[1234] 5. Emotion Recognition and Adaptive Manipulation

[1235] The server analyzes the user's facial expressions and voice to recognize their emotional state. For example, if it determines that the user is confused, it will take adaptive action, such as slowing down the video playback speed.

[1236] Specific examples

[1237] For example, consider the case of generating a video showing assembly procedures for a robot arm in a factory. When a worker uploads instructions, OCR and image recognition technology analyzes the instruction data and generates a video of the assembly procedures based on that. As the worker watches the video and performs the work, an emotion recognition engine monitors their emotions and adjusts the video playback speed or supplementary explanations as needed.

[1238] Prompt Sentence Examples

[1239] Upload a PDF of your furniture assembly instructions and we'll generate a video of the assembly process using adaptive control based on emotion recognition.

[1240] This system makes it easier for users to understand instructions and provides appropriate support according to their emotions, improving work efficiency and accuracy.

[1241] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1242] Step 1:

[1243] The user uploads the assembly instruction manual as a PDF or image file from the terminal to the system. The terminal sends the uploaded file to the server. The input is the assembly instruction file, and the output is the file sent to the server.

[1244] Step 2:

[1245] The server analyzes the received instruction manual file using OCR technology (e.g., Tesseract OCR) and extracts text data. It also uses image recognition technology (e.g., OpenCV or TensorFlow) to extract part information from diagrams and illustrations in the manual. The input is the instruction manual file, and the output is text data and image data.

[1246] Step 3:

[1247] The server runs the extracted text data and image data through a semantic analysis engine to generate assembly procedure information. This uses a natural language processing library (e.g., NLTK or spaCy). The input is text data and image data, and the output is assembly procedure information.

[1248] Step 4:

[1249] The server uses 3D modeling software (e.g., Blender) based on the generated assembly procedure information to simulate each assembly step, and generates an assembly video using a video generation engine. The input is the assembly procedure information, and the output is the assembly video.

[1250] Step 5:

[1251] The server uses an emotion recognition engine (e.g., a deep learning model using Keras) to analyze the user's real-time facial expressions and voice. The server recognizes the user's emotional state and adjusts the video playback speed or displays supplementary explanations as needed. The input is the user's facial expression data and voice data, and the output is adjusted video playback and supplementary explanations.

[1252] Step 6:

[1253] The server delivers the generated assembled video to the user terminal. The terminal displays the video URL sent from the server on the user interface, allowing the user to stream or download. The input is the assembled video file, and the output is the generation and delivery of the video URL.

[1254] Step 7:

[1255] The user watches the assembly video on a device and actually performs the assembly work according to the displayed steps. While the video is playing, an emotion recognition engine monitors the user's emotional state and adjusts the playback speed or displays supplementary explanations as necessary. The input is emotion data, and the output is the adjusted assembly video and supplementary explanations.

[1256] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1257] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1258] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1259] [Fourth embodiment]

[1260] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1261] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1262] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1263] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1264] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1265] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1266] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1267] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1268] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1269] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1270] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1271] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1272] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1273] This invention relates to a system that automatically generates assembly animations based on instructions. The operation and configuration of the entire system will be explained below with specific procedures and examples.

[1274] Overall system configuration

[1275] 1. User Device:

[1276] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[1277] 2. Server:

[1278] The server is a central device with key functions such as instruction processing, video generation, analysis, etc. Multiple modules and engines work together.

[1279] Invention Implementation Process

[1280] 1. Upload the instructions

[1281] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[1282] Terminal: Receives the uploaded instruction file and sends it to the server.

[1283] 2. Analysis of Instructions

[1284] Server: The server analyzes the received instruction file.

[1285] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[1286] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[1287] Example: When a PDF of furniture instructions uploaded by a user arrives at the server, the OCR function is activated and extracts assembly steps such as "fix part A to part B." At the same time, image recognition technology identifies the diagrams of parts A and B and stores them in a database.

[1288] 3. Interpretation of instructional information

[1289] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[1290] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[1291] Example: The server uses a semantic analysis engine to interpret the instruction "Secure part A to part B with screws" and clarifies the specific alignment of the parts, the type of screw, and how to use it.

[1292] 4. Generating assembly videos

[1293] Server: Generates assembly animation based on assembly procedure information.

[1294] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[1295] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, screw rotation motion, etc.

[1296] Example: The server simulates the steps for assembling furniture using 3D modeling software and animates the process. The generated video reproduces the steps, such as "Place part A here, then place part B on top of it and fasten it with screws."

[1297] 5. Video distribution

[1298] Server: Stores the completed assembly video and generates streaming and download links.

[1299] Terminal: Receives the video link sent from the server and displays it to the user.

[1300] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[1301] summary

[1302] This system allows users to simply upload instructions and easily understand the visual assembly steps, allowing them to assemble products accurately, significantly reducing the time and effort required for assembly and improving the user experience.

[1303] The processing flow will be explained below.

[1304] Step 1:

[1305] User: Logs in to the system and uploads instructions for the product to be assembled in image or PDF format.

[1306] Step 2:

[1307] Terminal: Receives the uploaded instruction file and sends it to the server.

[1308] Step 3:

[1309] Server: Receives the instruction manual file. After receiving it, it starts the OCR engine and converts the text information in the manual into digital text data.

[1310] Step 4:

[1311] Server: Extracts the text data after OCR processing, and simultaneously uses an image recognition algorithm to identify part information from diagrams and illustrations in the manual.

[1312] Step 5:

[1313] Server: The extracted text data and image data are input into a semantic analysis engine to analyze the relationships and meaning between procedures.

[1314] Step 6:

[1315] Server: Generates detailed assembly procedure information from the analysis results using a semantic analysis engine.

[1316] Step 7:

[1317] Server: Based on the assembly procedure information, a 3D model of the part is generated using 3D modeling software.

[1318] Step 8:

[1319] Server: The generated 3D model is input into a video generation engine, which records the assembly procedure as a video while simulating it.

[1320] Step 9:

[1321] Server: Renders the final assembled video, saves it, and generates a URL for the video file.

[1322] Step 10:

[1323] Device: Displays the video URL received from the server in the user interface, allowing the user to stream or download.

[1324] Step 11:

[1325] Users: Click on the video link to watch the video and then follow the instructions in the video to assemble the product. During playback, users can pause, rewind, fast forward, and more.

[1326] Example 1

[1327] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1328] Traditional assembly instructions consist of only text and drawings, making them difficult for users to understand and often resulting in time-consuming assembly tasks. Misunderstandings and mistakes at certain steps can easily occur, leading to assembly failures and frustration. Furthermore, manually creating assembly videos requires specialized knowledge and time, resulting in significant effort.

[1329] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1330] In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animations based on the generated assembly procedure information, means for delivering the generated assembly animations to users, and means for identifying specific software modules (OCR technology, semantic analysis engine, 3D modeling software, etc.) used in each processing step, thereby enabling users to easily understand the visual assembly procedures and accurately assemble the product.

[1331] "Means for inputting instructions" refers to a function that allows users to upload image files or PDF format instructions to the system via their terminals.

[1332] "Means for extracting text data and image data" is a processing function for extracting text information and image information from the uploaded instructions.

[1333] The "means for analyzing the extracted text data and image data and generating assembly procedure information" is an analysis function for understanding the assembly procedure based on the extracted data and generating specific assembly instructions.

[1334] The "means for generating assembly animation" is a function for creating animation that visually represents assembly work using 3D models of parts based on the generated assembly procedure information.

[1335] The "means for distributing assembly animation to users" is a function for saving the generated assembly animation and providing it so that users can view or download it.

[1336] "OCR technology" is an abbreviation for Optical Character Recognition, and is a technology that converts image-formatted characters into digital text data.

[1337] A "semantic analysis engine" is an analysis engine that analyzes text and image data and understands their meaning.

[1338] "3D modeling software" is software for creating and editing three-dimensional objects and scenes on a computer.

[1339] This invention relates to a system that automatically generates assembly animations based on instructions. The operation and configuration of the entire system will be explained below with specific procedures and examples.

[1340] System configuration

[1341] User terminal

[1342] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[1343] server

[1344] The server is a central device with key functions such as instruction processing, video generation, analysis, etc. Multiple modules and engines work together.

[1345] Feature details

[1346] Upload instructions

[1347] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[1348] Terminal: Receives the uploaded instruction file and sends it to the server.

[1349] Analysis of the instructions

[1350] Server: The server analyzes the received instruction file.

[1351] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[1352] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[1353] Interpreting the instruction manual information

[1354] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[1355] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[1356] Assembly video generation

[1357] Server: Generates assembly animation based on assembly procedure information.

[1358] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[1359] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, and screw rotation motion.

[1360] Video distribution

[1361] Server: Stores the completed assembly video and generates streaming and download links.

[1362] Terminal: Receives the video link sent from the server and displays it to the user.

[1363] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[1364] Specific examples

[1365] When using furniture assembly instructions

[1366] The user uploads a PDF file of furniture instructions from their device. The server uses OCR to extract the text information, such as "Secure part A to part B with screws," and uses image recognition to identify the diagrams of parts A and B. The semantic analysis engine analyzes the text and interprets the specific steps (such as aligning the parts and the type of screws to use). The steps are animated using 3D modeling software, and the server sends the generated video link to the user's device. The user clicks the link and watches the video to accurately assemble the furniture.

[1367] Example prompts for generative AI models

[1368] I have uploaded assembly instructions for a piece of furniture. These instructions include the step of fastening part A to part B with screws. Please generate a 3D model and an assembly video based on these instructions.

[1369] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1370] Step 1: Upload your instructions

[1371] Input: Instructions (image file or PDF format) that come with items that users assemble, such as furniture or plastic models

[1372] Specific operation: The user clicks the "Upload instructions" button on the device, a local file selection dialog appears, the user selects the assembly instructions file and presses the "Open" button, and the file is uploaded to the system.

[1373] Output: The device displays a notification that the upload is complete and sends the instruction file to the server.

[1374] Step 2: Parse the instructions

[1375] Input: Uploaded instruction file

[1376] Specific operation: The server inputs the received instruction manual file into the analysis module. Using OCR, the text in the manual is converted into digital text data. At the same time, image recognition is performed to extract part information from diagrams and illustrations.

[1377] Output: Extracted text and image data

[1378] Step 3: Interpret the information in the instructions

[1379] Input: Extracted text and image data

[1380] Specific operation: The server inputs the data obtained through OCR and image recognition into a semantic analysis engine to generate assembly procedure information. For example, a command such as "Secure part A to part B with screws" is analyzed and the specific steps, tools required, and part alignment methods are interpreted.

[1381] Output: Assembly procedure information

[1382] Step 4: Generate assembly videos

[1383] Input: Assembly procedure information

[1384] Specific operation: The server uses 3D modeling software to generate a 3D model of the specified part. It then uses the generated 3D model to simulate the assembly procedure, and animates the simulation results using a video generation engine. The video includes the part selection procedure, precise alignment, screw rotation, and more.

[1385] Output: Assembly video file

[1386] Step 5: Publish your video

[1387] Input: Assembly video file

[1388] Specific operation: The server saves the assembled video and generates a URL or download link for streaming the video file. The server then sends the URL or download link to the device and notifies the user.

[1389] Output: User-accessible video URL and download link

[1390] Step 6: Watch the video and assemble

[1391] Input: Video URL or download link

[1392] Specific operation: The user clicks on the provided URL on their device to watch the assembly video. The video is streamed and the user can pause, rewind, fast forward, etc. The user then performs the actual assembly work by following the video.

[1393] Output: Understanding the assembled product and the exact assembly steps along the way

[1394] (Application example 1)

[1395] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1396] In modern manufacturing factories, efficient execution of complex assembly tasks is important, but they often require a great deal of effort and time. In particular, assembling new products requires workers to understand the manual and accurately grasp the procedures. However, differences in the ability to read and understand instructions and the experience of actually assembling by hand vary from person to person, leading to variations in quality and errors. Furthermore, as automation using robots advances, it is necessary for the robots to properly understand and reliably execute assembly procedures. To solve these problems and achieve efficient and uniform assembly work, a system is needed that clarifies assembly procedures and enables robots to accurately execute them.

[1397] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1398] In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animation based on the generated assembly procedure information, means for distributing the generated assembly animation to a user, and means for generating and executing operation instructions for a robot based on the assembly procedure information. This not only provides assembly procedures that are easy to understand visually, but also enables the robot to perform assembly work accurately.

[1399] "Instructions" are documents that show how to assemble or operate a product.

[1400] "Text data" is text information extracted from the instructions.

[1401] "Image data" refers to visual information such as diagrams and illustrations extracted from the manual.

[1402] "Assembly procedure information" is information about a specific method for assembling a product, generated by analyzing text data and image data.

[1403] "OCR (Optical Character Recognition) technology" is a technology that converts characters in analog format, such as on paper or in images, into digital format.

[1404] A "semantic analysis engine" is software that analyzes the meaning of text and image data and understands their relationships.

[1405] "Video generation" is the process of creating a visually easy-to-view video based on assembly procedure information.

[1406] "Distribution" is the process of sending the generated video to users.

[1407] A "robot" is a machine that operates automatically according to a program and can perform assembly tasks.

[1408] The "operation instruction" is a specific instruction for the robot to perform an assembly task.

[1409] This invention relates to a system that automatically generates assembly animations based on instructions and provides operational instructions to a robot. The operation and configuration of the entire system will be explained below, along with specific procedures and examples.

[1410] Overall system configuration

[1411] 1. User Device

[1412] The user terminal is a device used to upload instructions and view the generated videos. Users access the system using a smartphone or tablet and operate it.

[1413] 2. Server

[1414] The server is a central device with key functions such as processing instructions, generating videos, analyzing them, and generating instructions for the robot. Multiple modules and engines work together.

[1415] Invention Implementation Process

[1416] Upload instructions

[1417] Users upload instructions for products such as furniture and plastic models to the system as image files or PDFs from their devices. The devices receive the uploaded instruction files and send them to the server.

[1418] Analysis of the instructions

[1419] The server uses OCR and image recognition technology to analyze the received instruction manual file. OCR technology (e.g., Tesseract) is used to convert the text into digital text data, and image recognition technology (e.g., OpenCV) is used to extract part information from diagrams and illustrations.

[1420] Interpreting the instruction manual information

[1421] The server analyzes the extracted text and image data to generate assembly procedure information. This analysis is performed using a semantic analysis engine, which analyzes the meaning of the text and image data and understands their relationships.

[1422] Assembly video generation

[1423] The server generates an assembly video based on the assembly procedure information. It uses 3D modeling software (e.g., Blender) to generate a 3D model of the part and performs a simulation. It uses a video generation engine (e.g., FFmpeg) to create an assembly video based on the generated 3D model and simulation results.

[1424] Video distribution

[1425] The server stores the completed assembly video and generates a streaming distribution or download link. The device receives the video link sent from the server and displays it to the user. The user can watch the video and actually perform the assembly work by following the displayed steps.

[1426] Instructions for the robot

[1427] Based on the assembly procedure information, the server generates and executes operation instructions for the robot, allowing the robot to accurately assemble the parts according to the generated instructions.

[1428] Specific examples

[1429] Let's say a user takes a photo of a piece of furniture instruction manual with their smartphone and uploads it to the application. The uploaded manual is converted into text using an OCR engine, and diagrams and illustrations are analyzed using an image recognition engine. Based on the analysis results, the server generates assembly procedures and simulates them using 3D modeling software. The generated video is sent to the user's device, and the user can proceed with the assembly work while watching it. Furthermore, a robot accurately performs the assembly work based on the operating instructions generated by the server.

[1430] Prompt Sentence Examples

[1431] Based on the furniture instructions below, please automatically analyze the assembly steps and generate the final assembly video.

[1432] "Secure part A to part B, then attach part C. Then place part D according to the instructions in step 3, and finally secure all parts together with screws."

[1433] Following this prompt, specific steps and component instructions are provided to ensure the model can be analyzed accurately.

[1434] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1435] Step 1: Upload your instructions

[1436] The user takes a photo of the instruction manual that comes with the product to be assembled using a smartphone or tablet, or saves it as a PDF file on the device. The device receives the photo or saved instruction file and sends it to the server. The input is an image file or PDF file of the instruction manual, and the output is the instruction data that is sent to the server. The specific operation is for the user to operate the application to select the file and press the upload button.

[1437] Step 2: Parse the instructions

[1438] The server analyzes the received instruction manual file using OCR and image recognition technology. Specifically, it uses an OCR engine (e.g., Tesseract) to convert the text into digital text data, and an image recognition engine (e.g., OpenCV) to extract part information from diagrams and illustrations. The input is the instruction manual data, and the output is text data and image data. Specifically, the server starts the OCR engine, scans the characters in the image, and converts them into text.

[1439] Step 3: Interpret the information in the instructions

[1440] The server analyzes the extracted text data and image data to generate assembly procedure information. A semantic analysis engine is used for this analysis. The input is text data and image data, and the output is assembly procedure information. Specifically, the semantic analysis engine analyzes the meaning of the text and understands the relationships between parts. For example, it interprets in detail the instruction "Secure part A to part B with a screw."

[1441] Step 4: Generate assembly videos

[1442] The server generates an assembly video based on the assembly procedure information. It uses 3D modeling software (e.g., Blender) to generate a 3D model of the part and performs a simulation. It uses a video generation engine (e.g., FFmpeg) to create an assembly video based on the generated 3D model and simulation results. The input is the assembly procedure information and the output is the assembly video. Specifically, the server starts the 3D modeling software, simulates the operation of the part, and encodes it as a video.

[1443] Step 5: Publish your video

[1444] The server stores the completed assembly video and generates a streaming distribution or download link. The terminal receives the video link sent from the server and displays it to the user. The input is the assembly video, and the output is the video link to the user's terminal. Specifically, the server uploads the video file to cloud storage and sends the link to the terminal.

[1445] Step 6: Instruct the robot to move

[1446] The server generates and executes operational instructions for the robot based on the generated assembly procedure information. Specifically, the instructions include specific operational instructions for the robot to accurately position and fix parts. The input is the assembly procedure information, and the output is the robot's operational instructions. The specific operation is that the server sends the operational instructions to the robot's control program, and the robot starts operating in accordance with the instructions.

[1447] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1448] This invention relates to a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations based on those emotions. The operation and configuration of the entire system are explained below, along with specific procedures and examples.

[1449] Overall system configuration

[1450] 1. User Device:

[1451] The user terminal is a device used to upload instructions and view the generated videos. The user accesses the system from the terminal and operates it.

[1452] 2. Server:

[1453] The server is a central device with key functions such as instruction processing, video generation, analysis, emotion recognition, etc. Multiple modules and engines work together.

[1454] 3. Emotion Engine:

[1455] This engine analyzes the user's facial expressions and voice to recognize their emotional state, and adapts the system's behavior according to the user's emotions.

[1456] Invention Implementation Process

[1457] 1. Upload the instructions

[1458] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDFs.

[1459] Terminal: Receives the uploaded instruction file and sends it to the server.

[1460] 2. Analysis of Instructions

[1461] Server: The server analyzes the received instruction file.

[1462] OCR function: The text in the manual is converted into digital text data using OCR (Optical Character Recognition) technology.

[1463] Image Recognition: Extracting part information from diagrams and illustrations in the manual. This step identifies the part's shape, position, and associated text information.

[1464] Example: When a PDF of furniture instructions uploaded by a user arrives at the server, the OCR function is activated and extracts assembly steps such as "fix part A to part B." At the same time, image recognition technology identifies the diagrams of parts A and B and stores them in a database.

[1465] 3. Interpretation of instructional information

[1466] Server: Analyzes the extracted text data and image data and generates assembly procedure information.

[1467] Semantic analysis engine: Text and image data are fed into the semantic analysis engine to understand the relationships between steps, for example, to interpret in detail how "Part A" and "Part B" are fastened together.

[1468] Example: The server uses a semantic analysis engine to interpret the instruction "Secure part A to part B with screws" and clarifies the specific alignment of the parts, the type of screw, and how to use it.

[1469] 4. Generating assembly videos

[1470] Server: Generates assembly animation based on assembly procedure information.

[1471] 3D modeling software: Generates 3D models of parts and performs simulations, accurately reproducing the movements of each step.

[1472] Video generation engine: Creates assembly videos based on the generated 3D model and simulation results, including part selection procedures, precise alignment, screw rotation motion, etc.

[1473] Example: The server simulates the steps for assembling furniture using 3D modeling software and animates the process. The generated video reproduces the steps, such as "Place part A here, then place part B on top of it and fasten it with screws."

[1474] 5. Operational adaptation using emotion engine

[1475] Server: The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotional state.

[1476] Example: If the server recognizes the user's facial expression and determines that the user is confused, it will take appropriate action, such as slowing down the video playback speed.

[1477] 6. Video distribution

[1478] Server: Stores the completed assembly video and generates streaming and download links.

[1479] Device: Displays the video URL sent from the server in the user interface, allowing the user to stream or download.

[1480] User: Watch the video and follow the instructions to assemble the product. While the video is playing, you can pause, rewind, fast forward, and more.

[1481] Example: As a user plays a video, an emotion engine monitors the user's emotional state and takes appropriate action, such as adjusting the playback speed or displaying additional information.

[1482] summary

[1483] This system allows users to easily understand the visual assembly procedures and assemble products accurately by simply uploading instructions. Furthermore, by using an emotion engine, it can provide optimal operations according to the user's emotions, improving the efficiency and satisfaction of assembly work.

[1484] The processing flow will be explained below.

[1485] Step 1:

[1486] User: The user uploads instructions that come with assembly products such as furniture and plastic models to the system from their terminal as image files or PDF format.

[1487] Step 2:

[1488] Terminal: Receives the uploaded instruction file and sends it to the server.

[1489] Step 3:

[1490] Server: The server analyzes the received instruction file.

[1491] Step 4:

[1492] Server: Starts the OCR engine and converts the text information in the manual into digital text data.

[1493] Step 5:

[1494] Server: Extracts the text data after OCR processing, and simultaneously uses an image recognition algorithm to identify part information from diagrams and illustrations in the manual.

[1495] Step 6:

[1496] Server: The extracted text data and image data are input into a semantic analysis engine to analyze the relationships and meaning between procedures.

[1497] Step 7:

[1498] Server: Generates detailed assembly procedure information from the analysis results using a semantic analysis engine.

[1499] Step 8:

[1500] Server: Based on the assembly procedure information, a 3D model of the part is generated using 3D modeling software.

[1501] Step 9:

[1502] Server: The generated 3D model is input into a video generation engine, which records the assembly procedure as a video while simulating it.

[1503] Step 10:

[1504] Server: Renders the final assembled video, saves it, and generates a URL for the video file.

[1505] Step 11:

[1506] Server: Activates the emotion engine, analyzes the user's facial expressions and voice in real time, and recognizes their emotional state.

[1507] Step 12:

[1508] Server: Adapts the pace and content of video playback based on emotion recognition results from the emotion engine. For example, if a confused expression is detected, commentary may be added or playback speed may be slowed down.

[1509] Step 13:

[1510] Device: Displays the video URL received from the server in the user interface, allowing the user to stream or download.

[1511] Step 14:

[1512] Users: Click on the video link to watch the video and then assemble the parts by following the instructions in the video. During playback, users can pause, rewind, fast forward, and more.

[1513] Step 15:

[1514] Server: The emotion engine continuously monitors the user's emotions while the video is playing and adapts in real time if the user becomes confused.

[1515] Specific examples

[1516] When a user uploads a PDF instruction manual for the furniture they purchased to the system, the server receives the PDF and extracts the text data using OCR technology. At the same time, an image recognition algorithm identifies the parts diagrams in the manual. A semantic analysis engine analyzes the text and image data to generate specific assembly instructions. 3D modeling software then creates 3D models of each part, and uses them to generate assembly instructions as videos.

[1517] The generated video is then delivered to the user, who then watches the video while performing the assembly task. While the video is playing, the emotion engine monitors the user's facial expressions, and if it detects that the user is confused, it slows down the video playback speed or displays additional explanations to help the user understand. This approach allows the user to understand the instructions more intuitively and efficiently, and perform the assembly task accurately.

[1518] Example 2

[1519] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1520] Conventional assembly manuals require a lot of time and effort for users to understand the procedures, placing a significant burden on beginners and those who have difficulty reading manuals. Furthermore, because they do not adapt to the user's emotional state, they lack support for difficult-to-understand parts. This results in problems such as incorrect assembly methods and increased stress during the process.

[1521] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animation based on the generated assembly procedure information, means for recognizing the user's emotional state and adapting operations based on the recognized emotional state, and means for delivering the generated assembly animation to the user. This allows the user to easily understand the visual assembly procedures simply by uploading the instructions, and makes it possible to provide optimal support according to the user's emotional state.

[1522] "Instructions" are documents that explain the assembly procedures and usage of a product or part.

[1523] "Input means" refers to the device or software used to load instructions into the system.

[1524] "Text data" is a digital representation of the textual information contained in the instructions.

[1525] "Image data" means a digital representation of the diagrams and illustrations contained in the instructions.

[1526] The "extracting means" refers to a method or device for extracting text data and image data from the instructions.

[1527] "Means for analyzing" refers to methods or devices for processing extracted digital data to make sense of it and derive information.

[1528] "Assembly procedure information" is digital information that indicates the specific steps and methods required to assemble a product or part.

[1529] "Generating means" refers to a method or device for creating new data or information based on the analysis results.

[1530] "Assembly animation" is content in the form of animation that visually shows assembly procedures.

[1531] A "user" is a person who uses the system to read the instructions and watch the assembly animations.

[1532] "Emotional state" refers to a user's current mental and emotional state.

[1533] The "recognition means" is a method or device for identifying the emotional state of a user from their facial expression or voice.

[1534] "Means for adapting operations" are methods or devices for modifying system behavior or settings based on a recognized emotional state.

[1535] The "distribution means" refers to a method or device for providing the generated assembled animation to a user.

[1536] This invention relates to a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations based on those emotions.

[1537] Overall system configuration

[1538] User terminal

[1539] The user terminal is a device that users use to upload instructions and view generated assembly videos. Examples include personal computers (PCs) and smartphones. Users access the system from their terminals and perform various operations (e.g., uploading files, viewing videos).

[1540] server

[1541] The server is the hardware that manages the core functions of the system, such as data processing of instructions, video generation, analysis, and emotion recognition. The following software and engines run on the server:

[1542] Optical Character Recognition (OCR) software (e.g., Tesseract): converts the text in the instructions into digital data.

[1543] Image recognition engine (e.g., OpenCV, YOLO): Extracts component information from diagrams and illustrations in the manual.

[1544] Natural language processing engine (e.g., SpaCy, BERT): Analyzes the extracted data and generates assembly instructions.

[1545] 3D modeling software (e.g. Blender): Create 3D models of parts and perform simulations.

[1546] Video generation engine: Creates assembly videos based on the generated 3D models.

[1547] Emotion recognition software (e.g., facial expression analysis, voice analysis): Recognizes the user's emotional state and performs corresponding adaptive processing.

[1548] Emotion Engine

[1549] The emotion engine is software that analyzes the user's facial expressions and voice to recognize the user's emotional state and adapt the system's behavior accordingly. For example, if the user is confused, it can adjust the video playback speed or display additional explanations.

[1550] Invention Implementation Process

[1551] Specific example explanation

[1552] Let's say a user uploads a PDF of a piece of furniture's instructions from their computer. Here's what happens:

[1553] 1. Upload the instructions

[1554] The user selects the assembly manual PDF from the file manager on their PC and clicks the "Upload" button. The device then sends this file to the server as an HTTP POST request.

[1555] 2. Analysis of Instructions

[1556] The server processes the received instruction manual file. First, the OCR engine converts the text into digital data, and then the image recognition engine extracts part information from diagrams and illustrations. For example, it extracts the text "Fix part A to part B" and recognizes the diagrams of parts A and B.

[1557] 3. Interpretation of instructional information

[1558] The server uses a natural language processing engine to analyze text and image data and generate assembly instructions. It then interprets the extracted instruction, "Secure part A to part B with screws," in detail to identify the specific positions of the parts and the types of screws.

[1559] 4. Generating assembly videos

[1560] The server uses 3D modeling software to generate a 3D model of the part based on the assembly procedure information, and then simulates it.The animation generation engine uses the simulation results to create an assembly animation that shows the specific steps.

[1561] 5. Operational adaptation using emotion engine

[1562] While the user is watching the video, the server analyzes their facial expressions and voice to recognize their emotional state. If it determines that the user is confused, it will take appropriate action, such as slowing down the video playback speed or displaying supplementary explanations.

[1563] 6. Video distribution

[1564] The generated assembly video is saved on the server and a streaming URL is generated. The URL is sent to the device, and the user can click the "Play Video" button to watch the video. The video can also be downloaded if necessary.

[1565] Prompt Sentence Examples

[1566] Here are some examples of prompts for generative AI models:

[1567] A user uploaded a PDF instruction manual for assembling furniture. The server then used OCR technology to convert the text in the manual into digital text data and image recognition technology to extract component information. It then used a natural language processing engine to interpret the relationships between steps and generated an assembly video using 3D modeling software. It also used emotion recognition software to analyze the user's emotions and adapt the playback speed and supplementary explanations accordingly. If the user showed a confused expression, the server slowed down the video playback speed and displayed supplementary explanations at key points. Please provide an overview of this system and its steps, with examples.

[1568] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1569] Step 1:

[1570] Upload instructions

[1571] (User)

[1572] The user selects the assembly manual as an image file (JPEG, PNG) or PDF file on their device and presses the upload button. This operation selects the manual file.

[1573] (Terminal)

[1574] The terminal sends the instruction file selected by the user to the server as an HTTP POST request.

[1575] The input is a user-selected description file (JPEG, PNG or PDF) and the output is that file is sent to the server.

[1576] Specific behavior:

[1577] The user selects the instruction manual PDF from the file manager on their PC and clicks the upload button on the web application. The device sends the file to the server.

[1578] Step 2:

[1579] Analysis of the instructions

[1580] (server)

[1581] The server analyzes the received instruction manual file. First, it starts an OCR engine (e.g., Tesseract) to digitize the text in the manual. Next, it uses an image recognition engine (e.g., OpenCV or YOLO) to extract part information from the diagrams and illustrations in the manual.

[1582] The input is a manual file, and the output is text data extracted by OCR and part information identified by an image recognition engine.

[1583] Specific behavior:

[1584] The server uses OCR to extract the text from the instructions, "Fix part A to part B," and at the same time, the image recognition engine identifies the diagrams of parts A and B.

[1585] Step 3:

[1586] Interpreting the instruction manual information

[1587] (server)

[1588] Based on the extracted text data and image data, the server uses a natural language processing engine (e.g., SpaCy or BERT) to perform a detailed analysis of the text data and diagram data and generate assembly instructions.

[1589] The inputs include text data and image data from the OCR and image recognition engines, and the output is the generated assembly procedure information.

[1590] Specific behavior:

[1591] The server uses a natural language processing engine to interpret the instruction "fix part A to part B with a screw" in detail, identifying the exact part placement and type of screw, and generating digital assembly instructions.

[1592] Step 4:

[1593] Assembly video generation

[1594] (server)

[1595] Based on the generated assembly procedure information, the server uses 3D modeling software (such as Blender) to create a 3D model of the part. Based on this 3D model, the animation generation engine simulates the assembly procedure and generates an assembly animation that shows the specific movements.

[1596] The input is assembly procedure information from a natural language processing engine, and the output is a generated assembly video.

[1597] Specific behavior:

[1598] The 3D modeling software creates detailed 3D models of parts A and B, and then the video generation engine simulates these to generate a video of specific steps, such as "Place part A here, place part B on top of it, and secure it with screws."

[1599] Step 5:

[1600] Emotional engine for adaptive operation

[1601] (server)

[1602] As users watch videos, the server uses an emotion engine to analyze their facial expressions and voice to recognize their emotional state. If the user is confused, the server will adjust the video playback speed or display additional explanations.

[1603] Inputs include facial expression data and voice data of the user, and outputs include adjusting the playback speed and displaying supplementary explanations.

[1604] Specific behavior:

[1605] If a user shows a confused expression while watching a video, the server slows the video playback speed by 1.5 times and displays explanatory text on the screen at key points.

[1606] Step 6:

[1607] Video distribution

[1608] (server)

[1609] The completed assembly video is saved, and the server generates a URL for streaming or a download link.

[1610] (Terminal)

[1611] The terminal displays the video URL sent from the server on the user interface, allowing the user to stream or download the video.

[1612] (User)

[1613] Users can click the "Play Video" button on their device to watch the streamed video, pausing, rewinding, fast-forwarding, and other operations as needed.

[1614] The input is the generated assembled video, and the output is a streaming URL or download link provided to the user.

[1615] Specific behavior:

[1616] The user clicks the "Play Video" button on the device to play the generated video. If the user shows a confused expression during playback, the emotion engine adjusts the playback speed.

[1617] (Application example 2)

[1618] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1619] Understanding assembly procedures based on instructions is often difficult, especially for beginners or those working for the first time. Furthermore, the confusion and stress felt by users during the process can reduce work efficiency and accuracy. Conventional systems are unable to provide real-time assistance that responds to the user's emotions, making it difficult to improve work efficiency.

[1620] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting instructions, means for extracting text data and image data from the input instructions, means for analyzing the extracted text data and image data and generating assembly procedure information, means for generating assembly animations based on the generated assembly procedure information, means for recognizing the user's emotions and adapting operations based on the emotions, and means for delivering the generated assembly animations to the user. This not only makes it easier for the user to understand the instructions, but also makes it possible to improve work efficiency and accuracy by providing support according to the user's emotions.

[1621] "Instructions" are documents that describe how to assemble or use a product or part.

[1622] "Input means" refers to a device or function that allows a user to load instructions into the system.

[1623] "Extraction means" refers to the technology or device used to extract text data or image data from the instructions.

[1624] The "means for analyzing and generating assembly procedure information" refers to a technique or device for understanding the extracted data and determining specific assembly procedures based on that data.

[1625] The "means for generating assembly animation" refers to a technique or device for creating a visual animation based on assembly procedure information.

[1626] "Distribution means" refers to the technology and devices used to deliver the generated assembly animation to the user's terminal.

[1627] "Means for recognizing user emotions" refers to technology or devices that analyze the user's facial expressions and voice and determine their emotional state.

[1628] "Operational adaptation" refers to techniques and devices for modifying system behavior based on perceived user emotions.

[1629] Overall system configuration

[1630] This invention is a system that combines a system that automatically generates assembly animations based on instructions with an emotion engine that recognizes the user's emotions and adapts operations accordingly. Specifically, it includes the following components:

[1631] 1. User Device:

[1632] A device used by users to upload instructions to the system and view the generated videos. Typically, a PC, smartphone, tablet, or other device is used.

[1633] 2. Server:

[1634] It is a central device with key functions such as instruction processing, video generation, analysis, and emotion recognition. Specifically, it includes OCR technology, image recognition technology, semantic analysis engine, 3D modeling software, and emotion recognition engine.

[1635] 3. Emotion Recognition Engine:

[1636] This technology analyzes a user's facial expressions and voice to recognize their emotional state. For example, it can use deep learning models such as Keras.

[1637] Operation explanation

[1638] 1. Upload the instructions

[1639] Users upload instructions for products such as furniture and plastic models to the system as image files or PDFs from their devices. This operation is performed through the user interface.

[1640] 2. Analysis of Instructions

[1641] The server analyzes the received instruction manual file, converting the text in the manual into digital data using OCR technology, and extracting part information from diagrams and illustrations using image recognition technology.

[1642] 3. Generating assembly procedure information

[1643] The extracted text data and image data are subjected to semantic analysis to generate assembly procedure information. The software used for this process includes a semantic analysis engine and a natural language processing library.

[1644] 4. Generating assembly videos

[1645] The server creates a 3D model based on the generated assembly procedure information, performs simulations, and finally generates an assembly video. This process is supported by a video generation engine.

[1646] 5. Emotion Recognition and Adaptive Manipulation

[1647] The server analyzes the user's facial expressions and voice to recognize their emotional state. For example, if it determines that the user is confused, it will take adaptive action, such as slowing down the video playback speed.

[1648] Specific examples

[1649] For example, consider the case of generating a video showing assembly procedures for a robot arm in a factory. When a worker uploads instructions, OCR and image recognition technology analyzes the instruction data and generates a video of the assembly procedures based on that. As the worker watches the video and performs the work, an emotion recognition engine monitors their emotions and adjusts the video playback speed or supplementary explanations as needed.

[1650] Prompt Sentence Examples

[1651] Upload a PDF of your furniture assembly instructions and we'll generate a video of the assembly process using adaptive control based on emotion recognition.

[1652] This system makes it easier for users to understand instructions and provides appropriate support according to their emotions, improving work efficiency and accuracy.

[1653] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1654] Step 1:

[1655] The user uploads the assembly instruction manual as a PDF or image file from the terminal to the system. The terminal sends the uploaded file to the server. The input is the assembly instruction file, and the output is the file sent to the server.

[1656] Step 2:

[1657] The server analyzes the received instruction manual file using OCR technology (e.g., Tesseract OCR) and extracts text data. It also uses image recognition technology (e.g., OpenCV or TensorFlow) to extract part information from diagrams and illustrations in the manual. The input is the instruction manual file, and the output is text data and image data.

[1658] Step 3:

[1659] The server runs the extracted text data and image data through a semantic analysis engine to generate assembly procedure information. This uses a natural language processing library (e.g., NLTK or spaCy). The input is text data and image data, and the output is assembly procedure information.

[1660] Step 4:

[1661] The server uses 3D modeling software (e.g., Blender) based on the generated assembly procedure information to simulate each assembly step, and generates an assembly video using a video generation engine. The input is the assembly procedure information, and the output is the assembly video.

[1662] Step 5:

[1663] The server uses an emotion recognition engine (e.g., a deep learning model using Keras) to analyze the user's real-time facial expressions and voice. The server recognizes the user's emotional state and adjusts the video playback speed or displays supplementary explanations as needed. The input is the user's facial expression data and voice data, and the output is adjusted video playback and supplementary explanations.

[1664] Step 6:

[1665] The server delivers the generated assembled video to the user terminal. The terminal displays the video URL sent from the server on the user interface, allowing the user to stream or download. The input is the assembled video file, and the output is the generation and delivery of the video URL.

[1666] Step 7:

[1667] The user watches the assembly video on a device and actually performs the assembly work according to the displayed steps. While the video is playing, an emotion recognition engine monitors the user's emotional state and adjusts the playback speed or displays supplementary explanations as necessary. The input is emotion data, and the output is the adjusted assembly video and supplementary explanations.

[1668] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1669] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1670] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1671] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1672] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1673] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1674] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1675] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1676] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1677] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1678] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1679] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1680] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1681] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1682] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1683] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1684] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1685] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1686] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1687] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1688] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1689] The following is further disclosed regarding the above embodiment.

[1690] (Claim 1)

[1691] a means for inputting instructions;

[1692] means for extracting text data and image data from the input instructions;

[1693] a means for analyzing the extracted text data and image data and generating assembly procedure information;

[1694] a means for generating an assembly animation based on the generated assembly procedure information;

[1695] means for distributing the generated assembly animation to a user;

[1696] A system including:

[1697] (Claim 2)

[1698] 2. The system according to claim 1, wherein the text data of the manual is extracted using OCR (Optical Character Recognition) technology.

[1699] (Claim 3)

[1700] 10. The system of claim 1, wherein the extracted textual and image data is analyzed using a semantic analysis engine.

[1701] "Example 1"

[1702] (Claim 1)

[1703] a means for inputting instructions;

[1704] means for extracting text data and image data from the input instructions;

[1705] a means for analyzing the extracted text data and image data and generating assembly procedure information;

[1706] a means for generating an assembly animation based on the generated assembly procedure information;

[1707] means for distributing the generated assembly animation to a user;

[1708] A means to identify the specific software modules (OCR technology, semantic analysis engine, 3D modeling software, etc.) used at each processing step;

[1709] A system including:

[1710] (Claim 2)

[1711] 2. The system according to claim 1, wherein the text data of the manual is extracted using OCR (Optical Character Recognition) technology.

[1712] (Claim 3)

[1713] 10. The system of claim 1, wherein the extracted textual and image data is analyzed using a semantic analysis engine.

[1714] "Application Example 1"

[1715] (Claim 1)

[1716] a means for inputting instructions;

[1717] means for extracting text data and image data from the input instructions;

[1718] a means for analyzing the extracted text data and image data and generating assembly procedure information;

[1719] a means for generating an assembly animation based on the generated assembly procedure information;

[1720] means for distributing the generated assembly animation to a user;

[1721] means for generating and executing operation instructions for a robot based on assembly procedure information;

[1722] A system including:

[1723] (Claim 2)

[1724] 2. The system according to claim 1, wherein the text data of the manual is extracted using OCR (Optical Character Recognition) technology.

[1725] (Claim 3)

[1726] 10. The system of claim 1, wherein the extracted textual and image data is analyzed using a semantic analysis engine.

[1727] "Example 2: Combining Emotion Engines"

[1728] (Claim 1)

[1729] a means for inputting instructions;

[1730] means for extracting text data and image data from the input instructions;

[1731] a means for analyzing the extracted text data and image data and generating assembly procedure information;

[1732] a means for generating an assembly animation based on the generated assembly procedure information;

[1733] means for recognizing an emotional state of a user and adapting operations based on the recognized emotional state;

[1734] means for distributing the generated assembly animation to a user;

[1735] A system including:

[1736] (Claim 2)

[1737] 10. The system of claim 1, wherein the text data of the instructions is extracted using optical character recognition technology.

[1738] (Claim 3)

[1739] 10. The system of claim 1, wherein the extracted text data and image data are analyzed using a natural language processing engine.

[1740] "Application example 2 when combining emotion engines"

[1741] (Claim 1)

[1742] a means for inputting instructions;

[1743] means for extracting text data and image data from the input instructions;

[1744] a means for analyzing the extracted text data and image data and generating assembly procedure information;

[1745] a means for generating an assembly animation based on the generated assembly procedure information;

[1746] means for recognizing a user's emotions and adapting operations accordingly;

[1747] means for distributing the generated assembly animation to a user;

[1748] A system including:

[1749] (Claim 2)

[1750] 2. The system according to claim 1, wherein the text data of the manual is extracted using OCR (Optical Character Recognition) technology.

[1751] (Claim 3)

[1752] 10. The system of claim 1, wherein the extracted textual and image data is analyzed using a semantic analysis engine. [Explanation of symbols]

[1753] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for inputting instructions; means for extracting text data and image data from the input instructions; a means for analyzing the extracted text data and image data and generating assembly procedure information; a means for generating an assembly animation based on the generated assembly procedure information; means for distributing the generated assembly animation to a user; A system including:

2. 2. The system of claim 1, wherein the text data of the manual is extracted using OCR technology.

3. The system of claim 1 , wherein the extracted textual and image data is analyzed using a semantic analysis engine.

Images