system

Abstract

Provide a system. A means for receiving and recording data from a motion capture device; means for analyzing the recorded motion capture data and recognizing specific movement patterns; A means for selecting an optimal industrial robot based on the analyzed motion pattern; means for automatically generating an operation program corresponding to the selected robot; means for verifying the generated operating program in a simulation environment; means for providing a verified operating program to a terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The implementation time for motion programs for industrial robots is long, making it difficult to build efficient systems. While the introduction of robots is increasing to address labor shortages and rising labor costs, generating motion programs requires advanced technology and time. Furthermore, in order to have robots accurately replicate human work movements, programs must be created manually, a very tedious process. This creates a need for rapid and accurate motion program generation and the selection of optimal robots. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for receiving and recording data from a motion capture device, a means for analyzing the recorded motion capture data and recognizing specific motion patterns, a means for selecting an optimal industrial robot based on the analyzed motion patterns, a means for automatically generating a motion program corresponding to the selected robot, a means for verifying the generated motion program in a simulation environment, and a means for providing the verified motion program to a terminal. This shortens the implementation time for industrial robot motion programs and enables rapid and accurate program generation. Furthermore, by automating motion analysis and robot selection, the system can address labor shortages and rising labor costs. Furthermore, the inclusion of a means for verification in a simulation environment allows the accuracy of the program to be confirmed and the program to be corrected if problems are discovered.

[0006] A "motion capture device" is a device that records the movements of people and objects as digital data in real time.

[0007] The "means for receiving and recording data" refers to a means having the function of receiving data transmitted from the motion capture device and storing it.

[0008] "Motion capture data" is digital data representing the movements of a person or object recorded by a motion capture device.

[0009] The "means for analyzing and recognizing specific movement patterns" refers to a means having the function of analyzing received motion capture data, identifying specific movements, and extracting information such as their start time, end time, and joint angles.

[0010] The "means for selecting the optimal industrial robot" is a means that has the function of selecting an industrial robot that meets specifications such as the required degree of freedom of movement, payload capacity, and working range based on the analyzed movement pattern.

[0011] The "means for automatically generating an operating program" is a means having a function for automatically generating an operating program that is compatible with the control language of the selected industrial robot.

[0012] "Means for verifying in a simulation environment" refers to means having the function of virtually executing a generated operating program and checking its accuracy and safety.

[0013] The "means for providing to the terminal" refers to a means having a function for transferring the verified operating program to the terminal either remotely or locally, so that the terminal can use the program. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention is a system that receives and analyzes data from a motion capture device and then automatically generates an operation program for an industrial robot. This system is composed of a motion capture device, a server, and a terminal, and operates as follows.

[0036] 1. Acquiring motion capture data

[0037] User

[0038] A user wears a motion capture device and performs a specific task, such as welding or picking. As the user performs the task, the motion is recorded by the motion capture device.

[0039] server

[0040] The server receives and stores the data sent from the motion capture device in real time. The data is recorded in a digital format (e.g., BVH, FBX).

[0041] 2. Motion analysis

[0042] server

[0043] The server reads the saved motion capture data and analyzes specific movement patterns, including the start and end times of the movement, joint angles, and speed, etc. Based on this, key frames are generated and sequential movement analysis is performed.

[0044] 3. Industrial robot selection

[0045] server

[0046] The server extracts data on the required degree of freedom of movement, payload capacity, and working range based on the analyzed movement patterns, then compares this data with an internal database to create a list of suitable industrial robots, and selects the most suitable robot from among them.

[0047] 4. Generating the operating program

[0048] server

[0049] The server loads a template corresponding to the selected robot's control language and applies the analyzed motion data to the template, thereby generating a specific set of control instructions, which are then compiled into a robot's motion program.

[0050] 5. Verification of the operating program

[0051] server

[0052] The server runs the generated motion program in a simulation environment to check its accuracy and safety. It also automatically inspects the simulation results for collision detection, timing accuracy, and other errors.

[0053] 6. Program Offering

[0054] server

[0055] In order to provide the terminal with the verified operating program, the server prepares the program file and transfers it to the terminal.

[0056] Terminal

[0057] The terminal stores the program file received from the server in local storage.

[0058] 7. Running the operation program

[0059] Terminal

[0060] The terminal transfers the saved program file to the industrial robot, loads it into the robot's control system, and then issues an instruction to the robot to execute the program.

[0061] User

[0062] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[0063] Specific examples

[0064] For example, when a user records a specific picking task, the motion capture device receives the motion as data. The server analyzes the data, selects the industrial robot best suited to the picking task, and generates a motion program for that robot. The accuracy of the program is checked in a simulation environment, and once no problems are confirmed, the program is provided to a terminal and installed on the robot. Finally, the user can monitor the robot's operations and make adjustments as necessary, completely automating the picking task.

[0065] This invention significantly reduces the time required to implement an industrial robot's operation program, enabling rapid and accurate program generation, thereby providing an effective system that can address labor shortages and rising labor costs.

[0066] The processing flow will be explained below.

[0067] Step 1: Acquiring motion capture data

[0068] User

[0069] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking). Once the task is initiated, the motion capture device records the user's movements in real time.

[0070] server

[0071] The server receives the data sent from the motion capture device and records the data in real time, which is then saved in a digital format such as BVH or FBX.

[0072] Step 2: Behavior analysis

[0073] server

[0074] The server reads the recorded motion capture data and begins analyzing the movements. Specifically, it analyzes the data frame by frame to recognize specific movement patterns. It extracts information such as the start and end times of the movements, joint angles, and speed, and generates key frames based on this information. Finally, sequential movement analysis is performed, and the movement patterns are clearly defined.

[0075] Step 3: Selecting an industrial robot

[0076] server

[0077] Based on the analyzed motion patterns, the server identifies requirements such as the required degrees of freedom of movement, payload capacity, and working range. It then references an internal database to create a list of industrial robots that match these requirements. From the list, it selects the most suitable robot and obtains its specification data.

[0078] Step 4: Generate the operating program

[0079] server

[0080] The server loads a template file corresponding to the selected robot's control language (e.g., Karel, RAPID, TP), applies the analyzed motion data to the template, and generates a specific set of control instructions. Based on the generated instruction set, it compiles a motion program for the robot.

[0081] Step 5: Verify the program

[0082] server

[0083] The server loads the generated motion program into the simulation environment. The simulation is run to check the accuracy of the motion program. Specifically, collision detection and timing checks are performed in the simulation to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[0084] Step 6: Program delivery

[0085] server

[0086] The server prepares to provide the verified operating program to the terminal, specifically by converting the program file into an appropriate format and preparing to transfer it to the terminal.

[0087] Terminal

[0088] The terminal receives the program file sent from the server and stores it in local storage.

[0089] Step 7: Run the action program

[0090] Terminal

[0091] The terminal transfers the saved program file to the industrial robot, which then loads the program into the robot's control system and instructs it to run.

[0092] User

[0093] Users can monitor the robot's operations to ensure they are working correctly, and can make fine adjustments or additional program modifications as needed, allowing specific tasks to be accurately replicated and automated.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Creating operational programs for conventional industrial equipment required a significant amount of time and specialized knowledge, which could result in delays in the automation process for certain tasks. Furthermore, there was a lack of means to verify the accuracy and safety of operational programs in advance, which created the risk of malfunctions during execution. This made it difficult to address labor shortages and rising labor costs.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific motion patterns, means for selecting the optimum industrial equipment based on the analyzed motion patterns, means for automatically generating an operation program corresponding to the selected equipment, means for verifying the generated operation program in a simulation environment before execution, means for providing the verified operation program to an information terminal, means for the information terminal to transfer the provided operation program to a control target and issue an execution instruction, and means for monitoring the operation of the control target and making adjustments as necessary. This makes it possible to quickly and accurately generate, verify, and provide operation programs for industrial equipment, thereby facilitating the automation of work.

[0099] A "motion capture device" is a device that records the movements of a human body or object using sensors or cameras and converts that data into a digital format.

[0100] A "server" is a device or software system that receives data from a motion capture device, analyzes, stores, generates and verifies motion programs.

[0101] "Digital format" refers to a file format such as BVH or FBX that is used to digitize and record movement data.

[0102] "Movement pattern" refers to the characteristics and route of a series of movements recognized based on motion capture data.

[0103] "Industrial equipment" is a general term for robots and other automation devices used in production and manufacturing processes.

[0104] An "operation program" refers to a set of instructions for controlling the operation of industrial equipment, and is written in a specific control language.

[0105] A "simulation environment" is a virtual space or software environment in which a generated operating program is executed and its accuracy and safety are virtually confirmed.

[0106] An "information terminal" is a device or system used to receive, store, display, and control data from a server or other device.

[0107] The term "controlled object" refers to industrial equipment that is controlled using an operating program.

[0108] The "internal database" is a database system that stores specification data and operation history data for industrial equipment.

[0109] A "key frame" refers to a frame that is an important point or turning point in a movement pattern, and detailed movements are analyzed based on this frame.

[0110] "Weight capacity" refers to the maximum weight that industrial equipment can lift and carry.

[0111] "Degrees of freedom of movement" is an indicator that shows the direction and range of movement of each joint and axis of industrial equipment and robots.

[0112] A "prompt" refers to an instruction or question that is input into a generative AI model, and functions as an input to obtain a specific output.

[0113] This invention is a system that records the movements of people and objects as data, analyzes the data, and automatically generates operating programs suitable for industrial equipment. This system consists of a motion capture device, a server, and a terminal. The detailed configuration and operation of this system are described below.

[0114] Hardware and Software Configuration

[0115] motion capture equipment

[0116] A motion capture device records a user's movements in real time and transmits the data to a server. A typical example of such a device is a motion capture suit equipped with cameras and sensors. The data is recorded in digital formats such as BVH and FBX.

[0117] server

[0118] The server receives, stores, analyzes, generates, and verifies the data sent from the motion capture device. The server has the following functions:

[0119] Data Reception and Storage: Motion capture data is received in real time and stored in a digital format.

[0120] Movement Analysis: Analyzes stored data and recognizes specific movement patterns, including identifying joint angles, movement speed, and start and end times.

[0121] Industrial equipment selection: Based on the analyzed movement patterns, the system selects equipment suitable for the required degree of freedom of movement, payload capacity, and working range from an internal database.

[0122] Generation of operation program: A specific set of control instructions is generated using a template corresponding to the control language of the selected device.

[0123] Verification in a simulation environment: The generated behavioral program is executed in a simulation environment to verify the correctness and safety of the program.

[0124] Terminal

[0125] The terminal receives the operation program sent from the server and transfers it to the industrial equipment. The terminal has the following functions:

[0126] Receiving and saving programs: Saves program files provided by the server in local storage.

[0127] Linking with industrial equipment: Transfers saved program files to industrial equipment and loads them into the control system. Executes the program according to user instructions.

[0128] Specific examples

[0129] For example, when a user records a specific picking task, the motion capture device receives the motion as data. The server analyzes the data and selects the industrial equipment best suited to the picking task. It then generates a motion program for that equipment and checks the accuracy of the program in a simulation environment. After verifying that there are no problems, the program is provided to a terminal and ultimately implemented in the robot. The user can monitor the robot's operations and make adjustments as necessary to fully automate the picking task.

[0130] Prompt Sentence Examples

[0131] "Please explain the detailed process of a system in which a user wears a motion capture device and performs a picking task, receives the motion data, analyzes the data, selects the most suitable industrial equipment, and generates and provides an operating program for the equipment. We would like a detailed explanation of each step of this system, including the specific processing content and algorithms used."

[0132] This invention enables the rapid and accurate generation and verification of operating programs for industrial equipment, facilitating the automation of work, thereby providing an efficient system that can cope with labor shortages and rising labor costs.

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] Step 1:

[0135] Acquiring motion capture data

[0136] The user wears a motion capture device and performs specific task movements. The device records the user's movements in real time and transmits the data to a server in a digital format (e.g., BVH, FBX).

[0137] Specifically, when a user performs picking work, the series of actions is captured by sensors and cameras.

[0138] Input: User action (e.g. picking work)

[0139] Output: Motion capture data (e.g. BVH, FBX)

[0140] Step 2:

[0141] Receiving and storing data

[0142] The server receives and stores the data transmitted from the motion capture device in real time, and the data is recorded in digital format.

[0143] Input: Motion capture data

[0144] Output: Saved motion capture data

[0145] Specifically, the server saves the data to disk in the specified format (e.g., BVH, FBX).

[0146] Step 3:

[0147] Motion analysis

[0148] The server reads the saved motion capture data and analyzes specific movement patterns, extracting the start and end times of the movement, as well as the joint angles and speeds, and generates keyframes.

[0149] Input: Stored motion capture data

[0150] Output: Analyzed movement patterns and keyframes

[0151] The server uses motion analysis algorithms to extract key motion points, for example, identifying specific motion patterns based on joint angles and movement speed.

[0152] Step 4:

[0153] Industrial Equipment Selection

[0154] Based on the analyzed motion patterns, the server selects from its internal database the industrial equipment that is suitable for the required degree of freedom of movement, payload capacity, and working range.

[0155] Input: Analyzed movement patterns and internal database

[0156] Output: List of selected industrial devices

[0157] Specifically, the server lists devices that meet pre-set filtering conditions and ranks the most suitable devices among them.

[0158] Step 5:

[0159] Generating the operating program

[0160] The server loads a template corresponding to the control language (e.g., URScript, KRL) of the selected industrial equipment and generates an operating program by applying the analyzed operating data to the template.

[0161] Input: Selected industrial equipment, analyzed operation patterns, control language template

[0162] Output: Generated behavior program

[0163] Specifically, the server integrates an instruction set based on an operation pattern into a template, and compiles it to generate a complete program.

[0164] Step 6:

[0165] Verification of the operating program

[0166] The server runs the generated motion program in a simulation environment to verify the accuracy and safety of the motion, which also includes collision detection and timing adjustment.

[0167] Input: Generated behavior program

[0168] Output: Verification results (verification of accuracy and safety)

[0169] Specifically, the server runs an operating program in a virtual environment and checks for any errors or malfunctions.

[0170] Step 7:

[0171] Program Offering

[0172] The server prepares a program file and transfers it to the terminal in order to provide the terminal with the verified operating program.

[0173] Input: Verified working program

[0174] Output: Transferred program file

[0175] Specifically, the server compresses the program file, encrypts it if necessary, and transmits it to the terminal.

[0176] Step 8:

[0177] Execution of the operating program

[0178] The terminal transfers the stored program file to the industrial equipment, loads it into the control system, and then issues an instruction to run the program.

[0179] Input: Transferred program file

[0180] Output: The device behavior caused by the executed program

[0181] Specifically, the terminal uses the control interface of the industrial equipment to load the program and send an execution instruction.

[0182] Step 9:

[0183] Monitor and adjust behavior

[0184] Users monitor the operation of industrial equipment and make fine adjustments or additional program modifications as needed.

[0185] Input: Running industrial equipment operations

[0186] Output: Adjusted behavior, modified program

[0187] Specifically, the user observes the operation in real time and manually corrects or reprograms any malfunctions.

[0188] (Application example 1)

[0189] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0190] In modern factory automation, many procedures rely on manual labor and robot operation programming. With conventional systems, it takes time to generate robot operation programs, and fine-tuning the programs is not easy. As a result, production efficiency can decrease and production can be halted due to errors. There is a need for a system that can solve this problem and enable rapid generation of operation programs and adjustments in real time.

[0191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0192] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific movement patterns, means for selecting an optimum industrial robot based on the analyzed movement patterns, means for automatically generating an operation program corresponding to the selected robot, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a mobile communication terminal so that a worker can fine-tune it in real time. This enables rapid and accurate generation of an operation program and adjustment in real time.

[0193] Key Word Definitions

[0194] ---

[0195] A "motion capture device" is a device that records the movements of people or objects as digital data.

[0196] The "means for receiving and recording data" is a system that has the function of receiving data transmitted from the motion capture device and storing it.

[0197] "Means for analyzing recorded motion capture data and recognizing specific movement patterns" refers to a system that has the function of analyzing stored data and identifying work movements and types of movements.

[0198] The "means for selecting the optimal industrial robot" is a system that has the function of selecting the robot that can perform the most appropriate operation based on the analyzed operation pattern.

[0199] The "means for automatically generating an operating program" is a system that has the function of automatically creating a program corresponding to the selected robot.

[0200] "Means for verifying in a simulation environment" refers to a system that has the function of testing in a virtual environment whether the generated program is accurate.

[0201] A "mobile communication terminal" is a portable communication device such as a smartphone or tablet.

[0202] "Means that allow workers to make fine adjustments in real time" refers to a system that has the function of immediately making necessary changes and fine adjustments to a verified program.

[0203] A "database" is a system that stores robot specification data and other related information and manages it so that necessary data can be retrieved quickly.

[0204] "Movement freedom" is a concept that refers to the number and range of movements that a robot can perform.

[0205] "Weight capacity" refers to the maximum weight that an industrial robot can safely lift and carry.

[0206] "Work envelope" refers to the physical area within which an industrial robot can move.

[0207] "Means for verifying accuracy and correcting the program if problems are found" refers to a system that has the ability to verify whether the generated program operates correctly and make appropriate corrections when errors or problems are found.

[0208] MODE FOR CARRYING OUT THE INVENTION

[0209] ---

[0210] This invention relates to a system that automatically creates an operating program for an industrial robot by receiving and analyzing motion capture data. This system is comprised of a motion capture device, a server, and a mobile communication terminal, and operates as follows.

[0211] 1. Acquiring motion capture data

[0212] User

[0213] A user wears a motion capture device and performs specific task movements, which are recorded as digital data by the motion capture device.

[0214] server

[0215] The server receives and stores the data sent from the motion capture equipment in real time, and stores this data in a digital format (e.g., BVH, FBX).

[0216] 2. Motion analysis

[0217] server

[0218] The server reads the stored motion capture data and analyzes specific movement patterns, including the start and end times of the movement, joint angles and speeds, etc. Keyframes are generated based on the analyzed data, and sequential movement analysis is performed.

[0219] 3. Industrial robot selection

[0220] server

[0221] Based on the analysis results, the server compares the robot specifications retrieved from the database to select the optimal industrial robot, including the required degrees of freedom of movement, payload capacity, and working range.

[0222] 4. Generating the operating program

[0223] server

[0224] The server loads a template corresponding to the selected robot's control language and applies the analyzed motion data, thereby generating a specific set of control instructions and compiling them into a robot's motion program.

[0225] 5. Verification of the operating program

[0226] server

[0227] The server runs the generated program in a simulation environment to verify its accuracy and safety. Based on the results of the simulation, the program is automatically modified if necessary.

[0228] 6. Program provision and real-time adjustment

[0229] server

[0230] In order to provide the verified operating program to the mobile communication terminal, the server prepares a program file and transfers it to the mobile communication terminal.

[0231] Mobile communication terminal

[0232] The mobile communication terminal stores the program files received from the server and provides an interface that allows the user to fine-tune the operation in real time.

[0233] User

[0234] The user monitors the robot's movements using a mobile communication terminal and fine-tunes the movement program in real time to finalize it. Additional program modifications are also possible as needed.

[0235] Hardware and software used

[0236] Hardware: Mobile communication devices (smartphones and tablets), motion capture devices (e.g., motion sensors), industrial robots

[0237] Software: Server-side technologies (e.g., Django, Flask), simulation environments (e.g., ROS, Gazebo), data parsing libraries (e.g., Python requests, json)

[0238] Specific examples

[0239] Factory workers operate smartphones and use motion capture devices to record specific picking movements. The recorded data is sent to a server, where it is analyzed and the server selects the robot best suited to the picking movement. A motion program for that robot is automatically generated and verified through simulation. After verification, the generated program is sent to a mobile communication device (e.g., a smartphone), where the worker can fine-tune the movement in real time.

[0240] Example prompts to input to the generative AI model

[0241] "We are seeking advice on the design and implementation of a system that analyzes motion capture data and automatically generates motion programs for industrial robots."

[0242] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0243] Program processing flow

[0244] ---

[0245] Step 1:

[0246] A user wears a motion capture device and performs a specific task. The input data here is the user's motion information, and the output is the motion data recorded by the motion capture device.

[0247] Specific behavior:

[0248] The user performs picking operations and welding parts, and the series of movements is recorded as digital data by a motion capture device.

[0249] Step 2:

[0250] The recorded motion capture data is sent to the server in real time. The input data is the digital format data (e.g. BVH, FBX) sent from the motion capture device, and the output is the motion capture data stored on the server.

[0251] Specific behavior:

[0252] The data collected by the motion capture device is sent via wireless or wired communication to a server, which stores it in a database.

[0253] Step 3:

[0254] The server analyzes the stored motion capture data and recognizes specific movement patterns. The input data is the motion capture data stored on the server, and the output is the analyzed movement pattern and related parameters (start time, end time of movement, joint angles, etc.).

[0255] Specific behavior:

[0256] The server uses data analysis algorithms (e.g., machine learning models) to extract specific actions or keyframes within the data.

[0257] Step 4:

[0258] Based on the analysis results, the server selects the optimal industrial robot. The input data is the analyzed motion pattern and the corresponding robot specification data (degree of freedom of motion, payload capacity, working range, etc.), and the output is the selected robot.

[0259] Specific behavior:

[0260] The server checks its internal database, lists the robots that best meet the required specifications, and selects the most suitable robot.

[0261] Step 5:

[0262] The server automatically generates an operating program for the selected robot. The input data is a template based on the control language of the selected robot and analyzed operation patterns, and the output is an operating program containing a specific set of control instructions.

[0263] Specific behavior:

[0264] The server uses a program generation algorithm to generate a control instruction set by applying an operation pattern to a template.

[0265] Step 6:

[0266] The generated behavioral program is verified in a simulation environment. The input data is the generated behavioral program, and the output is the simulation results and the verified program.

[0267] Specific behavior:

[0268] The server runs the generated program using a robot simulation environment (e.g., ROS, Gazebo) to check its safety and accuracy, and automatically corrects the program if necessary.

[0269] Step 7:

[0270] The verified operating program is transferred to the mobile communication terminal, the input data is the verified program file, and the output is the program file stored in the mobile communication terminal.

[0271] Specific behavior:

[0272] The server transmits the program file to the mobile communication terminal, which stores the file in its local memory.

[0273] Step 8:

[0274] The user fine-tunes the robot's movements in real time using a mobile communication terminal. The input data is a verified program file and the user's instructions, and the output is the final modified movement program.

[0275] Specific behavior:

[0276] Workers can fine-tune the operation program via a smartphone or tablet interface to determine the optimal operation program.

[0277] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0278] This invention is a system that combines a system that receives and analyzes data from a motion capture device and automatically generates an operating program for an industrial robot with an emotion engine that recognizes the user's emotions. This system is composed of a motion capture device, a server, a terminal, and an emotion engine, and operates as follows.

[0279] 1. Acquiring motion capture data

[0280] User

[0281] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking). Once the task is initiated, the motion capture device records the user's movements in real time.

[0282] server

[0283] The server receives and records the data sent from the motion capture device in real time, and saves it in formats such as BVH or FBX.

[0284] 2. Motion analysis

[0285] server

[0286] The server loads the saved motion capture data and begins analysis, including the start and end times of the motion, joint angles, and velocity. Based on this, key frames are generated and sequential motion analysis is performed.

[0287] 3. Emotion Recognition by Emotion Engine

[0288] server

[0289] The server utilizes an emotion engine to recognize the user's emotional state. The emotion engine detects the user's facial expressions, voice, and biometric signals, and identifies the user's emotions based on these data. For example, if the user is feeling stressed, the emotional state is recorded.

[0290] 4. Adjusting the movement data

[0291] server

[0292] The server adjusts the analyzed movement patterns based on the emotional state recognized by the emotion engine. For example, if the user is nervous and their movements are awkward, the server corrects those parts to generate a smooth movement pattern.

[0293] 5. Industrial Robot Selection

[0294] server

[0295] The server determines the required degrees of freedom of movement, payload capacity, and working range based on the adjusted motion pattern. It then references an internal database to create a list of industrial robots that meet these requirements. It then selects the most suitable robot from the list and obtains its specification data.

[0296] 6. Generating the operating program

[0297] server

[0298] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, and generates a specific set of control instructions, which are then compiled into the robot's motion program.

[0299] 7. Verification of the operating program

[0300] server

[0301] The server loads the generated motion program into the simulation environment. The simulation is run to verify the accuracy of the motion program. Specifically, collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[0302] 8. Program Offering

[0303] server

[0304] The server prepares to provide the verified operating program to the terminal, converts the program file into an appropriate format, and prepares to transfer it to the terminal.

[0305] Terminal

[0306] The terminal receives the program file sent from the server and stores it in local storage.

[0307] 9. Running the Operation Program

[0308] Terminal

[0309] The terminal transfers the saved program file to the industrial robot, loads the program into the robot's control system, and then issues an instruction to the robot to execute the program.

[0310] User

[0311] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[0312] Specific examples

[0313] For example, when a user records a specific welding task, the motion capture device receives the motion as data. The server analyzes the data and uses an emotion engine to identify areas where the user is feeling stressed. Next, it corrects any motion irregularities caused by stress, selects the most suitable industrial robot based on the adjusted motion data, and generates a motion program for that robot. After verifying the accuracy of the program in a simulation environment, it is provided to the terminal and finally implemented in the robot. The user monitors the robot's motion and makes adjustments as necessary, automating the welding task.

[0314] This invention significantly shortens the implementation time for industrial robot operation programs, enabling rapid and accurate program generation. Furthermore, by taking the user's emotional state into consideration, it is possible to provide more accurate operation programs. This system can provide an effective means of addressing labor shortages and rising labor costs.

[0315] The processing flow will be explained below.

[0316] Step 1: Acquiring motion capture data

[0317] User

[0318] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking), which records the user's movements in real time.

[0319] server

[0320] The server receives and records the data sent from the motion capture equipment in real time, and the data is saved in digital formats such as BVH and FBX.

[0321] Step 2: Behavior analysis

[0322] server

[0323] The server reads the stored motion capture data and analyzes it frame by frame, extracting the start and end times of the movements, joint angles, speed, etc., and generates keyframes based on this information. Specific movement patterns are identified through the analysis.

[0324] Step 3: Emotion recognition by the emotion engine

[0325] server

[0326] The server utilizes an emotion engine to recognize the user's emotional state. The emotion engine detects the user's facial expressions, voice, and biometric signals, and identifies the user's emotions based on these data. For example, if the user is feeling stressed, the emotional state is recorded.

[0327] Step 4: Adjusting the behavior data

[0328] server

[0329] The server adjusts the analyzed movement patterns based on the recognized emotional state: if the user is nervous and their movements are awkward, it corrects those parts and converts them into smoother movement patterns.

[0330] Step 5: Selecting an industrial robot

[0331] server

[0332] The server determines the required degree of freedom of movement, payload capacity, and working range based on the adjusted motion pattern, and then references an internal database to create a list of industrial robots that match these requirements and select the most suitable robot.

[0333] Step 6: Generate the operating program

[0334] server

[0335] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, generates a specific set of control instructions, and compiles them into a robot motion program.

[0336] Step 7: Verify the program

[0337] server

[0338] The server loads the generated motion program into the simulation environment. The simulation is run to check the accuracy and safety of the motion program. Collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[0339] Step 8: Program Delivery

[0340] server

[0341] The server prepares to provide the verified operating program to the terminal, converts the program file into an appropriate format, and prepares to transfer it to the terminal.

[0342] Terminal

[0343] The terminal receives the program file sent from the server and stores it in local storage.

[0344] Step 9: Run the action program

[0345] Terminal

[0346] The terminal transfers the saved program file to the industrial robot, which then loads the program into the robot's control system and issues an instruction to run it.

[0347] User

[0348] The user monitors the robot's operation to ensure it is working correctly. If necessary, they can make fine adjustments or additional program modifications. Once proper operation is confirmed, the robot will automate the task.

[0349] Specific examples

[0350] For example, when a user records a specific picking task, the motion capture device receives the movement as data. The server analyzes the data and uses an emotion engine to determine whether the user is feeling stressed. The server then corrects any movement irregularities caused by stress and selects the optimal industrial robot based on the adjusted movement data. A movement program appropriate for the selected robot is generated, its accuracy confirmed through simulation, and then provided to the terminal and installed on the robot. The user monitors the robot's movements and makes adjustments as necessary, allowing the picking task to be accurately automated.

[0351] Example 2

[0352] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0353] Conventional industrial robot motion program generation systems have the risk of motion accuracy decreasing due to the user's emotional state, such as stress or tension. Furthermore, generating and verifying motion programs takes time, resulting in inefficiency. This has led to problems with automation in industrial settings and productivity not improving.

[0354] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0355] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific movement patterns, means for utilizing an emotion engine for recognizing the emotional state of a user, means for adjusting the movement patterns based on the recognized emotional state, means for selecting an optimum industrial robot based on the adjusted movement patterns, means for automatically generating an operation program corresponding to the selected robot, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a terminal. This enables the generation and verification of an operation program quickly and accurately while taking the emotional state of a user into consideration.

[0356] A "motion capture device" is a device that records a user's body movements in real time and transmits the data to a server.

[0357] The "means for receiving and recording data" is a component of the server that has the function of receiving data transmitted from the motion capture device and recording that data.

[0358] The "means for analyzing motion capture data" is a server component that has the function of analyzing the start time, end time, joint angles, speed, etc. of a movement based on recorded motion capture data, and recognizing a specific movement pattern.

[0359] An "emotion engine" is software or hardware that analyzes a user's facial expressions, voice, and biometric signals to identify the user's emotional state.

[0360] The "means for adjusting the movement pattern" is a component of the server that has the function of correcting unnatural parts of the analyzed movement pattern and adjusting it to a smooth movement pattern based on the recognized emotional state of the user.

[0361] An "industrial robot" is an automated device used in industrial settings that requires an operating program to perform a specific task (e.g., welding, assembly, picking).

[0362] The "means for automatically generating an operation program" is a component of the server that has the function of automatically generating an operation program in a control language corresponding to the selected robot based on the adjusted operation data.

[0363] A "simulation environment" is a computer-based system for executing generated operating programs in a virtual space and verifying their accuracy and errors.

[0364] The "means for verifying the operation program" is a component of the server that has the function of checking the accuracy of the operation program generated using the simulation environment and making any necessary corrections.

[0365] A "terminal" is an electronic device that receives an operating program provided by a server, stores it in local storage, and transfers it to an industrial robot for execution.

[0366] This invention is a system that combines a system that receives and analyzes data from a motion capture device and automatically generates an operating program for an industrial robot with an emotion engine that recognizes the user's emotions. This system is composed of a motion capture device, a server, a terminal, and an emotion engine, and operates as follows.

[0367] 1. Generate a program for this system. Specifically, create a program to receive and record data sent from the motion capture device in real time. This data is saved in formats such as BVH or FBX. Next, create a program to analyze the saved motion capture data and extract information such as the start and end times of the movement, joint angles, and speed. This also includes a program that uses an emotion engine to recognize the user's emotional state from facial expressions, voice, and biometric signals. These programs will use a generative AI model to adjust movement patterns and automatically generate optimal movement programs for industrial robots.

[0368] 2. The processing of the generated program is explained in natural language. Specifically, a server is used to receive and record data from the motion capture device. The server saves the recorded data in formats such as BVH or FBX and analyzes the movements. It uses an emotion engine to recognize the user's emotional state and adjusts the movement pattern as needed. Based on the adjusted movement data, the server selects the optimal industrial robot and automatically generates a movement program for that robot. The generated program is verified in a simulation environment to confirm there are no problems before being provided to the terminal. The terminal transfers the provided program to the industrial robot and executes it. The user can monitor the robot's movement and make fine adjustments or additional program modifications as needed.

[0369] 3. Specific examples are added to sentences that explain the program's processing. For example, a user wears a motion capture device and performs a specific welding task. The server receives and records this data in real time in BVH format. The server then analyzes the data, calculating the start and end times of the motion, joint angles, speed, and other parameters, and generates keyframes. The server then uses an emotion engine to analyze the user's facial expressions, voice, and biometric signals to identify stress levels. Based on the stress levels identified by the emotion engine, the server adjusts the analyzed movement patterns. For example, it modifies the user's tense movements to smoother movements. Based on the adjusted movement patterns, the server then identifies the required degrees of freedom of movement, payload capacity, and working range, and selects the optimal industrial robot from a database. The server generates a movement program using a template file corresponding to the selected robot's control language (e.g., Karel). The generated movement program is then verified in a simulation environment to check timing and detect collisions. If verification is successful, the program is transferred to the terminal, which then loads it onto the industrial robot. The user then monitors the robot's movements and makes adjustments as necessary, automating the welding process.

[0370] Specific prompt examples:

[0371] "Please explain in detail, step by step, the procedure for generating an operation program for an industrial robot that modifies its operation pattern based on the stress felt by the user during welding work."

[0372] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0373] Step 1:

[0374] Acquiring motion capture data

[0375] User

[0376] The user wears a motion capture device and performs a specific work motion (e.g., welding). Once the motion is initiated, the motion capture device records the user's movements in real time.

[0377] Input: User's physical movements

[0378] Output: Real-time motion capture data

[0379] Step 2:

[0380] Receiving and recording data

[0381] server

[0382] The server receives the data sent from the motion capture device in real time and records it in BVH or FBX format.

[0383] Input: Motion capture data

[0384] Output: BVH or FBX format files

[0385] Step 3:

[0386] Motion analysis

[0387] server

[0388] The server reads the recorded motion capture data and analyzes the start and end times of the movements, joint angles, speed, etc. Based on this, key frames are generated and detailed movement analysis is performed.

[0389] Input: BVH or FBX format files

[0390] Output: Analysis results (start time, end time, joint angles, speed, keyframes)

[0391] Step 4:

[0392] Emotion recognition by emotion engine

[0393] server

[0394] The server uses an emotion engine to analyze the user's facial expressions, voice, and biometric signals to identify the user's emotional state. If the user is feeling stressed or tense, the server records that emotional state.

[0395] Input: User's facial expressions, voice, and biometric signals

[0396] Output: Emotional state (e.g., stress, tension)

[0397] Step 5:

[0398] Adjusting the movement data

[0399] server

[0400] The server adjusts the analyzed movement patterns based on the recognized emotional state. For example, if the user is nervous and their movements are awkward, it will correct those parts to make them smoother.

[0401] Input: Analysis results, emotional state

[0402] Output: Coordinated movement pattern

[0403] Step 6:

[0404] Industrial robot selection

[0405] server

[0406] The server determines the required degree of freedom of movement, payload capacity, and working range based on the adjusted motion pattern, and then references an internal database to create a list of industrial robots that match these requirements and select the most suitable robot.

[0407] Input: Adjusted movement pattern

[0408] Output: Selected industrial robot

[0409] Step 7:

[0410] Generating the operating program

[0411] server

[0412] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, and generates a specific set of control instructions, which are then compiled into the robot's motion program.

[0413] Input: Selected industrial robot, adjusted motion pattern

[0414] Output: Working program

[0415] Step 8:

[0416] Verification of the operating program

[0417] server

[0418] The server loads the generated motion program into the simulation environment. The simulation is run to verify the accuracy of the motion program. Specifically, collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[0419] Input: Action program

[0420] Output: Verification results (correctness, presence of errors)

[0421] Step 9:

[0422] Program Offering

[0423] server

[0424] The server provides the verified operating program to the terminal, converts the program file into a format compatible with the robot control system, and prepares it for transfer to the terminal.

[0425] Input: Verified working program

[0426] Output: Program file for distribution

[0427] Terminal

[0428] The terminal receives the program file sent from the server and stores it in local storage.

[0429] Input: Program file to be provided

[0430] Output: Program files in local storage

[0431] Step 10:

[0432] Execution of the operating program

[0433] Terminal

[0434] The terminal transfers the program file stored in the local storage to the industrial robot, loads the program into the robot's control system, and then issues an instruction to the robot to execute the program.

[0435] Input: Program files in local storage

[0436] Output: Running program

[0437] User

[0438] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[0439] Input: Running action program

[0440] Output: Monitoring results and fine-tuning

[0441] (Application example 2)

[0442] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0443] The rapid generation of efficient and accurate motion programs for industrial robots is a high requirement in many factories. However, conventional systems require a great deal of time and effort to generate and verify motion programs. Furthermore, because they do not take into account the impact of the worker's emotional state on motion, the motions of unskilled or fatigued workers are directly reflected in the program, resulting in the generation of inefficient motion programs. Furthermore, modifying motion programs is complex, making it difficult to generate programs optimized for specific work conditions.

[0444] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing a specific movement pattern, means for selecting an optimal industrial robot based on the analyzed movement pattern, means for automatically generating an operation program corresponding to the selected robot, means for analyzing the emotional state of a user in real time and adjusting the movement pattern based on the emotional state of the user, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a terminal. This enables rapid automatic generation and verification of accurate and efficient operation programs that take into account the emotional state of a worker.

[0445] Key word definitions

[0446] 1. "Motion capture device" means a device that records a user's movements in real time and transmits the data.

[0447] 2. "Emotional state" refers to emotions such as stress, tension, and joy recognized from the user's facial expressions, voice, and biometric signals.

[0448] 3. "Server" refers to the central processing unit that receives and analyzes motion capture data and emotional data, and generates and verifies the motion program.

[0449] 4. "Movement pattern" refers to the details of a series of movements recorded by a motion capture device, including the start and end times, joint angles, and speed.

[0450] 5. An "industrial robot" is an automated mechanical device that is programmed to perform a specific task.

[0451] 6. An "operation program" is a set of control instructions that enable an industrial robot to perform a specific operation.

[0452] 7. "Simulation environment" is a system for executing and verifying generated operating programs in a virtual space.

[0453] 8. "Terminal" means a device that receives an operating program sent from a server and stores it in a form that can be executed in a local environment.

[0454] 9. "Database" refers to an information collection system that stores robot specification data (such as degrees of freedom of movement, weight capacity, and working range).

[0455] MODE FOR CARRYING OUT THE INVENTION

[0456] Overall system overview

[0457] A system for implementing this invention comprises a motion capture device, a server, a terminal, and an emotion engine. A user wears smart glasses and performs a task, and the motion capture device records motion data in real time. This data is sent to the server, where it is analyzed. The server then uses the emotion engine to recognize the user's emotional state and adjusts the motion data based on this. Based on the adjusted motion data, the server selects an optimal industrial robot and generates an operation program. The generated operation program is verified in a simulation environment, and after any necessary modifications are made based on the results, it is provided to the terminal. Finally, the program is transferred from the terminal to the industrial robot and executed.

[0458] Hardware and software used

[0459] Hardware:

[0460] Smart glasses: Motion capture devices for recording the user's work movements (e.g., Google® Glass®, Vuzix M400).

[0461] Server: A central processing unit equipped with a high-performance GPU for analyzing motion capture data and emotion data, and for generating and verifying movement programs.

[0462] Industrial robot: An automated mechanical device for performing specific tasks (e.g. FANUC, Yaskawa).

[0463] Terminal: A device that receives an operating program sent from a server.

[0464] software:

[0465] Motion capture software: Software used to capture user movement data (e.g., Vicon Nexus).

[0466] Emotion Analysis Engine (Emotion Engine): Software for recognizing the user's emotional state (e.g., Affectiva SDK).

[0467] Motion analysis software: Software for analyzing and adjusting motion data (e.g., ROS, OpenCV).

[0468] Compiler for industrial robot control languages: Software for compiling the generated operation programs (e.g., FANUC RoboGuide, Yaskawa Motoman).

[0469] Simulation environment: A system for verifying operating programs.

[0470] Processing flow

[0471] 1. Acquiring motion capture data:

[0472] The user wears the smart glasses and performs specific tasks, while sensors built into the glasses record the movement data in real time.

[0473] 2. Sending operational data to the server:

[0474] The movement data is sent to the server, which records it in BVH or FBX format.

[0475] 3. Analysis of behavioral data:

[0476] The server analyzes the recorded data and generates keyframes based on the start time, end time, joint angle, and velocity.

[0477] 4. Recognition of emotional states:

[0478] The server uses an emotion engine to recognize the user's emotional state and analyzes the emotional data.

[0479] 5. Adjusting operational data:

[0480] The server adjusts the movement data based on the recognized emotional state to generate smooth movement patterns.

[0481] 6. Industrial robot selection:

[0482] The server selects the most suitable industrial robot based on the required degree of freedom of movement, payload capacity, and working range.

[0483] 7. Generating the operating program:

[0484] The server automatically generates an operating program for the selected robot.

[0485] 8. Verifying the operation program:

[0486] The server verifies the generated operating program in a simulation environment and makes any necessary modifications.

[0487] 9. Provision of operating programs:

[0488] The modified operating program is transferred to the terminal, which stores the program in its local environment.

[0489] 10. Running the operating program:

[0490] The terminal transfers the program to the industrial robot and causes the robot to execute the program.

[0491] Specific examples

[0492] For example, a worker wears smart glasses and performs welding work, recording his or her movements in real time using motion capture. The server uses an emotion engine to analyze the worker's emotional state (e.g., stress, tension) during the work, corrects for irregularities in movement, and generates an optimal movement pattern. A factory robot's movement program is then automatically generated based on this movement pattern, verified in a simulation environment, and implemented on the factory robot. The worker monitors the final robot movement and makes adjustments as necessary.

[0493] Prompt Sentence Examples

[0494] Prompt the generative AI model:

[0495] "Please explain in detail how to automatically generate a factory robot's motion program based on the motion and emotion data of a worker wearing smart glasses and verify it in a simulation environment."

[0496] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0497] Program processing steps

[0498] Step 1

[0499] Acquiring motion capture data

[0500] A user wears smart glasses and performs specific tasks. The motion capture device (smart glasses) records the user's movements in real time and generates data.

[0501] Input: User action.

[0502] Data processing: Convert motion data into BVH or FBX format.

[0503] Output: Recorded motion data.

[0504] Step 2

[0505] Sending operation data to the server

[0506] The user wears the smart glasses and transmits the acquired data to the server.

[0507] Input: Recorded motion data.

[0508] Data processing: Converting data into a transferable format.

[0509] Output: The operation data sent to the server.

[0510] Step 3

[0511] Analysis of behavioral data

[0512] The server analyzes the received motion data and generates key frames based on information such as start time, end time, joint angle, and speed.

[0513] Input: The motion data sent.

[0514] Data calculation: Analyzing and processing joint angles, speeds, etc.

[0515] Output: Analyzed behavior patterns.

[0516] Step 4

[0517] Recognition of emotional states

[0518] The server uses an emotion engine to recognize the user's emotional state and analyze the emotion data.

[0519] Input: User's facial, voice, and biometric data.

[0520] Data Computing: Identifying and classifying emotional states using a sentiment analysis engine.

[0521] Output: Recognized emotional state data.

[0522] Step 5

[0523] Adjusting the movement data

[0524] The server adjusts the analyzed motion data based on the recognized emotional state, for example, if the user is nervous and making awkward movements, it will correct those parts.

[0525] Input: movement patterns, emotional state data.

[0526] Data processing: Correcting movement patterns based on emotional state.

[0527] Output: Coordinated movement pattern.

[0528] Step 6

[0529] Industrial robot selection

[0530] The server refers to an internal database and selects the most suitable industrial robot based on the required degree of freedom of movement, payload capacity, and working range.

[0531] Input: Calibrated motion pattern, robot specification data in database.

[0532] Data calculation: Algorithm for selecting the best robot.

[0533] Output: Specifications of the selected industrial robot.

[0534] Step 7

[0535] Generating the operating program

[0536] The server generates a motion program for the selected robot, applying the adjusted motion data to an existing template to create a set of control instructions.

[0537] Input: Coordinated movement pattern, control template for the robot.

[0538] Data processing: Applying motion data to a template and generating a set of control instructions.

[0539] Output: The generated behavior program.

[0540] Step 8

[0541] Verification of the operating program

[0542] The server then verifies the generated motion program in a simulation environment, specifically by checking collision detection and timing to ensure there are no errors.

[0543] Input: The generated behavior program.

[0544] Data calculation: Operation verification and error detection through simulation.

[0545] Output: Verified working program, error report.

[0546] Step 9

[0547] Providing operation programs

[0548] The server prepares to provide the verified operating program to the terminal, and transfers the program file to the terminal.

[0549] Input: A verified working program.

[0550] Data processing: Converting program files into the appropriate format.

[0551] Output: The action program file sent to the terminal.

[0552] Step 10

[0553] Execution of the operating program

[0554] The terminal transfers the saved program file to the industrial robot, causing the robot to load and execute the program.

[0555] Input: The operating program file stored on the device.

[0556] Data processing: Application of motion programs to robot control systems.

[0557] Output: A running industrial robot.

[0558] Prompt the generative AI model:

[0559] "Please explain in detail how to automatically generate a factory robot's motion program based on the motion and emotion data of a worker wearing smart glasses and verify it in a simulation environment."

[0560] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0561] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0562] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0563] [Second embodiment]

[0564] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0565] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0566] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0567] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0568] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0569] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0570] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0571] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0572] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0573] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0574] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0575] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0576] This invention is a system that receives and analyzes data from a motion capture device and then automatically generates an operation program for an industrial robot. This system is composed of a motion capture device, a server, and a terminal, and operates as follows.

[0577] 1. Acquiring motion capture data

[0578] User

[0579] A user wears a motion capture device and performs a specific task, such as welding or picking. As the user performs the task, the motion is recorded by the motion capture device.

[0580] server

[0581] The server receives and stores the data sent from the motion capture device in real time. The data is recorded in a digital format (e.g., BVH, FBX).

[0582] 2. Motion analysis

[0583] server

[0584] The server reads the saved motion capture data and analyzes specific movement patterns, including the start and end times of the movement, joint angles, and speed, etc. Based on this, key frames are generated and sequential movement analysis is performed.

[0585] 3. Industrial robot selection

[0586] server

[0587] The server extracts data on the required degree of freedom of movement, payload capacity, and working range based on the analyzed movement patterns, then compares this data with an internal database to create a list of suitable industrial robots, and selects the most suitable robot from among them.

[0588] 4. Generating the operating program

[0589] server

[0590] The server loads a template corresponding to the selected robot's control language and applies the analyzed motion data to the template, thereby generating a specific set of control instructions, which are then compiled into a robot's motion program.

[0591] 5. Verification of the operating program

[0592] server

[0593] The server runs the generated motion program in a simulation environment to check its accuracy and safety. It also automatically inspects the simulation results for collision detection, timing accuracy, and other errors.

[0594] 6. Program Offering

[0595] server

[0596] In order to provide the terminal with the verified operating program, the server prepares the program file and transfers it to the terminal.

[0597] Terminal

[0598] The terminal stores the program file received from the server in local storage.

[0599] 7. Running the operation program

[0600] Terminal

[0601] The terminal transfers the saved program file to the industrial robot, loads it into the robot's control system, and then issues an instruction to the robot to execute the program.

[0602] User

[0603] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[0604] Specific examples

[0605] For example, when a user records a specific picking task, the motion capture device receives the motion as data. The server analyzes the data, selects the industrial robot best suited to the picking task, and generates a motion program for that robot. The accuracy of the program is checked in a simulation environment, and once no problems are confirmed, the program is provided to a terminal and installed on the robot. Finally, the user can monitor the robot's operations and make adjustments as necessary, completely automating the picking task.

[0606] This invention significantly reduces the time required to implement an industrial robot's operation program, enabling rapid and accurate program generation, thereby providing an effective system that can address labor shortages and rising labor costs.

[0607] The processing flow will be explained below.

[0608] Step 1: Acquiring motion capture data

[0609] User

[0610] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking). Once the task is initiated, the motion capture device records the user's movements in real time.

[0611] server

[0612] The server receives the data sent from the motion capture device and records the data in real time, which is then saved in a digital format such as BVH or FBX.

[0613] Step 2: Behavior analysis

[0614] server

[0615] The server reads the recorded motion capture data and begins analyzing the movements. Specifically, it analyzes the data frame by frame to recognize specific movement patterns. It extracts information such as the start and end times of the movements, joint angles, and speed, and generates key frames based on this information. Finally, sequential movement analysis is performed, and the movement patterns are clearly defined.

[0616] Step 3: Selecting an industrial robot

[0617] server

[0618] Based on the analyzed motion patterns, the server identifies requirements such as the required degrees of freedom of movement, payload capacity, and working range. It then references an internal database to create a list of industrial robots that match these requirements. From the list, it selects the most suitable robot and obtains its specification data.

[0619] Step 4: Generate the operating program

[0620] server

[0621] The server loads a template file corresponding to the selected robot's control language (e.g., Karel, RAPID, TP), applies the analyzed motion data to the template, and generates a specific set of control instructions. Based on the generated instruction set, it compiles a motion program for the robot.

[0622] Step 5: Verify the program

[0623] server

[0624] The server loads the generated motion program into the simulation environment. The simulation is run to check the accuracy of the motion program. Specifically, collision detection and timing checks are performed in the simulation to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[0625] Step 6: Program delivery

[0626] server

[0627] The server prepares to provide the verified operating program to the terminal, specifically by converting the program file into an appropriate format and preparing to transfer it to the terminal.

[0628] Terminal

[0629] The terminal receives the program file sent from the server and stores it in local storage.

[0630] Step 7: Run the action program

[0631] Terminal

[0632] The terminal transfers the saved program file to the industrial robot, which then loads the program into the robot's control system and instructs it to run.

[0633] User

[0634] Users can monitor the robot's operations to ensure they are working correctly, and can make fine adjustments or additional program modifications as needed, allowing specific tasks to be accurately replicated and automated.

[0635] Example 1

[0636] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0637] Creating operational programs for conventional industrial equipment required a significant amount of time and specialized knowledge, which could result in delays in the automation process for certain tasks. Furthermore, there was a lack of means to verify the accuracy and safety of operational programs in advance, which created the risk of malfunctions during execution. This made it difficult to address labor shortages and rising labor costs.

[0638] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0639] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific motion patterns, means for selecting the optimum industrial equipment based on the analyzed motion patterns, means for automatically generating an operation program corresponding to the selected equipment, means for verifying the generated operation program in a simulation environment before execution, means for providing the verified operation program to an information terminal, means for the information terminal to transfer the provided operation program to a control target and issue an execution instruction, and means for monitoring the operation of the control target and making adjustments as necessary. This makes it possible to quickly and accurately generate, verify, and provide operation programs for industrial equipment, thereby facilitating the automation of work.

[0640] A "motion capture device" is a device that records the movements of a human body or object using sensors or cameras and converts that data into a digital format.

[0641] A "server" is a device or software system that receives data from a motion capture device, analyzes, stores, generates and verifies motion programs.

[0642] "Digital format" refers to a file format such as BVH or FBX that is used to digitize and record movement data.

[0643] "Movement pattern" refers to the characteristics and route of a series of movements recognized based on motion capture data.

[0644] "Industrial equipment" is a general term for robots and other automation devices used in production and manufacturing processes.

[0645] An "operation program" refers to a set of instructions for controlling the operation of industrial equipment, and is written in a specific control language.

[0646] A "simulation environment" is a virtual space or software environment in which a generated operating program is executed and its accuracy and safety are virtually confirmed.

[0647] An "information terminal" is a device or system used to receive, store, display, and control data from a server or other device.

[0648] The term "controlled object" refers to industrial equipment that is controlled using an operating program.

[0649] The "internal database" is a database system that stores specification data and operation history data for industrial equipment.

[0650] A "key frame" refers to a frame that is an important point or turning point in a movement pattern, and detailed movements are analyzed based on this frame.

[0651] "Weight capacity" refers to the maximum weight that industrial equipment can lift and carry.

[0652] "Degrees of freedom of movement" is an indicator that shows the direction and range of movement of each joint and axis of industrial equipment and robots.

[0653] A "prompt" refers to an instruction or question that is input into a generative AI model, and functions as an input to obtain a specific output.

[0654] This invention is a system that records the movements of people and objects as data, analyzes the data, and automatically generates operating programs suitable for industrial equipment. This system consists of a motion capture device, a server, and a terminal. The detailed configuration and operation of this system are described below.

[0655] Hardware and Software Configuration

[0656] motion capture equipment

[0657] A motion capture device records a user's movements in real time and transmits the data to a server. A typical example of such a device is a motion capture suit equipped with cameras and sensors. The data is recorded in digital formats such as BVH and FBX.

[0658] server

[0659] The server receives, stores, analyzes, generates, and verifies the data sent from the motion capture device. The server has the following functions:

[0660] Data Reception and Storage: Motion capture data is received in real time and stored in a digital format.

[0661] Movement Analysis: Analyzes stored data and recognizes specific movement patterns, including identifying joint angles, movement speed, and start and end times.

[0662] Industrial equipment selection: Based on the analyzed movement patterns, the system selects equipment suitable for the required degree of freedom of movement, payload capacity, and working range from an internal database.

[0663] Generation of operation program: A specific set of control instructions is generated using a template corresponding to the control language of the selected device.

[0664] Verification in a simulation environment: The generated behavioral program is executed in a simulation environment to verify the correctness and safety of the program.

[0665] Terminal

[0666] The terminal receives the operation program sent from the server and transfers it to the industrial equipment. The terminal has the following functions:

[0667] Receiving and saving programs: Saves program files provided by the server in local storage.

[0668] Linking with industrial equipment: Transfers saved program files to industrial equipment and loads them into the control system. Executes the program according to user instructions.

[0669] Specific examples

[0670] For example, when a user records a specific picking task, the motion capture device receives the motion as data. The server analyzes the data and selects the industrial equipment best suited to the picking task. It then generates a motion program for that equipment and checks the accuracy of the program in a simulation environment. After verifying that there are no problems, the program is provided to a terminal and ultimately implemented in the robot. The user can monitor the robot's operations and make adjustments as necessary to fully automate the picking task.

[0671] Prompt Sentence Examples

[0672] "Please explain the detailed process of a system in which a user wears a motion capture device and performs a picking task, receives the motion data, analyzes the data, selects the most suitable industrial equipment, and generates and provides an operating program for the equipment. We would like a detailed explanation of each step of this system, including the specific processing content and algorithms used."

[0673] This invention enables the rapid and accurate generation and verification of operating programs for industrial equipment, facilitating the automation of work, thereby providing an efficient system that can cope with labor shortages and rising labor costs.

[0674] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0675] Step 1:

[0676] Acquiring motion capture data

[0677] The user wears a motion capture device and performs specific task movements. The device records the user's movements in real time and transmits the data to a server in a digital format (e.g., BVH, FBX).

[0678] Specifically, when a user performs picking work, the series of actions is captured by sensors and cameras.

[0679] Input: User action (e.g. picking work)

[0680] Output: Motion capture data (e.g. BVH, FBX)

[0681] Step 2:

[0682] Receiving and storing data

[0683] The server receives and stores the data transmitted from the motion capture device in real time, and the data is recorded in digital format.

[0684] Input: Motion capture data

[0685] Output: Saved motion capture data

[0686] Specifically, the server saves the data to disk in the specified format (e.g., BVH, FBX).

[0687] Step 3:

[0688] Motion analysis

[0689] The server reads the saved motion capture data and analyzes specific movement patterns, extracting the start and end times of the movement, as well as the joint angles and speeds, and generates keyframes.

[0690] Input: Stored motion capture data

[0691] Output: Analyzed movement patterns and keyframes

[0692] The server uses motion analysis algorithms to extract key motion points, for example, identifying specific motion patterns based on joint angles and movement speed.

[0693] Step 4:

[0694] Industrial Equipment Selection

[0695] Based on the analyzed motion patterns, the server selects from its internal database the industrial equipment that is suitable for the required degree of freedom of movement, payload capacity, and working range.

[0696] Input: Analyzed movement patterns and internal database

[0697] Output: List of selected industrial devices

[0698] Specifically, the server lists devices that meet pre-set filtering conditions and ranks the most suitable devices among them.

[0699] Step 5:

[0700] Generating the operating program

[0701] The server loads a template corresponding to the control language (e.g., URScript, KRL) of the selected industrial equipment and generates an operating program by applying the analyzed operating data to the template.

[0702] Input: Selected industrial equipment, analyzed operation patterns, control language template

[0703] Output: Generated behavior program

[0704] Specifically, the server integrates an instruction set based on an operation pattern into a template, and compiles it to generate a complete program.

[0705] Step 6:

[0706] Verification of the operating program

[0707] The server runs the generated motion program in a simulation environment to verify the accuracy and safety of the motion, which also includes collision detection and timing adjustment.

[0708] Input: Generated behavior program

[0709] Output: Verification results (verification of accuracy and safety)

[0710] Specifically, the server runs an operating program in a virtual environment and checks for any errors or malfunctions.

[0711] Step 7:

[0712] Program Offering

[0713] The server prepares a program file and transfers it to the terminal in order to provide the terminal with the verified operating program.

[0714] Input: Verified working program

[0715] Output: Transferred program file

[0716] Specifically, the server compresses the program file, encrypts it if necessary, and transmits it to the terminal.

[0717] Step 8:

[0718] Execution of the operating program

[0719] The terminal transfers the stored program file to the industrial equipment, loads it into the control system, and then issues an instruction to run the program.

[0720] Input: Transferred program file

[0721] Output: The device behavior caused by the executed program

[0722] Specifically, the terminal uses the control interface of the industrial equipment to load the program and send an execution instruction.

[0723] Step 9:

[0724] Monitor and adjust behavior

[0725] Users monitor the operation of industrial equipment and make fine adjustments or additional program modifications as needed.

[0726] Input: Running industrial equipment operations

[0727] Output: Adjusted behavior, modified program

[0728] Specifically, the user observes the operation in real time and manually corrects or reprograms any malfunctions.

[0729] (Application example 1)

[0730] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0731] In modern factory automation, many procedures rely on manual labor and robot operation programming. With conventional systems, it takes time to generate robot operation programs, and fine-tuning the programs is not easy. As a result, production efficiency can decrease and production can be halted due to errors. There is a need for a system that can solve this problem and enable rapid generation of operation programs and adjustments in real time.

[0732] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0733] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific movement patterns, means for selecting an optimum industrial robot based on the analyzed movement patterns, means for automatically generating an operation program corresponding to the selected robot, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a mobile communication terminal so that a worker can fine-tune it in real time. This enables rapid and accurate generation of an operation program and adjustment in real time.

[0734] Key Word Definitions

[0735] ---

[0736] A "motion capture device" is a device that records the movements of people or objects as digital data.

[0737] The "means for receiving and recording data" is a system that has the function of receiving data transmitted from the motion capture device and storing it.

[0738] "Means for analyzing recorded motion capture data and recognizing specific movement patterns" refers to a system that has the function of analyzing stored data and identifying work movements and types of movements.

[0739] The "means for selecting the optimal industrial robot" is a system that has the function of selecting the robot that can perform the most appropriate operation based on the analyzed operation pattern.

[0740] The "means for automatically generating an operating program" is a system that has the function of automatically creating a program corresponding to the selected robot.

[0741] "Means for verifying in a simulation environment" refers to a system that has the function of testing in a virtual environment whether the generated program is accurate.

[0742] A "mobile communication terminal" is a portable communication device such as a smartphone or tablet.

[0743] "Means that allow workers to make fine adjustments in real time" refers to a system that has the function of immediately making necessary changes and fine adjustments to a verified program.

[0744] A "database" is a system that stores robot specification data and other related information and manages it so that necessary data can be retrieved quickly.

[0745] "Movement freedom" is a concept that refers to the number and range of movements that a robot can perform.

[0746] "Weight capacity" refers to the maximum weight that an industrial robot can safely lift and carry.

[0747] "Work envelope" refers to the physical area within which an industrial robot can move.

[0748] "Means for verifying accuracy and correcting the program if problems are found" refers to a system that has the ability to verify whether the generated program operates correctly and make appropriate corrections when errors or problems are found.

[0749] MODE FOR CARRYING OUT THE INVENTION

[0750] ---

[0751] This invention relates to a system that automatically creates an operating program for an industrial robot by receiving and analyzing motion capture data. This system is comprised of a motion capture device, a server, and a mobile communication terminal, and operates as follows.

[0752] 1. Acquiring motion capture data

[0753] User

[0754] A user wears a motion capture device and performs specific task movements, which are recorded as digital data by the motion capture device.

[0755] server

[0756] The server receives and stores the data sent from the motion capture equipment in real time, and stores this data in a digital format (e.g., BVH, FBX).

[0757] 2. Motion analysis

[0758] server

[0759] The server reads the stored motion capture data and analyzes specific movement patterns, including the start and end times of the movement, joint angles and speeds, etc. Keyframes are generated based on the analyzed data, and sequential movement analysis is performed.

[0760] 3. Industrial robot selection

[0761] server

[0762] Based on the analysis results, the server compares the robot specifications retrieved from the database to select the optimal industrial robot, including the required degrees of freedom of movement, payload capacity, and working range.

[0763] 4. Generating the operating program

[0764] server

[0765] The server loads a template corresponding to the selected robot's control language and applies the analyzed motion data, thereby generating a specific set of control instructions and compiling them into a robot's motion program.

[0766] 5. Verification of the operating program

[0767] server

[0768] The server runs the generated program in a simulation environment to verify its accuracy and safety. Based on the results of the simulation, the program is automatically modified if necessary.

[0769] 6. Program provision and real-time adjustment

[0770] server

[0771] In order to provide the verified operating program to the mobile communication terminal, the server prepares a program file and transfers it to the mobile communication terminal.

[0772] Mobile communication terminal

[0773] The mobile communication terminal stores the program files received from the server and provides an interface that allows the user to fine-tune the operation in real time.

[0774] User

[0775] The user monitors the robot's movements using a mobile communication terminal and fine-tunes the movement program in real time to finalize it. Additional program modifications are also possible as needed.

[0776] Hardware and software used

[0777] Hardware: Mobile communication devices (smartphones and tablets), motion capture devices (e.g., motion sensors), industrial robots

[0778] Software: Server-side technologies (e.g., Django, Flask), simulation environments (e.g., ROS, Gazebo), data parsing libraries (e.g., Python requests, json)

[0779] Specific examples

[0780] Factory workers operate smartphones and use motion capture devices to record specific picking movements. The recorded data is sent to a server, where it is analyzed and the server selects the robot best suited to the picking movement. A motion program for that robot is automatically generated and verified through simulation. After verification, the generated program is sent to a mobile communication device (e.g., a smartphone), where the worker can fine-tune the movement in real time.

[0781] Example prompts to input to the generative AI model

[0782] "We are seeking advice on the design and implementation of a system that analyzes motion capture data and automatically generates motion programs for industrial robots."

[0783] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0784] Program processing flow

[0785] ---

[0786] Step 1:

[0787] A user wears a motion capture device and performs a specific task. The input data here is the user's motion information, and the output is the motion data recorded by the motion capture device.

[0788] Specific behavior:

[0789] The user performs picking operations and welding parts, and the series of movements is recorded as digital data by a motion capture device.

[0790] Step 2:

[0791] The recorded motion capture data is sent to the server in real time. The input data is the digital format data (e.g. BVH, FBX) sent from the motion capture device, and the output is the motion capture data stored on the server.

[0792] Specific behavior:

[0793] The data collected by the motion capture device is sent via wireless or wired communication to a server, which stores it in a database.

[0794] Step 3:

[0795] The server analyzes the stored motion capture data and recognizes specific movement patterns. The input data is the motion capture data stored on the server, and the output is the analyzed movement pattern and related parameters (start time, end time of movement, joint angles, etc.).

[0796] Specific behavior:

[0797] The server uses data analysis algorithms (e.g., machine learning models) to extract specific actions or keyframes within the data.

[0798] Step 4:

[0799] Based on the analysis results, the server selects the optimal industrial robot. The input data is the analyzed motion pattern and the corresponding robot specification data (degree of freedom of motion, payload capacity, working range, etc.), and the output is the selected robot.

[0800] Specific behavior:

[0801] The server checks its internal database, lists the robots that best meet the required specifications, and selects the most suitable robot.

[0802] Step 5:

[0803] The server automatically generates an operating program for the selected robot. The input data is a template based on the control language of the selected robot and analyzed operation patterns, and the output is an operating program containing a specific set of control instructions.

[0804] Specific behavior:

[0805] The server uses a program generation algorithm to generate a control instruction set by applying an operation pattern to a template.

[0806] Step 6:

[0807] The generated behavioral program is verified in a simulation environment. The input data is the generated behavioral program, and the output is the simulation results and the verified program.

[0808] Specific behavior:

[0809] The server runs the generated program using a robot simulation environment (e.g., ROS, Gazebo) to check its safety and accuracy, and automatically corrects the program if necessary.

[0810] Step 7:

[0811] The verified operating program is transferred to the mobile communication terminal, the input data is the verified program file, and the output is the program file stored in the mobile communication terminal.

[0812] Specific behavior:

[0813] The server transmits the program file to the mobile communication terminal, which stores the file in its local memory.

[0814] Step 8:

[0815] The user fine-tunes the robot's movements in real time using a mobile communication terminal. The input data is a verified program file and the user's instructions, and the output is the final modified movement program.

[0816] Specific behavior:

[0817] Workers can fine-tune the operation program via a smartphone or tablet interface to determine the optimal operation program.

[0818] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0819] This invention is a system that combines a system that receives and analyzes data from a motion capture device and automatically generates an operating program for an industrial robot with an emotion engine that recognizes the user's emotions. This system is composed of a motion capture device, a server, a terminal, and an emotion engine, and operates as follows.

[0820] 1. Acquiring motion capture data

[0821] User

[0822] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking). Once the task is initiated, the motion capture device records the user's movements in real time.

[0823] server

[0824] The server receives and records the data sent from the motion capture device in real time, and saves it in formats such as BVH or FBX.

[0825] 2. Motion analysis

[0826] server

[0827] The server loads the saved motion capture data and begins analysis, including the start and end times of the motion, joint angles, and velocity. Based on this, key frames are generated and sequential motion analysis is performed.

[0828] 3. Emotion Recognition by Emotion Engine

[0829] server

[0830] The server utilizes an emotion engine to recognize the user's emotional state. The emotion engine detects the user's facial expressions, voice, and biometric signals, and identifies the user's emotions based on these data. For example, if the user is feeling stressed, the emotional state is recorded.

[0831] 4. Adjusting the movement data

[0832] server

[0833] The server adjusts the analyzed movement patterns based on the emotional state recognized by the emotion engine. For example, if the user is nervous and their movements are awkward, the server corrects those parts to generate a smooth movement pattern.

[0834] 5. Industrial Robot Selection

[0835] server

[0836] The server determines the required degrees of freedom of movement, payload capacity, and working range based on the adjusted motion pattern. It then references an internal database to create a list of industrial robots that meet these requirements. It then selects the most suitable robot from the list and obtains its specification data.

[0837] 6. Generating the operating program

[0838] server

[0839] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, and generates a specific set of control instructions, which are then compiled into the robot's motion program.

[0840] 7. Verification of the operating program

[0841] server

[0842] The server loads the generated motion program into the simulation environment. The simulation is run to verify the accuracy of the motion program. Specifically, collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[0843] 8. Program Offering

[0844] server

[0845] The server prepares to provide the verified operating program to the terminal, converts the program file into an appropriate format, and prepares to transfer it to the terminal.

[0846] Terminal

[0847] The terminal receives the program file sent from the server and stores it in local storage.

[0848] 9. Running the Operation Program

[0849] Terminal

[0850] The terminal transfers the saved program file to the industrial robot, loads the program into the robot's control system, and then issues an instruction to the robot to execute the program.

[0851] User

[0852] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[0853] Specific examples

[0854] For example, when a user records a specific welding task, the motion capture device receives the motion as data. The server analyzes the data and uses an emotion engine to identify areas where the user is feeling stressed. Next, it corrects any motion irregularities caused by stress, selects the most suitable industrial robot based on the adjusted motion data, and generates a motion program for that robot. After verifying the accuracy of the program in a simulation environment, it is provided to the terminal and finally implemented in the robot. The user monitors the robot's motion and makes adjustments as necessary, automating the welding task.

[0855] This invention significantly shortens the implementation time for industrial robot operation programs, enabling rapid and accurate program generation. Furthermore, by taking the user's emotional state into consideration, it is possible to provide more accurate operation programs. This system can provide an effective means of addressing labor shortages and rising labor costs.

[0856] The processing flow will be explained below.

[0857] Step 1: Acquiring motion capture data

[0858] User

[0859] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking), which records the user's movements in real time.

[0860] server

[0861] The server receives and records the data sent from the motion capture equipment in real time, and the data is saved in digital formats such as BVH and FBX.

[0862] Step 2: Behavior analysis

[0863] server

[0864] The server reads the stored motion capture data and analyzes it frame by frame, extracting the start and end times of the movements, joint angles, speed, etc., and generates keyframes based on this information. Specific movement patterns are identified through the analysis.

[0865] Step 3: Emotion recognition by the emotion engine

[0866] server

[0867] The server utilizes an emotion engine to recognize the user's emotional state. The emotion engine detects the user's facial expressions, voice, and biometric signals, and identifies the user's emotions based on these data. For example, if the user is feeling stressed, the emotional state is recorded.

[0868] Step 4: Adjusting the behavior data

[0869] server

[0870] The server adjusts the analyzed movement patterns based on the recognized emotional state: if the user is nervous and their movements are awkward, it corrects those parts and converts them into smoother movement patterns.

[0871] Step 5: Selecting an industrial robot

[0872] server

[0873] The server determines the required degree of freedom of movement, payload capacity, and working range based on the adjusted motion pattern, and then references an internal database to create a list of industrial robots that match these requirements and select the most suitable robot.

[0874] Step 6: Generate the operating program

[0875] server

[0876] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, generates a specific set of control instructions, and compiles them into a robot motion program.

[0877] Step 7: Verify the program

[0878] server

[0879] The server loads the generated motion program into the simulation environment. The simulation is run to check the accuracy and safety of the motion program. Collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[0880] Step 8: Program Delivery

[0881] server

[0882] The server prepares to provide the verified operating program to the terminal, converts the program file into an appropriate format, and prepares to transfer it to the terminal.

[0883] Terminal

[0884] The terminal receives the program file sent from the server and stores it in local storage.

[0885] Step 9: Run the action program

[0886] Terminal

[0887] The terminal transfers the saved program file to the industrial robot, which then loads the program into the robot's control system and issues an instruction to run it.

[0888] User

[0889] The user monitors the robot's operation to ensure it is working correctly. If necessary, they can make fine adjustments or additional program modifications. Once proper operation is confirmed, the robot will automate the task.

[0890] Specific examples

[0891] For example, when a user records a specific picking task, the motion capture device receives the movement as data. The server analyzes the data and uses an emotion engine to determine whether the user is feeling stressed. The server then corrects any movement irregularities caused by stress and selects the optimal industrial robot based on the adjusted movement data. A movement program appropriate for the selected robot is generated, its accuracy confirmed through simulation, and then provided to the terminal and installed on the robot. The user monitors the robot's movements and makes adjustments as necessary, allowing the picking task to be accurately automated.

[0892] Example 2

[0893] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0894] Conventional industrial robot motion program generation systems have the risk of motion accuracy decreasing due to the user's emotional state, such as stress or tension. Furthermore, generating and verifying motion programs takes time, resulting in inefficiency. This has led to problems with automation in industrial settings and productivity not improving.

[0895] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0896] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific movement patterns, means for utilizing an emotion engine for recognizing the emotional state of a user, means for adjusting the movement patterns based on the recognized emotional state, means for selecting an optimum industrial robot based on the adjusted movement patterns, means for automatically generating an operation program corresponding to the selected robot, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a terminal. This enables the generation and verification of an operation program quickly and accurately while taking the emotional state of a user into consideration.

[0897] A "motion capture device" is a device that records a user's body movements in real time and transmits the data to a server.

[0898] The "means for receiving and recording data" is a component of the server that has the function of receiving data transmitted from the motion capture device and recording that data.

[0899] The "means for analyzing motion capture data" is a server component that has the function of analyzing the start time, end time, joint angles, speed, etc. of a movement based on recorded motion capture data, and recognizing a specific movement pattern.

[0900] An "emotion engine" is software or hardware that analyzes a user's facial expressions, voice, and biometric signals to identify the user's emotional state.

[0901] The "means for adjusting the movement pattern" is a component of the server that has the function of correcting unnatural parts of the analyzed movement pattern and adjusting it to a smooth movement pattern based on the recognized emotional state of the user.

[0902] An "industrial robot" is an automated device used in industrial settings that requires an operating program to perform a specific task (e.g., welding, assembly, picking).

[0903] The "means for automatically generating an operation program" is a component of the server that has the function of automatically generating an operation program in a control language corresponding to the selected robot based on the adjusted operation data.

[0904] A "simulation environment" is a computer-based system for executing generated operating programs in a virtual space and verifying their accuracy and errors.

[0905] The "means for verifying the operation program" is a component of the server that has the function of checking the accuracy of the operation program generated using the simulation environment and making any necessary corrections.

[0906] A "terminal" is an electronic device that receives an operating program provided by a server, stores it in local storage, and transfers it to an industrial robot for execution.

[0907] This invention is a system that combines a system that receives and analyzes data from a motion capture device and automatically generates an operating program for an industrial robot with an emotion engine that recognizes the user's emotions. This system is composed of a motion capture device, a server, a terminal, and an emotion engine, and operates as follows.

[0908] 1. Generate a program for this system. Specifically, create a program to receive and record data sent from the motion capture device in real time. This data is saved in formats such as BVH or FBX. Next, create a program to analyze the saved motion capture data and extract information such as the start and end times of the movement, joint angles, and speed. This also includes a program that uses an emotion engine to recognize the user's emotional state from facial expressions, voice, and biometric signals. These programs will use a generative AI model to adjust movement patterns and automatically generate optimal movement programs for industrial robots.

[0909] 2. The processing of the generated program is explained in natural language. Specifically, a server is used to receive and record data from the motion capture device. The server saves the recorded data in formats such as BVH or FBX and analyzes the movements. It uses an emotion engine to recognize the user's emotional state and adjusts the movement pattern as needed. Based on the adjusted movement data, the server selects the optimal industrial robot and automatically generates a movement program for that robot. The generated program is verified in a simulation environment to confirm there are no problems before being provided to the terminal. The terminal transfers the provided program to the industrial robot and executes it. The user can monitor the robot's movement and make fine adjustments or additional program modifications as needed.

[0910] 3. Specific examples are added to sentences that explain the program's processing. For example, a user wears a motion capture device and performs a specific welding task. The server receives and records this data in real time in BVH format. The server then analyzes the data, calculating the start and end times of the motion, joint angles, speed, and other parameters, and generates keyframes. The server then uses an emotion engine to analyze the user's facial expressions, voice, and biometric signals to identify stress levels. Based on the stress levels identified by the emotion engine, the server adjusts the analyzed movement patterns. For example, it modifies the user's tense movements to smoother movements. Based on the adjusted movement patterns, the server then identifies the required degrees of freedom of movement, payload capacity, and working range, and selects the optimal industrial robot from a database. The server generates a movement program using a template file corresponding to the selected robot's control language (e.g., Karel). The generated movement program is then verified in a simulation environment to check timing and detect collisions. If verification is successful, the program is transferred to the terminal, which then loads it onto the industrial robot. The user then monitors the robot's movements and makes adjustments as necessary, automating the welding process.

[0911] Specific prompt examples:

[0912] "Please explain in detail, step by step, the procedure for generating an operation program for an industrial robot that modifies its operation pattern based on the stress felt by the user during welding work."

[0913] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0914] Step 1:

[0915] Acquiring motion capture data

[0916] User

[0917] The user wears a motion capture device and performs a specific work motion (e.g., welding). Once the motion is initiated, the motion capture device records the user's movements in real time.

[0918] Input: User's physical movements

[0919] Output: Real-time motion capture data

[0920] Step 2:

[0921] Receiving and recording data

[0922] server

[0923] The server receives the data sent from the motion capture device in real time and records it in BVH or FBX format.

[0924] Input: Motion capture data

[0925] Output: BVH or FBX format files

[0926] Step 3:

[0927] Motion analysis

[0928] server

[0929] The server reads the recorded motion capture data and analyzes the start and end times of the movements, joint angles, speed, etc. Based on this, key frames are generated and detailed movement analysis is performed.

[0930] Input: BVH or FBX format files

[0931] Output: Analysis results (start time, end time, joint angles, speed, keyframes)

[0932] Step 4:

[0933] Emotion recognition by emotion engine

[0934] server

[0935] The server uses an emotion engine to analyze the user's facial expressions, voice, and biometric signals to identify the user's emotional state. If the user is feeling stressed or tense, the server records that emotional state.

[0936] Input: User's facial expressions, voice, and biometric signals

[0937] Output: Emotional state (e.g., stress, tension)

[0938] Step 5:

[0939] Adjusting the movement data

[0940] server

[0941] The server adjusts the analyzed movement patterns based on the recognized emotional state. For example, if the user is nervous and their movements are awkward, it will correct those parts to make them smoother.

[0942] Input: Analysis results, emotional state

[0943] Output: Coordinated movement pattern

[0944] Step 6:

[0945] Industrial robot selection

[0946] server

[0947] The server determines the required degree of freedom of movement, payload capacity, and working range based on the adjusted motion pattern, and then references an internal database to create a list of industrial robots that match these requirements and select the most suitable robot.

[0948] Input: Adjusted movement pattern

[0949] Output: Selected industrial robot

[0950] Step 7:

[0951] Generating the operating program

[0952] server

[0953] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, and generates a specific set of control instructions, which are then compiled into the robot's motion program.

[0954] Input: Selected industrial robot, adjusted motion pattern

[0955] Output: Working program

[0956] Step 8:

[0957] Verification of the operating program

[0958] server

[0959] The server loads the generated motion program into the simulation environment. The simulation is run to verify the accuracy of the motion program. Specifically, collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[0960] Input: Action program

[0961] Output: Verification results (correctness, presence of errors)

[0962] Step 9:

[0963] Program Offering

[0964] server

[0965] The server provides the verified operating program to the terminal, converts the program file into a format compatible with the robot control system, and prepares it for transfer to the terminal.

[0966] Input: Verified working program

[0967] Output: Program file for distribution

[0968] Terminal

[0969] The terminal receives the program file sent from the server and stores it in local storage.

[0970] Input: Program file to be provided

[0971] Output: Program files in local storage

[0972] Step 10:

[0973] Execution of the operating program

[0974] Terminal

[0975] The terminal transfers the program file stored in the local storage to the industrial robot, loads the program into the robot's control system, and then issues an instruction to the robot to execute the program.

[0976] Input: Program files in local storage

[0977] Output: Running program

[0978] User

[0979] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[0980] Input: Running action program

[0981] Output: Monitoring results and fine-tuning

[0982] (Application example 2)

[0983] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0984] The rapid generation of efficient and accurate motion programs for industrial robots is a high requirement in many factories. However, conventional systems require a great deal of time and effort to generate and verify motion programs. Furthermore, because they do not take into account the impact of the worker's emotional state on motion, the motions of unskilled or fatigued workers are directly reflected in the program, resulting in the generation of inefficient motion programs. Furthermore, modifying motion programs is complex, making it difficult to generate programs optimized for specific work conditions.

[0985] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing a specific movement pattern, means for selecting an optimal industrial robot based on the analyzed movement pattern, means for automatically generating an operation program corresponding to the selected robot, means for analyzing the emotional state of a user in real time and adjusting the movement pattern based on the emotional state of the user, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a terminal. This enables rapid automatic generation and verification of accurate and efficient operation programs that take into account the emotional state of a worker.

[0986] Key word definitions

[0987] 1. "Motion capture device" means a device that records a user's movements in real time and transmits the data.

[0988] 2. "Emotional state" refers to emotions such as stress, tension, and joy recognized from the user's facial expressions, voice, and biometric signals.

[0989] 3. "Server" refers to the central processing unit that receives and analyzes motion capture data and emotional data, and generates and verifies the motion program.

[0990] 4. "Movement pattern" refers to the details of a series of movements recorded by a motion capture device, including the start and end times, joint angles, and speed.

[0991] 5. An "industrial robot" is an automated mechanical device that is programmed to perform a specific task.

[0992] 6. An "operation program" is a set of control instructions that enable an industrial robot to perform a specific operation.

[0993] 7. "Simulation environment" is a system for executing and verifying generated operating programs in a virtual space.

[0994] 8. "Terminal" means a device that receives an operating program sent from a server and stores it in a form that can be executed in a local environment.

[0995] 9. "Database" refers to an information collection system that stores robot specification data (such as degrees of freedom of movement, weight capacity, and working range).

[0996] MODE FOR CARRYING OUT THE INVENTION

[0997] Overall system overview

[0998] A system for implementing this invention comprises a motion capture device, a server, a terminal, and an emotion engine. A user wears smart glasses and performs a task, and the motion capture device records motion data in real time. This data is sent to the server, where it is analyzed. The server then uses the emotion engine to recognize the user's emotional state and adjusts the motion data based on this. Based on the adjusted motion data, the server selects an optimal industrial robot and generates an operation program. The generated operation program is verified in a simulation environment, and after any necessary modifications are made based on the results, it is provided to the terminal. Finally, the program is transferred from the terminal to the industrial robot and executed.

[0999] Hardware and software used

[1000] Hardware:

[1001] Smart glasses: Motion capture devices for recording the user's work movements (e.g., Google Glass, Vuzix M400).

[1002] Server: A central processing unit equipped with a high-performance GPU for analyzing motion capture data and emotion data, and for generating and verifying movement programs.

[1003] Industrial robot: An automated mechanical device for performing specific tasks (e.g. FANUC, Yaskawa).

[1004] Terminal: A device that receives an operating program sent from a server.

[1005] software:

[1006] Motion capture software: Software used to capture user movement data (e.g., Vicon Nexus).

[1007] Emotion Analysis Engine (Emotion Engine): Software for recognizing the user's emotional state (e.g., Affectiva SDK).

[1008] Motion analysis software: Software for analyzing and adjusting motion data (e.g., ROS, OpenCV).

[1009] Compiler for industrial robot control languages: Software for compiling the generated operation programs (e.g., FANUC RoboGuide, Yaskawa Motoman).

[1010] Simulation environment: A system for verifying operating programs.

[1011] Processing flow

[1012] 1. Acquiring motion capture data:

[1013] The user wears the smart glasses and performs specific tasks, while sensors built into the glasses record the movement data in real time.

[1014] 2. Sending operational data to the server:

[1015] The movement data is sent to the server, which records it in BVH or FBX format.

[1016] 3. Analysis of behavioral data:

[1017] The server analyzes the recorded data and generates keyframes based on the start time, end time, joint angle, and velocity.

[1018] 4. Recognition of emotional states:

[1019] The server uses an emotion engine to recognize the user's emotional state and analyzes the emotional data.

[1020] 5. Adjusting operational data:

[1021] The server adjusts the movement data based on the recognized emotional state to generate smooth movement patterns.

[1022] 6. Industrial robot selection:

[1023] The server selects the most suitable industrial robot based on the required degree of freedom of movement, payload capacity, and working range.

[1024] 7. Generating the operating program:

[1025] The server automatically generates an operating program for the selected robot.

[1026] 8. Verifying the operation program:

[1027] The server verifies the generated operating program in a simulation environment and makes any necessary modifications.

[1028] 9. Provision of operating programs:

[1029] The modified operating program is transferred to the terminal, which stores the program in its local environment.

[1030] 10. Running the operating program:

[1031] The terminal transfers the program to the industrial robot and causes the robot to execute the program.

[1032] Specific examples

[1033] For example, a worker wears smart glasses and performs welding work, recording his or her movements in real time using motion capture. The server uses an emotion engine to analyze the worker's emotional state (e.g., stress, tension) during the work, corrects for irregularities in movement, and generates an optimal movement pattern. A factory robot's movement program is then automatically generated based on this movement pattern, verified in a simulation environment, and implemented on the factory robot. The worker monitors the final robot movement and makes adjustments as necessary.

[1034] Prompt Sentence Examples

[1035] Prompt the generative AI model:

[1036] "Please explain in detail how to automatically generate a factory robot's motion program based on the motion and emotion data of a worker wearing smart glasses and verify it in a simulation environment."

[1037] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1038] Program processing steps

[1039] Step 1

[1040] Acquiring motion capture data

[1041] A user wears smart glasses and performs specific tasks. The motion capture device (smart glasses) records the user's movements in real time and generates data.

[1042] Input: User action.

[1043] Data processing: Convert motion data into BVH or FBX format.

[1044] Output: Recorded motion data.

[1045] Step 2

[1046] Sending operation data to the server

[1047] The user wears the smart glasses and transmits the acquired data to the server.

[1048] Input: Recorded motion data.

[1049] Data processing: Converting data into a transferable format.

[1050] Output: The operation data sent to the server.

[1051] Step 3

[1052] Analysis of behavioral data

[1053] The server analyzes the received motion data and generates key frames based on information such as start time, end time, joint angle, and speed.

[1054] Input: The motion data sent.

[1055] Data calculation: Analyzing and processing joint angles, speeds, etc.

[1056] Output: Analyzed behavior patterns.

[1057] Step 4

[1058] Recognition of emotional states

[1059] The server uses an emotion engine to recognize the user's emotional state and analyze the emotion data.

[1060] Input: User's facial, voice, and biometric data.

[1061] Data Computing: Identifying and classifying emotional states using a sentiment analysis engine.

[1062] Output: Recognized emotional state data.

[1063] Step 5

[1064] Adjusting the movement data

[1065] The server adjusts the analyzed motion data based on the recognized emotional state, for example, if the user is nervous and making awkward movements, it will correct those parts.

[1066] Input: movement patterns, emotional state data.

[1067] Data processing: Correcting movement patterns based on emotional state.

[1068] Output: Coordinated movement pattern.

[1069] Step 6

[1070] Industrial robot selection

[1071] The server refers to an internal database and selects the most suitable industrial robot based on the required degree of freedom of movement, payload capacity, and working range.

[1072] Input: Calibrated motion pattern, robot specification data in database.

[1073] Data calculation: Algorithm for selecting the best robot.

[1074] Output: Specifications of the selected industrial robot.

[1075] Step 7

[1076] Generating the operating program

[1077] The server generates a motion program for the selected robot, applying the adjusted motion data to an existing template to create a set of control instructions.

[1078] Input: Coordinated movement pattern, control template for the robot.

[1079] Data processing: Applying motion data to a template and generating a set of control instructions.

[1080] Output: The generated behavior program.

[1081] Step 8

[1082] Verification of the operating program

[1083] The server then verifies the generated motion program in a simulation environment, specifically by checking collision detection and timing to ensure there are no errors.

[1084] Input: The generated behavior program.

[1085] Data calculation: Operation verification and error detection through simulation.

[1086] Output: Verified working program, error report.

[1087] Step 9

[1088] Providing operation programs

[1089] The server prepares to provide the verified operating program to the terminal, and transfers the program file to the terminal.

[1090] Input: A verified working program.

[1091] Data processing: Converting program files into the appropriate format.

[1092] Output: The action program file sent to the terminal.

[1093] Step 10

[1094] Execution of the operating program

[1095] The terminal transfers the saved program file to the industrial robot, causing the robot to load and execute the program.

[1096] Input: The operating program file stored on the device.

[1097] Data processing: Application of motion programs to robot control systems.

[1098] Output: A running industrial robot.

[1099] Prompt the generative AI model:

[1100] "Please explain in detail how to automatically generate a factory robot's motion program based on the motion and emotion data of a worker wearing smart glasses and verify it in a simulation environment."

[1101] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1102] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1103] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1104] [Third embodiment]

[1105] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1106] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1107] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1108] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1109] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1110] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1111] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1112] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1113] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1114] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1115] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1116] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1117] This invention is a system that receives and analyzes data from a motion capture device and then automatically generates an operation program for an industrial robot. This system is composed of a motion capture device, a server, and a terminal, and operates as follows.

[1118] 1. Acquiring motion capture data

[1119] User

[1120] A user wears a motion capture device and performs a specific task, such as welding or picking. As the user performs the task, the motion is recorded by the motion capture device.

[1121] server

[1122] The server receives and stores the data sent from the motion capture device in real time. The data is recorded in a digital format (e.g., BVH, FBX).

[1123] 2. Motion analysis

[1124] server

[1125] The server reads the saved motion capture data and analyzes specific movement patterns, including the start and end times of the movement, joint angles, and speed, etc. Based on this, key frames are generated and sequential movement analysis is performed.

[1126] 3. Industrial robot selection

[1127] server

[1128] The server extracts data on the required degree of freedom of movement, payload capacity, and working range based on the analyzed movement patterns, then compares this data with an internal database to create a list of suitable industrial robots, and selects the most suitable robot from among them.

[1129] 4. Generating the operating program

[1130] server

[1131] The server loads a template corresponding to the selected robot's control language and applies the analyzed motion data to the template, thereby generating a specific set of control instructions, which are then compiled into a robot's motion program.

[1132] 5. Verification of the operating program

[1133] server

[1134] The server runs the generated motion program in a simulation environment to check its accuracy and safety. It also automatically inspects the simulation results for collision detection, timing accuracy, and other errors.

[1135] 6. Program Offering

[1136] server

[1137] In order to provide the terminal with the verified operating program, the server prepares the program file and transfers it to the terminal.

[1138] Terminal

[1139] The terminal stores the program file received from the server in local storage.

[1140] 7. Running the operation program

[1141] Terminal

[1142] The terminal transfers the saved program file to the industrial robot, loads it into the robot's control system, and then issues an instruction to the robot to execute the program.

[1143] User

[1144] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[1145] Specific examples

[1146] For example, when a user records a specific picking task, the motion capture device receives the motion as data. The server analyzes the data, selects the industrial robot best suited to the picking task, and generates a motion program for that robot. The accuracy of the program is checked in a simulation environment, and once no problems are confirmed, the program is provided to a terminal and installed on the robot. Finally, the user can monitor the robot's operations and make adjustments as necessary, completely automating the picking task.

[1147] This invention significantly reduces the time required to implement an industrial robot's operation program, enabling rapid and accurate program generation, thereby providing an effective system that can address labor shortages and rising labor costs.

[1148] The processing flow will be explained below.

[1149] Step 1: Acquiring motion capture data

[1150] User

[1151] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking). Once the task is initiated, the motion capture device records the user's movements in real time.

[1152] server

[1153] The server receives the data sent from the motion capture device and records the data in real time, which is then saved in a digital format such as BVH or FBX.

[1154] Step 2: Behavior analysis

[1155] server

[1156] The server reads the recorded motion capture data and begins analyzing the movements. Specifically, it analyzes the data frame by frame to recognize specific movement patterns. It extracts information such as the start and end times of the movements, joint angles, and speed, and generates key frames based on this information. Finally, sequential movement analysis is performed, and the movement patterns are clearly defined.

[1157] Step 3: Selecting an industrial robot

[1158] server

[1159] Based on the analyzed motion patterns, the server identifies requirements such as the required degrees of freedom of movement, payload capacity, and working range. It then references an internal database to create a list of industrial robots that match these requirements. From the list, it selects the most suitable robot and obtains its specification data.

[1160] Step 4: Generate the operating program

[1161] server

[1162] The server loads a template file corresponding to the selected robot's control language (e.g., Karel, RAPID, TP), applies the analyzed motion data to the template, and generates a specific set of control instructions. Based on the generated instruction set, it compiles a motion program for the robot.

[1163] Step 5: Verify the program

[1164] server

[1165] The server loads the generated motion program into the simulation environment. The simulation is run to check the accuracy of the motion program. Specifically, collision detection and timing checks are performed in the simulation to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[1166] Step 6: Program delivery

[1167] server

[1168] The server prepares to provide the verified operating program to the terminal, specifically by converting the program file into an appropriate format and preparing to transfer it to the terminal.

[1169] Terminal

[1170] The terminal receives the program file sent from the server and stores it in local storage.

[1171] Step 7: Run the action program

[1172] Terminal

[1173] The terminal transfers the saved program file to the industrial robot, which then loads the program into the robot's control system and instructs it to run.

[1174] User

[1175] Users can monitor the robot's operations to ensure they are working correctly, and can make fine adjustments or additional program modifications as needed, allowing specific tasks to be accurately replicated and automated.

[1176] Example 1

[1177] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1178] Creating operational programs for conventional industrial equipment required a significant amount of time and specialized knowledge, which could result in delays in the automation process for certain tasks. Furthermore, there was a lack of means to verify the accuracy and safety of operational programs in advance, which created the risk of malfunctions during execution. This made it difficult to address labor shortages and rising labor costs.

[1179] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1180] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific motion patterns, means for selecting the optimum industrial equipment based on the analyzed motion patterns, means for automatically generating an operation program corresponding to the selected equipment, means for verifying the generated operation program in a simulation environment before execution, means for providing the verified operation program to an information terminal, means for the information terminal to transfer the provided operation program to a control target and issue an execution instruction, and means for monitoring the operation of the control target and making adjustments as necessary. This makes it possible to quickly and accurately generate, verify, and provide operation programs for industrial equipment, thereby facilitating the automation of work.

[1181] A "motion capture device" is a device that records the movements of a human body or object using sensors or cameras and converts that data into a digital format.

[1182] A "server" is a device or software system that receives data from a motion capture device, analyzes, stores, generates and verifies motion programs.

[1183] "Digital format" refers to a file format such as BVH or FBX that is used to digitize and record movement data.

[1184] "Movement pattern" refers to the characteristics and route of a series of movements recognized based on motion capture data.

[1185] "Industrial equipment" is a general term for robots and other automation devices used in production and manufacturing processes.

[1186] An "operation program" refers to a set of instructions for controlling the operation of industrial equipment, and is written in a specific control language.

[1187] A "simulation environment" is a virtual space or software environment in which a generated operating program is executed and its accuracy and safety are virtually confirmed.

[1188] An "information terminal" is a device or system used to receive, store, display, and control data from a server or other device.

[1189] The term "controlled object" refers to industrial equipment that is controlled using an operating program.

[1190] The "internal database" is a database system that stores specification data and operation history data for industrial equipment.

[1191] A "key frame" refers to a frame that is an important point or turning point in a movement pattern, and detailed movements are analyzed based on this frame.

[1192] "Weight capacity" refers to the maximum weight that industrial equipment can lift and carry.

[1193] "Degrees of freedom of movement" is an indicator that shows the direction and range of movement of each joint and axis of industrial equipment and robots.

[1194] A "prompt" refers to an instruction or question that is input into a generative AI model, and functions as an input to obtain a specific output.

[1195] This invention is a system that records the movements of people and objects as data, analyzes the data, and automatically generates operating programs suitable for industrial equipment. This system consists of a motion capture device, a server, and a terminal. The detailed configuration and operation of this system are described below.

[1196] Hardware and Software Configuration

[1197] motion capture equipment

[1198] A motion capture device records a user's movements in real time and transmits the data to a server. A typical example of such a device is a motion capture suit equipped with cameras and sensors. The data is recorded in digital formats such as BVH and FBX.

[1199] server

[1200] The server receives, stores, analyzes, generates, and verifies the data sent from the motion capture device. The server has the following functions:

[1201] Data Reception and Storage: Motion capture data is received in real time and stored in a digital format.

[1202] Movement Analysis: Analyzes stored data and recognizes specific movement patterns, including identifying joint angles, movement speed, and start and end times.

[1203] Industrial equipment selection: Based on the analyzed movement patterns, the system selects equipment suitable for the required degree of freedom of movement, payload capacity, and working range from an internal database.

[1204] Generation of operation program: A specific set of control instructions is generated using a template corresponding to the control language of the selected device.

[1205] Verification in a simulation environment: The generated behavioral program is executed in a simulation environment to verify the correctness and safety of the program.

[1206] Terminal

[1207] The terminal receives the operation program sent from the server and transfers it to the industrial equipment. The terminal has the following functions:

[1208] Receiving and saving programs: Saves program files provided by the server in local storage.

[1209] Linking with industrial equipment: Transfers saved program files to industrial equipment and loads them into the control system. Executes the program according to user instructions.

[1210] Specific examples

[1211] For example, when a user records a specific picking task, the motion capture device receives the motion as data. The server analyzes the data and selects the industrial equipment best suited to the picking task. It then generates a motion program for that equipment and checks the accuracy of the program in a simulation environment. After verifying that there are no problems, the program is provided to a terminal and ultimately implemented in the robot. The user can monitor the robot's operations and make adjustments as necessary to fully automate the picking task.

[1212] Prompt Sentence Examples

[1213] "Please explain the detailed process of a system in which a user wears a motion capture device and performs a picking task, receives the motion data, analyzes the data, selects the most suitable industrial equipment, and generates and provides an operating program for the equipment. We would like a detailed explanation of each step of this system, including the specific processing content and algorithms used."

[1214] This invention enables the rapid and accurate generation and verification of operating programs for industrial equipment, facilitating the automation of work, thereby providing an efficient system that can cope with labor shortages and rising labor costs.

[1215] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1216] Step 1:

[1217] Acquiring motion capture data

[1218] The user wears a motion capture device and performs specific task movements. The device records the user's movements in real time and transmits the data to a server in a digital format (e.g., BVH, FBX).

[1219] Specifically, when a user performs picking work, the series of actions is captured by sensors and cameras.

[1220] Input: User action (e.g. picking work)

[1221] Output: Motion capture data (e.g. BVH, FBX)

[1222] Step 2:

[1223] Receiving and storing data

[1224] The server receives and stores the data transmitted from the motion capture device in real time, and the data is recorded in digital format.

[1225] Input: Motion capture data

[1226] Output: Saved motion capture data

[1227] Specifically, the server saves the data to disk in the specified format (e.g., BVH, FBX).

[1228] Step 3:

[1229] Motion analysis

[1230] The server reads the saved motion capture data and analyzes specific movement patterns, extracting the start and end times of the movement, as well as the joint angles and speeds, and generates keyframes.

[1231] Input: Stored motion capture data

[1232] Output: Analyzed movement patterns and keyframes

[1233] The server uses motion analysis algorithms to extract key motion points, for example, identifying specific motion patterns based on joint angles and movement speed.

[1234] Step 4:

[1235] Industrial Equipment Selection

[1236] Based on the analyzed motion patterns, the server selects from its internal database the industrial equipment that is suitable for the required degree of freedom of movement, payload capacity, and working range.

[1237] Input: Analyzed movement patterns and internal database

[1238] Output: List of selected industrial devices

[1239] Specifically, the server lists devices that meet pre-set filtering conditions and ranks the most suitable devices among them.

[1240] Step 5:

[1241] Generating the operating program

[1242] The server loads a template corresponding to the control language (e.g., URScript, KRL) of the selected industrial equipment and generates an operating program by applying the analyzed operating data to the template.

[1243] Input: Selected industrial equipment, analyzed operation patterns, control language template

[1244] Output: Generated behavior program

[1245] Specifically, the server integrates an instruction set based on an operation pattern into a template, and compiles it to generate a complete program.

[1246] Step 6:

[1247] Verification of the operating program

[1248] The server runs the generated motion program in a simulation environment to verify the accuracy and safety of the motion, which also includes collision detection and timing adjustment.

[1249] Input: Generated behavior program

[1250] Output: Verification results (verification of accuracy and safety)

[1251] Specifically, the server runs an operating program in a virtual environment and checks for any errors or malfunctions.

[1252] Step 7:

[1253] Program Offering

[1254] The server prepares a program file and transfers it to the terminal in order to provide the terminal with the verified operating program.

[1255] Input: Verified working program

[1256] Output: Transferred program file

[1257] Specifically, the server compresses the program file, encrypts it if necessary, and transmits it to the terminal.

[1258] Step 8:

[1259] Execution of the operating program

[1260] The terminal transfers the stored program file to the industrial equipment, loads it into the control system, and then issues an instruction to run the program.

[1261] Input: Transferred program file

[1262] Output: The device behavior caused by the executed program

[1263] Specifically, the terminal uses the control interface of the industrial equipment to load the program and send an execution instruction.

[1264] Step 9:

[1265] Monitor and adjust behavior

[1266] Users monitor the operation of industrial equipment and make fine adjustments or additional program modifications as needed.

[1267] Input: Running industrial equipment operations

[1268] Output: Adjusted behavior, modified program

[1269] Specifically, the user observes the operation in real time and manually corrects or reprograms any malfunctions.

[1270] (Application example 1)

[1271] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1272] In modern factory automation, many procedures rely on manual labor and robot operation programming. With conventional systems, it takes time to generate robot operation programs, and fine-tuning the programs is not easy. As a result, production efficiency can decrease and production can be halted due to errors. There is a need for a system that can solve this problem and enable rapid generation of operation programs and adjustments in real time.

[1273] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1274] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific movement patterns, means for selecting an optimum industrial robot based on the analyzed movement patterns, means for automatically generating an operation program corresponding to the selected robot, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a mobile communication terminal so that a worker can fine-tune it in real time. This enables rapid and accurate generation of an operation program and adjustment in real time.

[1275] Key Word Definitions

[1276] ---

[1277] A "motion capture device" is a device that records the movements of people or objects as digital data.

[1278] The "means for receiving and recording data" is a system that has the function of receiving data transmitted from the motion capture device and storing it.

[1279] "Means for analyzing recorded motion capture data and recognizing specific movement patterns" refers to a system that has the function of analyzing stored data and identifying work movements and types of movements.

[1280] The "means for selecting the optimal industrial robot" is a system that has the function of selecting the robot that can perform the most appropriate operation based on the analyzed operation pattern.

[1281] The "means for automatically generating an operating program" is a system that has the function of automatically creating a program corresponding to the selected robot.

[1282] "Means for verifying in a simulation environment" refers to a system that has the function of testing in a virtual environment whether the generated program is accurate.

[1283] A "mobile communication terminal" is a portable communication device such as a smartphone or tablet.

[1284] "Means that allow workers to make fine adjustments in real time" refers to a system that has the function of immediately making necessary changes and fine adjustments to a verified program.

[1285] A "database" is a system that stores robot specification data and other related information and manages it so that necessary data can be retrieved quickly.

[1286] "Movement freedom" is a concept that refers to the number and range of movements that a robot can perform.

[1287] "Weight capacity" refers to the maximum weight that an industrial robot can safely lift and carry.

[1288] "Work envelope" refers to the physical area within which an industrial robot can move.

[1289] "Means for verifying accuracy and correcting the program if problems are found" refers to a system that has the ability to verify whether the generated program operates correctly and make appropriate corrections when errors or problems are found.

[1290] MODE FOR CARRYING OUT THE INVENTION

[1291] ---

[1292] This invention relates to a system that automatically creates an operating program for an industrial robot by receiving and analyzing motion capture data. This system is comprised of a motion capture device, a server, and a mobile communication terminal, and operates as follows.

[1293] 1. Acquiring motion capture data

[1294] User

[1295] A user wears a motion capture device and performs specific task movements, which are recorded as digital data by the motion capture device.

[1296] server

[1297] The server receives and stores the data sent from the motion capture equipment in real time, and stores this data in a digital format (e.g., BVH, FBX).

[1298] 2. Motion analysis

[1299] server

[1300] The server reads the stored motion capture data and analyzes specific movement patterns, including the start and end times of the movement, joint angles and speeds, etc. Keyframes are generated based on the analyzed data, and sequential movement analysis is performed.

[1301] 3. Industrial robot selection

[1302] server

[1303] Based on the analysis results, the server compares the robot specifications retrieved from the database to select the optimal industrial robot, including the required degrees of freedom of movement, payload capacity, and working range.

[1304] 4. Generating the operating program

[1305] server

[1306] The server loads a template corresponding to the selected robot's control language and applies the analyzed motion data, thereby generating a specific set of control instructions and compiling them into a robot's motion program.

[1307] 5. Verification of the operating program

[1308] server

[1309] The server runs the generated program in a simulation environment to verify its accuracy and safety. Based on the results of the simulation, the program is automatically modified if necessary.

[1310] 6. Program provision and real-time adjustment

[1311] server

[1312] In order to provide the verified operating program to the mobile communication terminal, the server prepares a program file and transfers it to the mobile communication terminal.

[1313] Mobile communication terminal

[1314] The mobile communication terminal stores the program files received from the server and provides an interface that allows the user to fine-tune the operation in real time.

[1315] User

[1316] The user monitors the robot's movements using a mobile communication terminal and fine-tunes the movement program in real time to finalize it. Additional program modifications are also possible as needed.

[1317] Hardware and software used

[1318] Hardware: Mobile communication devices (smartphones and tablets), motion capture devices (e.g., motion sensors), industrial robots

[1319] Software: Server-side technologies (e.g., Django, Flask), simulation environments (e.g., ROS, Gazebo), data parsing libraries (e.g., Python requests, json)

[1320] Specific examples

[1321] Factory workers operate smartphones and use motion capture devices to record specific picking movements. The recorded data is sent to a server, where it is analyzed and the server selects the robot best suited to the picking movement. A motion program for that robot is automatically generated and verified through simulation. After verification, the generated program is sent to a mobile communication device (e.g., a smartphone), where the worker can fine-tune the movement in real time.

[1322] Example prompts to input to the generative AI model

[1323] "We are seeking advice on the design and implementation of a system that analyzes motion capture data and automatically generates motion programs for industrial robots."

[1324] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1325] Program processing flow

[1326] ---

[1327] Step 1:

[1328] A user wears a motion capture device and performs a specific task. The input data here is the user's motion information, and the output is the motion data recorded by the motion capture device.

[1329] Specific behavior:

[1330] The user performs picking operations and welding parts, and the series of movements is recorded as digital data by a motion capture device.

[1331] Step 2:

[1332] The recorded motion capture data is sent to the server in real time. The input data is the digital format data (e.g. BVH, FBX) sent from the motion capture device, and the output is the motion capture data stored on the server.

[1333] Specific behavior:

[1334] The data collected by the motion capture device is sent via wireless or wired communication to a server, which stores it in a database.

[1335] Step 3:

[1336] The server analyzes the stored motion capture data and recognizes specific movement patterns. The input data is the motion capture data stored on the server, and the output is the analyzed movement pattern and related parameters (start time, end time of movement, joint angles, etc.).

[1337] Specific behavior:

[1338] The server uses data analysis algorithms (e.g., machine learning models) to extract specific actions or keyframes within the data.

[1339] Step 4:

[1340] Based on the analysis results, the server selects the optimal industrial robot. The input data is the analyzed motion pattern and the corresponding robot specification data (degree of freedom of motion, payload capacity, working range, etc.), and the output is the selected robot.

[1341] Specific behavior:

[1342] The server checks its internal database, lists the robots that best meet the required specifications, and selects the most suitable robot.

[1343] Step 5:

[1344] The server automatically generates an operating program for the selected robot. The input data is a template based on the control language of the selected robot and analyzed operation patterns, and the output is an operating program containing a specific set of control instructions.

[1345] Specific behavior:

[1346] The server uses a program generation algorithm to generate a control instruction set by applying an operation pattern to a template.

[1347] Step 6:

[1348] The generated behavioral program is verified in a simulation environment. The input data is the generated behavioral program, and the output is the simulation results and the verified program.

[1349] Specific behavior:

[1350] The server runs the generated program using a robot simulation environment (e.g., ROS, Gazebo) to check its safety and accuracy, and automatically corrects the program if necessary.

[1351] Step 7:

[1352] The verified operating program is transferred to the mobile communication terminal, the input data is the verified program file, and the output is the program file stored in the mobile communication terminal.

[1353] Specific behavior:

[1354] The server transmits the program file to the mobile communication terminal, which stores the file in its local memory.

[1355] Step 8:

[1356] The user fine-tunes the robot's movements in real time using a mobile communication terminal. The input data is a verified program file and the user's instructions, and the output is the final modified movement program.

[1357] Specific behavior:

[1358] Workers can fine-tune the operation program via a smartphone or tablet interface to determine the optimal operation program.

[1359] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1360] This invention is a system that combines a system that receives and analyzes data from a motion capture device and automatically generates an operating program for an industrial robot with an emotion engine that recognizes the user's emotions. This system is composed of a motion capture device, a server, a terminal, and an emotion engine, and operates as follows.

[1361] 1. Acquiring motion capture data

[1362] User

[1363] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking). Once the task is initiated, the motion capture device records the user's movements in real time.

[1364] server

[1365] The server receives and records the data sent from the motion capture device in real time, and saves it in formats such as BVH or FBX.

[1366] 2. Motion analysis

[1367] server

[1368] The server loads the saved motion capture data and begins analysis, including the start and end times of the motion, joint angles, and velocity. Based on this, key frames are generated and sequential motion analysis is performed.

[1369] 3. Emotion Recognition by Emotion Engine

[1370] server

[1371] The server utilizes an emotion engine to recognize the user's emotional state. The emotion engine detects the user's facial expressions, voice, and biometric signals, and identifies the user's emotions based on these data. For example, if the user is feeling stressed, the emotional state is recorded.

[1372] 4. Adjusting the movement data

[1373] server

[1374] The server adjusts the analyzed movement patterns based on the emotional state recognized by the emotion engine. For example, if the user is nervous and their movements are awkward, the server corrects those parts to generate a smooth movement pattern.

[1375] 5. Industrial Robot Selection

[1376] server

[1377] The server determines the required degrees of freedom of movement, payload capacity, and working range based on the adjusted motion pattern. It then references an internal database to create a list of industrial robots that meet these requirements. It then selects the most suitable robot from the list and obtains its specification data.

[1378] 6. Generating the operating program

[1379] server

[1380] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, and generates a specific set of control instructions, which are then compiled into the robot's motion program.

[1381] 7. Verification of the operating program

[1382] server

[1383] The server loads the generated motion program into the simulation environment. The simulation is run to verify the accuracy of the motion program. Specifically, collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[1384] 8. Program Offering

[1385] server

[1386] The server prepares to provide the verified operating program to the terminal, converts the program file into an appropriate format, and prepares to transfer it to the terminal.

[1387] Terminal

[1388] The terminal receives the program file sent from the server and stores it in local storage.

[1389] 9. Running the Operation Program

[1390] Terminal

[1391] The terminal transfers the saved program file to the industrial robot, loads the program into the robot's control system, and then issues an instruction to the robot to execute the program.

[1392] User

[1393] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[1394] Specific examples

[1395] For example, when a user records a specific welding task, the motion capture device receives the motion as data. The server analyzes the data and uses an emotion engine to identify areas where the user is feeling stressed. Next, it corrects any motion irregularities caused by stress, selects the most suitable industrial robot based on the adjusted motion data, and generates a motion program for that robot. After verifying the accuracy of the program in a simulation environment, it is provided to the terminal and finally implemented in the robot. The user monitors the robot's motion and makes adjustments as necessary, automating the welding task.

[1396] This invention significantly shortens the implementation time for industrial robot operation programs, enabling rapid and accurate program generation. Furthermore, by taking the user's emotional state into consideration, it is possible to provide more accurate operation programs. This system can provide an effective means of addressing labor shortages and rising labor costs.

[1397] The processing flow will be explained below.

[1398] Step 1: Acquiring motion capture data

[1399] User

[1400] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking), which records the user's movements in real time.

[1401] server

[1402] The server receives and records the data sent from the motion capture equipment in real time, and the data is saved in digital formats such as BVH and FBX.

[1403] Step 2: Behavior analysis

[1404] server

[1405] The server reads the stored motion capture data and analyzes it frame by frame, extracting the start and end times of the movements, joint angles, speed, etc., and generates keyframes based on this information. Specific movement patterns are identified through the analysis.

[1406] Step 3: Emotion recognition by the emotion engine

[1407] server

[1408] The server utilizes an emotion engine to recognize the user's emotional state. The emotion engine detects the user's facial expressions, voice, and biometric signals, and identifies the user's emotions based on these data. For example, if the user is feeling stressed, the emotional state is recorded.

[1409] Step 4: Adjusting the behavior data

[1410] server

[1411] The server adjusts the analyzed movement patterns based on the recognized emotional state: if the user is nervous and their movements are awkward, it corrects those parts and converts them into smoother movement patterns.

[1412] Step 5: Selecting an industrial robot

[1413] server

[1414] The server determines the required degree of freedom of movement, payload capacity, and working range based on the adjusted motion pattern, and then references an internal database to create a list of industrial robots that match these requirements and select the most suitable robot.

[1415] Step 6: Generate the operating program

[1416] server

[1417] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, generates a specific set of control instructions, and compiles them into a robot motion program.

[1418] Step 7: Verify the program

[1419] server

[1420] The server loads the generated motion program into the simulation environment. The simulation is run to check the accuracy and safety of the motion program. Collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[1421] Step 8: Program Delivery

[1422] server

[1423] The server prepares to provide the verified operating program to the terminal, converts the program file into an appropriate format, and prepares to transfer it to the terminal.

[1424] Terminal

[1425] The terminal receives the program file sent from the server and stores it in local storage.

[1426] Step 9: Run the action program

[1427] Terminal

[1428] The terminal transfers the saved program file to the industrial robot, which then loads the program into the robot's control system and issues an instruction to run it.

[1429] User

[1430] The user monitors the robot's operation to ensure it is working correctly. If necessary, they can make fine adjustments or additional program modifications. Once proper operation is confirmed, the robot will automate the task.

[1431] Specific examples

[1432] For example, when a user records a specific picking task, the motion capture device receives the movement as data. The server analyzes the data and uses an emotion engine to determine whether the user is feeling stressed. The server then corrects any movement irregularities caused by stress and selects the optimal industrial robot based on the adjusted movement data. A movement program appropriate for the selected robot is generated, its accuracy confirmed through simulation, and then provided to the terminal and installed on the robot. The user monitors the robot's movements and makes adjustments as necessary, allowing the picking task to be accurately automated.

[1433] Example 2

[1434] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1435] Conventional industrial robot motion program generation systems have the risk of motion accuracy decreasing due to the user's emotional state, such as stress or tension. Furthermore, generating and verifying motion programs takes time, resulting in inefficiency. This has led to problems with automation in industrial settings and productivity not improving.

[1436] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1437] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific movement patterns, means for utilizing an emotion engine for recognizing the emotional state of a user, means for adjusting the movement patterns based on the recognized emotional state, means for selecting an optimum industrial robot based on the adjusted movement patterns, means for automatically generating an operation program corresponding to the selected robot, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a terminal. This enables the generation and verification of an operation program quickly and accurately while taking the emotional state of a user into consideration.

[1438] A "motion capture device" is a device that records a user's body movements in real time and transmits the data to a server.

[1439] The "means for receiving and recording data" is a component of the server that has the function of receiving data transmitted from the motion capture device and recording that data.

[1440] The "means for analyzing motion capture data" is a server component that has the function of analyzing the start time, end time, joint angles, speed, etc. of a movement based on recorded motion capture data, and recognizing a specific movement pattern.

[1441] An "emotion engine" is software or hardware that analyzes a user's facial expressions, voice, and biometric signals to identify the user's emotional state.

[1442] The "means for adjusting the movement pattern" is a component of the server that has the function of correcting unnatural parts of the analyzed movement pattern and adjusting it to a smooth movement pattern based on the recognized emotional state of the user.

[1443] An "industrial robot" is an automated device used in industrial settings that requires an operating program to perform a specific task (e.g., welding, assembly, picking).

[1444] The "means for automatically generating an operation program" is a component of the server that has the function of automatically generating an operation program in a control language corresponding to the selected robot based on the adjusted operation data.

[1445] A "simulation environment" is a computer-based system for executing generated operating programs in a virtual space and verifying their accuracy and errors.

[1446] The "means for verifying the operation program" is a component of the server that has the function of checking the accuracy of the operation program generated using the simulation environment and making any necessary corrections.

[1447] A "terminal" is an electronic device that receives an operating program provided by a server, stores it in local storage, and transfers it to an industrial robot for execution.

[1448] This invention is a system that combines a system that receives and analyzes data from a motion capture device and automatically generates an operating program for an industrial robot with an emotion engine that recognizes the user's emotions. This system is composed of a motion capture device, a server, a terminal, and an emotion engine, and operates as follows.

[1449] 1. Generate a program for this system. Specifically, create a program to receive and record data sent from the motion capture device in real time. This data is saved in formats such as BVH or FBX. Next, create a program to analyze the saved motion capture data and extract information such as the start and end times of the movement, joint angles, and speed. This also includes a program that uses an emotion engine to recognize the user's emotional state from facial expressions, voice, and biometric signals. These programs will use a generative AI model to adjust movement patterns and automatically generate optimal movement programs for industrial robots.

[1450] 2. The processing of the generated program is explained in natural language. Specifically, a server is used to receive and record data from the motion capture device. The server saves the recorded data in formats such as BVH or FBX and analyzes the movements. It uses an emotion engine to recognize the user's emotional state and adjusts the movement pattern as needed. Based on the adjusted movement data, the server selects the optimal industrial robot and automatically generates a movement program for that robot. The generated program is verified in a simulation environment to confirm there are no problems before being provided to the terminal. The terminal transfers the provided program to the industrial robot and executes it. The user can monitor the robot's movement and make fine adjustments or additional program modifications as needed.

[1451] 3. Specific examples are added to sentences that explain the program's processing. For example, a user wears a motion capture device and performs a specific welding task. The server receives and records this data in real time in BVH format. The server then analyzes the data, calculating the start and end times of the motion, joint angles, speed, and other parameters, and generates keyframes. The server then uses an emotion engine to analyze the user's facial expressions, voice, and biometric signals to identify stress levels. Based on the stress levels identified by the emotion engine, the server adjusts the analyzed movement patterns. For example, it modifies the user's tense movements to smoother movements. Based on the adjusted movement patterns, the server then identifies the required degrees of freedom of movement, payload capacity, and working range, and selects the optimal industrial robot from a database. The server generates a movement program using a template file corresponding to the selected robot's control language (e.g., Karel). The generated movement program is then verified in a simulation environment to check timing and detect collisions. If verification is successful, the program is transferred to the terminal, which then loads it onto the industrial robot. The user then monitors the robot's movements and makes adjustments as necessary, automating the welding process.

[1452] Specific prompt examples:

[1453] "Please explain in detail, step by step, the procedure for generating an operation program for an industrial robot that modifies its operation pattern based on the stress felt by the user during welding work."

[1454] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1455] Step 1:

[1456] Acquiring motion capture data

[1457] User

[1458] The user wears a motion capture device and performs a specific work motion (e.g., welding). Once the motion is initiated, the motion capture device records the user's movements in real time.

[1459] Input: User's physical movements

[1460] Output: Real-time motion capture data

[1461] Step 2:

[1462] Receiving and recording data

[1463] server

[1464] The server receives the data sent from the motion capture device in real time and records it in BVH or FBX format.

[1465] Input: Motion capture data

[1466] Output: BVH or FBX format files

[1467] Step 3:

[1468] Motion analysis

[1469] server

[1470] The server reads the recorded motion capture data and analyzes the start and end times of the movements, joint angles, speed, etc. Based on this, key frames are generated and detailed movement analysis is performed.

[1471] Input: BVH or FBX format files

[1472] Output: Analysis results (start time, end time, joint angles, speed, keyframes)

[1473] Step 4:

[1474] Emotion recognition by emotion engine

[1475] server

[1476] The server uses an emotion engine to analyze the user's facial expressions, voice, and biometric signals to identify the user's emotional state. If the user is feeling stressed or tense, the server records that emotional state.

[1477] Input: User's facial expressions, voice, and biometric signals

[1478] Output: Emotional state (e.g., stress, tension)

[1479] Step 5:

[1480] Adjusting the movement data

[1481] server

[1482] The server adjusts the analyzed movement patterns based on the recognized emotional state. For example, if the user is nervous and their movements are awkward, it will correct those parts to make them smoother.

[1483] Input: Analysis results, emotional state

[1484] Output: Coordinated movement pattern

[1485] Step 6:

[1486] Industrial robot selection

[1487] server

[1488] The server determines the required degree of freedom of movement, payload capacity, and working range based on the adjusted motion pattern, and then references an internal database to create a list of industrial robots that match these requirements and select the most suitable robot.

[1489] Input: Adjusted movement pattern

[1490] Output: Selected industrial robot

[1491] Step 7:

[1492] Generating the operating program

[1493] server

[1494] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, and generates a specific set of control instructions, which are then compiled into the robot's motion program.

[1495] Input: Selected industrial robot, adjusted motion pattern

[1496] Output: Working program

[1497] Step 8:

[1498] Verification of the operating program

[1499] server

[1500] The server loads the generated motion program into the simulation environment. The simulation is run to verify the accuracy of the motion program. Specifically, collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[1501] Input: Action program

[1502] Output: Verification results (correctness, presence of errors)

[1503] Step 9:

[1504] Program Offering

[1505] server

[1506] The server provides the verified operating program to the terminal, converts the program file into a format compatible with the robot control system, and prepares it for transfer to the terminal.

[1507] Input: Verified working program

[1508] Output: Program file for distribution

[1509] Terminal

[1510] The terminal receives the program file sent from the server and stores it in local storage.

[1511] Input: Program file to be provided

[1512] Output: Program files in local storage

[1513] Step 10:

[1514] Execution of the operating program

[1515] Terminal

[1516] The terminal transfers the program file stored in the local storage to the industrial robot, loads the program into the robot's control system, and then issues an instruction to the robot to execute the program.

[1517] Input: Program files in local storage

[1518] Output: Running program

[1519] User

[1520] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[1521] Input: Running action program

[1522] Output: Monitoring results and fine-tuning

[1523] (Application example 2)

[1524] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1525] The rapid generation of efficient and accurate motion programs for industrial robots is a high requirement in many factories. However, conventional systems require a great deal of time and effort to generate and verify motion programs. Furthermore, because they do not take into account the impact of the worker's emotional state on motion, the motions of unskilled or fatigued workers are directly reflected in the program, resulting in the generation of inefficient motion programs. Furthermore, modifying motion programs is complex, making it difficult to generate programs optimized for specific work conditions.

[1526] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing a specific movement pattern, means for selecting an optimal industrial robot based on the analyzed movement pattern, means for automatically generating an operation program corresponding to the selected robot, means for analyzing the emotional state of a user in real time and adjusting the movement pattern based on the emotional state of the user, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a terminal. This enables rapid automatic generation and verification of accurate and efficient operation programs that take into account the emotional state of a worker.

[1527] Key word definitions

[1528] 1. "Motion capture device" means a device that records a user's movements in real time and transmits the data.

[1529] 2. "Emotional state" refers to emotions such as stress, tension, and joy recognized from the user's facial expressions, voice, and biometric signals.

[1530] 3. "Server" refers to the central processing unit that receives and analyzes motion capture data and emotional data, and generates and verifies the motion program.

[1531] 4. "Movement pattern" refers to the details of a series of movements recorded by a motion capture device, including the start and end times, joint angles, and speed.

[1532] 5. An "industrial robot" is an automated mechanical device that is programmed to perform a specific task.

[1533] 6. An "operation program" is a set of control instructions that enable an industrial robot to perform a specific operation.

[1534] 7. "Simulation environment" is a system for executing and verifying generated operating programs in a virtual space.

[1535] 8. "Terminal" means a device that receives an operating program sent from a server and stores it in a form that can be executed in a local environment.

[1536] 9. "Database" refers to an information collection system that stores robot specification data (such as degrees of freedom of movement, weight capacity, and working range).

[1537] MODE FOR CARRYING OUT THE INVENTION

[1538] Overall system overview

[1539] A system for implementing this invention comprises a motion capture device, a server, a terminal, and an emotion engine. A user wears smart glasses and performs a task, and the motion capture device records motion data in real time. This data is sent to the server, where it is analyzed. The server then uses the emotion engine to recognize the user's emotional state and adjusts the motion data based on this. Based on the adjusted motion data, the server selects an optimal industrial robot and generates an operation program. The generated operation program is verified in a simulation environment, and after any necessary modifications are made based on the results, it is provided to the terminal. Finally, the program is transferred from the terminal to the industrial robot and executed.

[1540] Hardware and software used

[1541] Hardware:

[1542] Smart glasses: Motion capture devices for recording the user's work movements (e.g., Google Glass, Vuzix M400).

[1543] Server: A central processing unit equipped with a high-performance GPU for analyzing motion capture data and emotion data, and for generating and verifying movement programs.

[1544] Industrial robot: An automated mechanical device for performing specific tasks (e.g. FANUC, Yaskawa).

[1545] Terminal: A device that receives an operating program sent from a server.

[1546] software:

[1547] Motion capture software: Software used to capture user movement data (e.g., Vicon Nexus).

[1548] Emotion Analysis Engine (Emotion Engine): Software for recognizing the user's emotional state (e.g., Affectiva SDK).

[1549] Motion analysis software: Software for analyzing and adjusting motion data (e.g., ROS, OpenCV).

[1550] Compiler for industrial robot control languages: Software for compiling the generated operation programs (e.g., FANUC RoboGuide, Yaskawa Motoman).

[1551] Simulation environment: A system for verifying operating programs.

[1552] Processing flow

[1553] 1. Acquiring motion capture data:

[1554] The user wears the smart glasses and performs specific tasks, while sensors built into the glasses record the movement data in real time.

[1555] 2. Sending operational data to the server:

[1556] The movement data is sent to the server, which records it in BVH or FBX format.

[1557] 3. Analysis of behavioral data:

[1558] The server analyzes the recorded data and generates keyframes based on the start time, end time, joint angle, and velocity.

[1559] 4. Recognition of emotional states:

[1560] The server uses an emotion engine to recognize the user's emotional state and analyzes the emotional data.

[1561] 5. Adjusting operational data:

[1562] The server adjusts the movement data based on the recognized emotional state to generate smooth movement patterns.

[1563] 6. Industrial robot selection:

[1564] The server selects the most suitable industrial robot based on the required degree of freedom of movement, payload capacity, and working range.

[1565] 7. Generating the operating program:

[1566] The server automatically generates an operating program for the selected robot.

[1567] 8. Verifying the operation program:

[1568] The server verifies the generated operating program in a simulation environment and makes any necessary modifications.

[1569] 9. Provision of operating programs:

[1570] The modified operating program is transferred to the terminal, which stores the program in its local environment.

[1571] 10. Running the operating program:

[1572] The terminal transfers the program to the industrial robot and causes the robot to execute the program.

[1573] Specific examples

[1574] For example, a worker wears smart glasses and performs welding work, recording his or her movements in real time using motion capture. The server uses an emotion engine to analyze the worker's emotional state (e.g., stress, tension) during the work, corrects for irregularities in movement, and generates an optimal movement pattern. A factory robot's movement program is then automatically generated based on this movement pattern, verified in a simulation environment, and implemented on the factory robot. The worker monitors the final robot movement and makes adjustments as necessary.

[1575] Prompt Sentence Examples

[1576] Prompt the generative AI model:

[1577] "Please explain in detail how to automatically generate a factory robot's motion program based on the motion and emotion data of a worker wearing smart glasses and verify it in a simulation environment."

[1578] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1579] Program processing steps

[1580] Step 1

[1581] Acquiring motion capture data

[1582] A user wears smart glasses and performs specific tasks. The motion capture device (smart glasses) records the user's movements in real time and generates data.

[1583] Input: User action.

[1584] Data processing: Convert motion data into BVH or FBX format.

[1585] Output: Recorded motion data.

[1586] Step 2

[1587] Sending operation data to the server

[1588] The user wears the smart glasses and transmits the acquired data to the server.

[1589] Input: Recorded motion data.

[1590] Data processing: Converting data into a transferable format.

[1591] Output: The operation data sent to the server.

[1592] Step 3

[1593] Analysis of behavioral data

[1594] The server analyzes the received motion data and generates key frames based on information such as start time, end time, joint angle, and speed.

[1595] Input: The motion data sent.

[1596] Data calculation: Analyzing and processing joint angles, speeds, etc.

[1597] Output: Analyzed behavior patterns.

[1598] Step 4

[1599] Recognition of emotional states

[1600] The server uses an emotion engine to recognize the user's emotional state and analyze the emotion data.

[1601] Input: User's facial, voice, and biometric data.

[1602] Data Computing: Identifying and classifying emotional states using a sentiment analysis engine.

[1603] Output: Recognized emotional state data.

[1604] Step 5

[1605] Adjusting the movement data

[1606] The server adjusts the analyzed motion data based on the recognized emotional state, for example, if the user is nervous and making awkward movements, it will correct those parts.

[1607] Input: movement patterns, emotional state data.

[1608] Data processing: Correcting movement patterns based on emotional state.

[1609] Output: Coordinated movement pattern.

[1610] Step 6

[1611] Industrial robot selection

[1612] The server refers to an internal database and selects the most suitable industrial robot based on the required degree of freedom of movement, payload capacity, and working range.

[1613] Input: Calibrated motion pattern, robot specification data in database.

[1614] Data calculation: Algorithm for selecting the best robot.

[1615] Output: Specifications of the selected industrial robot.

[1616] Step 7

[1617] Generating the operating program

[1618] The server generates a motion program for the selected robot, applying the adjusted motion data to an existing template to create a set of control instructions.

[1619] Input: Coordinated movement pattern, control template for the robot.

[1620] Data processing: Applying motion data to a template and generating a set of control instructions.

[1621] Output: The generated behavior program.

[1622] Step 8

[1623] Verification of the operating program

[1624] The server then verifies the generated motion program in a simulation environment, specifically by checking collision detection and timing to ensure there are no errors.

[1625] Input: The generated behavior program.

[1626] Data calculation: Operation verification and error detection through simulation.

[1627] Output: Verified working program, error report.

[1628] Step 9

[1629] Providing operation programs

[1630] The server prepares to provide the verified operating program to the terminal, and transfers the program file to the terminal.

[1631] Input: A verified working program.

[1632] Data processing: Converting program files into the appropriate format.

[1633] Output: The action program file sent to the terminal.

[1634] Step 10

[1635] Execution of the operating program

[1636] The terminal transfers the saved program file to the industrial robot, causing the robot to load and execute the program.

[1637] Input: The operating program file stored on the device.

[1638] Data processing: Application of motion programs to robot control systems.

[1639] Output: A running industrial robot.

[1640] Prompt the generative AI model:

[1641] "Please explain in detail how to automatically generate a factory robot's motion program based on the motion and emotion data of a worker wearing smart glasses and verify it in a simulation environment."

[1642] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1643] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1644] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1645] [Fourth embodiment]

[1646] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1647] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1648] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1649] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1650] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1651] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1652] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1653] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1654] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1655] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1656] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1657] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1658] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1659] This invention is a system that receives and analyzes data from a motion capture device and then automatically generates an operation program for an industrial robot. This system is composed of a motion capture device, a server, and a terminal, and operates as follows.

[1660] 1. Acquiring motion capture data

[1661] User

[1662] A user wears a motion capture device and performs a specific task, such as welding or picking. As the user performs the task, the motion is recorded by the motion capture device.

[1663] server

[1664] The server receives and stores the data sent from the motion capture device in real time. The data is recorded in a digital format (e.g., BVH, FBX).

[1665] 2. Motion analysis

[1666] server

[1667] The server reads the saved motion capture data and analyzes specific movement patterns, including the start and end times of the movement, joint angles, and speed, etc. Based on this, key frames are generated and sequential movement analysis is performed.

[1668] 3. Industrial robot selection

[1669] server

[1670] The server extracts data on the required degree of freedom of movement, payload capacity, and working range based on the analyzed movement patterns, then compares this data with an internal database to create a list of suitable industrial robots, and selects the most suitable robot from among them.

[1671] 4. Generating the operating program

[1672] server

[1673] The server loads a template corresponding to the selected robot's control language and applies the analyzed motion data to the template, thereby generating a specific set of control instructions, which are then compiled into a robot's motion program.

[1674] 5. Verification of the operating program

[1675] server

[1676] The server runs the generated motion program in a simulation environment to check its accuracy and safety. It also automatically inspects the simulation results for collision detection, timing accuracy, and other errors.

[1677] 6. Program Offering

[1678] server

[1679] In order to provide the terminal with the verified operating program, the server prepares the program file and transfers it to the terminal.

[1680] Terminal

[1681] The terminal stores the program file received from the server in local storage.

[1682] 7. Running the operation program

[1683] Terminal

[1684] The terminal transfers the saved program file to the industrial robot, loads it into the robot's control system, and then issues an instruction to the robot to execute the program.

[1685] User

[1686] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[1687] Specific examples

[1688] For example, when a user records a specific picking task, the motion capture device receives the motion as data. The server analyzes the data, selects the industrial robot best suited to the picking task, and generates a motion program for that robot. The accuracy of the program is checked in a simulation environment, and once no problems are confirmed, the program is provided to a terminal and installed on the robot. Finally, the user can monitor the robot's operations and make adjustments as necessary, completely automating the picking task.

[1689] This invention significantly reduces the time required to implement an industrial robot's operation program, enabling rapid and accurate program generation, thereby providing an effective system that can address labor shortages and rising labor costs.

[1690] The processing flow will be explained below.

[1691] Step 1: Acquiring motion capture data

[1692] User

[1693] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking). Once the task is initiated, the motion capture device records the user's movements in real time.

[1694] server

[1695] The server receives the data sent from the motion capture device and records the data in real time, which is then saved in a digital format such as BVH or FBX.

[1696] Step 2: Behavior analysis

[1697] server

[1698] The server reads the recorded motion capture data and begins analyzing the movements. Specifically, it analyzes the data frame by frame to recognize specific movement patterns. It extracts information such as the start and end times of the movements, joint angles, and speed, and generates key frames based on this information. Finally, sequential movement analysis is performed, and the movement patterns are clearly defined.

[1699] Step 3: Selecting an industrial robot

[1700] server

[1701] Based on the analyzed motion patterns, the server identifies requirements such as the required degrees of freedom of movement, payload capacity, and working range. It then references an internal database to create a list of industrial robots that match these requirements. From the list, it selects the most suitable robot and obtains its specification data.

[1702] Step 4: Generate the operating program

[1703] server

[1704] The server loads a template file corresponding to the selected robot's control language (e.g., Karel, RAPID, TP), applies the analyzed motion data to the template, and generates a specific set of control instructions. Based on the generated instruction set, it compiles a motion program for the robot.

[1705] Step 5: Verify the program

[1706] server

[1707] The server loads the generated motion program into the simulation environment. The simulation is run to check the accuracy of the motion program. Specifically, collision detection and timing checks are performed in the simulation to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[1708] Step 6: Program delivery

[1709] server

[1710] The server prepares to provide the verified operating program to the terminal, specifically by converting the program file into an appropriate format and preparing to transfer it to the terminal.

[1711] Terminal

[1712] The terminal receives the program file sent from the server and stores it in local storage.

[1713] Step 7: Run the action program

[1714] Terminal

[1715] The terminal transfers the saved program file to the industrial robot, which then loads the program into the robot's control system and instructs it to run.

[1716] User

[1717] Users can monitor the robot's operations to ensure they are working correctly, and can make fine adjustments or additional program modifications as needed, allowing specific tasks to be accurately replicated and automated.

[1718] Example 1

[1719] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1720] Creating operational programs for conventional industrial equipment required a significant amount of time and specialized knowledge, which could result in delays in the automation process for certain tasks. Furthermore, there was a lack of means to verify the accuracy and safety of operational programs in advance, which created the risk of malfunctions during execution. This made it difficult to address labor shortages and rising labor costs.

[1721] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1722] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific motion patterns, means for selecting the optimum industrial equipment based on the analyzed motion patterns, means for automatically generating an operation program corresponding to the selected equipment, means for verifying the generated operation program in a simulation environment before execution, means for providing the verified operation program to an information terminal, means for the information terminal to transfer the provided operation program to a control target and issue an execution instruction, and means for monitoring the operation of the control target and making adjustments as necessary. This makes it possible to quickly and accurately generate, verify, and provide operation programs for industrial equipment, thereby facilitating the automation of work.

[1723] A "motion capture device" is a device that records the movements of a human body or object using sensors or cameras and converts that data into a digital format.

[1724] A "server" is a device or software system that receives data from a motion capture device, analyzes, stores, generates and verifies motion programs.

[1725] "Digital format" refers to a file format such as BVH or FBX that is used to digitize and record movement data.

[1726] "Movement pattern" refers to the characteristics and route of a series of movements recognized based on motion capture data.

[1727] "Industrial equipment" is a general term for robots and other automation devices used in production and manufacturing processes.

[1728] An "operation program" refers to a set of instructions for controlling the operation of industrial equipment, and is written in a specific control language.

[1729] A "simulation environment" is a virtual space or software environment in which a generated operating program is executed and its accuracy and safety are virtually confirmed.

[1730] An "information terminal" is a device or system used to receive, store, display, and control data from a server or other device.

[1731] The term "controlled object" refers to industrial equipment that is controlled using an operating program.

[1732] The "internal database" is a database system that stores specification data and operation history data for industrial equipment.

[1733] A "key frame" refers to a frame that is an important point or turning point in a movement pattern, and detailed movements are analyzed based on this frame.

[1734] "Weight capacity" refers to the maximum weight that industrial equipment can lift and carry.

[1735] "Degrees of freedom of movement" is an indicator that shows the direction and range of movement of each joint and axis of industrial equipment and robots.

[1736] A "prompt" refers to an instruction or question that is input into a generative AI model, and functions as an input to obtain a specific output.

[1737] This invention is a system that records the movements of people and objects as data, analyzes the data, and automatically generates operating programs suitable for industrial equipment. This system consists of a motion capture device, a server, and a terminal. The detailed configuration and operation of this system are described below.

[1738] Hardware and Software Configuration

[1739] motion capture equipment

[1740] A motion capture device records a user's movements in real time and transmits the data to a server. A typical example of such a device is a motion capture suit equipped with cameras and sensors. The data is recorded in digital formats such as BVH and FBX.

[1741] server

[1742] The server receives, stores, analyzes, generates, and verifies the data sent from the motion capture device. The server has the following functions:

[1743] Data Reception and Storage: Motion capture data is received in real time and stored in a digital format.

[1744] Movement Analysis: Analyzes stored data and recognizes specific movement patterns, including identifying joint angles, movement speed, and start and end times.

[1745] Industrial equipment selection: Based on the analyzed movement patterns, the system selects equipment suitable for the required degree of freedom of movement, payload capacity, and working range from an internal database.

[1746] Generation of operation program: A specific set of control instructions is generated using a template corresponding to the control language of the selected device.

[1747] Verification in a simulation environment: The generated behavioral program is executed in a simulation environment to verify the correctness and safety of the program.

[1748] Terminal

[1749] The terminal receives the operation program sent from the server and transfers it to the industrial equipment. The terminal has the following functions:

[1750] Receiving and saving programs: Saves program files provided by the server in local storage.

[1751] Linking with industrial equipment: Transfers saved program files to industrial equipment and loads them into the control system. Executes the program according to user instructions.

[1752] Specific examples

[1753] For example, when a user records a specific picking task, the motion capture device receives the motion as data. The server analyzes the data and selects the industrial equipment best suited to the picking task. It then generates a motion program for that equipment and checks the accuracy of the program in a simulation environment. After verifying that there are no problems, the program is provided to a terminal and ultimately implemented in the robot. The user can monitor the robot's operations and make adjustments as necessary to fully automate the picking task.

[1754] Prompt Sentence Examples

[1755] "Please explain the detailed process of a system in which a user wears a motion capture device and performs a picking task, receives the motion data, analyzes the data, selects the most suitable industrial equipment, and generates and provides an operating program for the equipment. We would like a detailed explanation of each step of this system, including the specific processing content and algorithms used."

[1756] This invention enables the rapid and accurate generation and verification of operating programs for industrial equipment, facilitating the automation of work, thereby providing an efficient system that can cope with labor shortages and rising labor costs.

[1757] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1758] Step 1:

[1759] Acquiring motion capture data

[1760] The user wears a motion capture device and performs specific task movements. The device records the user's movements in real time and transmits the data to a server in a digital format (e.g., BVH, FBX).

[1761] Specifically, when a user performs picking work, the series of actions is captured by sensors and cameras.

[1762] Input: User action (e.g. picking work)

[1763] Output: Motion capture data (e.g. BVH, FBX)

[1764] Step 2:

[1765] Receiving and storing data

[1766] The server receives and stores the data transmitted from the motion capture device in real time, and the data is recorded in digital format.

[1767] Input: Motion capture data

[1768] Output: Saved motion capture data

[1769] Specifically, the server saves the data to disk in the specified format (e.g., BVH, FBX).

[1770] Step 3:

[1771] Motion analysis

[1772] The server reads the saved motion capture data and analyzes specific movement patterns, extracting the start and end times of the movement, as well as the joint angles and speeds, and generates keyframes.

[1773] Input: Stored motion capture data

[1774] Output: Analyzed movement patterns and keyframes

[1775] The server uses motion analysis algorithms to extract key motion points, for example, identifying specific motion patterns based on joint angles and movement speed.

[1776] Step 4:

[1777] Industrial Equipment Selection

[1778] Based on the analyzed motion patterns, the server selects from its internal database the industrial equipment that is suitable for the required degree of freedom of movement, payload capacity, and working range.

[1779] Input: Analyzed movement patterns and internal database

[1780] Output: List of selected industrial devices

[1781] Specifically, the server lists devices that meet pre-set filtering conditions and ranks the most suitable devices among them.

[1782] Step 5:

[1783] Generating the operating program

[1784] The server loads a template corresponding to the control language (e.g., URScript, KRL) of the selected industrial equipment and generates an operating program by applying the analyzed operating data to the template.

[1785] Input: Selected industrial equipment, analyzed operation patterns, control language template

[1786] Output: Generated behavior program

[1787] Specifically, the server integrates an instruction set based on an operation pattern into a template, and compiles it to generate a complete program.

[1788] Step 6:

[1789] Verification of the operating program

[1790] The server runs the generated motion program in a simulation environment to verify the accuracy and safety of the motion, which also includes collision detection and timing adjustment.

[1791] Input: Generated behavior program

[1792] Output: Verification results (verification of accuracy and safety)

[1793] Specifically, the server runs an operating program in a virtual environment and checks for any errors or malfunctions.

[1794] Step 7:

[1795] Program Offering

[1796] The server prepares a program file and transfers it to the terminal in order to provide the terminal with the verified operating program.

[1797] Input: Verified working program

[1798] Output: Transferred program file

[1799] Specifically, the server compresses the program file, encrypts it if necessary, and transmits it to the terminal.

[1800] Step 8:

[1801] Execution of the operating program

[1802] The terminal transfers the stored program file to the industrial equipment, loads it into the control system, and then issues an instruction to run the program.

[1803] Input: Transferred program file

[1804] Output: The device behavior caused by the executed program

[1805] Specifically, the terminal uses the control interface of the industrial equipment to load the program and send an execution instruction.

[1806] Step 9:

[1807] Monitor and adjust behavior

[1808] Users monitor the operation of industrial equipment and make fine adjustments or additional program modifications as needed.

[1809] Input: Running industrial equipment operations

[1810] Output: Adjusted behavior, modified program

[1811] Specifically, the user observes the operation in real time and manually corrects or reprograms any malfunctions.

[1812] (Application example 1)

[1813] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1814] In modern factory automation, many procedures rely on manual labor and robot operation programming. With conventional systems, it takes time to generate robot operation programs, and fine-tuning the programs is not easy. As a result, production efficiency can decrease and production can be halted due to errors. There is a need for a system that can solve this problem and enable rapid generation of operation programs and adjustments in real time.

[1815] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1816] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific movement patterns, means for selecting an optimum industrial robot based on the analyzed movement patterns, means for automatically generating an operation program corresponding to the selected robot, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a mobile communication terminal so that a worker can fine-tune it in real time. This enables rapid and accurate generation of an operation program and adjustment in real time.

[1817] Key Word Definitions

[1818] ---

[1819] A "motion capture device" is a device that records the movements of people or objects as digital data.

[1820] The "means for receiving and recording data" is a system that has the function of receiving data transmitted from the motion capture device and storing it.

[1821] "Means for analyzing recorded motion capture data and recognizing specific movement patterns" refers to a system that has the function of analyzing stored data and identifying work movements and types of movements.

[1822] The "means for selecting the optimal industrial robot" is a system that has the function of selecting the robot that can perform the most appropriate operation based on the analyzed operation pattern.

[1823] The "means for automatically generating an operating program" is a system that has the function of automatically creating a program corresponding to the selected robot.

[1824] "Means for verifying in a simulation environment" refers to a system that has the function of testing in a virtual environment whether the generated program is accurate.

[1825] A "mobile communication terminal" is a portable communication device such as a smartphone or tablet.

[1826] "Means that allow workers to make fine adjustments in real time" refers to a system that has the function of immediately making necessary changes and fine adjustments to a verified program.

[1827] A "database" is a system that stores robot specification data and other related information and manages it so that necessary data can be retrieved quickly.

[1828] "Movement freedom" is a concept that refers to the number and range of movements that a robot can perform.

[1829] "Weight capacity" refers to the maximum weight that an industrial robot can safely lift and carry.

[1830] "Work envelope" refers to the physical area within which an industrial robot can move.

[1831] "Means for verifying accuracy and correcting the program if problems are found" refers to a system that has the ability to verify whether the generated program operates correctly and make appropriate corrections when errors or problems are found.

[1832] MODE FOR CARRYING OUT THE INVENTION

[1833] ---

[1834] This invention relates to a system that automatically creates an operating program for an industrial robot by receiving and analyzing motion capture data. This system is comprised of a motion capture device, a server, and a mobile communication terminal, and operates as follows.

[1835] 1. Acquiring motion capture data

[1836] User

[1837] A user wears a motion capture device and performs specific task movements, which are recorded as digital data by the motion capture device.

[1838] server

[1839] The server receives and stores the data sent from the motion capture equipment in real time, and stores this data in a digital format (e.g., BVH, FBX).

[1840] 2. Motion analysis

[1841] server

[1842] The server reads the stored motion capture data and analyzes specific movement patterns, including the start and end times of the movement, joint angles and speeds, etc. Keyframes are generated based on the analyzed data, and sequential movement analysis is performed.

[1843] 3. Industrial robot selection

[1844] server

[1845] Based on the analysis results, the server compares the robot specifications retrieved from the database to select the optimal industrial robot, including the required degrees of freedom of movement, payload capacity, and working range.

[1846] 4. Generating the operating program

[1847] server

[1848] The server loads a template corresponding to the selected robot's control language and applies the analyzed motion data, thereby generating a specific set of control instructions and compiling them into a robot's motion program.

[1849] 5. Verification of the operating program

[1850] server

[1851] The server runs the generated program in a simulation environment to verify its accuracy and safety. Based on the results of the simulation, the program is automatically modified if necessary.

[1852] 6. Program provision and real-time adjustment

[1853] server

[1854] In order to provide the verified operating program to the mobile communication terminal, the server prepares a program file and transfers it to the mobile communication terminal.

[1855] Mobile communication terminal

[1856] The mobile communication terminal stores the program files received from the server and provides an interface that allows the user to fine-tune the operation in real time.

[1857] User

[1858] The user monitors the robot's movements using a mobile communication terminal and fine-tunes the movement program in real time to finalize it. Additional program modifications are also possible as needed.

[1859] Hardware and software used

[1860] Hardware: Mobile communication devices (smartphones and tablets), motion capture devices (e.g., motion sensors), industrial robots

[1861] Software: Server-side technologies (e.g., Django, Flask), simulation environments (e.g., ROS, Gazebo), data parsing libraries (e.g., Python requests, json)

[1862] Specific examples

[1863] Factory workers operate smartphones and use motion capture devices to record specific picking movements. The recorded data is sent to a server, where it is analyzed and the server selects the robot best suited to the picking movement. A motion program for that robot is automatically generated and verified through simulation. After verification, the generated program is sent to a mobile communication device (e.g., a smartphone), where the worker can fine-tune the movement in real time.

[1864] Example prompts to input to the generative AI model

[1865] "We are seeking advice on the design and implementation of a system that analyzes motion capture data and automatically generates motion programs for industrial robots."

[1866] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1867] Program processing flow

[1868] ---

[1869] Step 1:

[1870] A user wears a motion capture device and performs a specific task. The input data here is the user's motion information, and the output is the motion data recorded by the motion capture device.

[1871] Specific behavior:

[1872] The user performs picking operations and welding parts, and the series of movements is recorded as digital data by a motion capture device.

[1873] Step 2:

[1874] The recorded motion capture data is sent to the server in real time. The input data is the digital format data (e.g. BVH, FBX) sent from the motion capture device, and the output is the motion capture data stored on the server.

[1875] Specific behavior:

[1876] The data collected by the motion capture device is sent via wireless or wired communication to a server, which stores it in a database.

[1877] Step 3:

[1878] The server analyzes the stored motion capture data and recognizes specific movement patterns. The input data is the motion capture data stored on the server, and the output is the analyzed movement pattern and related parameters (start time, end time of movement, joint angles, etc.).

[1879] Specific behavior:

[1880] The server uses data analysis algorithms (e.g., machine learning models) to extract specific actions or keyframes within the data.

[1881] Step 4:

[1882] Based on the analysis results, the server selects the optimal industrial robot. The input data is the analyzed motion pattern and the corresponding robot specification data (degree of freedom of motion, payload capacity, working range, etc.), and the output is the selected robot.

[1883] Specific behavior:

[1884] The server checks its internal database, lists the robots that best meet the required specifications, and selects the most suitable robot.

[1885] Step 5:

[1886] The server automatically generates an operating program for the selected robot. The input data is a template based on the control language of the selected robot and analyzed operation patterns, and the output is an operating program containing a specific set of control instructions.

[1887] Specific behavior:

[1888] The server uses a program generation algorithm to generate a control instruction set by applying an operation pattern to a template.

[1889] Step 6:

[1890] The generated behavioral program is verified in a simulation environment. The input data is the generated behavioral program, and the output is the simulation results and the verified program.

[1891] Specific behavior:

[1892] The server runs the generated program using a robot simulation environment (e.g., ROS, Gazebo) to check its safety and accuracy, and automatically corrects the program if necessary.

[1893] Step 7:

[1894] The verified operating program is transferred to the mobile communication terminal, the input data is the verified program file, and the output is the program file stored in the mobile communication terminal.

[1895] Specific behavior:

[1896] The server transmits the program file to the mobile communication terminal, which stores the file in its local memory.

[1897] Step 8:

[1898] The user fine-tunes the robot's movements in real time using a mobile communication terminal. The input data is a verified program file and the user's instructions, and the output is the final modified movement program.

[1899] Specific behavior:

[1900] Workers can fine-tune the operation program via a smartphone or tablet interface to determine the optimal operation program.

[1901] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1902] This invention is a system that combines a system that receives and analyzes data from a motion capture device and automatically generates an operating program for an industrial robot with an emotion engine that recognizes the user's emotions. This system is composed of a motion capture device, a server, a terminal, and an emotion engine, and operates as follows.

[1903] 1. Acquiring motion capture data

[1904] User

[1905] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking). Once the task is initiated, the motion capture device records the user's movements in real time.

[1906] server

[1907] The server receives and records the data sent from the motion capture device in real time, and saves it in formats such as BVH or FBX.

[1908] 2. Motion analysis

[1909] server

[1910] The server loads the saved motion capture data and begins analysis, including the start and end times of the motion, joint angles, and velocity. Based on this, key frames are generated and sequential motion analysis is performed.

[1911] 3. Emotion Recognition by Emotion Engine

[1912] server

[1913] The server utilizes an emotion engine to recognize the user's emotional state. The emotion engine detects the user's facial expressions, voice, and biometric signals, and identifies the user's emotions based on these data. For example, if the user is feeling stressed, the emotional state is recorded.

[1914] 4. Adjusting the movement data

[1915] server

[1916] The server adjusts the analyzed movement patterns based on the emotional state recognized by the emotion engine. For example, if the user is nervous and their movements are awkward, the server corrects those parts to generate a smooth movement pattern.

[1917] 5. Industrial Robot Selection

[1918] server

[1919] The server determines the required degrees of freedom of movement, payload capacity, and working range based on the adjusted motion pattern. It then references an internal database to create a list of industrial robots that meet these requirements. It then selects the most suitable robot from the list and obtains its specification data.

[1920] 6. Generating the operating program

[1921] server

[1922] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, and generates a specific set of control instructions, which are then compiled into the robot's motion program.

[1923] 7. Verification of the operating program

[1924] server

[1925] The server loads the generated motion program into the simulation environment. The simulation is run to verify the accuracy of the motion program. Specifically, collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[1926] 8. Program Offering

[1927] server

[1928] The server prepares to provide the verified operating program to the terminal, converts the program file into an appropriate format, and prepares to transfer it to the terminal.

[1929] Terminal

[1930] The terminal receives the program file sent from the server and stores it in local storage.

[1931] 9. Running the Operation Program

[1932] Terminal

[1933] The terminal transfers the saved program file to the industrial robot, loads the program into the robot's control system, and then issues an instruction to the robot to execute the program.

[1934] User

[1935] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[1936] Specific examples

[1937] For example, when a user records a specific welding task, the motion capture device receives the motion as data. The server analyzes the data and uses an emotion engine to identify areas where the user is feeling stressed. Next, it corrects any motion irregularities caused by stress, selects the most suitable industrial robot based on the adjusted motion data, and generates a motion program for that robot. After verifying the accuracy of the program in a simulation environment, it is provided to the terminal and finally implemented in the robot. The user monitors the robot's motion and makes adjustments as necessary, automating the welding task.

[1938] This invention significantly shortens the implementation time for industrial robot operation programs, enabling rapid and accurate program generation. Furthermore, by taking the user's emotional state into consideration, it is possible to provide more accurate operation programs. This system can provide an effective means of addressing labor shortages and rising labor costs.

[1939] The processing flow will be explained below.

[1940] Step 1: Acquiring motion capture data

[1941] User

[1942] The user wears a motion capture device and performs a specific task (e.g., assembly, welding, picking), which records the user's movements in real time.

[1943] server

[1944] The server receives and records the data sent from the motion capture equipment in real time, and the data is saved in digital formats such as BVH and FBX.

[1945] Step 2: Behavior analysis

[1946] server

[1947] The server reads the stored motion capture data and analyzes it frame by frame, extracting the start and end times of the movements, joint angles, speed, etc., and generates keyframes based on this information. Specific movement patterns are identified through the analysis.

[1948] Step 3: Emotion recognition by the emotion engine

[1949] server

[1950] The server utilizes an emotion engine to recognize the user's emotional state. The emotion engine detects the user's facial expressions, voice, and biometric signals, and identifies the user's emotions based on these data. For example, if the user is feeling stressed, the emotional state is recorded.

[1951] Step 4: Adjusting the behavior data

[1952] server

[1953] The server adjusts the analyzed movement patterns based on the recognized emotional state: if the user is nervous and their movements are awkward, it corrects those parts and converts them into smoother movement patterns.

[1954] Step 5: Selecting an industrial robot

[1955] server

[1956] The server determines the required degree of freedom of movement, payload capacity, and working range based on the adjusted motion pattern, and then references an internal database to create a list of industrial robots that match these requirements and select the most suitable robot.

[1957] Step 6: Generate the operating program

[1958] server

[1959] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, generates a specific set of control instructions, and compiles them into a robot motion program.

[1960] Step 7: Verify the program

[1961] server

[1962] The server loads the generated motion program into the simulation environment. The simulation is run to check the accuracy and safety of the motion program. Collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[1963] Step 8: Program Delivery

[1964] server

[1965] The server prepares to provide the verified operating program to the terminal, converts the program file into an appropriate format, and prepares to transfer it to the terminal.

[1966] Terminal

[1967] The terminal receives the program file sent from the server and stores it in local storage.

[1968] Step 9: Run the action program

[1969] Terminal

[1970] The terminal transfers the saved program file to the industrial robot, which then loads the program into the robot's control system and issues an instruction to run it.

[1971] User

[1972] The user monitors the robot's operation to ensure it is working correctly. If necessary, they can make fine adjustments or additional program modifications. Once proper operation is confirmed, the robot will automate the task.

[1973] Specific examples

[1974] For example, when a user records a specific picking task, the motion capture device receives the movement as data. The server analyzes the data and uses an emotion engine to determine whether the user is feeling stressed. The server then corrects any movement irregularities caused by stress and selects the optimal industrial robot based on the adjusted movement data. A movement program appropriate for the selected robot is generated, its accuracy confirmed through simulation, and then provided to the terminal and installed on the robot. The user monitors the robot's movements and makes adjustments as necessary, allowing the picking task to be accurately automated.

[1975] Example 2

[1976] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1977] Conventional industrial robot motion program generation systems have the risk of motion accuracy decreasing due to the user's emotional state, such as stress or tension. Furthermore, generating and verifying motion programs takes time, resulting in inefficiency. This has led to problems with automation in industrial settings and productivity not improving.

[1978] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1979] In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing specific movement patterns, means for utilizing an emotion engine for recognizing the emotional state of a user, means for adjusting the movement patterns based on the recognized emotional state, means for selecting an optimum industrial robot based on the adjusted movement patterns, means for automatically generating an operation program corresponding to the selected robot, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a terminal. This enables the generation and verification of an operation program quickly and accurately while taking the emotional state of a user into consideration.

[1980] A "motion capture device" is a device that records a user's body movements in real time and transmits the data to a server.

[1981] The "means for receiving and recording data" is a component of the server that has the function of receiving data transmitted from the motion capture device and recording that data.

[1982] The "means for analyzing motion capture data" is a server component that has the function of analyzing the start time, end time, joint angles, speed, etc. of a movement based on recorded motion capture data, and recognizing a specific movement pattern.

[1983] An "emotion engine" is software or hardware that analyzes a user's facial expressions, voice, and biometric signals to identify the user's emotional state.

[1984] The "means for adjusting the movement pattern" is a component of the server that has the function of correcting unnatural parts of the analyzed movement pattern and adjusting it to a smooth movement pattern based on the recognized emotional state of the user.

[1985] An "industrial robot" is an automated device used in industrial settings that requires an operating program to perform a specific task (e.g., welding, assembly, picking).

[1986] The "means for automatically generating an operation program" is a component of the server that has the function of automatically generating an operation program in a control language corresponding to the selected robot based on the adjusted operation data.

[1987] A "simulation environment" is a computer-based system for executing generated operating programs in a virtual space and verifying their accuracy and errors.

[1988] The "means for verifying the operation program" is a component of the server that has the function of checking the accuracy of the operation program generated using the simulation environment and making any necessary corrections.

[1989] A "terminal" is an electronic device that receives an operating program provided by a server, stores it in local storage, and transfers it to an industrial robot for execution.

[1990] This invention is a system that combines a system that receives and analyzes data from a motion capture device and automatically generates an operating program for an industrial robot with an emotion engine that recognizes the user's emotions. This system is composed of a motion capture device, a server, a terminal, and an emotion engine, and operates as follows.

[1991] 1. Generate a program for this system. Specifically, create a program to receive and record data sent from the motion capture device in real time. This data is saved in formats such as BVH or FBX. Next, create a program to analyze the saved motion capture data and extract information such as the start and end times of the movement, joint angles, and speed. This also includes a program that uses an emotion engine to recognize the user's emotional state from facial expressions, voice, and biometric signals. These programs will use a generative AI model to adjust movement patterns and automatically generate optimal movement programs for industrial robots.

[1992] 2. The processing of the generated program is explained in natural language. Specifically, a server is used to receive and record data from the motion capture device. The server saves the recorded data in formats such as BVH or FBX and analyzes the movements. It uses an emotion engine to recognize the user's emotional state and adjusts the movement pattern as needed. Based on the adjusted movement data, the server selects the optimal industrial robot and automatically generates a movement program for that robot. The generated program is verified in a simulation environment to confirm there are no problems before being provided to the terminal. The terminal transfers the provided program to the industrial robot and executes it. The user can monitor the robot's movement and make fine adjustments or additional program modifications as needed.

[1993] 3. Specific examples are added to sentences that explain the program's processing. For example, a user wears a motion capture device and performs a specific welding task. The server receives and records this data in real time in BVH format. The server then analyzes the data, calculating the start and end times of the motion, joint angles, speed, and other parameters, and generates keyframes. The server then uses an emotion engine to analyze the user's facial expressions, voice, and biometric signals to identify stress levels. Based on the stress levels identified by the emotion engine, the server adjusts the analyzed movement patterns. For example, it modifies the user's tense movements to smoother movements. Based on the adjusted movement patterns, the server then identifies the required degrees of freedom of movement, payload capacity, and working range, and selects the optimal industrial robot from a database. The server generates a movement program using a template file corresponding to the selected robot's control language (e.g., Karel). The generated movement program is then verified in a simulation environment to check timing and detect collisions. If verification is successful, the program is transferred to the terminal, which then loads it onto the industrial robot. The user then monitors the robot's movements and makes adjustments as necessary, automating the welding process.

[1994] Specific prompt examples:

[1995] "Please explain in detail, step by step, the procedure for generating an operation program for an industrial robot that modifies its operation pattern based on the stress felt by the user during welding work."

[1996] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1997] Step 1:

[1998] Acquiring motion capture data

[1999] User

[2000] The user wears a motion capture device and performs a specific work motion (e.g., welding). Once the motion is initiated, the motion capture device records the user's movements in real time.

[2001] Input: User's physical movements

[2002] Output: Real-time motion capture data

[2003] Step 2:

[2004] Receiving and recording data

[2005] server

[2006] The server receives the data sent from the motion capture device in real time and records it in BVH or FBX format.

[2007] Input: Motion capture data

[2008] Output: BVH or FBX format files

[2009] Step 3:

[2010] Motion analysis

[2011] server

[2012] The server reads the recorded motion capture data and analyzes the start and end times of the movements, joint angles, speed, etc. Based on this, key frames are generated and detailed movement analysis is performed.

[2013] Input: BVH or FBX format files

[2014] Output: Analysis results (start time, end time, joint angles, speed, keyframes)

[2015] Step 4:

[2016] Emotion recognition by emotion engine

[2017] server

[2018] The server uses an emotion engine to analyze the user's facial expressions, voice, and biometric signals to identify the user's emotional state. If the user is feeling stressed or tense, the server records that emotional state.

[2019] Input: User's facial expressions, voice, and biometric signals

[2020] Output: Emotional state (e.g., stress, tension)

[2021] Step 5:

[2022] Adjusting the movement data

[2023] server

[2024] The server adjusts the analyzed movement patterns based on the recognized emotional state. For example, if the user is nervous and their movements are awkward, it will correct those parts to make them smoother.

[2025] Input: Analysis results, emotional state

[2026] Output: Coordinated movement pattern

[2027] Step 6:

[2028] Industrial robot selection

[2029] server

[2030] The server determines the required degree of freedom of movement, payload capacity, and working range based on the adjusted motion pattern, and then references an internal database to create a list of industrial robots that match these requirements and select the most suitable robot.

[2031] Input: Adjusted movement pattern

[2032] Output: Selected industrial robot

[2033] Step 7:

[2034] Generating the operating program

[2035] server

[2036] The server loads a template file corresponding to the selected robot control language (e.g., Karel, RAPID, TP), applies the adjusted motion data to the template, and generates a specific set of control instructions, which are then compiled into the robot's motion program.

[2037] Input: Selected industrial robot, adjusted motion pattern

[2038] Output: Working program

[2039] Step 8:

[2040] Verification of the operating program

[2041] server

[2042] The server loads the generated motion program into the simulation environment. The simulation is run to verify the accuracy of the motion program. Specifically, collision detection and timing checks are performed to verify that there are no errors. If any problems are found, the necessary corrections are made and the simulation is run again.

[2043] Input: Action program

[2044] Output: Verification results (correctness, presence of errors)

[2045] Step 9:

[2046] Program Offering

[2047] server

[2048] The server provides the verified operating program to the terminal, converts the program file into a format compatible with the robot control system, and prepares it for transfer to the terminal.

[2049] Input: Verified working program

[2050] Output: Program file for distribution

[2051] Terminal

[2052] The terminal receives the program file sent from the server and stores it in local storage.

[2053] Input: Program file to be provided

[2054] Output: Program files in local storage

[2055] Step 10:

[2056] Execution of the operating program

[2057] Terminal

[2058] The terminal transfers the program file stored in the local storage to the industrial robot, loads the program into the robot's control system, and then issues an instruction to the robot to execute the program.

[2059] Input: Program files in local storage

[2060] Output: Running program

[2061] User

[2062] Users can monitor the robot's operation to ensure it is working correctly, and can make fine adjustments or additional program modifications as needed.

[2063] Input: Running action program

[2064] Output: Monitoring results and fine-tuning

[2065] (Application example 2)

[2066] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2067] The rapid generation of efficient and accurate motion programs for industrial robots is a high requirement in many factories. However, conventional systems require a great deal of time and effort to generate and verify motion programs. Furthermore, because they do not take into account the impact of the worker's emotional state on motion, the motions of unskilled or fatigued workers are directly reflected in the program, resulting in the generation of inefficient motion programs. Furthermore, modifying motion programs is complex, making it difficult to generate programs optimized for specific work conditions.

[2068] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and recording data from a motion capture device, means for analyzing the recorded motion capture data and recognizing a specific movement pattern, means for selecting an optimal industrial robot based on the analyzed movement pattern, means for automatically generating an operation program corresponding to the selected robot, means for analyzing the emotional state of a user in real time and adjusting the movement pattern based on the emotional state of the user, means for verifying the generated operation program in a simulation environment, and means for providing the verified operation program to a terminal. This enables rapid automatic generation and verification of accurate and efficient operation programs that take into account the emotional state of a worker.

[2069] Key word definitions

[2070] 1. "Motion capture device" means a device that records a user's movements in real time and transmits the data.

[2071] 2. "Emotional state" refers to emotions such as stress, tension, and joy recognized from the user's facial expressions, voice, and biometric signals.

[2072] 3. "Server" refers to the central processing unit that receives and analyzes motion capture data and emotional data, and generates and verifies the motion program.

[2073] 4. "Movement pattern" refers to the details of a series of movements recorded by a motion capture device, including the start and end times, joint angles, and speed.

[2074] 5. An "industrial robot" is an automated mechanical device that is programmed to perform a specific task.

[2075] 6. An "operation program" is a set of control instructions that enable an industrial robot to perform a specific operation.

[2076] 7. "Simulation environment" is a system for executing and verifying generated operating programs in a virtual space.

[2077] 8. "Terminal" means a device that receives an operating program sent from a server and stores it in a form that can be executed in a local environment.

[2078] 9. "Database" refers to an information collection system that stores robot specification data (such as degrees of freedom of movement, weight capacity, and working range).

[2079] MODE FOR CARRYING OUT THE INVENTION

[2080] Overall system overview

[2081] A system for implementing this invention comprises a motion capture device, a server, a terminal, and an emotion engine. A user wears smart glasses and performs a task, and the motion capture device records motion data in real time. This data is sent to the server, where it is analyzed. The server then uses the emotion engine to recognize the user's emotional state and adjusts the motion data based on this. Based on the adjusted motion data, the server selects an optimal industrial robot and generates an operation program. The generated operation program is verified in a simulation environment, and after any necessary modifications are made based on the results, it is provided to the terminal. Finally, the program is transferred from the terminal to the industrial robot and executed.

[2082] Hardware and software used

[2083] Hardware:

[2084] Smart glasses: Motion capture devices for recording the user's work movements (e.g., Google Glass, Vuzix M400).

[2085] Server: A central processing unit equipped with a high-performance GPU for analyzing motion capture data and emotion data, and for generating and verifying movement programs.

[2086] Industrial robot: An automated mechanical device for performing specific tasks (e.g. FANUC, Yaskawa).

[2087] Terminal: A device that receives an operating program sent from a server.

[2088] software:

[2089] Motion capture software: Software used to capture user movement data (e.g., Vicon Nexus).

[2090] Emotion Analysis Engine (Emotion Engine): Software for recognizing the user's emotional state (e.g., Affectiva SDK).

[2091] Motion analysis software: Software for analyzing and adjusting motion data (e.g., ROS, OpenCV).

[2092] Compiler for industrial robot control languages: Software for compiling the generated operation programs (e.g., FANUC RoboGuide, Yaskawa Motoman).

[2093] Simulation environment: A system for verifying operating programs.

[2094] Processing flow

[2095] 1. Acquiring motion capture data:

[2096] The user wears the smart glasses and performs specific tasks, while sensors built into the glasses record the movement data in real time.

[2097] 2. Sending operational data to the server:

[2098] The movement data is sent to the server, which records it in BVH or FBX format.

[2099] 3. Analysis of behavioral data:

[2100] The server analyzes the recorded data and generates keyframes based on the start time, end time, joint angle, and velocity.

[2101] 4. Recognition of emotional states:

[2102] The server uses an emotion engine to recognize the user's emotional state and analyzes the emotional data.

[2103] 5. Adjusting operational data:

[2104] The server adjusts the movement data based on the recognized emotional state to generate smooth movement patterns.

[2105] 6. Industrial robot selection:

[2106] The server selects the most suitable industrial robot based on the required degree of freedom of movement, payload capacity, and working range.

[2107] 7. Generating the operating program:

[2108] The server automatically generates an operating program for the selected robot.

[2109] 8. Verifying the operation program:

[2110] The server verifies the generated operating program in a simulation environment and makes any necessary modifications.

[2111] 9. Provision of operating programs:

[2112] The modified operating program is transferred to the terminal, which stores the program in its local environment.

[2113] 10. Running the operating program:

[2114] The terminal transfers the program to the industrial robot and causes the robot to execute the program.

[2115] Specific examples

[2116] For example, a worker wears smart glasses and performs welding work, recording his or her movements in real time using motion capture. The server uses an emotion engine to analyze the worker's emotional state (e.g., stress, tension) during the work, corrects for irregularities in movement, and generates an optimal movement pattern. A factory robot's movement program is then automatically generated based on this movement pattern, verified in a simulation environment, and implemented on the factory robot. The worker monitors the final robot movement and makes adjustments as necessary.

[2117] Prompt Sentence Examples

[2118] Prompt the generative AI model:

[2119] "Please explain in detail how to automatically generate a factory robot's motion program based on the motion and emotion data of a worker wearing smart glasses and verify it in a simulation environment."

[2120] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2121] Program processing steps

[2122] Step 1

[2123] Acquiring motion capture data

[2124] A user wears smart glasses and performs specific tasks. The motion capture device (smart glasses) records the user's movements in real time and generates data.

[2125] Input: User action.

[2126] Data processing: Convert motion data into BVH or FBX format.

[2127] Output: Recorded motion data.

[2128] Step 2

[2129] Sending operation data to the server

[2130] The user wears the smart glasses and transmits the acquired data to the server.

[2131] Input: Recorded motion data.

[2132] Data processing: Converting data into a transferable format.

[2133] Output: The operation data sent to the server.

[2134] Step 3

[2135] Analysis of behavioral data

[2136] The server analyzes the received motion data and generates key frames based on information such as start time, end time, joint angle, and speed.

[2137] Input: The motion data sent.

[2138] Data calculation: Analyzing and processing joint angles, speeds, etc.

[2139] Output: Analyzed behavior patterns.

[2140] Step 4

[2141] Recognition of emotional states

[2142] The server uses an emotion engine to recognize the user's emotional state and analyze the emotion data.

[2143] Input: User's facial, voice, and biometric data.

[2144] Data Computing: Identifying and classifying emotional states using a sentiment analysis engine.

[2145] Output: Recognized emotional state data.

[2146] Step 5

[2147] Adjusting the movement data

[2148] The server adjusts the analyzed motion data based on the recognized emotional state, for example, if the user is nervous and making awkward movements, it will correct those parts.

[2149] Input: movement patterns, emotional state data.

[2150] Data processing: Correcting movement patterns based on emotional state.

[2151] Output: Coordinated movement pattern.

[2152] Step 6

[2153] Industrial robot selection

[2154] The server refers to an internal database and selects the most suitable industrial robot based on the required degree of freedom of movement, payload capacity, and working range.

[2155] Input: Calibrated motion pattern, robot specification data in database.

[2156] Data calculation: Algorithm for selecting the best robot.

[2157] Output: Specifications of the selected industrial robot.

[2158] Step 7

[2159] Generating the operating program

[2160] The server generates a motion program for the selected robot, applying the adjusted motion data to an existing template to create a set of control instructions.

[2161] Input: Coordinated movement pattern, control template for the robot.

[2162] Data processing: Applying motion data to a template and generating a set of control instructions.

[2163] Output: The generated behavior program.

[2164] Step 8

[2165] Verification of the operating program

[2166] The server then verifies the generated motion program in a simulation environment, specifically by checking collision detection and timing to ensure there are no errors.

[2167] Input: The generated behavior program.

[2168] Data calculation: Operation verification and error detection through simulation.

[2169] Output: Verified working program, error report.

[2170] Step 9

[2171] Providing operation programs

[2172] The server prepares to provide the verified operating program to the terminal, and transfers the program file to the terminal.

[2173] Input: A verified working program.

[2174] Data processing: Converting program files into the appropriate format.

[2175] Output: The action program file sent to the terminal.

[2176] Step 10

[2177] Execution of the operating program

[2178] The terminal transfers the saved program file to the industrial robot, causing the robot to load and execute the program.

[2179] Input: The operating program file stored on the device.

[2180] Data processing: Application of motion programs to robot control systems.

[2181] Output: A running industrial robot.

[2182] Prompt the generative AI model:

[2183] "Please explain in detail how to automatically generate a factory robot's motion program based on the motion and emotion data of a worker wearing smart glasses and verify it in a simulation environment."

[2184] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2185] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2186] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2187] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2188] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2189] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2190] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2191] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2192] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2193] The emotion identification model 59 inputs user input into a pre-trained neural network, obtain...

Claims

1. means for receiving and recording data from a motion capture device; means for analyzing the recorded motion capture data and recognizing specific movement patterns; A means for selecting an optimal industrial robot based on the analyzed motion pattern; means for automatically generating an operation program corresponding to the selected robot; means for verifying the generated operating program in a simulation environment; means for providing a verified operating program to a terminal; A system including:

2. 2. The system according to claim 1, wherein specification data of the robot is acquired from a database, and a robot is selected based on the required degree of freedom of movement, payload capacity, and working range.

3. 2. The system according to claim 1, further comprising means for verifying the accuracy of the operation program generated in the simulation environment and correcting the program if a problem is found.

Images