system

Abstract

We provide the system. [Solution] A means for filming the actions of an individual performing a task and acquiring video data of said actions, A means for analyzing the video data to extract the characteristics of the movements of an individual's hands, fingers, and tools, A means for generating instructional information to make the procedure of operation reproducible based on the analysis results, Means for converting the aforementioned instruction information into control information for execution in machine operations, Means for controlling a work device that automatically performs work based on the aforementioned control information, A means for providing the aforementioned guidance information and control information to users through an online platform, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a problem that the inheritance of traditional technologies has become difficult due to aging and a shortage of successors. In particular, it is difficult to inherit skilled techniques such as the movements and actions of craftsmen's hands, which may lead to a break in culture. It is necessary to solve this problem and provide a means to effectively transmit technologies to the next generation.

Means for Solving the Problems

[0005] This invention provides a system that captures video data of an individual performing a task, analyzes that video data using AI, and extracts the characteristics of hand, finger, and tool movements. Based on the analysis results, it generates instructional information that makes the procedure of the action reproducible and provides it to users through an online platform, thereby enabling efficient knowledge transfer. Furthermore, it can continuously improve the transfer of skills by collecting feedback from users and updating the system.

[0006] "Movement" refers to the hand and body movements an individual makes when performing a specific action or task.

[0007] "Video data" refers to video information that records an individual's actions in high resolution.

[0008] "Analysis" refers to data processing performed to extract and understand the characteristics of individual movements from video data.

[0009] "Procedure" refers to a series of steps necessary to complete a particular task.

[0010] "Instructional information" refers to guidance and explanations provided to make specific actions or procedures reproducible.

[0011] An "online platform" refers to a digital environment that provides instructional information via the internet and is accessible to users.

[0012] "Feedback" refers to information provided by system users, including opinions, evaluations, and suggestions for improvement. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] ​​​​​​​​​​​​​​​​In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention relates to a system that utilizes generative AI for the purpose of passing on traditional techniques. This system records the traditional work performed by an individual, analyzes the movements using an AI model, and generates instructional information, thereby enabling successors to acquire the same techniques.

[0035] Program processing and explanation

[0036] 1. Data collection (server)

[0037] The server uses a high-resolution camera to record the individual's movements. This is done to capture in detail the series of work processes performed by the craftsman. The server saves the video data to storage and prepares it for analysis.

[0038] 2. Data analysis (server)

[0039] The collected video data is analyzed using a generative AI model installed on the server. The AI ​​model analyzes how individuals manipulate their hands and tools in the video and extracts the characteristics of these movements as numerical data.

[0040] 3. Generation of instructional information (server)

[0041] The server generates reproducible instructional information based on the AI ​​analysis results. This information includes specific explanations of technical procedures, key points of operation, and necessary tools and materials.

[0042] 4. Information provision (terminal)

[0043] Users can access the online platform using their devices. The server uploads generated instructional information to the platform, which users can then use to learn the techniques. The user interface includes video playback functionality, allowing users to learn while visually confirming the actual movements.

[0044] Specific example

[0045] For example, if someone wants to preserve traditional Japanese pottery making, a high-resolution camera would film the entire process, from the craftsman kneading the clay to firing it in the kiln. A server would then analyze this footage, extracting information on the clay's condition, hand pressure and movement, and important points to note. Next, the server would generate a detailed manual based on the information obtained and provide it to the user via a terminal. The user could then learn the technique by reviewing the steps on the platform and imitating the same actions according to the instructions. In this way, the present invention facilitates the transmission of traditional techniques and enables the continuation of culture.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] The server controls high-resolution cameras to capture the craftsman's work process in real time. The server periodically saves the acquired video data to storage and prepares it for analysis.

[0049] Step 2:

[0050] The server preprocesses the video data, removing noise and improving video quality. This process includes image stabilization and adjustment of brightness and contrast.

[0051] Step 3:

[0052] The server activates the AI, which analyzes the pre-processed video data. The AI ​​analyzes each frame to recognize the movements of hands and tools, and extracts the characteristics of these movements as numerical data.

[0053] Step 4:

[0054] The server transcribes the actions and procedures based on the analyzed data. Instructional information includes specific steps, key operational points, and a guide to the materials and tools to be used.

[0055] Step 5:

[0056] The server formats the generated instructional information and uploads it to the online platform. This makes the information accessible from anywhere.

[0057] Step 6:

[0058] Users can log in to the online platform using their devices and search for and view necessary instructional information. Users can learn the technology by viewing the provided manuals and videos.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] In the transmission of traditional techniques, a challenge is to accurately record and analyze individual physical movements and provide an environment that enables effective instruction to the next generation of learners. Conventional methods have problems such as difficulty in adequately conveying the finer points of the techniques and inefficiency in the transmission process.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes means for acquiring high-resolution video data, means for analyzing individual movements using a generative AI model and extracting numerical data, and means for generating reproducible instructional information and providing it through an online learning platform. This makes it possible to accurately record and analyze the detailed movements of traditional techniques and provide learners with specific and visual instructional information.

[0064] "High-resolution video data" refers to high-quality video information that can capture the movements of an individual performing a task in detail.

[0065] A "generative AI model" refers to an artificial intelligence algorithm used to analyze video data and extract characteristics of individual movements.

[0066] "Numerical data" refers to the numerical representation of behavioral characteristics extracted by the generative AI model, and is used for analyzing those behaviors.

[0067] "Instructional information" refers to educational information that includes specific explanations of reproducible technical procedures, key points of movement, and necessary tools.

[0068] An "online learning platform" refers to a digital environment where users can access instructional information and learn via the internet.

[0069] This invention is a system aimed at the transmission of traditional techniques, in which a server, terminal, and user work in cooperation with each other. The server first uses a high-resolution camera to film the actions of an individual performing a traditional task. The video data requires detailed image quality to accurately record the movements of the craftsman's hands and the operation of tools. The filmed data is stored on the server. Next, the server analyzes this video data using a generative AI model. The generative AI model extracts the characteristics of the actions in the video as numerical data and analyzes the essential elements of the technique.

[0070] Next, the server generates reproducible instructional information based on the AI ​​analysis results. This instructional information includes specific steps for reproducing the technique, key points of operation, and information on necessary tools and materials. This allows users to acquire the technique in a detailed and easy-to-understand manner.

[0071] The generated instructional information is uploaded to an online learning platform by the server. Users access this platform via their devices and learn from the provided instructional information. Users can visually confirm the operation of the technology using the video playback function and progress through their learning.

[0072] Specific example

[0073] For example, when learning the techniques of Japanese pottery making, the server films the process of a craftsman kneading clay and firing it in a kiln. The generative AI model analyzes this footage and extracts detailed characteristics of the clay's condition, hand pressure, and movement. Based on this, the server generates instructional information on specific steps for shaping the clay and the tools to be used, and provides this information to users through an online platform, enabling users to learn pottery making techniques.

[0074] An example of a prompt message is: "We will build a system that uses AI to analyze the hand movements and pressure when kneading clay for pottery making, and generate detailed instructional information." This will enable users to learn traditional techniques precisely and efficiently.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The server activates a high-resolution camera and films the actions of an individual performing a traditional technique. The input is the individual's actions, and the output is high-resolution video data. The server uses this video data to obtain detailed action information necessary for later analysis. Specifically, the camera tracks the person's hands and the movement of tools, continuously collecting data.

[0078] Step 2:

[0079] The server saves the captured video data to storage. The input is the high-resolution video data obtained in step 1, and the output is the video file stored in the database. The purpose of this step is to ensure that the video data necessary for analysis is reliably saved and prepared for subsequent processing.

[0080] Step 3:

[0081] The server prepares a generation AI model and loads the stored video data. The input is video data stored in storage. The generation AI model analyzes the video data and extracts important features of the movements as numerical data. Specifically, the AI ​​model identifies and analyzes the position of hands and tools, the speed of movement, the angle, etc., for each frame.

[0082] Step 4:

[0083] The server generates instructional information based on the numerical data it receives. The input is the analysis results from the AI ​​model, and the output is reproducible instructional information. This instructional information includes specific steps for the technique, points to note when operating it, and a list of necessary tools and materials. The server combines this information and presents it in a format that is easy for learners to understand.

[0084] Step 5:

[0085] The server uploads the generated instructional information to the online platform. The input is the generated instructional information, and the output is instructional information in a format viewable on the platform. The server optimizes the system so that the information is displayed accurately and is easily accessible to users.

[0086] Step 6:

[0087] Users access an online platform using their devices and view instructional information. The input is online instructional information, and the output is the learner's acquisition of skills. Users visually confirm actions through video playback and practice and master skills based on the provided instruction. This allows users to progress in their learning while referring to actual actions.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] The transmission of traditional techniques faces the challenge of difficulty in incorporating them into modern, efficient mass production technologies. Furthermore, in modern companies where standardization and automation are increasingly promoted, there is a demand to efficiently replicate the manual work of highly skilled traditional craftsmen using robots and machinery. However, mechanically reproducing the movements of traditional techniques requires accurately capturing, analyzing, and standardizing the intuitive skills of human beings, and methods for achieving this are needed.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes means for capturing the movements of an individual performing a task and acquiring video data of said movements; means for analyzing the video data to extract characteristics of the individual's hand, finger, and tool movements; means for generating instructional information to make the procedure of the work reproducible based on the analysis results; means for converting the instructional information into control information for execution in machine-based work; means for controlling a work device that automatically performs the work based on the control information; and means for providing the instructional information and control information to users through an online platform. This enables the efficient inheritance of traditional techniques and the realization of work standardization and automation in factories.

[0093] "Work" refers to a series of actions or operations performed by a human or machine to achieve a specific purpose.

[0094] "Motion data" refers to video information that records the movements of an individual's hands and fingers, as well as the process of operating tools while performing a task.

[0095] "Video analysis" is the process of analyzing acquired motion data and numerically extracting the characteristics of individual movements.

[0096] "Instructional information" refers to specific guidelines and manuals generated based on analysis results to make the operation procedures reproducible.

[0097] "Control information" refers to specific operating instructions created based on guidance information, which enable machines and work devices to automatically perform their actions.

[0098] A "working device" refers to a machine or equipment that automatically performs a specific task based on control information.

[0099] An "online platform" is a digital infrastructure that provides information in a format that users can access via the internet.

[0100] To implement this invention, the entire system consists of several main components. First, the server uses a high-resolution camera to capture the movements of an individual performing a task, thereby acquiring movement data. This data is processed within the server via video analysis software and used to extract numerical data on the characteristics of the hands, fingers, and tools. This analysis utilizes existing generative AI models to generate instructional information for automatically reproducing the steps of the movement.

[0101] Next, the server converts the instruction information into control information and automatically operates the work equipment based on it. Specific hardware examples include factory robots and programmable work machines for motion control. These precisely execute the actions instructed by the control information, recreating traditional techniques.

[0102] Furthermore, the generated instructional and control information is uploaded to an online platform for user access. Users can connect to this platform using their devices to view and learn from the provided information. Specific examples include providing operation manuals for replicating pottery molding techniques with robots, and standard action sequences of craftsmanship as programs.

[0103] By utilizing generative AI models, traditional techniques can be adapted to modern manufacturing environments, providing a system that supports the transmission of diverse technologies. Specifically, one example involves instructing the AI ​​model to perform a task using a prompt such as, "Analyze the movements of manual weaving from this high-resolution video and generate a reproducible robot program."

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The server uses a high-resolution camera to capture the actions of the individual performing the task and acquires a video file as action data. This video data is the input, and the server saves this data to storage for use in subsequent processing steps.

[0107] Step 2:

[0108] The server inputs the acquired video data into a generating AI model to analyze the characteristics of hand, finger, and tool movements. In this step, data processing is performed to extract feature quantities for each part of the movement from the input video data, and feature data is obtained as output.

[0109] Step 3:

[0110] The server generates instructional information based on the analyzed feature data. Here, data calculations are performed to generate guidelines and operation manuals that allow for the reproduction of work procedures based on the feature data, and this content is output as instructional information.

[0111] Step 4:

[0112] The server converts the generated instruction information into control information. This process involves converting the content of the instruction information into a machine-executable format and performing data conversion to generate control data. The output is a specific set of instructions for controlling the work device.

[0113] Step 5:

[0114] The server controls the work equipment based on control information, automatically executing the actual work. Using control data as input, the equipment starts the specified operation and performs the desired task. The output is the result of the performed work.

[0115] Step 6:

[0116] The server uploads the generated instructional and control information to an online platform, and users access this information using their terminals. After the data is uploaded, the information is provided to support the user's learning, and the output is the user's ability to view the information and learn how to operate it.

[0117] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0118] This invention combines a system that analyzes individual actions and generates instructional information necessary for the transmission of skills with an emotion engine that recognizes the user's emotions and provides instruction tailored to their state. By combining advanced generative AI and emotion recognition technology, this system aims to efficiently and effectively transmit traditional skills to the next generation.

[0119] Program processing and explanation

[0120] 1. Acquisition of video data (server)

[0121] The server uses high-resolution cameras to film the craftsman at work and acquires video data. This data captures the movements of the craftsman's hands and tools in detail.

[0122] 2. Analysis of operation (server)

[0123] The server uses a generated AI model to analyze video data and extract motion characteristics. The sequence and key points of the motion are recorded as numerical data, and standardized motion patterns are created.

[0124] 3. Generation of instructional information (server)

[0125] The server generates instructional information based on the analysis results. This information is presented to the user as learning material delivered via the terminal, making it possible to reproduce traditional techniques.

[0126] 4. Recognition of emotions (device)

[0127] The emotion engine built into the device acquires emotional information through the user's video or audio. This is achieved, for example, by analyzing the user's facial expressions and tone of voice using the camera and microphone.

[0128] 5. Optimization of instructional information (server)

[0129] Based on the analysis results of the emotion engine, the server dynamically adjusts the guidance information according to the user's current emotional state. For example, if it determines that the user is not understanding the information, it will provide additional explanations or video clips.

[0130] 6. Information provision and learning (user)

[0131] Users access instructional information from their devices and view implementation procedures and details. Information appropriately tailored by an emotion engine allows users to acquire skills more effectively.

[0132] Specific example

[0133] For example, if a user wants to learn the procedures of the Japanese tea ceremony, this system first films the hands of a tea ceremony master with a high-resolution camera and analyzes their movements. Then, using the generated instructional information, the user begins learning on their home device. During learning, the emotion engine scans the user's face with the camera, and if it detects low concentration, the server immediately provides additional videos and explanations to facilitate learning. In this way, the system has comprehensive functions to maximize the effectiveness of skill acquisition.

[0134] The following describes the processing flow.

[0135] Step 1:

[0136] The server activates a high-resolution camera and records the craftsman's movements as a video in real time. The acquired video data is saved to the server's storage.

[0137] Step 2:

[0138] The server preprocesses the stored video data, removing noise and stabilizing the frames to prepare it for analysis.

[0139] Step 3:

[0140] The server analyzes pre-processed video data using an AI model it has generated. It identifies the movements of hands, fingers, and tools, and extracts the characteristics of these movements as numerical data.

[0141] Step 4:

[0142] The server extracts data to create standardized patterns of operation and specific procedures. Based on this, customized instructional information is generated.

[0143] Step 5:

[0144] The emotion engine built into the device acquires the user's facial expressions and voice as input and analyzes their emotions in real time. The user's emotional state is converted into data and sent to the server.

[0145] Step 6:

[0146] The server receives analysis data from the emotion engine and dynamically adjusts the content and format of the instructional information according to the user's emotional state. This maximizes learning effectiveness.

[0147] Step 7:

[0148] Users access customized instructional information through their devices and learn skills using videos and guides. As users progress, they can provide feedback to the system with additional questions and requests.

[0149] (Example 2)

[0150] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0151] In modern times, the transmission of traditional techniques and skills is becoming increasingly difficult as those with specialized knowledge and experience age, making it challenging to properly pass them on to the next generation. Furthermore, accurately understanding and responding to the emotional states of individual learners during the learning process is difficult, hindering effective learning outcomes. Moreover, systems for effectively utilizing real-time feedback from learners to improve teaching content and models are not yet fully established.

[0152] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0153] In this invention, the server includes means for capturing video information of the actions of a subject performing a task, means for analyzing the video information and extracting the characteristics of the subject's movements, means for generating instructional information based on the analysis results, and means for monitoring the user's emotional state using an emotion analysis function and optimizing the instructional information. This enables the effective transmission of traditional techniques and skills to the next generation, provides appropriate instruction tailored to the individual emotional state of each learner, and is expected to improve learning efficiency.

[0154] "The entity performing the work" refers to an individual or group that implements a specific skill or technique, and is the subject of recording and analysis of their actions and techniques.

[0155] "Visual information" refers to visual data captured by cameras and video equipment, used for visually analyzing technology and actions.

[0156] "Analysis" is the process of extracting the subject's movements and tool movement patterns from acquired video information, and includes the process of quantifying technical characteristics.

[0157] "Instructional information" refers to information generated based on analysis results to enable the reproduction of technical and skill procedures, and is provided in text, audio, and video formats.

[0158] The "emotion analysis function" is a technology that analyzes the user's facial expressions and tone of voice in real time to determine their emotional state, and is used to optimize instructional information.

[0159] "Communication technology" refers to technology for transmitting data between one or more devices, and is used as a means of providing instructional information to users in remote locations.

[0160] "Response" refers to data obtained from user feedback and reactions, which is used to improve the system's guidance information and AI models.

[0161] This invention is a system for efficiently transmitting traditional techniques and skills held by users to the next generation, and includes information exchange between servers, terminals, and users. To implement this, the coordination of various hardware and software components is essential.

[0162] First, the server uses a high-resolution camera to film the worker's work and acquire video information. This captures the details of the work and collects data necessary for subsequent analysis. This video data is a crucial element for motion analysis, and the analysis is performed using a generative AI model. The generative AI model processes the captured video, extracts and standardizes motion features and patterns. For example, it can analyze the hand movements when shaping pottery and provide a guide for beginners to reproduce the procedure.

[0163] Next, the server generates instructional information based on the analysis results obtained. This instructional information is expressed in various formats such as text, images, audio, and video, and is transmitted to the terminal using communication technology. Users can access this instructional information through their terminal and learn the necessary skills.

[0164] Furthermore, the device incorporates an emotion analysis function. This is achieved by the device capturing the user's facial expressions and voice tone through its camera and microphone, and analyzing the resulting emotion data. This emotion data is sent to a server and used to adjust instructional information to improve the user's learning efficiency. If the server determines that the user is tired or lacking understanding, it can automatically provide additional explanations or specific examples.

[0165] As a concrete example, when assisting with the learning of the tea ceremony, the system first films the movements of the tea ceremony instructor, analyzes them, and generates a procedure. Then, when the user begins learning this procedure at home, if they show signs of confusion, the system provides additional detailed explanations. In this way, the user can be expected to learn the movements of the tea ceremony efficiently and effectively.

[0166] An example of a prompt message is, "Analyze high-resolution motion video and generate standardized motion patterns." This instructs the system to begin the analysis process using a generative AI model.

[0167] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0168] Step 1:

[0169] The server uses a high-resolution camera to record the technician's work in real time. The input is the video data captured by the camera. The server saves this data to storage. The output is video information that captures the detailed actions. At this stage, specific actions such as adjusting the camera's position and angle during shooting to create the optimal shooting environment are taken.

[0170] Step 2:

[0171] The server inputs the stored video information into the generating AI model and begins the analysis. The input is the acquired video information. The server uses the generating AI model to analyze each frame and extract the characteristics of the movements. The output is numerical data such as the trajectory and speed of the movements and the angles of the joints. Specifically, it identifies hand movements and tool manipulations in the video and quantifies them as data.

[0172] Step 3:

[0173] The server generates instructional information based on the analysis results. The input is numerical data extracted through analysis. The server processes this data using an algorithm to generate learning materials such as text, images, audio, and video. The output is instructional material that shows specific procedures. Specific actions here include text editing, audio narration creation, and diagram preparation.

[0174] Step 4:

[0175] The device uses the user's camera and microphone to collect the user's facial expressions and voice through emotion analysis. The input is real-time facial images and voice data. The device activates an emotion engine to determine the user's emotional state. The output is an emotion label such as smile or confusion. Specifically, it analyzes and classifies whether the user is smiling or has a stern expression.

[0176] Step 5:

[0177] The server receives the results of sentiment analysis and dynamically adjusts the instructional information. The input is the user's sentiment label. The server changes the content and format of the instructional information according to the user's level of understanding and emotional state. The output is the adjusted instructional information. For example, if a lack of understanding is detected, the server selects and provides supplementary explanations or additional videos.

[0178] Step 6:

[0179] Users view instructional information provided on their devices and engage in learning. The input is the instructional information itself. Users refer to this at their own pace and acquire skills. The output is the acquired techniques and skills. Specific actions include users playing videos on the screen or reading text.

[0180] (Application Example 2)

[0181] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0182] In factory and other industrial settings, there is a challenge in effectively transferring the movements of skilled workers to machines. Furthermore, there is a lack of means to flexibly optimize machine operation according to the situation at hand, thus creating a demand for efficient and safe work practices.

[0183] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0184] In this invention, the server includes means for capturing an individual's movements and acquiring video data of those movements, means for analyzing the video data to extract characteristics of the movements, and means for dynamically optimizing the machine's movements based on the acquired emotional information. This makes it possible to transfer the skills of skilled workers to the machine while optimizing the movements according to the work environment.

[0185] "Action recording" refers to the act of recording an individual's work process as video footage.

[0186] "Acquiring video data" refers to the process of saving the captured video footage of the action and sending it to a server in a format that can be analyzed.

[0187] "Extracting movement characteristics" refers to the act of analyzing video data to identify specific movement patterns or elements from an individual's body movements and how they use equipment.

[0188] "Generating instructional information" is the process of creating information and procedures based on the characteristics of an action, enabling others to reproduce similar actions.

[0189] "Emotional information" refers to data that represents an individual's psychological state and mood, and is analyzed from facial expressions and voice tone acquired through cameras and microphones.

[0190] "Dynamic optimization of operation" refers to using acquired emotional information to adjust machine operation in real time according to the work environment and situation, thereby improving efficiency and safety.

[0191] The system for implementing this invention consists of multiple elements. First, the server uses a high-resolution camera to capture an individual's movements and acquire video data. Next, the server analyzes this video data using a generative AI model to extract key features of the movements. Based on the results of this analysis, information for movement instruction is generated and provided to the user through an online platform.

[0192] Furthermore, the terminal is equipped with emotion recognition capabilities, which analyze the user's facial expressions and voice through the camera and microphone to obtain emotional information. This emotional information is sent to a server, which dynamically optimizes the machine's operation based on it. For example, if the server determines that a worker's attention is distracted in the work environment, it will temporarily slow down the operation of the automated machine to ensure safety. In addition, newly generated operation patterns are continuously adjusted based on the results of the emotional information analysis.

[0193] For example, in the assembly of precision equipment, the highly accurate procedures of skilled workers are captured by cameras and learned as robotic movement patterns. On the other hand, if the emotion engine detects that a worker is fatigued, the work speed is adjusted as needed, thereby achieving both production efficiency and safety.

[0194] An example of a prompt statement would be: "Develop a system that allows a robot to learn the movements of skilled factory workers and perform tasks safely and efficiently. The robot is required to read the worker's emotions and optimize the workflow."

[0195] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0196] Step 1:

[0197] The server uses high-resolution cameras to film the movements of skilled workers in the factory. The input is real-time video of the work, and this video data is acquired by the server. This data includes the movements of the workers and detailed operation of the equipment they use.

[0198] Step 2:

[0199] The server analyzes video data acquired using a generative AI model. The input is video data, and the AI ​​model extracts features such as body movements and equipment handling. The output is represented as numerical data in the form of standardized motion patterns. Here, the flow of movement and key points are quantified.

[0200] Step 3:

[0201] The server generates instructional information based on the analyzed behavioral patterns. The input is the extracted behavioral characteristics, and the output is step-by-step instructional information to help other users reproduce similar tasks. This information is provided online through the platform.

[0202] Step 4:

[0203] The device uses a camera and microphone to acquire user emotional information. Input consists of the user's facial expressions and voice, and an emotion engine analyzes this data. Output is emotional information indicating the user's psychological state. This serves as an important indicator of work-related stress levels and concentration.

[0204] Step 5:

[0205] The server receives emotional information transmitted from the terminal and dynamically optimizes its operation patterns. The input is emotional information, and the output is optimized machine operation commands. For example, if the user indicates fatigue, the server adjusts the machine's operation to ensure a safe and comfortable working environment.

[0206] Step 6:

[0207] The user can perform tasks based on optimized operational commands. The input is optimized guidance information and machine operation, and the output is the result of efficient and safe work.

[0208] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0209] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0210] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0211] [Second Embodiment]

[0212] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0213] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0214] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0215] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0216] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0217] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0218] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0219] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0220] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0221] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0222] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0223] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0224] This invention relates to a system that utilizes generative AI for the purpose of passing on traditional techniques. This system records the traditional work performed by an individual, analyzes the movements using an AI model, and generates instructional information, thereby enabling successors to acquire the same techniques.

[0225] Program processing and explanation

[0226] 1. Data collection (server)

[0227] The server uses a high-resolution camera to record the individual's movements. This is done to capture in detail the series of work processes performed by the craftsman. The server saves the video data to storage and prepares it for analysis.

[0228] 2. Data analysis (server)

[0229] The collected video data is analyzed using a generative AI model installed on the server. The AI ​​model analyzes how individuals manipulate their hands and tools in the video and extracts the characteristics of these movements as numerical data.

[0230] 3. Generation of instructional information (server)

[0231] The server generates reproducible instructional information based on the AI ​​analysis results. This information includes specific explanations of technical procedures, key points of operation, and necessary tools and materials.

[0232] 4. Information provision (terminal)

[0233] Users can access the online platform using their devices. The server uploads generated instructional information to the platform, which users can then use to learn the techniques. The user interface includes video playback functionality, allowing users to learn while visually confirming the actual movements.

[0234] Specific example

[0235] For example, if someone wants to preserve traditional Japanese pottery making, a high-resolution camera would film the entire process, from the craftsman kneading the clay to firing it in the kiln. A server would then analyze this footage, extracting information on the clay's condition, hand pressure and movement, and important points to note. Next, the server would generate a detailed manual based on the information obtained and provide it to the user via a terminal. The user could then learn the technique by reviewing the steps on the platform and imitating the same actions according to the instructions. In this way, the present invention facilitates the transmission of traditional techniques and enables the continuation of culture.

[0236] The following describes the processing flow.

[0237] Step 1:

[0238] The server controls high-resolution cameras to capture the craftsman's work process in real time. The server periodically saves the acquired video data to storage and prepares it for analysis.

[0239] Step 2:

[0240] The server preprocesses the video data, removing noise and improving video quality. This process includes image stabilization and adjustment of brightness and contrast.

[0241] Step 3:

[0242] The server activates the AI, which analyzes the pre-processed video data. The AI ​​analyzes each frame to recognize the movements of hands and tools, and extracts the characteristics of these movements as numerical data.

[0243] Step 4:

[0244] The server transcribes the actions and procedures based on the analyzed data. Instructional information includes specific steps, key operational points, and a guide to the materials and tools to be used.

[0245] Step 5:

[0246] The server formats the generated instructional information and uploads it to the online platform. This makes the information accessible from anywhere.

[0247] Step 6:

[0248] Users can log in to the online platform using their devices and search for and view necessary instructional information. Users can learn the technology by viewing the provided manuals and videos.

[0249] (Example 1)

[0250] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0251] In the transmission of traditional techniques, a challenge is to accurately record and analyze individual physical movements and provide an environment that enables effective instruction to the next generation of learners. Conventional methods have problems such as difficulty in adequately conveying the finer points of the techniques and inefficiency in the transmission process.

[0252] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0253] In this invention, the server includes means for acquiring high-resolution video data, means for analyzing individual movements using a generative AI model and extracting numerical data, and means for generating reproducible instructional information and providing it through an online learning platform. This makes it possible to accurately record and analyze the detailed movements of traditional techniques and provide learners with specific and visual instructional information.

[0254] "High-resolution video data" refers to high-quality video information that can capture the movements of an individual performing a task in detail.

[0255] A "generative AI model" refers to an artificial intelligence algorithm used to analyze video data and extract characteristics of individual movements.

[0256] "Numerical data" refers to the numerical representation of behavioral characteristics extracted by the generative AI model, and is used for analyzing those behaviors.

[0257] "Instructional information" refers to educational information that includes specific explanations of reproducible technical procedures, key points of movement, and necessary tools.

[0258] An "online learning platform" refers to a digital environment where users can access instructional information and learn via the internet.

[0259] This invention is a system aimed at the transmission of traditional techniques, in which a server, terminal, and user work in cooperation with each other. The server first uses a high-resolution camera to film the actions of an individual performing a traditional task. The video data requires detailed image quality to accurately record the movements of the craftsman's hands and the operation of tools. The filmed data is stored on the server. Next, the server analyzes this video data using a generative AI model. The generative AI model extracts the characteristics of the actions in the video as numerical data and analyzes the essential elements of the technique.

[0260] Next, the server generates reproducible instructional information based on the AI ​​analysis results. This instructional information includes specific steps for reproducing the technique, key points of operation, and information on necessary tools and materials. This allows users to acquire the technique in a detailed and easy-to-understand manner.

[0261] The generated instructional information is uploaded to an online learning platform by the server. Users access this platform via their devices and learn from the provided instructional information. Users can visually confirm the operation of the technology using the video playback function and progress through their learning.

[0262] Specific example

[0263] For example, when learning the techniques of Japanese pottery making, the server films the process of a craftsman kneading clay and firing it in a kiln. The generative AI model analyzes this footage and extracts detailed characteristics of the clay's condition, hand pressure, and movement. Based on this, the server generates instructional information on specific steps for shaping the clay and the tools to be used, and provides this information to users through an online platform, enabling users to learn pottery making techniques.

[0264] An example of a prompt message is: "We will build a system that uses AI to analyze the hand movements and pressure when kneading clay for pottery making, and generate detailed instructional information." This will enable users to learn traditional techniques precisely and efficiently.

[0265] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0266] Step 1:

[0267] The server activates a high-resolution camera and films the actions of an individual performing a traditional technique. The input is the individual's actions, and the output is high-resolution video data. The server uses this video data to obtain detailed action information necessary for later analysis. Specifically, the camera tracks the person's hands and the movement of tools, continuously collecting data.

[0268] Step 2:

[0269] The server saves the captured video data to storage. The input is the high-resolution video data obtained in step 1, and the output is the video file stored in the database. The purpose of this step is to ensure that the video data necessary for analysis is reliably saved and prepared for subsequent processing.

[0270] Step 3:

[0271] The server prepares a generation AI model and loads the stored video data. The input is video data stored in storage. The generation AI model analyzes the video data and extracts important features of the movements as numerical data. Specifically, the AI ​​model identifies and analyzes the position of hands and tools, the speed of movement, the angle, etc., for each frame.

[0272] Step 4:

[0273] The server generates instructional information based on the numerical data it receives. The input is the analysis results from the AI ​​model, and the output is reproducible instructional information. This instructional information includes specific steps for the technique, points to note when operating it, and a list of necessary tools and materials. The server combines this information and presents it in a format that is easy for learners to understand.

[0274] Step 5:

[0275] The server uploads the generated instructional information to the online platform. The input is the generated instructional information, and the output is instructional information in a format viewable on the platform. The server optimizes the system so that the information is displayed accurately and is easily accessible to users.

[0276] Step 6:

[0277] Users access an online platform using their devices and view instructional information. The input is online instructional information, and the output is the learner's acquisition of skills. Users visually confirm actions through video playback and practice and master skills based on the provided instruction. This allows users to progress in their learning while referring to actual actions.

[0278] (Application Example 1)

[0279] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0280] The inheritance of traditional technologies has the problem that it is difficult to incorporate them into modern efficient mass production technologies. Also, in modern enterprises where the promotion of work standardization and automation is progressing, there is a demand to efficiently reproduce the manual work of craftsmen with advanced traditional technologies by robots or mechanical devices. However, in order to mechanically reproduce the operation of traditional technologies, it is necessary to accurately capture, analyze, and standardize the human sensory technologies, and a method to achieve these is required.

[0281] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0282] In this invention, the server includes means for photographing the actions of an individual performing work and acquiring video data of the actions, means for analyzing the video data to extract the characteristics of the movements of the individual's hands, fingers, and tools, means for generating guidance information for making the operation procedure reproducible based on the analysis result, means for converting the guidance information into control information for execution in work by a machine, means for controlling a work device that automatically performs work based on the control information, and means for providing the guidance information and control information to a user through an online platform. Thereby, traditional technologies can be efficiently inherited, and work standardization and automation in a factory can be realized.

[0283] "Work" refers to a series of actions and operations performed by humans or machines to achieve a specific purpose.

[0284] "Motion data" refers to video information that records the movements of an individual's hands and fingers and the operation process of tools when performing work. <0,

[0285] "Video analysis" is a process of analyzing the acquired motion data and numerically extracting the characteristics of individual movements.

[0286] "Guidance information" refers to specific guidelines and manuals generated based on the analysis result for making the operation procedure reproducible.

[0287] "Control information" refers to specific operation instructions for a machine or working device to automatically execute operations, created based on guiding information.

[0288] "Working device" refers to a machine or equipment for automatically executing a specific task based on control information.

[0289] "Online platform" refers to a digital infrastructure for providing information in a form accessible to users via the Internet.

[0290] To implement this invention, the entire system is composed of several main components. First, the server uses a high-resolution camera to capture the actions of an individual performing a task, thereby obtaining action data. This data is processed within the server via video analysis software and used to extract the characteristics of the hands, fingers, and tools as numerical data. Existing generative AI models are utilized in this analysis to generate guiding information for automatically reproducing the action procedures.

[0291] Next, the server converts the guiding information into control information and automatically operates the working device based on it. Specific hardware includes factory robots for motion control and programmable working machines. These precisely execute the actions instructed by the control information and reproduce traditional techniques.

[0292] Furthermore, the generated guiding information and control information are uploaded to an online platform so that users can access them. Users can connect to this platform using a terminal and view and learn the provided information. Specific examples include an operation manual for a robot to reproduce pottery shaping techniques and providing the standard action sequences of craftsman skills as programs.

[0293] By utilizing generative AI models, traditional techniques can be adapted to modern manufacturing environments, providing a system that supports the transmission of diverse technologies. Specifically, one example involves instructing the AI ​​model to perform a task using a prompt such as, "Analyze the movements of manual weaving from this high-resolution video and generate a reproducible robot program."

[0294] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0295] Step 1:

[0296] The server uses a high-resolution camera to capture the actions of the individual performing the task and acquires a video file as action data. This video data is the input, and the server saves this data to storage for use in subsequent processing steps.

[0297] Step 2:

[0298] The server inputs the acquired video data into a generating AI model to analyze the characteristics of hand, finger, and tool movements. In this step, data processing is performed to extract feature quantities for each part of the movement from the input video data, and feature data is obtained as output.

[0299] Step 3:

[0300] The server generates instructional information based on the analyzed feature data. Here, data calculations are performed to generate guidelines and operation manuals that allow for the reproduction of work procedures based on the feature data, and this content is output as instructional information.

[0301] Step 4:

[0302] The server converts the generated instruction information into control information. This process involves converting the content of the instruction information into a machine-executable format and performing data conversion to generate control data. The output is a specific set of instructions for controlling the work device.

[0303] Step 5:

[0304] The server controls the working device based on the control information and automatically executes the actual work. Using the control data as input, the device starts the specified operation and performs the target work. The output is the result of the performed work.

[0305] Step 6:

[0306] The server uploads the generated guidance information and control information to the online platform, and the user accesses this information using the terminal. After the data is uploaded, information is provided for the purpose of learning support on the user side, and the output is the user's viewing of the information and acquisition of the operation method.

[0307] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0308] In addition to a system that analyzes an individual's actions and generates the guidance information necessary for inheriting the technology, the present invention combines an emotion engine that recognizes the user's emotion and provides guidance according to the state. This system aims to efficiently and effectively convey traditional technologies to the next generation by combining advanced generation AI and emotion recognition technology.

[0309] Processing and Explanation of the Program

[0310] 1. Acquisition of Video Data (Server)

[0311] The server shoots the work of the craftsman with a high-resolution camera and acquires video data. This data captures the movements of the craftsman's hands and tools in detail.

[0312] 2. Analysis of Movements (Server)

[0313] The server uses a generated AI model to analyze video data and extract motion characteristics. The sequence and key points of the motion are recorded as numerical data, and standardized motion patterns are created.

[0314] 3. Generation of instructional information (server)

[0315] The server generates instructional information based on the analysis results. This information is presented to the user as learning material delivered via the terminal, making it possible to reproduce traditional techniques.

[0316] 4. Recognition of emotions (device)

[0317] The emotion engine built into the device acquires emotional information through the user's video or audio. This is achieved, for example, by analyzing the user's facial expressions and tone of voice using the camera and microphone.

[0318] 5. Optimization of instructional information (server)

[0319] Based on the analysis results of the emotion engine, the server dynamically adjusts the guidance information according to the user's current emotional state. For example, if it determines that the user is not understanding the information, it will provide additional explanations or video clips.

[0320] 6. Information provision and learning (user)

[0321] Users access instructional information from their devices and view implementation procedures and details. Information appropriately tailored by an emotion engine allows users to acquire skills more effectively.

[0322] Specific example

[0323] For example, if a user wants to learn the procedures of the Japanese tea ceremony, this system first films the hands of a tea ceremony master with a high-resolution camera and analyzes their movements. Then, using the generated instructional information, the user begins learning on their home device. During learning, the emotion engine scans the user's face with the camera, and if it detects low concentration, the server immediately provides additional videos and explanations to facilitate learning. In this way, the system has comprehensive functions to maximize the effectiveness of skill acquisition.

[0324] The following describes the processing flow.

[0325] Step 1:

[0326] The server activates a high-resolution camera and records the craftsman's movements as a video in real time. The acquired video data is saved to the server's storage.

[0327] Step 2:

[0328] The server preprocesses the stored video data, removing noise and stabilizing the frames to prepare it for analysis.

[0329] Step 3:

[0330] The server analyzes pre-processed video data using an AI model it has generated. It identifies the movements of hands, fingers, and tools, and extracts the characteristics of these movements as numerical data.

[0331] Step 4:

[0332] The server extracts data to create standardized patterns of operation and specific procedures. Based on this, customized instructional information is generated.

[0333] Step 5:

[0334] The emotion engine built into the device acquires the user's facial expressions and voice as input and analyzes their emotions in real time. The user's emotional state is converted into data and sent to the server.

[0335] Step 6:

[0336] The server receives analysis data from the emotion engine and dynamically adjusts the content and format of the instructional information according to the user's emotional state. This maximizes learning effectiveness.

[0337] Step 7:

[0338] Users access customized instructional information through their devices and learn skills using videos and guides. As users progress, they can provide feedback to the system with additional questions and requests.

[0339] (Example 2)

[0340] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0341] In modern times, the transmission of traditional techniques and skills is becoming increasingly difficult as those with specialized knowledge and experience age, making it challenging to properly pass them on to the next generation. Furthermore, accurately understanding and responding to the emotional states of individual learners during the learning process is difficult, hindering effective learning outcomes. Moreover, systems for effectively utilizing real-time feedback from learners to improve teaching content and models are not yet fully established.

[0342] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0343] In this invention, the server includes means for capturing video information of the actions of a subject performing a task, means for analyzing the video information and extracting the characteristics of the subject's movements, means for generating instructional information based on the analysis results, and means for monitoring the user's emotional state using an emotion analysis function and optimizing the instructional information. This enables the effective transmission of traditional techniques and skills to the next generation, provides appropriate instruction tailored to the individual emotional state of each learner, and is expected to improve learning efficiency.

[0344] "The entity performing the work" refers to an individual or group that implements a specific skill or technique, and is the subject of recording and analysis of their actions and techniques.

[0345] "Visual information" refers to visual data captured by cameras and video equipment, used for visually analyzing technology and actions.

[0346] "Analysis" is the process of extracting the subject's movements and tool movement patterns from acquired video information, and includes the process of quantifying technical characteristics.

[0347] "Instructional information" refers to information generated based on analysis results to enable the reproduction of technical and skill procedures, and is provided in text, audio, and video formats.

[0348] The "emotion analysis function" is a technology that analyzes the user's facial expressions and tone of voice in real time to determine their emotional state, and is used to optimize instructional information.

[0349] "Communication technology" refers to technology for transmitting data between one or more devices, and is used as a means of providing instructional information to users in remote locations.

[0350] "Response" refers to data obtained from user feedback and reactions, which is used to improve the system's guidance information and AI models.

[0351] This invention is a system for efficiently transmitting traditional techniques and skills held by users to the next generation, and includes information exchange between servers, terminals, and users. To implement this, the coordination of various hardware and software components is essential.

[0352] First, the server uses a high-resolution camera to film the worker's work and acquire video information. This captures the details of the work and collects data necessary for subsequent analysis. This video data is a crucial element for motion analysis, and the analysis is performed using a generative AI model. The generative AI model processes the captured video, extracts and standardizes motion features and patterns. For example, it can analyze the hand movements when shaping pottery and provide a guide for beginners to reproduce the procedure.

[0353] Next, the server generates instructional information based on the analysis results obtained. This instructional information is expressed in various formats such as text, images, audio, and video, and is transmitted to the terminal using communication technology. Users can access this instructional information through their terminal and learn the necessary skills.

[0354] Furthermore, the device incorporates an emotion analysis function. This is achieved by the device capturing the user's facial expressions and voice tone through its camera and microphone, and analyzing the resulting emotion data. This emotion data is sent to a server and used to adjust instructional information to improve the user's learning efficiency. If the server determines that the user is tired or lacking understanding, it can automatically provide additional explanations or specific examples.

[0355] As a concrete example, when assisting with the learning of the tea ceremony, the system first films the movements of the tea ceremony instructor, analyzes them, and generates a procedure. Then, when the user begins learning this procedure at home, if they show signs of confusion, the system provides additional detailed explanations. In this way, the user can be expected to learn the movements of the tea ceremony efficiently and effectively.

[0356] An example of a prompt message is, "Analyze high-resolution motion video and generate standardized motion patterns." This instructs the system to begin the analysis process using a generative AI model.

[0357] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0358] Step 1:

[0359] The server uses a high-resolution camera to record the technician's work in real time. The input is the video data captured by the camera. The server saves this data to storage. The output is video information that captures the detailed actions. At this stage, specific actions such as adjusting the camera's position and angle during shooting to create the optimal shooting environment are taken.

[0360] Step 2:

[0361] The server inputs the stored video information into the generating AI model and begins the analysis. The input is the acquired video information. The server uses the generating AI model to analyze each frame and extract the characteristics of the movements. The output is numerical data such as the trajectory and speed of the movements and the angles of the joints. Specifically, it identifies hand movements and tool manipulations in the video and quantifies them as data.

[0362] Step 3:

[0363] The server generates instructional information based on the analysis results. The input is numerical data extracted through analysis. The server processes this data using an algorithm to generate learning materials such as text, images, audio, and video. The output is instructional material that shows specific procedures. Specific actions here include text editing, audio narration creation, and diagram preparation.

[0364] Step 4:

[0365] The device uses the user's camera and microphone to collect the user's facial expressions and voice through emotion analysis. The input is real-time facial images and voice data. The device activates an emotion engine to determine the user's emotional state. The output is an emotion label such as smile or confusion. Specifically, it analyzes and classifies whether the user is smiling or has a stern expression.

[0366] Step 5:

[0367] The server receives the results of sentiment analysis and dynamically adjusts the instructional information. The input is the user's sentiment label. The server changes the content and format of the instructional information according to the user's level of understanding and emotional state. The output is the adjusted instructional information. For example, if a lack of understanding is detected, the server selects and provides supplementary explanations or additional videos.

[0368] Step 6:

[0369] Users view instructional information provided on their devices and engage in learning. The input is the instructional information itself. Users refer to this at their own pace and acquire skills. The output is the acquired techniques and skills. Specific actions include users playing videos on the screen or reading text.

[0370] (Application Example 2)

[0371] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0372] In factory and other industrial settings, there is a challenge in effectively transferring the movements of skilled workers to machines. Furthermore, there is a lack of means to flexibly optimize machine operation according to the situation at hand, thus creating a demand for efficient and safe work practices.

[0373] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0374] In this invention, the server includes means for capturing an individual's movements and acquiring video data of those movements, means for analyzing the video data to extract characteristics of the movements, and means for dynamically optimizing the machine's movements based on the acquired emotional information. This makes it possible to transfer the skills of skilled workers to the machine while optimizing the movements according to the work environment.

[0375] "Action recording" refers to the act of recording an individual's work process as video footage.

[0376] "Acquiring video data" refers to the process of saving the captured video footage of the action and sending it to a server in a format that can be analyzed.

[0377] "Extracting movement characteristics" refers to the act of analyzing video data to identify specific movement patterns or elements from an individual's body movements and how they use equipment.

[0378] "Generating instructional information" is the process of creating information and procedures based on the characteristics of an action, enabling others to reproduce similar actions.

[0379] "Emotional information" refers to data that represents an individual's psychological state and mood, and is analyzed from facial expressions and voice tone acquired through cameras and microphones.

[0380] "Dynamic optimization of operation" refers to using acquired emotional information to adjust machine operation in real time according to the work environment and situation, thereby improving efficiency and safety.

[0381] The system for implementing this invention consists of multiple elements. First, the server uses a high-resolution camera to capture an individual's movements and acquire video data. Next, the server analyzes this video data using a generative AI model to extract key features of the movements. Based on the results of this analysis, information for movement instruction is generated and provided to the user through an online platform.

[0382] Furthermore, the terminal is equipped with emotion recognition capabilities, which analyze the user's facial expressions and voice through the camera and microphone to obtain emotional information. This emotional information is sent to a server, which dynamically optimizes the machine's operation based on it. For example, if the server determines that a worker's attention is distracted in the work environment, it will temporarily slow down the operation of the automated machine to ensure safety. In addition, newly generated operation patterns are continuously adjusted based on the results of the emotional information analysis.

[0383] For example, in the assembly of precision equipment, the highly accurate procedures of skilled workers are captured by cameras and learned as robotic movement patterns. On the other hand, if the emotion engine detects that a worker is fatigued, the work speed is adjusted as needed, thereby achieving both production efficiency and safety.

[0384] An example of a prompt statement would be: "Develop a system that allows a robot to learn the movements of skilled factory workers and perform tasks safely and efficiently. The robot is required to read the worker's emotions and optimize the workflow."

[0385] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0386] Step 1:

[0387] The server uses high-resolution cameras to film the movements of skilled workers in the factory. The input is real-time video of the work, and this video data is acquired by the server. This data includes the movements of the workers and detailed operation of the equipment they use.

[0388] Step 2:

[0389] The server analyzes video data acquired using a generative AI model. The input is video data, and the AI ​​model extracts features such as body movements and equipment handling. The output is represented as numerical data in the form of standardized motion patterns. Here, the flow of movement and key points are quantified.

[0390] Step 3:

[0391] The server generates instructional information based on the analyzed behavioral patterns. The input is the extracted behavioral characteristics, and the output is step-by-step instructional information to help other users reproduce similar tasks. This information is provided online through the platform.

[0392] Step 4:

[0393] The device uses a camera and microphone to acquire user emotional information. Input consists of the user's facial expressions and voice, and an emotion engine analyzes this data. Output is emotional information indicating the user's psychological state. This serves as an important indicator of work-related stress levels and concentration.

[0394] Step 5:

[0395] The server receives emotional information transmitted from the terminal and dynamically optimizes its operation patterns. The input is emotional information, and the output is optimized machine operation commands. For example, if the user indicates fatigue, the server adjusts the machine's operation to ensure a safe and comfortable working environment.

[0396] Step 6:

[0397] The user can perform tasks based on optimized operational commands. The input is optimized guidance information and machine operation, and the output is the result of efficient and safe work.

[0398] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0399] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0400] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0401] [Third Embodiment]

[0402] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0403] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0404] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0405] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0406] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0407] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0408] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0409] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0410] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0411] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0412] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0413] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0414] This invention relates to a system that utilizes generative AI for the purpose of passing on traditional techniques. This system records the traditional work performed by an individual, analyzes the movements using an AI model, and generates instructional information, thereby enabling successors to acquire the same techniques.

[0415] Program processing and explanation

[0416] 1. Data collection (server)

[0417] The server uses a high-resolution camera to record the individual's movements. This is done to capture in detail the series of work processes performed by the craftsman. The server saves the video data to storage and prepares it for analysis.

[0418] 2. Data analysis (server)

[0419] The collected video data is analyzed using a generative AI model installed on the server. The AI ​​model analyzes how individuals manipulate their hands and tools in the video and extracts the characteristics of these movements as numerical data.

[0420] 3. Generation of instructional information (server)

[0421] The server generates reproducible instructional information based on the AI ​​analysis results. This information includes specific explanations of technical procedures, key points of operation, and necessary tools and materials.

[0422] 4. Information provision (terminal)

[0423] Users can access the online platform using their devices. The server uploads generated instructional information to the platform, which users can then use to learn the techniques. The user interface includes video playback functionality, allowing users to learn while visually confirming the actual movements.

[0424] Specific example

[0425] For example, if someone wants to preserve traditional Japanese pottery making, a high-resolution camera would film the entire process, from the craftsman kneading the clay to firing it in the kiln. A server would then analyze this footage, extracting information on the clay's condition, hand pressure and movement, and important points to note. Next, the server would generate a detailed manual based on the information obtained and provide it to the user via a terminal. The user could then learn the technique by reviewing the steps on the platform and imitating the same actions according to the instructions. In this way, the present invention facilitates the transmission of traditional techniques and enables the continuation of culture.

[0426] The following describes the processing flow.

[0427] Step 1:

[0428] The server controls high-resolution cameras to capture the craftsman's work process in real time. The server periodically saves the acquired video data to storage and prepares it for analysis.

[0429] Step 2:

[0430] The server preprocesses the video data, removing noise and improving video quality. This process includes image stabilization and adjustment of brightness and contrast.

[0431] Step 3:

[0432] The server activates the AI, which analyzes the pre-processed video data. The AI ​​analyzes each frame to recognize the movements of hands and tools, and extracts the characteristics of these movements as numerical data.

[0433] Step 4:

[0434] The server transcribes the actions and procedures based on the analyzed data. Instructional information includes specific steps, key operational points, and a guide to the materials and tools to be used.

[0435] Step 5:

[0436] The server formats the generated instructional information and uploads it to the online platform. This makes the information accessible from anywhere.

[0437] Step 6:

[0438] Users can log in to the online platform using their devices and search for and view necessary instructional information. Users can learn the technology by viewing the provided manuals and videos.

[0439] (Example 1)

[0440] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0441] In the transmission of traditional techniques, a challenge is to accurately record and analyze individual physical movements and provide an environment that enables effective instruction to the next generation of learners. Conventional methods have problems such as difficulty in adequately conveying the finer points of the techniques and inefficiency in the transmission process.

[0442] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0443] In this invention, the server includes means for acquiring high-resolution video data, means for analyzing individual movements using a generative AI model and extracting numerical data, and means for generating reproducible instructional information and providing it through an online learning platform. This makes it possible to accurately record and analyze the detailed movements of traditional techniques and provide learners with specific and visual instructional information.

[0444] "High-resolution video data" refers to high-quality video information that can capture the movements of an individual performing a task in detail.

[0445] A "generative AI model" refers to an artificial intelligence algorithm used to analyze video data and extract characteristics of individual movements.

[0446] "Numerical data" refers to the numerical representation of behavioral characteristics extracted by the generative AI model, and is used for analyzing those behaviors.

[0447] "Instructional information" refers to educational information that includes specific explanations of reproducible technical procedures, key points of movement, and necessary tools.

[0448] An "online learning platform" refers to a digital environment where users can access instructional information and learn via the internet.

[0449] This invention is a system aimed at the transmission of traditional techniques, in which a server, terminal, and user work in cooperation with each other. The server first uses a high-resolution camera to film the actions of an individual performing a traditional task. The video data requires detailed image quality to accurately record the movements of the craftsman's hands and the operation of tools. The filmed data is stored on the server. Next, the server analyzes this video data using a generative AI model. The generative AI model extracts the characteristics of the actions in the video as numerical data and analyzes the essential elements of the technique.

[0450] Next, the server generates reproducible instructional information based on the AI ​​analysis results. This instructional information includes specific steps for reproducing the technique, key points of operation, and information on necessary tools and materials. This allows users to acquire the technique in a detailed and easy-to-understand manner.

[0451] The generated instructional information is uploaded to an online learning platform by the server. Users access this platform via their devices and learn from the provided instructional information. Users can visually confirm the operation of the technology using the video playback function and progress through their learning.

[0452] Specific example

[0453] For example, when learning the techniques of Japanese pottery making, the server films the process of a craftsman kneading clay and firing it in a kiln. The generative AI model analyzes this footage and extracts detailed characteristics of the clay's condition, hand pressure, and movement. Based on this, the server generates instructional information on specific steps for shaping the clay and the tools to be used, and provides this information to users through an online platform, enabling users to learn pottery making techniques.

[0454] An example of a prompt message is: "We will build a system that uses AI to analyze the hand movements and pressure when kneading clay for pottery making, and generate detailed instructional information." This will enable users to learn traditional techniques precisely and efficiently.

[0455] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0456] Step 1:

[0457] The server activates a high-resolution camera and films the actions of an individual performing a traditional technique. The input is the individual's actions, and the output is high-resolution video data. The server uses this video data to obtain detailed action information necessary for later analysis. Specifically, the camera tracks the person's hands and the movement of tools, continuously collecting data.

[0458] Step 2:

[0459] The server saves the captured video data to storage. The input is the high-resolution video data obtained in step 1, and the output is the video file stored in the database. The purpose of this step is to ensure that the video data necessary for analysis is reliably saved and prepared for subsequent processing.

[0460] Step 3:

[0461] The server prepares a generation AI model and loads the stored video data. The input is video data stored in storage. The generation AI model analyzes the video data and extracts important features of the movements as numerical data. Specifically, the AI ​​model identifies and analyzes the position of hands and tools, the speed of movement, the angle, etc., for each frame.

[0462] Step 4:

[0463] The server generates instructional information based on the numerical data it receives. The input is the analysis results from the AI ​​model, and the output is reproducible instructional information. This instructional information includes specific steps for the technique, points to note when operating it, and a list of necessary tools and materials. The server combines this information and presents it in a format that is easy for learners to understand.

[0464] Step 5:

[0465] The server uploads the generated instructional information to the online platform. The input is the generated instructional information, and the output is instructional information in a format viewable on the platform. The server optimizes the system so that the information is displayed accurately and is easily accessible to users.

[0466] Step 6:

[0467] Users access an online platform using their devices and view instructional information. The input is online instructional information, and the output is the learner's acquisition of skills. Users visually confirm actions through video playback and practice and master skills based on the provided instruction. This allows users to progress in their learning while referring to actual actions.

[0468] (Application Example 1)

[0469] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0470] The transmission of traditional techniques faces the challenge of difficulty in incorporating them into modern, efficient mass production technologies. Furthermore, in modern companies where standardization and automation are increasingly promoted, there is a demand to efficiently replicate the manual work of highly skilled traditional craftsmen using robots and machinery. However, mechanically reproducing the movements of traditional techniques requires accurately capturing, analyzing, and standardizing the intuitive skills of human beings, and methods for achieving this are needed.

[0471] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0472] In this invention, the server includes means for capturing the movements of an individual performing a task and acquiring video data of said movements; means for analyzing the video data to extract characteristics of the individual's hand, finger, and tool movements; means for generating instructional information to make the procedure of the work reproducible based on the analysis results; means for converting the instructional information into control information for execution in machine-based work; means for controlling a work device that automatically performs the work based on the control information; and means for providing the instructional information and control information to users through an online platform. This enables the efficient inheritance of traditional techniques and the realization of work standardization and automation in factories.

[0473] "Work" refers to a series of actions or operations performed by a human or machine to achieve a specific purpose.

[0474] "Motion data" refers to video information that records the movements of an individual's hands and fingers, as well as the process of operating tools while performing a task.

[0475] "Video analysis" is the process of analyzing acquired motion data and numerically extracting the characteristics of individual movements.

[0476] "Instructional information" refers to specific guidelines and manuals generated based on analysis results to make the operation procedures reproducible.

[0477] "Control information" refers to specific operating instructions created based on guidance information, which enable machines and work devices to automatically perform their actions.

[0478] A "working device" refers to a machine or equipment that automatically performs a specific task based on control information.

[0479] An "online platform" is a digital infrastructure that provides information in a format that users can access via the internet.

[0480] To implement this invention, the entire system consists of several main components. First, the server uses a high-resolution camera to capture the movements of an individual performing a task, thereby acquiring movement data. This data is processed within the server via video analysis software and used to extract numerical data on the characteristics of the hands, fingers, and tools. This analysis utilizes existing generative AI models to generate instructional information for automatically reproducing the steps of the movement.

[0481] Next, the server converts the instruction information into control information and automatically operates the work equipment based on it. Specific hardware examples include factory robots and programmable work machines for motion control. These precisely execute the actions instructed by the control information, recreating traditional techniques.

[0482] Furthermore, the generated instructional and control information is uploaded to an online platform for user access. Users can connect to this platform using their devices to view and learn from the provided information. Specific examples include providing operation manuals for replicating pottery molding techniques with robots, and standard action sequences of craftsmanship as programs.

[0483] By utilizing generative AI models, traditional techniques can be adapted to modern manufacturing environments, providing a system that supports the transmission of diverse technologies. Specifically, one example involves instructing the AI ​​model to perform a task using a prompt such as, "Analyze the movements of manual weaving from this high-resolution video and generate a reproducible robot program."

[0484] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0485] Step 1:

[0486] The server uses a high-resolution camera to capture the actions of the individual performing the task and acquires a video file as action data. This video data is the input, and the server saves this data to storage for use in subsequent processing steps.

[0487] Step 2:

[0488] The server inputs the acquired video data into a generating AI model to analyze the characteristics of hand, finger, and tool movements. In this step, data processing is performed to extract feature quantities for each part of the movement from the input video data, and feature data is obtained as output.

[0489] Step 3:

[0490] The server generates instructional information based on the analyzed feature data. Here, data calculations are performed to generate guidelines and operation manuals that allow for the reproduction of work procedures based on the feature data, and this content is output as instructional information.

[0491] Step 4:

[0492] The server converts the generated instruction information into control information. This process involves converting the content of the instruction information into a machine-executable format and performing data conversion to generate control data. The output is a specific set of instructions for controlling the work device.

[0493] Step 5:

[0494] The server controls the work equipment based on control information, automatically executing the actual work. Using control data as input, the equipment starts the specified operation and performs the desired task. The output is the result of the performed work.

[0495] Step 6:

[0496] The server uploads the generated instructional and control information to an online platform, and users access this information using their terminals. After the data is uploaded, the information is provided to support the user's learning, and the output is the user's ability to view the information and learn how to operate it.

[0497] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0498] This invention combines a system that analyzes individual actions and generates instructional information necessary for the transmission of skills with an emotion engine that recognizes the user's emotions and provides instruction tailored to their state. By combining advanced generative AI and emotion recognition technology, this system aims to efficiently and effectively transmit traditional skills to the next generation.

[0499] Program processing and explanation

[0500] 1. Acquisition of video data (server)

[0501] The server uses high-resolution cameras to film the craftsman at work and acquires video data. This data captures the movements of the craftsman's hands and tools in detail.

[0502] 2. Analysis of operation (server)

[0503] The server uses a generated AI model to analyze video data and extract motion characteristics. The sequence and key points of the motion are recorded as numerical data, and standardized motion patterns are created.

[0504] 3. Generation of instructional information (server)

[0505] The server generates instructional information based on the analysis results. This information is presented to the user as learning material delivered via the terminal, making it possible to reproduce traditional techniques.

[0506] 4. Recognition of emotions (device)

[0507] The emotion engine built into the device acquires emotional information through the user's video or audio. This is achieved, for example, by analyzing the user's facial expressions and tone of voice using the camera and microphone.

[0508] 5. Optimization of instructional information (server)

[0509] Based on the analysis results of the emotion engine, the server dynamically adjusts the guidance information according to the user's current emotional state. For example, if it determines that the user is not understanding the information, it will provide additional explanations or video clips.

[0510] 6. Information provision and learning (user)

[0511] Users access instructional information from their devices and view implementation procedures and details. Information appropriately tailored by an emotion engine allows users to acquire skills more effectively.

[0512] Specific example

[0513] For example, if a user wants to learn the procedures of the Japanese tea ceremony, this system first films the hands of a tea ceremony master with a high-resolution camera and analyzes their movements. Then, using the generated instructional information, the user begins learning on their home device. During learning, the emotion engine scans the user's face with the camera, and if it detects low concentration, the server immediately provides additional videos and explanations to facilitate learning. In this way, the system has comprehensive functions to maximize the effectiveness of skill acquisition.

[0514] The following describes the processing flow.

[0515] Step 1:

[0516] The server activates a high-resolution camera and records the craftsman's movements as a video in real time. The acquired video data is saved to the server's storage.

[0517] Step 2:

[0518] The server preprocesses the stored video data, removing noise and stabilizing the frames to prepare it for analysis.

[0519] Step 3:

[0520] The server analyzes pre-processed video data using an AI model it has generated. It identifies the movements of hands, fingers, and tools, and extracts the characteristics of these movements as numerical data.

[0521] Step 4:

[0522] The server extracts data to create standardized patterns of operation and specific procedures. Based on this, customized instructional information is generated.

[0523] Step 5:

[0524] The emotion engine built into the device acquires the user's facial expressions and voice as input and analyzes their emotions in real time. The user's emotional state is converted into data and sent to the server.

[0525] Step 6:

[0526] The server receives analysis data from the emotion engine and dynamically adjusts the content and format of the instructional information according to the user's emotional state. This maximizes learning effectiveness.

[0527] Step 7:

[0528] Users access customized instructional information through their devices and learn skills using videos and guides. As users progress, they can provide feedback to the system with additional questions and requests.

[0529] (Example 2)

[0530] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0531] In modern times, the transmission of traditional techniques and skills is becoming increasingly difficult as those with specialized knowledge and experience age, making it challenging to properly pass them on to the next generation. Furthermore, accurately understanding and responding to the emotional states of individual learners during the learning process is difficult, hindering effective learning outcomes. Moreover, systems for effectively utilizing real-time feedback from learners to improve teaching content and models are not yet fully established.

[0532] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0533] In this invention, the server includes means for capturing video information of the actions of a subject performing a task, means for analyzing the video information and extracting the characteristics of the subject's movements, means for generating instructional information based on the analysis results, and means for monitoring the user's emotional state using an emotion analysis function and optimizing the instructional information. This enables the effective transmission of traditional techniques and skills to the next generation, provides appropriate instruction tailored to the individual emotional state of each learner, and is expected to improve learning efficiency.

[0534] "The entity performing the work" refers to an individual or group that implements a specific skill or technique, and is the subject of recording and analysis of their actions and techniques.

[0535] "Visual information" refers to visual data captured by cameras and video equipment, used for visually analyzing technology and actions.

[0536] "Analysis" is the process of extracting the subject's movements and tool movement patterns from acquired video information, and includes the process of quantifying technical characteristics.

[0537] "Instructional information" refers to information generated based on analysis results to enable the reproduction of technical and skill procedures, and is provided in text, audio, and video formats.

[0538] The "emotion analysis function" is a technology that analyzes the user's facial expressions and tone of voice in real time to determine their emotional state, and is used to optimize instructional information.

[0539] "Communication technology" refers to technology for transmitting data between one or more devices, and is used as a means of providing instructional information to users in remote locations.

[0540] "Response" refers to data obtained from user feedback and reactions, which is used to improve the system's guidance information and AI models.

[0541] This invention is a system for efficiently transmitting traditional techniques and skills held by users to the next generation, and includes information exchange between servers, terminals, and users. To implement this, the coordination of various hardware and software components is essential.

[0542] First, the server uses a high-resolution camera to film the worker's work and acquire video information. This captures the details of the work and collects data necessary for subsequent analysis. This video data is a crucial element for motion analysis, and the analysis is performed using a generative AI model. The generative AI model processes the captured video, extracts and standardizes motion features and patterns. For example, it can analyze the hand movements when shaping pottery and provide a guide for beginners to reproduce the procedure.

[0543] Next, the server generates instructional information based on the analysis results obtained. This instructional information is expressed in various formats such as text, images, audio, and video, and is transmitted to the terminal using communication technology. Users can access this instructional information through their terminal and learn the necessary skills.

[0544] Furthermore, the device incorporates an emotion analysis function. This is achieved by the device capturing the user's facial expressions and voice tone through its camera and microphone, and analyzing the resulting emotion data. This emotion data is sent to a server and used to adjust instructional information to improve the user's learning efficiency. If the server determines that the user is tired or lacking understanding, it can automatically provide additional explanations or specific examples.

[0545] As a concrete example, when assisting with the learning of the tea ceremony, the system first films the movements of the tea ceremony instructor, analyzes them, and generates a procedure. Then, when the user begins learning this procedure at home, if they show signs of confusion, the system provides additional detailed explanations. In this way, the user can be expected to learn the movements of the tea ceremony efficiently and effectively.

[0546] An example of a prompt message is, "Analyze high-resolution motion video and generate standardized motion patterns." This instructs the system to begin the analysis process using a generative AI model.

[0547] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0548] Step 1:

[0549] The server uses a high-resolution camera to record the technician's work in real time. The input is the video data captured by the camera. The server saves this data to storage. The output is video information that captures the detailed actions. At this stage, specific actions such as adjusting the camera's position and angle during shooting to create the optimal shooting environment are taken.

[0550] Step 2:

[0551] The server inputs the stored video information into the generating AI model and begins the analysis. The input is the acquired video information. The server uses the generating AI model to analyze each frame and extract the characteristics of the movements. The output is numerical data such as the trajectory and speed of the movements and the angles of the joints. Specifically, it identifies hand movements and tool manipulations in the video and quantifies them as data.

[0552] Step 3:

[0553] The server generates instructional information based on the analysis results. The input is numerical data extracted through analysis. The server processes this data using an algorithm to generate learning materials such as text, images, audio, and video. The output is instructional material that shows specific procedures. Specific actions here include text editing, audio narration creation, and diagram preparation.

[0554] Step 4:

[0555] The device uses the user's camera and microphone to collect the user's facial expressions and voice through emotion analysis. The input is real-time facial images and voice data. The device activates an emotion engine to determine the user's emotional state. The output is an emotion label such as smile or confusion. Specifically, it analyzes and classifies whether the user is smiling or has a stern expression.

[0556] Step 5:

[0557] The server receives the results of sentiment analysis and dynamically adjusts the instructional information. The input is the user's sentiment label. The server changes the content and format of the instructional information according to the user's level of understanding and emotional state. The output is the adjusted instructional information. For example, if a lack of understanding is detected, the server selects and provides supplementary explanations or additional videos.

[0558] Step 6:

[0559] Users view instructional information provided on their devices and engage in learning. The input is the instructional information itself. Users refer to this at their own pace and acquire skills. The output is the acquired techniques and skills. Specific actions include users playing videos on the screen or reading text.

[0560] (Application Example 2)

[0561] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0562] In factory and other industrial settings, there is a challenge in effectively transferring the movements of skilled workers to machines. Furthermore, there is a lack of means to flexibly optimize machine operation according to the situation at hand, thus creating a demand for efficient and safe work practices.

[0563] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0564] In this invention, the server includes means for capturing an individual's movements and acquiring video data of those movements, means for analyzing the video data to extract characteristics of the movements, and means for dynamically optimizing the machine's movements based on the acquired emotional information. This makes it possible to transfer the skills of skilled workers to the machine while optimizing the movements according to the work environment.

[0565] "Action recording" refers to the act of recording an individual's work process as video footage.

[0566] "Acquiring video data" refers to the process of saving the captured video footage of the action and sending it to a server in a format that can be analyzed.

[0567] "Extracting movement characteristics" refers to the act of analyzing video data to identify specific movement patterns or elements from an individual's body movements and how they use equipment.

[0568] "Generating instructional information" is the process of creating information and procedures based on the characteristics of an action, enabling others to reproduce similar actions.

[0569] "Emotional information" refers to data that represents an individual's psychological state and mood, and is analyzed from facial expressions and voice tone acquired through cameras and microphones.

[0570] "Dynamic optimization of operation" refers to using acquired emotional information to adjust machine operation in real time according to the work environment and situation, thereby improving efficiency and safety.

[0571] The system for implementing this invention consists of multiple elements. First, the server uses a high-resolution camera to capture an individual's movements and acquire video data. Next, the server analyzes this video data using a generative AI model to extract key features of the movements. Based on the results of this analysis, information for movement instruction is generated and provided to the user through an online platform.

[0572] Furthermore, the terminal is equipped with emotion recognition capabilities, which analyze the user's facial expressions and voice through the camera and microphone to obtain emotional information. This emotional information is sent to a server, which dynamically optimizes the machine's operation based on it. For example, if the server determines that a worker's attention is distracted in the work environment, it will temporarily slow down the operation of the automated machine to ensure safety. In addition, newly generated operation patterns are continuously adjusted based on the results of the emotional information analysis.

[0573] For example, in the assembly of precision equipment, the highly accurate procedures of skilled workers are captured by cameras and learned as robotic movement patterns. On the other hand, if the emotion engine detects that a worker is fatigued, the work speed is adjusted as needed, thereby achieving both production efficiency and safety.

[0574] An example of a prompt statement would be: "Develop a system that allows a robot to learn the movements of skilled factory workers and perform tasks safely and efficiently. The robot is required to read the worker's emotions and optimize the workflow."

[0575] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0576] Step 1:

[0577] The server uses high-resolution cameras to film the movements of skilled workers in the factory. The input is real-time video of the work, and this video data is acquired by the server. This data includes the movements of the workers and detailed operation of the equipment they use.

[0578] Step 2:

[0579] The server analyzes video data acquired using a generative AI model. The input is video data, and the AI ​​model extracts features such as body movements and equipment handling. The output is represented as numerical data in the form of standardized motion patterns. Here, the flow of movement and key points are quantified.

[0580] Step 3:

[0581] The server generates instructional information based on the analyzed behavioral patterns. The input is the extracted behavioral characteristics, and the output is step-by-step instructional information to help other users reproduce similar tasks. This information is provided online through the platform.

[0582] Step 4:

[0583] The device uses a camera and microphone to acquire user emotional information. Input consists of the user's facial expressions and voice, and an emotion engine analyzes this data. Output is emotional information indicating the user's psychological state. This serves as an important indicator of work-related stress levels and concentration.

[0584] Step 5:

[0585] The server receives emotional information transmitted from the terminal and dynamically optimizes its operation patterns. The input is emotional information, and the output is optimized machine operation commands. For example, if the user indicates fatigue, the server adjusts the machine's operation to ensure a safe and comfortable working environment.

[0586] Step 6:

[0587] The user can perform tasks based on optimized operational commands. The input is optimized guidance information and machine operation, and the output is the result of efficient and safe work.

[0588] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0589] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0590] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0591] [Fourth Embodiment]

[0592] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0593] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0594] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0595] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0596] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0597] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0598] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0599] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0600] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0601] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0602] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0603] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0604] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0605] This invention relates to a system that utilizes generative AI for the purpose of passing on traditional techniques. This system records the traditional work performed by an individual, analyzes the movements using an AI model, and generates instructional information, thereby enabling successors to acquire the same techniques.

[0606] Program processing and explanation

[0607] 1. Data collection (server)

[0608] The server uses a high-resolution camera to record the individual's movements. This is done to capture in detail the series of work processes performed by the craftsman. The server saves the video data to storage and prepares it for analysis.

[0609] 2. Data analysis (server)

[0610] The collected video data is analyzed using a generative AI model installed on the server. The AI ​​model analyzes how individuals manipulate their hands and tools in the video and extracts the characteristics of these movements as numerical data.

[0611] 3. Generation of instructional information (server)

[0612] The server generates reproducible instructional information based on the AI ​​analysis results. This information includes specific explanations of technical procedures, key points of operation, and necessary tools and materials.

[0613] 4. Information provision (terminal)

[0614] Users can access the online platform using their devices. The server uploads generated instructional information to the platform, which users can then use to learn the techniques. The user interface includes video playback functionality, allowing users to learn while visually confirming the actual movements.

[0615] Specific example

[0616] For example, if someone wants to preserve traditional Japanese pottery making, a high-resolution camera would film the entire process, from the craftsman kneading the clay to firing it in the kiln. A server would then analyze this footage, extracting information on the clay's condition, hand pressure and movement, and important points to note. Next, the server would generate a detailed manual based on the information obtained and provide it to the user via a terminal. The user could then learn the technique by reviewing the steps on the platform and imitating the same actions according to the instructions. In this way, the present invention facilitates the transmission of traditional techniques and enables the continuation of culture.

[0617] The following describes the processing flow.

[0618] Step 1:

[0619] The server controls high-resolution cameras to capture the craftsman's work process in real time. The server periodically saves the acquired video data to storage and prepares it for analysis.

[0620] Step 2:

[0621] The server preprocesses the video data, removing noise and improving video quality. This process includes image stabilization and adjustment of brightness and contrast.

[0622] Step 3:

[0623] The server activates the AI, which analyzes the pre-processed video data. The AI ​​analyzes each frame to recognize the movements of hands and tools, and extracts the characteristics of these movements as numerical data.

[0624] Step 4:

[0625] The server transcribes the actions and procedures based on the analyzed data. Instructional information includes specific steps, key operational points, and a guide to the materials and tools to be used.

[0626] Step 5:

[0627] The server formats the generated instructional information and uploads it to the online platform. This makes the information accessible from anywhere.

[0628] Step 6:

[0629] Users can log in to the online platform using their devices and search for and view necessary instructional information. Users can learn the technology by viewing the provided manuals and videos.

[0630] (Example 1)

[0631] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0632] In the transmission of traditional techniques, a challenge is to accurately record and analyze individual physical movements and provide an environment that enables effective instruction to the next generation of learners. Conventional methods have problems such as difficulty in adequately conveying the finer points of the techniques and inefficiency in the transmission process.

[0633] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0634] In this invention, the server includes means for acquiring high-resolution video data, means for analyzing individual movements using a generative AI model and extracting numerical data, and means for generating reproducible instructional information and providing it through an online learning platform. This makes it possible to accurately record and analyze the detailed movements of traditional techniques and provide learners with specific and visual instructional information.

[0635] "High-resolution video data" refers to high-quality video information that can capture the movements of an individual performing a task in detail.

[0636] A "generative AI model" refers to an artificial intelligence algorithm used to analyze video data and extract characteristics of individual movements.

[0637] "Numerical data" refers to the numerical representation of behavioral characteristics extracted by the generative AI model, and is used for analyzing those behaviors.

[0638] "Instructional information" refers to educational information that includes specific explanations of reproducible technical procedures, key points of movement, and necessary tools.

[0639] An "online learning platform" refers to a digital environment where users can access instructional information and learn via the internet.

[0640] This invention is a system aimed at the transmission of traditional techniques, in which a server, terminal, and user work in cooperation with each other. The server first uses a high-resolution camera to film the actions of an individual performing a traditional task. The video data requires detailed image quality to accurately record the movements of the craftsman's hands and the operation of tools. The filmed data is stored on the server. Next, the server analyzes this video data using a generative AI model. The generative AI model extracts the characteristics of the actions in the video as numerical data and analyzes the essential elements of the technique.

[0641] Next, the server generates reproducible instructional information based on the AI ​​analysis results. This instructional information includes specific steps for reproducing the technique, key points of operation, and information on necessary tools and materials. This allows users to acquire the technique in a detailed and easy-to-understand manner.

[0642] The generated instructional information is uploaded to an online learning platform by the server. Users access this platform via their devices and learn from the provided instructional information. Users can visually confirm the operation of the technology using the video playback function and progress through their learning.

[0643] Specific example

[0644] For example, when learning the techniques of Japanese pottery making, the server films the process of a craftsman kneading clay and firing it in a kiln. The generative AI model analyzes this footage and extracts detailed characteristics of the clay's condition, hand pressure, and movement. Based on this, the server generates instructional information on specific steps for shaping the clay and the tools to be used, and provides this information to users through an online platform, enabling users to learn pottery making techniques.

[0645] An example of a prompt message is: "We will build a system that uses AI to analyze the hand movements and pressure when kneading clay for pottery making, and generate detailed instructional information." This will enable users to learn traditional techniques precisely and efficiently.

[0646] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0647] Step 1:

[0648] The server activates a high-resolution camera and films the actions of an individual performing a traditional technique. The input is the individual's actions, and the output is high-resolution video data. The server uses this video data to obtain detailed action information necessary for later analysis. Specifically, the camera tracks the person's hands and the movement of tools, continuously collecting data.

[0649] Step 2:

[0650] The server saves the captured video data to storage. The input is the high-resolution video data obtained in step 1, and the output is the video file stored in the database. The purpose of this step is to ensure that the video data necessary for analysis is reliably saved and prepared for subsequent processing.

[0651] Step 3:

[0652] The server prepares a generation AI model and loads the stored video data. The input is video data stored in storage. The generation AI model analyzes the video data and extracts important features of the movements as numerical data. Specifically, the AI ​​model identifies and analyzes the position of hands and tools, the speed of movement, the angle, etc., for each frame.

[0653] Step 4:

[0654] The server generates instructional information based on the numerical data it receives. The input is the analysis results from the AI ​​model, and the output is reproducible instructional information. This instructional information includes specific steps for the technique, points to note when operating it, and a list of necessary tools and materials. The server combines this information and presents it in a format that is easy for learners to understand.

[0655] Step 5:

[0656] The server uploads the generated instructional information to the online platform. The input is the generated instructional information, and the output is instructional information in a format viewable on the platform. The server optimizes the system so that the information is displayed accurately and is easily accessible to users.

[0657] Step 6:

[0658] Users access an online platform using their devices and view instructional information. The input is online instructional information, and the output is the learner's acquisition of skills. Users visually confirm actions through video playback and practice and master skills based on the provided instruction. This allows users to progress in their learning while referring to actual actions.

[0659] (Application Example 1)

[0660] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0661] The transmission of traditional techniques faces the challenge of difficulty in incorporating them into modern, efficient mass production technologies. Furthermore, in modern companies where standardization and automation are increasingly promoted, there is a demand to efficiently replicate the manual work of highly skilled traditional craftsmen using robots and machinery. However, mechanically reproducing the movements of traditional techniques requires accurately capturing, analyzing, and standardizing the intuitive skills of human beings, and methods for achieving this are needed.

[0662] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0663] In this invention, the server includes means for capturing the movements of an individual performing a task and acquiring video data of said movements; means for analyzing the video data to extract characteristics of the individual's hand, finger, and tool movements; means for generating instructional information to make the procedure of the work reproducible based on the analysis results; means for converting the instructional information into control information for execution in machine-based work; means for controlling a work device that automatically performs the work based on the control information; and means for providing the instructional information and control information to users through an online platform. This enables the efficient inheritance of traditional techniques and the realization of work standardization and automation in factories.

[0664] "Work" refers to a series of actions or operations performed by a human or machine to achieve a specific purpose.

[0665] "Motion data" refers to video information that records the movements of an individual's hands and fingers, as well as the process of operating tools while performing a task.

[0666] "Video analysis" is the process of analyzing acquired motion data and numerically extracting the characteristics of individual movements.

[0667] "Instructional information" refers to specific guidelines and manuals generated based on analysis results to make the operation procedures reproducible.

[0668] "Control information" refers to specific operating instructions created based on guidance information, which enable machines and work devices to automatically perform their actions.

[0669] A "working device" refers to a machine or equipment that automatically performs a specific task based on control information.

[0670] An "online platform" is a digital infrastructure that provides information in a format that users can access via the internet.

[0671] To implement this invention, the entire system consists of several main components. First, the server uses a high-resolution camera to capture the movements of an individual performing a task, thereby acquiring movement data. This data is processed within the server via video analysis software and used to extract numerical data on the characteristics of the hands, fingers, and tools. This analysis utilizes existing generative AI models to generate instructional information for automatically reproducing the steps of the movement.

[0672] Next, the server converts the instruction information into control information and automatically operates the work equipment based on it. Specific hardware examples include factory robots and programmable work machines for motion control. These precisely execute the actions instructed by the control information, recreating traditional techniques.

[0673] Furthermore, the generated instructional and control information is uploaded to an online platform for user access. Users can connect to this platform using their devices to view and learn from the provided information. Specific examples include providing operation manuals for replicating pottery molding techniques with robots, and standard action sequences of craftsmanship as programs.

[0674] By utilizing generative AI models, traditional techniques can be adapted to modern manufacturing environments, providing a system that supports the transmission of diverse technologies. Specifically, one example involves instructing the AI ​​model to perform a task using a prompt such as, "Analyze the movements of manual weaving from this high-resolution video and generate a reproducible robot program."

[0675] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0676] Step 1:

[0677] The server uses a high-resolution camera to capture the actions of the individual performing the task and acquires a video file as action data. This video data is the input, and the server saves this data to storage for use in subsequent processing steps.

[0678] Step 2:

[0679] The server inputs the acquired video data into a generating AI model to analyze the characteristics of hand, finger, and tool movements. In this step, data processing is performed to extract feature quantities for each part of the movement from the input video data, and feature data is obtained as output.

[0680] Step 3:

[0681] The server generates instructional information based on the analyzed feature data. Here, data calculations are performed to generate guidelines and operation manuals that allow for the reproduction of work procedures based on the feature data, and this content is output as instructional information.

[0682] Step 4:

[0683] The server converts the generated instruction information into control information. This process involves converting the content of the instruction information into a machine-executable format and performing data conversion to generate control data. The output is a specific set of instructions for controlling the work device.

[0684] Step 5:

[0685] The server controls the work equipment based on control information, automatically executing the actual work. Using control data as input, the equipment starts the specified operation and performs the desired task. The output is the result of the performed work.

[0686] Step 6:

[0687] The server uploads the generated instructional and control information to an online platform, and users access this information using their terminals. After the data is uploaded, the information is provided to support the user's learning, and the output is the user's ability to view the information and learn how to operate it.

[0688] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0689] This invention combines a system that analyzes individual actions and generates instructional information necessary for the transmission of skills with an emotion engine that recognizes the user's emotions and provides instruction tailored to their state. By combining advanced generative AI and emotion recognition technology, this system aims to efficiently and effectively transmit traditional skills to the next generation.

[0690] Program processing and explanation

[0691] 1. Acquisition of video data (server)

[0692] The server uses high-resolution cameras to film the craftsman at work and acquires video data. This data captures the movements of the craftsman's hands and tools in detail.

[0693] 2. Analysis of operation (server)

[0694] The server uses a generated AI model to analyze video data and extract motion characteristics. The sequence and key points of the motion are recorded as numerical data, and standardized motion patterns are created.

[0695] 3. Generation of instructional information (server)

[0696] The server generates instructional information based on the analysis results. This information is presented to the user as learning material delivered via the terminal, making it possible to reproduce traditional techniques.

[0697] 4. Recognition of emotions (device)

[0698] The emotion engine built into the device acquires emotional information through the user's video or audio. This is achieved, for example, by analyzing the user's facial expressions and tone of voice using the camera and microphone.

[0699] 5. Optimization of instructional information (server)

[0700] Based on the analysis results of the emotion engine, the server dynamically adjusts the guidance information according to the user's current emotional state. For example, if it determines that the user is not understanding the information, it will provide additional explanations or video clips.

[0701] 6. Information provision and learning (user)

[0702] Users access instructional information from their devices and view implementation procedures and details. Information appropriately tailored by an emotion engine allows users to acquire skills more effectively.

[0703] Specific example

[0704] For example, if a user wants to learn the procedures of the Japanese tea ceremony, this system first films the hands of a tea ceremony master with a high-resolution camera and analyzes their movements. Then, using the generated instructional information, the user begins learning on their home device. During learning, the emotion engine scans the user's face with the camera, and if it detects low concentration, the server immediately provides additional videos and explanations to facilitate learning. In this way, the system has comprehensive functions to maximize the effectiveness of skill acquisition.

[0705] The following describes the processing flow.

[0706] Step 1:

[0707] The server activates a high-resolution camera and records the craftsman's movements as a video in real time. The acquired video data is saved to the server's storage.

[0708] Step 2:

[0709] The server preprocesses the stored video data, removing noise and stabilizing the frames to prepare it for analysis.

[0710] Step 3:

[0711] The server analyzes pre-processed video data using an AI model it has generated. It identifies the movements of hands, fingers, and tools, and extracts the characteristics of these movements as numerical data.

[0712] Step 4:

[0713] The server extracts data to create standardized patterns of operation and specific procedures. Based on this, customized instructional information is generated.

[0714] Step 5:

[0715] The emotion engine built into the device acquires the user's facial expressions and voice as input and analyzes their emotions in real time. The user's emotional state is converted into data and sent to the server.

[0716] Step 6:

[0717] The server receives analysis data from the emotion engine and dynamically adjusts the content and format of the instructional information according to the user's emotional state. This maximizes learning effectiveness.

[0718] Step 7:

[0719] Users access customized instructional information through their devices and learn skills using videos and guides. As users progress, they can provide feedback to the system with additional questions and requests.

[0720] (Example 2)

[0721] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0722] In modern times, the transmission of traditional techniques and skills is becoming increasingly difficult as those with specialized knowledge and experience age, making it challenging to properly pass them on to the next generation. Furthermore, accurately understanding and responding to the emotional states of individual learners during the learning process is difficult, hindering effective learning outcomes. Moreover, systems for effectively utilizing real-time feedback from learners to improve teaching content and models are not yet fully established.

[0723] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0724] In this invention, the server includes means for capturing video information of the actions of a subject performing a task, means for analyzing the video information and extracting the characteristics of the subject's movements, means for generating instructional information based on the analysis results, and means for monitoring the user's emotional state using an emotion analysis function and optimizing the instructional information. This enables the effective transmission of traditional techniques and skills to the next generation, provides appropriate instruction tailored to the individual emotional state of each learner, and is expected to improve learning efficiency.

[0725] "The entity performing the work" refers to an individual or group that implements a specific skill or technique, and is the subject of recording and analysis of their actions and techniques.

[0726] "Visual information" refers to visual data captured by cameras and video equipment, used for visually analyzing technology and actions.

[0727] "Analysis" is the process of extracting the subject's movements and tool movement patterns from acquired video information, and includes the process of quantifying technical characteristics.

[0728] "Instructional information" refers to information generated based on analysis results to enable the reproduction of technical and skill procedures, and is provided in text, audio, and video formats.

[0729] The "emotion analysis function" is a technology that analyzes the user's facial expressions and tone of voice in real time to determine their emotional state, and is used to optimize instructional information.

[0730] "Communication technology" refers to technology for transmitting data between one or more devices, and is used as a means of providing instructional information to users in remote locations.

[0731] "Response" refers to data obtained from user feedback and reactions, which is used to improve the system's guidance information and AI models.

[0732] This invention is a system for efficiently transmitting traditional techniques and skills held by users to the next generation, and includes information exchange between servers, terminals, and users. To implement this, the coordination of various hardware and software components is essential.

[0733] First, the server uses a high-resolution camera to film the worker's work and acquire video information. This captures the details of the work and collects data necessary for subsequent analysis. This video data is a crucial element for motion analysis, and the analysis is performed using a generative AI model. The generative AI model processes the captured video, extracts and standardizes motion features and patterns. For example, it can analyze the hand movements when shaping pottery and provide a guide for beginners to reproduce the procedure.

[0734] Next, the server generates instructional information based on the analysis results obtained. This instructional information is expressed in various formats such as text, images, audio, and video, and is transmitted to the terminal using communication technology. Users can access this instructional information through their terminal and learn the necessary skills.

[0735] Furthermore, the device incorporates an emotion analysis function. This is achieved by the device capturing the user's facial expressions and voice tone through its camera and microphone, and analyzing the resulting emotion data. This emotion data is sent to a server and used to adjust instructional information to improve the user's learning efficiency. If the server determines that the user is tired or lacking understanding, it can automatically provide additional explanations or specific examples.

[0736] As a concrete example, when assisting with the learning of the tea ceremony, the system first films the movements of the tea ceremony instructor, analyzes them, and generates a procedure. Then, when the user begins learning this procedure at home, if they show signs of confusion, the system provides additional detailed explanations. In this way, the user can be expected to learn the movements of the tea ceremony efficiently and effectively.

[0737] An example of a prompt message is, "Analyze high-resolution motion video and generate standardized motion patterns." This instructs the system to begin the analysis process using a generative AI model.

[0738] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0739] Step 1:

[0740] The server uses a high-resolution camera to record the technician's work in real time. The input is the video data captured by the camera. The server saves this data to storage. The output is video information that captures the detailed actions. At this stage, specific actions such as adjusting the camera's position and angle during shooting to create the optimal shooting environment are taken.

[0741] Step 2:

[0742] The server inputs the stored video information into the generating AI model and begins the analysis. The input is the acquired video information. The server uses the generating AI model to analyze each frame and extract the characteristics of the movements. The output is numerical data such as the trajectory and speed of the movements and the angles of the joints. Specifically, it identifies hand movements and tool manipulations in the video and quantifies them as data.

[0743] Step 3:

[0744] The server generates instructional information based on the analysis results. The input is numerical data extracted through analysis. The server processes this data using an algorithm to generate learning materials such as text, images, audio, and video. The output is instructional material that shows specific procedures. Specific actions here include text editing, audio narration creation, and diagram preparation.

[0745] Step 4:

[0746] The device uses the user's camera and microphone to collect the user's facial expressions and voice through emotion analysis. The input is real-time facial images and voice data. The device activates an emotion engine to determine the user's emotional state. The output is an emotion label such as smile or confusion. Specifically, it analyzes and classifies whether the user is smiling or has a stern expression.

[0747] Step 5:

[0748] The server receives the results of sentiment analysis and dynamically adjusts the instructional information. The input is the user's sentiment label. The server changes the content and format of the instructional information according to the user's level of understanding and emotional state. The output is the adjusted instructional information. For example, if a lack of understanding is detected, the server selects and provides supplementary explanations or additional videos.

[0749] Step 6:

[0750] Users view instructional information provided on their devices and engage in learning. The input is the instructional information itself. Users refer to this at their own pace and acquire skills. The output is the acquired techniques and skills. Specific actions include users playing videos on the screen or reading text.

[0751] (Application Example 2)

[0752] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0753] In factory and other industrial settings, there is a challenge in effectively transferring the movements of skilled workers to machines. Furthermore, there is a lack of means to flexibly optimize machine operation according to the situation at hand, thus creating a demand for efficient and safe work practices.

[0754] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0755] In this invention, the server includes means for capturing an individual's movements and acquiring video data of those movements, means for analyzing the video data to extract characteristics of the movements, and means for dynamically optimizing the machine's movements based on the acquired emotional information. This makes it possible to transfer the skills of skilled workers to the machine while optimizing the movements according to the work environment.

[0756] "Action recording" refers to the act of recording an individual's work process as video footage.

[0757] "Acquiring video data" refers to the process of saving the captured video footage of the action and sending it to a server in a format that can be analyzed.

[0758] "Extracting movement characteristics" refers to the act of analyzing video data to identify specific movement patterns or elements from an individual's body movements and how they use equipment.

[0759] "Generating instructional information" is the process of creating information and procedures based on the characteristics of an action, enabling others to reproduce similar actions.

[0760] "Emotional information" refers to data that represents an individual's psychological state and mood, and is analyzed from facial expressions and voice tone acquired through cameras and microphones.

[0761] "Dynamic optimization of operation" refers to using acquired emotional information to adjust machine operation in real time according to the work environment and situation, thereby improving efficiency and safety.

[0762] The system for implementing this invention consists of multiple elements. First, the server uses a high-resolution camera to capture an individual's movements and acquire video data. Next, the server analyzes this video data using a generative AI model to extract key features of the movements. Based on the results of this analysis, information for movement instruction is generated and provided to the user through an online platform.

[0763] Furthermore, the terminal is equipped with emotion recognition capabilities, which analyze the user's facial expressions and voice through the camera and microphone to obtain emotional information. This emotional information is sent to a server, which dynamically optimizes the machine's operation based on it. For example, if the server determines that a worker's attention is distracted in the work environment, it will temporarily slow down the operation of the automated machine to ensure safety. In addition, newly generated operation patterns are continuously adjusted based on the results of the emotional information analysis.

[0764] For example, in the assembly of precision equipment, the highly accurate procedures of skilled workers are captured by cameras and learned as robotic movement patterns. On the other hand, if the emotion engine detects that a worker is fatigued, the work speed is adjusted as needed, thereby achieving both production efficiency and safety.

[0765] An example of a prompt statement would be: "Develop a system that allows a robot to learn the movements of skilled factory workers and perform tasks safely and efficiently. The robot is required to read the worker's emotions and optimize the workflow."

[0766] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0767] Step 1:

[0768] The server uses high-resolution cameras to film the movements of skilled workers in the factory. The input is real-time video of the work, and this video data is acquired by the server. This data includes the movements of the workers and detailed operation of the equipment they use.

[0769] Step 2:

[0770] The server analyzes video data acquired using a generative AI model. The input is video data, and the AI ​​model extracts features such as body movements and equipment handling. The output is represented as numerical data in the form of standardized motion patterns. Here, the flow of movement and key points are quantified.

[0771] Step 3:

[0772] The server generates instructional information based on the analyzed behavioral patterns. The input is the extracted behavioral characteristics, and the output is step-by-step instructional information to help other users reproduce similar tasks. This information is provided online through the platform.

[0773] Step 4:

[0774] The device uses a camera and microphone to acquire user emotional information. Input consists of the user's facial expressions and voice, and an emotion engine analyzes this data. Output is emotional information indicating the user's psychological state. This serves as an important indicator of work-related stress levels and concentration.

[0775] Step 5:

[0776] The server receives emotional information transmitted from the terminal and dynamically optimizes its operation patterns. The input is emotional information, and the output is optimized machine operation commands. For example, if the user indicates fatigue, the server adjusts the machine's operation to ensure a safe and comfortable working environment.

[0777] Step 6:

[0778] The user can perform tasks based on optimized operational commands. The input is optimized guidance information and machine operation, and the output is the result of efficient and safe work.

[0779] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0780] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0781] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0782] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0783] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0784] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0785] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0786] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0787] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0788] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0789] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0790] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0791] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0792] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0793] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0794] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0795] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0796] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0797] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0798] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0799] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0800] The following is further disclosed regarding the embodiments described above.

[0801] (Claim 1)

[0802] A means for filming the actions of an individual performing a task and acquiring video data of said actions,

[0803] A means for analyzing the video data to extract the characteristics of the movements of an individual's hands, fingers, and tools,

[0804] A means for generating instructional information to make the procedure of operation reproducible based on the analysis results,

[0805] A means of providing the aforementioned instructional information to users through an online platform,

[0806] A system that includes this.

[0807] (Claim 2)

[0808] The system according to claim 1, which creates standardized patterns of movement by analyzing an individual's movements using an AI model.

[0809] (Claim 3)

[0810] The system according to claim 1, comprising means for collecting user feedback and updating AI models and instructional information.

[0811] "Example 1"

[0812] (Claim 1)

[0813] A means of capturing the actions of an individual performing a task and acquiring high-resolution video data,

[0814] A means for analyzing the video data using a generating AI model to extract the characteristics of an individual's hand and tool movements as numerical data,

[0815] A means for generating instructional information including reproducible operating procedures and necessary material information based on analysis results,

[0816] The aforementioned instructional information is provided to users through an online learning platform, and means are available to visually confirm the operation, including a video playback function.

[0817] A system that includes this.

[0818] (Claim 2)

[0819] The system according to claim 1, which analyzes an individual's movements using an AI model, creates standardized patterns of movement, and generates movement instruction information using these patterns.

[0820] (Claim 3)

[0821] The system according to claim 1, comprising means for collecting user feedback and continuously updating the generated AI model and instructional information.

[0822] "Application Example 1"

[0823] (Claim 1)

[0824] A means for filming the actions of an individual performing a task and acquiring video data of said actions,

[0825] A means for analyzing the video data to extract the characteristics of the movements of an individual's hands, fingers, and tools,

[0826] A means for generating instructional information to make the procedure of operation reproducible based on the analysis results,

[0827] Means for converting the aforementioned instruction information into control information for execution in machine operations,

[0828] Means for controlling a work device that automatically performs work based on the aforementioned control information,

[0829] A means for providing the aforementioned guidance information and control information to users through an online platform,

[0830] A system that includes this.

[0831] (Claim 2)

[0832] The system according to claim 1, which creates standardized patterns of movement by analyzing an individual's movements using an AI model, and makes these patterns applicable to machine work.

[0833] (Claim 3)

[0834] The system according to claim 1, comprising means for collecting user feedback and machine operation results, and updating AI models and instructional information.

[0835] "Example 2 of combining an emotion engine"

[0836] (Claim 1)

[0837] A means for photographing the actions of the person performing the work and acquiring video information of said actions,

[0838] A means for analyzing the video information to extract characteristics of the movement of the subject's body and tools,

[0839] A means for generating instructional information to make the procedure of operation reproducible based on the analysis results,

[0840] A means of monitoring the user's emotional state using an emotion analysis function and optimizing guidance information according to the user's state,

[0841] A means of providing the aforementioned instructional information to users through communication technology,

[0842] A system that includes this.

[0843] (Claim 2)

[0844] The system according to claim 1, which creates standardized patterns of action by analyzing the actions of the subject using a generative AI model.

[0845] (Claim 3)

[0846] The system according to claim 1, comprising means for collecting responses from users and updating AI models and instructional information.

[0847] "Application example 2 when combining with an emotional engine"

[0848] (Claim 1)

[0849] A means for filming the actions of an individual performing a task and acquiring video data of said actions,

[0850] A means for analyzing the video data to extract characteristics of the movement of an individual's limbs or equipment,

[0851] A means for generating instructional information to make the procedure of operation reproducible based on the analysis results,

[0852] A means of providing the aforementioned instructional information to users through an online platform,

[0853] A means for dynamically optimizing the operation of a machine performing a task based on acquired emotional information,

[0854] A system that includes this.

[0855] (Claim 2)

[0856] The system according to claim 1, which generates machine motion control information by creating standardized patterns of motion by analyzing an individual's movements using an AI model.

[0857] (Claim 3)

[0858] The system according to claim 1, which includes means for collecting emotional information from users and updating AI models, instructional information, and machine operation control information. [Explanation of Symbols]

[0859] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for filming the actions of an individual performing a task and acquiring video data of said actions, A means for analyzing the video data to extract the characteristics of the movements of an individual's hands, fingers, and tools, A means for generating instructional information to make the procedure of operation reproducible based on the analysis results, Means for converting the aforementioned instruction information into control information for execution in machine operations, Means for controlling a work device that automatically performs work based on the aforementioned control information, A means for providing the aforementioned guidance information and control information to users through an online platform, A system that includes this.

2. The system according to claim 1, which creates standardized patterns of movement by analyzing an individual's movements using an AI model, and makes these patterns applicable to machine work.

3. The system according to claim 1, comprising means for collecting user feedback and machine operation results, and updating AI models and instructional information.

Images