system

Abstract

We provide the system. [Solution] A terminal for recording work information, A server that aggregates and preprocesses work information transmitted from the terminal, The server provides a means for creating a summary of work information using a generative model and automatically generating a handover video, A means of distributing the handover video to the new person in charge's terminal, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a company, the handover of work often becomes inefficient due to different working methods and personalization of know-how for each person in charge, and it takes time and effort for a new person in charge to smoothly perform the work. Also, when the work content is complex, mistakes may occur due to insufficient handover, which may hinder the work. There is a need for an efficient and standardized handover method to solve such problems.

Means for Solving the Problems

[0005] This invention provides a system that records and collects work information on a terminal, and aggregates and preprocesses that information on a server. Furthermore, it summarizes the aggregated information using a generative model and automatically generates video content for handover. The generated video is distributed to the new person in charge, enabling them to understand the workflow visually and audibly, thereby facilitating a quick handover of tasks and preventing task dependency on specific individuals.

[0006] "Work information" refers to data such as the operations performed by the user, the time taken to execute them, and the tools used when performing a task.

[0007] "Terminal" refers to a computer or mobile device used by a user to record work information and send it to a server.

[0008] A "server" refers to a computer system that receives work information sent from terminals, aggregates and preprocesses it, and generates handover videos using a generative model.

[0009] A "generative model" refers to an algorithm that uses machine learning and natural language processing to analyze work information and generate summaries and handover content.

[0010] A "handover video" refers to automatically generated video content that uses a generative model to visually and audibly demonstrate the workflow and procedures to the new person in charge.

[0011] "New person in charge" refers to the person who will receive the handover of duties, and the user who will receive the handover video. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3]It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. <0OO0064>It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0014] First, the language used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This system's implementation primarily involves three functional elements: terminals, servers, and users.

[0034] 1. Recording and collecting work information

[0035] Terminal: The terminals used by users for their work are equipped with a dedicated tracking application that records daily work activities in detail. The application collects data such as programs used, operations performed, related file information, and work time.

[0036] 2. Data aggregation and preprocessing

[0037] Server: Work information collected by terminals is periodically sent to the server. The server receives this data and performs preprocessing such as deduplication, data formatting, and data imputation to improve data quality.

[0038] 3. Generating a handover video

[0039] Server: Preprocessed data is analyzed by a generative model, and the workflow and key points are automatically summarized. Based on this summary, the server generates a handover video that visually shows the steps necessary for the handover. The video includes narration and diagrams to ensure that the new person in charge can understand it intuitively.

[0040] 4. Video distribution and use

[0041] Server: The completed handover video is automatically delivered to the new person in charge's terminal. By watching the received video and learning the workflow and necessary procedures, the user can smoothly transition to their duties.

[0042] Specific example

[0043] For example, during the handover of accounting duties, the current employee's daily bookkeeping and data analysis procedures are recorded. The tracking system accurately records these operations, and uses the aggregated data on the server to generate a handover video. The video provides detailed explanations of how to use specific software and how to deal with errors to watch out for, allowing the new employee to learn the tricks of the trade and specific processing methods simply by watching the video.

[0044] In this way, this system streamlines the business handover process by efficiently recording and analyzing user work and providing that information in an easy-to-understand format for new personnel.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] Device: When a user begins their daily work, a tracking application on the device automatically launches. The app monitors the user's actions in the background, recording specific tasks, software used, data entered, and operation time in real time.

[0048] Step 2:

[0049] Terminal: The collected data is packaged at predetermined time intervals or after specific operational events and sent to the server via the internet. Before transmission, the data is encrypted using a secure protocol.

[0050] Step 3:

[0051] Server: Receives data, saves it to the database, and performs cleansing. Specifically, it removes duplicate data, standardizes the format, and fills in missing parts to ensure data integrity and quality.

[0052] Step 4:

[0053] Server: Inputs pre-processed data into a generative model to analyze the workflow. The generative model uses machine learning algorithms to analyze the data, extract important operations and frequent patterns, and create a summary.

[0054] Step 5:

[0055] Server: Based on the summarized data, it automatically generates a script for the handover video. This video details specific operating procedures, necessary tools, and points to note. Voice narration and annotations are added to the video to create visually easy-to-understand content.

[0056] Step 6:

[0057] Server: Once video generation is complete, the server will distribute the video to the new person in charge's terminal. The server will monitor the distribution status and confirm that the video was delivered without any problems.

[0058] Step 7:

[0059] User: New employees watch the received video and learn the tasks by following the steps explained in the video. If necessary, they can pause the video or replay specific steps to fully understand the tasks.

[0060] This process makes it possible to smoothly transfer responsibilities.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] In modern work environments, the handover process is inefficient, making it difficult for new employees to quickly grasp the necessary knowledge and procedures. In particular, accurately understanding the workflow requires manually organizing vast amounts of work information, which increases the likelihood of important procedures being lost during the handover process. Solving this problem is crucial for improving operational efficiency.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes means for aggregating and pre-processing work information transmitted from an information processing device, means for creating a summary of the work information using an artificial intelligence model and automatically generating visual content, and means for delivering the visual content to a new user's information processing device. This makes it possible to visually and intuitively communicate the workflow to a new person in charge.

[0066] An "information processing device" is an electronic device used by users to perform their work, and is equipped with the function of collecting and recording work information.

[0067] An "information processing server" is a computing device that receives data transmitted from an information processing device, performs aggregation and preprocessing, and has the function of processing that data for other purposes.

[0068] An "artificial intelligence model" refers to a computational algorithm that can analyze patterns from vast amounts of data and automatically generate results for a specific task.

[0069] "Visual content" refers to digital media that visually represents specific information, and includes images, videos, diagrams, etc.

[0070] "Hidden state" refers to a state that the user is not explicitly aware of, and it represents the operating mode in which the system collects data in the background.

[0071] "Audio descriptions" are audio explanations provided to complement visual content, and include information designed to help viewers better understand the content.

[0072] This invention is an information processing system for achieving efficient handover of tasks. It mainly consists of three elements: a terminal, a server, and a user.

[0073] terminal

[0074] The terminal refers to the information processing device used by the user for their daily work. A dedicated tracking program is installed on it, and work information is automatically recorded in an undisclosed manner. Specifically, this includes operation logs of the software the user uses, work time, and information on accessed files. This data is automatically collected at specific time intervals and sent to the server in an encrypted state.

[0075] server

[0076] The server is the central component that receives and analyzes data transmitted from the information processing device. Data aggregation and preprocessing are performed here. Preprocessing includes removing duplicate data, adjusting data formats, and imputing missing information. The preprocessed data is then analyzed by a generative AI model to create a summary of business procedures. This analysis uses prompts such as, for example, "Please create a video demonstrating the procedures and points to note when using accounting software X for bookkeeping." Based on this prompt, the generative AI model automatically generates a handover video as visual content.

[0077] User

[0078] The user, as the new person in charge of the task, receives visual content delivered from the server. The received video is played and viewable on the user's information processing device. The video includes not only a visual explanation of the operating procedure but also related audio commentary, allowing the new person in charge to intuitively understand the work procedure.

[0079] Thus, the system of the present invention has the effect of streamlining the business handover process and reducing the time and costs associated with the handover.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The terminal activates a tracking program when a user begins work, recording work information. This program discreetly collects logs of user actions, software used, accessed file information, and work time. Specifically, when data entry begins in accounting software, the operation steps and time are recorded. By acquiring this data, foundational information is obtained to understand the workflow in detail.

[0083] Step 2:

[0084] The terminal encrypts the collected work information at regular intervals and sends it to the server using a secure communication protocol. End-to-end encryption technology is used to maintain data confidentiality during transmission. The input is each user's work information log, and the output is encrypted data.

[0085] Step 3:

[0086] The server receives work information sent from the terminal and performs data preprocessing. First, it removes duplicate data and detects and corrects outliers. Next, it processes the data to fill in missing data and prepare it for a standard format. The input is encrypted work information, and by analyzing this, a clean and high-quality dataset is output. Through these processing steps, data suitable for subsequent analysis is prepared.

[0087] Step 4:

[0088] The server inputs pre-processed data into a generating AI model, which automatically summarizes the workflow and key points. Prompts such as "Summarize the main steps of the accounting process" are used. Based on this, the generating AI model creates a handover video to visually represent the work procedure. Here, the input is formatted data, and the output is video content based on the request.

[0089] Step 5:

[0090] The server compresses the generated handover video and automatically distributes it to the new employee's device. The distribution process utilizes technology to optimize data size and enable rapid downloads. Users receive notifications and learn the overview and procedures of the work by watching the video on their device. The output includes visual content provided to the new employee, which ultimately deepens their understanding of the work and supports a smooth handover.

[0091] (Application Example 1)

[0092] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0093] In modern manufacturing lines, efficient knowledge transfer to new workers and operators is essential. In particular, practical understanding of procedures and precautions is required when operating specialized machinery and robots, but this training is time-consuming and labor-intensive. There is a need to solve this problem and create a method for quickly and accurately transferring work responsibilities.

[0094] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0095] In this invention, the server includes means for aggregating and pre-processing work information, means for creating a summary of the work information using a generation algorithm and automatically generating a handover video, and means for analyzing records made using a visual information processing device and extracting points to note. As a result, new employees can smoothly learn complex work procedures and points to note by receiving visual and auditory explanations.

[0096] "Work information" refers to data collected during work activities, including records of operating procedures, the status of equipment used, and working hours.

[0097] An "information processing device" refers to a terminal used by workers to collect and record data.

[0098] An "information processing system" is a network system, such as a server, that aggregates and preprocesses data from information processing devices.

[0099] A "generation algorithm" is a set of procedures and calculation formulas used to generate summaries and handover videos based on data.

[0100] A "handover video" is a video that visually demonstrates the workflow and key points of a task, and may include audio narration or annotations.

[0101] A "visual information processing device" is a device used to record and analyze information obtained from the work environment and the worker's perspective.

[0102] "Application software" refers to applications installed on information processing devices that collect work information in the background.

[0103] To implement this invention, an information processing device for collecting and recording work information, an information processing system for pre-processing and analyzing the data, and a display device for viewing the generated video are required. The information processing device consists of a portable information terminal or visual information processing device (e.g., smart glasses) used by the worker, and records the procedures and machine operations during work in real time.

[0104] The server receives recorded work information at regular intervals and stores it in a database. The data is organized to avoid duplication and fill in missing information, and after quality assurance is achieved, it is summarized using a generation algorithm. This generation algorithm uses machine learning models such as PyTorch and TENSORFLOW®, and handover videos are automatically generated through data analysis.

[0105] The generated handover video includes visual explanations. This involves a process of incorporating visual information using OpenCV and FFmpeg, and it is also possible to add audio narration and annotations. The completed video is delivered to the new employee's information processing device using distribution technologies such as the Flask API.

[0106] As a concrete example, in a manufacturing plant, to teach new robot operation procedures, data recorded by experienced workers is analyzed to generate a video that visually and audibly explains the operation method to new workers. This video includes content designed to help viewers intuitively understand viewpoint control, operating procedures, and points to note. An example of a prompt message to be input into the generating AI model might be, "Use this program to generate a training video summarizing the necessary steps and points to note so that new factory workers can quickly learn specific robot operation procedures."

[0107] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0108] Step 1:

[0109] The terminal collects data in real time while the worker is performing their task. Specifically, it uses a visual information processing device to record video footage of the work and log information of operations. This data includes work procedures and machine operation history. Input is video and measurement data obtained from the worker's perspective, and output is the recorded raw data.

[0110] Step 2:

[0111] The server periodically receives work data sent from the terminal. After receiving the data, it performs preprocessing to remove duplicates, standardize the format, and fill in any missing data. The input is raw data sent from the terminal, and the output is normalized work data. Python or SQL can be used for this processing.

[0112] Step 3:

[0113] The server applies a generative AI model to preprocessed data to create a work summary. This model utilizes PyTorch and TensorFlow to extract and analyze important work steps and points to note. The input is normalized work data, and the output is a work summary based on the analysis results.

[0114] Step 4:

[0115] The server generates a handover video based on the summary data of the work. Using OpenCV and FFmpeg, it adds visual information and narration to the summary data to create easy-to-understand video content. The input is the summarized work data, and the output is the completed handover video.

[0116] Step 5:

[0117] The server distributes the generated handover video to the new employee's terminal. It uses the Flask API to send the video file, ensuring the user can access it at any time. The input is the handover video, and the output is the status of the video distribution to the terminal.

[0118] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0119] In this invention, in order to achieve efficient handover of tasks, a system is incorporated that records user work information and generates videos based on that information, as well as an emotion engine that recognizes the user's emotions.

[0120] 1. Recording of work information and emotional data

[0121] Device: As the user performs their tasks, a tracking application on the device records work information in the background. Simultaneously, an emotion engine acquires emotional data from the user's facial expressions and voice via the built-in camera and microphone.

[0122] 2. Sending work information and emotional data

[0123] Terminal: Emotional data and work information are sent to the server at regular intervals, and the data is aggregated on the server in a secure state.

[0124] 3. Data preprocessing and analysis

[0125] Server: The transmitted sentiment data and work information are pre-processed on the server to improve data consistency and accuracy. Here, the sentiment data is used to evaluate the workload and user stress levels.

[0126] 4. Generating a handover video

[0127] Server: A generative model that takes emotional data into account creates transition video content tailored to the user's emotional state. For example, if the user is experiencing stress, the model will highlight the details of that scene and create a concise and easy-to-understand guide.

[0128] 5. Video distribution and feedback

[0129] Server: The completed handover video is delivered to the new person in charge's terminal. Users watch the video and learn a methodology for handing over tasks based on their own emotional data. The system is also continuously improved based on feedback after viewing.

[0130] Specific example

[0131] For example, during the handover of customer service duties, the emotion engine records user smiles and frustrations to determine which stages require particular attention. This information is reflected in the handover video, which specifically explains problem-solving methods and customer service techniques. Through the video, the new employee can acquire the skills to appropriately respond to customer needs.

[0132] This invention enables the handover of tasks in conjunction with the analysis of emotional data, making it possible to provide an efficient and high-quality handover process.

[0133] The following describes the processing flow.

[0134] Step 1:

[0135] Terminal: When a user starts work, the tracking app and emotion engine automatically activate. The tracking app records user activity information (software used, activity content, time), while the emotion engine uses the camera and microphone to analyze emotional data (e.g., joy, anger, surprise) from the user's facial expressions and voice in real time.

[0136] Step 2:

[0137] Terminal: Collected work information and sentiment data are securely packaged and periodically sent to the server via a specified protocol. Data encryption is performed during this process.

[0138] Step 3:

[0139] Server: The received data is stored in the database, and the cleansing process begins immediately. Specifically, missing values ​​are imputed, the data format is corrected, and noisy data is removed. This generates a high-quality dataset.

[0140] Step 4:

[0141] Server: Pre-processed data is analyzed using sentiment analysis tools to determine the user's workload and stress level. The analysis results indicate which parts of the work were difficult for the user and are reflected in the generation of handover videos.

[0142] Step 5:

[0143] Server: Based on the analysis results, the generative model creates a specific handover video script. The video incorporates the workflow, points to note, and emotionally-sensitive responses. Furthermore, voice narration and text explanations are added to facilitate visual and auditory understanding.

[0144] Step 6:

[0145] Server: The completed handover video will be delivered to the new person in charge's terminal via streaming or download. The delivery status and viewing history will be logged and used for quality control.

[0146] Step 7:

[0147] User: New employees will watch the handover video they receive to acquire knowledge about work procedures. In particular, the inclusion of personalized advice based on sentiment analysis has the potential to improve the quality of work. Furthermore, by providing feedback, the system will be improved for future use.

[0148] In this way, an effective process for handing over responsibilities has been established.

[0149] (Example 2)

[0150] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0151] Traditional business handover processes rely on individual experience and subjectivity, leading to inaccuracies and inefficiencies. Furthermore, they often fail to consider the user's emotional state or stress level, making them difficult for new employees to understand. Therefore, there is a need to improve the efficiency and quality of business handover processes.

[0152] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0153] In this invention, the server includes means for aggregating and preprocessing work information and emotional data; means for creating a summary that takes work information and emotional data into consideration using a generative model and automatically generating a handover video; and means for distributing the handover video to the new person in charge and collecting feedback from the user. This enables highly accurate work handover incorporating emotional data.

[0154] "Work information" refers to detailed information such as operation data, application usage history, and work time recorded when a user performs a task.

[0155] "Emotional data" refers to information about a user's emotional state and stress level, obtained by analyzing changes in their facial expressions and vocal characteristics.

[0156] An "information processing device" is a terminal device that records work information and emotional data, and inputs and collects user-related data.

[0157] A "computer" is a central processing unit, such as a server, that aggregates, preprocesses, and analyzes data transmitted from information processing devices.

[0158] A "generative model" is a program that uses machine learning algorithms to automatically generate handover videos based on input data.

[0159] A "handover video" is an explanatory video for new employees, generated based on work information and emotional data, designed to clearly convey work procedures and points to note.

[0160] "Feedback" refers to opinions and evaluations regarding the understanding of the content and areas for improvement, collected from new staff members and users after they have watched the video.

[0161] This invention relates to a system for improving user work efficiency and facilitating smooth handovers. This system collects work information and emotional data from the user's terminal, and a server analyzes this data to generate a handover video.

[0162] The terminal is an information processing device on which the user performs their work, and it has an application installed that collects work information. This application records operation data related to the user's work in the background. In addition, it analyzes the user's facial expressions and voice through input devices such as cameras and microphones using an emotion engine to acquire emotion data.

[0163] The server receives work information and emotional data transmitted from the terminal. This data is preprocessed on the server to improve its accuracy and consistency. The server uses a generative AI model to analyze this data and automatically generate a handover video that takes into account the user's emotional state. The generated video clearly summarizes the work content and points to note and is provided to the new person in charge.

[0164] Users can learn about job handover procedures, including emotional data, by watching newly distributed handover videos. Users are encouraged to provide feedback on the video content, and this feedback will be used to improve the system.

[0165] As a concrete example, in customer service operations, an emotion engine may record a user's smiles and frustrations, and highlight particularly critical stages in a handover video. This allows new staff members to improve their customer service skills through the video.

[0166] An example of a prompt would be: "Use the emotion engine to record user smiles and frustrations during customer service interactions and generate a handover video based on that data. The video should include situations requiring special attention and their solutions."

[0167] This system is expected to enable efficient and emotionally conscious handover of tasks, thereby improving the quality of the handover process.

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] The device collects work information based on user actions. In this process, a tracking application installed on the device records the applications used, actions performed, and work time in the background. Simultaneously, an emotion engine analyzes the user's facial expressions and voice via the camera and microphone connected to the device to acquire emotion data. The inputs to this step are user actions and emotional changes, and the outputs are work information and emotion data.

[0171] Step 2:

[0172] The terminal transmits collected work information and sentiment data to the server at regular intervals. To ensure data security, encryption technology is used for communication, and a checksum is added to guarantee data integrity. The input to this step is the work information and sentiment data obtained in the previous step, and the output is an encrypted data packet.

[0173] Step 3:

[0174] The server receives data sent from the terminal and performs preprocessing. Specifically, it performs formatting to improve data consistency, filters outliers, and removes duplicate data. The input for this step is encrypted data received from the terminal, and the output is the preprocessed dataset.

[0175] Step 4:

[0176] The server uses a generation AI model based on pre-processed data to automatically generate a handover video summarizing the work content. Here, in addition to work information, the video content and tone are adjusted based on emotional data. For example, particularly detailed explanations are added in high-stress situations. The input for this step is the pre-processed data obtained in the previous step, and the output is the generated handover video.

[0177] Step 5:

[0178] The server delivers the completed video to the new person in charge's terminal. Streaming technology is used for delivery to ensure smooth video playback for the recipient. Furthermore, user feedback is collected after viewing the video to help improve the system. The input for this step is the generated handover video, and the output is the new person in charge's viewing and feedback data.

[0179] (Application Example 2)

[0180] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0181] In the traditional method of handing over on-site work, simply conveying the job content without considering the emotional state of the on-site staff makes efficient and high-quality handover difficult. Therefore, a system is needed that can convey to the new staff member the situation and emotions the previous staff member was experiencing. In particular, in on-site work where stress and anxiety are likely to occur, a lack of proper handover can lead to a decline in the quality and efficiency of work.

[0182] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0183] In this invention, the server includes an information processing device for recording work information and user emotional information; a computer device for aggregating and pre-processing the work information and emotional information transmitted from the information processing device; and means for creating a summary of the work information and emotional information using a generative model in the computer device and automatically generating a handover video corresponding to the emotional state. This makes it possible to create a detailed and situational handover video using emotional information as a starting point.

[0184] "Work information" refers to the entirety of user actions, operating procedures, and related data recorded during the performance of a task.

[0185] "Emotional information" refers to data that indicates the emotional state of a user, obtained from their facial expressions, tone of voice, gestures, etc.

[0186] An "information processing device" refers to a terminal device used for recording and transmitting data, such as a smartphone or dedicated device.

[0187] A "computer device" is a computer system that aggregates and preprocesses received work information and emotional information, and uses it for video generation.

[0188] A "generative model" is an algorithm that automatically generates meaningful summaries and content from received information.

[0189] A "handover video" is a visual content created based on work information and emotional information, intended to convey the details of the job to the next person in charge.

[0190] The system that realizes this application utilizes information processing equipment, computer devices, and generative models to efficiently collect, analyze, and generate handover videos of work and emotional information.

[0191] The information processing device is used by users as a smartphone or dedicated device while they perform their work, recording work information and emotional information in the background. This device acquires emotional data in real time using Azure® facial recognition APIs and other technologies, and stores the data.

[0192] The computer receives work information and emotional information periodically transmitted from the information processing device, and performs data aggregation and preprocessing. Here, processing is carried out to ensure data integrity and accuracy, and analysis is performed to evaluate workload and user stress levels. Furthermore, the data is visualized as a video using Google Cloud's video processing capabilities.

[0193] The generative model automatically generates a handover video for the next person in charge, based on analyzed work and emotional information. This video highlights important and emotionally challenging scenes and includes detailed visual and audio explanations. The generated video is then distributed to an information processing device, enabling the new person in charge to learn effectively.

[0194] As a concrete example, surveillance cameras record a security guard handling a large number of visitors in a shopping mall. A computer system identifies the moments when the guard is calmly handling the situation and generates a handover video highlighting those parts. This video is later used by new security guards for training.

[0195] An example of a prompt message is: "Generate a video demonstrating best practices for crowd management based on the emotional data of security guards during their work."

[0196] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0197] Step 1:

[0198] The device records work information and emotional information in real time while the user is performing their tasks. Work information includes user operation logs and activity patterns, while emotional information is based on facial and voice data acquired using the camera and microphone. Input data includes user operation data and sensor data. This data is converted into a predetermined format within the device and stored for subsequent processing.

[0199] Step 2:

[0200] The terminal transmits recorded work information and emotional information to the server at regular time intervals. During transmission, a secure protocol is used to maintain data integrity and confidentiality. The input is recorded data stored on the terminal, and the output generates data that is transmitted to the server via the network.

[0201] Step 3:

[0202] The server preprocesses the received data into a format suitable for analysis. This process involves data cleaning, such as removing noise and imputing missing values. The input is raw data received via the network, and the output is a refined dataset. This refined data is then used as input for a generative AI model.

[0203] Step 4:

[0204] The server uses pre-processed data to activate a generative AI model, summarizing work and emotional information and automatically generating a handover video. The model integrates elements based on emotional data into the video, particularly highlighting scenes where the user experienced stress. The input is a well-organized dataset, and the output is video data containing visual and audio explanations for the new employee.

[0205] Step 5:

[0206] The server distributes the generated handover video to the new employee's terminal. This distribution process also employs methods that prioritize security and efficiency. The input is the generated video data, and the output is the secure transfer of this data to a specific terminal. Based on this video, the new employee can learn and practice their duties.

[0207] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0208] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0209] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0210] [Second Embodiment]

[0211] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0212] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0213] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0214] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0215] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0216] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0217] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0218] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0219] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0220] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0221] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0222] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0223] This system's implementation primarily involves three functional elements: terminals, servers, and users.

[0224] 1. Recording and collecting work information

[0225] Terminal: The terminals used by users for their work are equipped with a dedicated tracking application that records daily work activities in detail. The application collects data such as programs used, operations performed, related file information, and work time.

[0226] 2. Data aggregation and preprocessing

[0227] Server: Work information collected by terminals is periodically sent to the server. The server receives this data and performs preprocessing such as deduplication, data formatting, and data imputation to improve data quality.

[0228] 3. Generating a handover video

[0229] Server: Preprocessed data is analyzed by a generative model, and the workflow and key points are automatically summarized. Based on this summary, the server generates a handover video that visually shows the steps necessary for the handover. The video includes narration and diagrams to ensure that the new person in charge can understand it intuitively.

[0230] 4. Video distribution and use

[0231] Server: The completed handover video is automatically delivered to the new person in charge's terminal. By watching the received video and learning the workflow and necessary procedures, the user can smoothly transition to their duties.

[0232] Specific example

[0233] For example, during the handover of accounting duties, the current employee's daily bookkeeping and data analysis procedures are recorded. The tracking system accurately records these operations, and uses the aggregated data on the server to generate a handover video. The video provides detailed explanations of how to use specific software and how to deal with errors to watch out for, allowing the new employee to learn the tricks of the trade and specific processing methods simply by watching the video.

[0234] In this way, this system streamlines the business handover process by efficiently recording and analyzing user work and providing that information in an easy-to-understand format for new personnel.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] Device: When a user begins their daily work, a tracking application on the device automatically launches. The app monitors the user's actions in the background, recording specific tasks, software used, data entered, and operation time in real time.

[0238] Step 2:

[0239] Terminal: The collected data is packaged at predetermined time intervals or after specific operational events and sent to the server via the internet. Before transmission, the data is encrypted using a secure protocol.

[0240] Step 3:

[0241] Server: Receives data, saves it to the database, and performs cleansing. Specifically, it removes duplicate data, standardizes the format, and fills in missing parts to ensure data integrity and quality.

[0242] Step 4:

[0243] Server: Inputs pre-processed data into a generative model to analyze the workflow. The generative model uses machine learning algorithms to analyze the data, extract important operations and frequent patterns, and create a summary.

[0244] Step 5:

[0245] Server: Based on the summarized data, it automatically generates a script for the handover video. This video details specific operating procedures, necessary tools, and points to note. Voice narration and annotations are added to the video to create visually easy-to-understand content.

[0246] Step 6:

[0247] Server: Once video generation is complete, the server will distribute the video to the new person in charge's terminal. The server will monitor the distribution status and confirm that the video was delivered without any problems.

[0248] Step 7:

[0249] User: New employees watch the received video and learn the tasks by following the steps explained in the video. If necessary, they can pause the video or replay specific steps to fully understand the tasks.

[0250] This process makes it possible to smoothly transfer responsibilities.

[0251] (Example 1)

[0252] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0253] In modern work environments, the handover process is inefficient, making it difficult for new employees to quickly grasp the necessary knowledge and procedures. In particular, accurately understanding the workflow requires manually organizing vast amounts of work information, which increases the likelihood of important procedures being lost during the handover process. Solving this problem is crucial for improving operational efficiency.

[0254] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0255] In this invention, the server includes means for aggregating and pre-processing work information transmitted from an information processing device, means for creating a summary of the work information using an artificial intelligence model and automatically generating visual content, and means for delivering the visual content to a new user's information processing device. This makes it possible to visually and intuitively communicate the workflow to a new person in charge.

[0256] An "information processing device" is an electronic device used by users to perform their work, and is equipped with the function of collecting and recording work information.

[0257] An "information processing server" is a computing device that receives data transmitted from an information processing device, performs aggregation and preprocessing, and has the function of processing that data for other purposes.

[0258] An "artificial intelligence model" refers to a computational algorithm that can analyze patterns from vast amounts of data and automatically generate results for a specific task.

[0259] "Visual content" refers to digital media that visually represents specific information, and includes images, videos, diagrams, etc.

[0260] "Hidden state" refers to a state that the user is not explicitly aware of, and it represents the operating mode in which the system collects data in the background.

[0261] "Audio descriptions" are audio explanations provided to complement visual content, and include information designed to help viewers better understand the content.

[0262] This invention is an information processing system for achieving efficient handover of tasks. It mainly consists of three elements: a terminal, a server, and a user.

[0263] terminal

[0264] The terminal refers to the information processing device used by the user for their daily work. A dedicated tracking program is installed on it, and work information is automatically recorded in an undisclosed manner. Specifically, this includes operation logs of the software the user uses, work time, and information on accessed files. This data is automatically collected at specific time intervals and sent to the server in an encrypted state.

[0265] server

[0266] The server is the central component that receives and analyzes data transmitted from the information processing device. Data aggregation and preprocessing are performed here. Preprocessing includes removing duplicate data, adjusting data formats, and imputing missing information. The preprocessed data is then analyzed by a generative AI model to create a summary of business procedures. This analysis uses prompts such as, for example, "Please create a video demonstrating the procedures and points to note when using accounting software X for bookkeeping." Based on this prompt, the generative AI model automatically generates a handover video as visual content.

[0267] User

[0268] The user, as the new person in charge of the task, receives visual content delivered from the server. The received video is played and viewable on the user's information processing device. The video includes not only a visual explanation of the operating procedure but also related audio commentary, allowing the new person in charge to intuitively understand the work procedure.

[0269] Thus, the system of the present invention has the effect of streamlining the business handover process and reducing the time and costs associated with the handover.

[0270] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0271] Step 1:

[0272] The terminal activates a tracking program when a user begins work, recording work information. This program discreetly collects logs of user actions, software used, accessed file information, and work time. Specifically, when data entry begins in accounting software, the operation steps and time are recorded. By acquiring this data, foundational information is obtained to understand the workflow in detail.

[0273] Step 2:

[0274] The terminal encrypts the collected work information at regular intervals and sends it to the server using a secure communication protocol. End-to-end encryption technology is used to maintain data confidentiality during transmission. The input is each user's work information log, and the output is encrypted data.

[0275] Step 3:

[0276] The server receives work information sent from the terminal and performs data preprocessing. First, it removes duplicate data and detects and corrects outliers. Next, it processes the data to fill in missing data and prepare it for a standard format. The input is encrypted work information, and by analyzing this, a clean and high-quality dataset is output. Through these processing steps, data suitable for subsequent analysis is prepared.

[0277] Step 4:

[0278] The server inputs pre-processed data into a generating AI model, which automatically summarizes the workflow and key points. Prompts such as "Summarize the main steps of the accounting process" are used. Based on this, the generating AI model creates a handover video to visually represent the work procedure. Here, the input is formatted data, and the output is video content based on the request.

[0279] Step 5:

[0280] The server compresses the generated handover video and automatically distributes it to the new employee's device. The distribution process utilizes technology to optimize data size and enable rapid downloads. Users receive notifications and learn the overview and procedures of the work by watching the video on their device. The output includes visual content provided to the new employee, which ultimately deepens their understanding of the work and supports a smooth handover.

[0281] (Application Example 1)

[0282] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0283] In modern manufacturing lines, efficient knowledge transfer to new workers and operators is essential. In particular, when operating specialized machinery and robots, it is required to practically understand the procedures and precautions, but the training for this takes time and effort. A method for solving this problem and quickly and accurately taking over the work content is required.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0285] In this invention, the server includes means for aggregating and preprocessing work information, means for creating a summary of the work information using a generation algorithm and automatically generating a handover video, and means for analyzing the records performed using a visual information processing device and extracting precautions. As a result, the new person in charge can smoothly acquire complex work procedures and precautions by receiving visual and audio explanations.

[0286] "Work information" is data collected during business activities and is a record including operation procedures, the status of used equipment, working hours, etc.

[0287] "Information processing device" is a terminal that collects and records data and refers to the device used by an operator.

[0288] "Information processing system" is a network system such as a server that aggregates data from an information processing device and performs preprocessing.

[0289] "Generation algorithm" is a set of procedures and calculation formulas used to generate a summary and handover video based on data.

[0290] A "handover video" is a video that visually demonstrates the workflow and key points of a task, and may include audio narration or annotations.

[0291] A "visual information processing device" is a device used to record and analyze information obtained from the work environment and the worker's perspective.

[0292] "Application software" refers to applications installed on information processing devices that collect work information in the background.

[0293] To implement this invention, an information processing device for collecting and recording work information, an information processing system for pre-processing and analyzing the data, and a display device for viewing the generated video are required. The information processing device consists of a portable information terminal or visual information processing device (e.g., smart glasses) used by the worker, and records the procedures and machine operations during work in real time.

[0294] The server receives recorded work information at regular intervals and stores it in a database. The data is organized to avoid duplication and fill in missing information, and after quality assurance is achieved, it is summarized using a generation algorithm. This generation algorithm uses machine learning models such as PyTorch and TensorFlow, and the handover video is automatically generated through data analysis.

[0295] The generated handover video includes visual explanations. This involves a process of incorporating visual information using OpenCV and FFmpeg, and it is also possible to add audio narration and annotations. The completed video is delivered to the new employee's information processing device using distribution technologies such as the Flask API.

[0296] As a concrete example, in a manufacturing plant, to teach new robot operation procedures, data recorded by experienced workers is analyzed to generate a video that visually and audibly explains the operation method to new workers. This video includes content designed to help viewers intuitively understand viewpoint control, operating procedures, and points to note. An example of a prompt message to be input into the generating AI model might be, "Use this program to generate a training video summarizing the necessary steps and points to note so that new factory workers can quickly learn specific robot operation procedures."

[0297] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0298] Step 1:

[0299] The terminal collects data in real time while the worker is performing their task. Specifically, it uses a visual information processing device to record video footage of the work and log information of operations. This data includes work procedures and machine operation history. Input is video and measurement data obtained from the worker's perspective, and output is the recorded raw data.

[0300] Step 2:

[0301] The server periodically receives work data sent from the terminal. After receiving the data, it performs preprocessing to remove duplicates, standardize the format, and fill in any missing data. The input is raw data sent from the terminal, and the output is normalized work data. Python or SQL can be used for this processing.

[0302] Step 3:

[0303] The server applies a generative AI model to preprocessed data to create a work summary. This model utilizes PyTorch and TensorFlow to extract and analyze important work steps and points to note. The input is normalized work data, and the output is a work summary based on the analysis results.

[0304] Step 4:

[0305] The server generates a handover video based on the summary data of the work. Using OpenCV or FFmpeg, visual information and narration are added to the summary data to create easy-to-understand video content. The input is the summarized work data, and the output is the completed handover video.

[0306] Step 5:

[0307] The server distributes the generated handover video to the terminal of the new person in charge. The Flask API is used to send the video file to ensure that the user can access it at any time. The input is the handover video, and the output is the video distribution status to the terminal.

[0308] Furthermore, an emotion engine for estimating the user's emotions may be combined. That is, the specific processing unit 290 may estimate the user's emotions using the emotion recognition model 59 and perform specific processing using the user's emotions.

[0309] In the present invention, in order to achieve efficient handover of work, in addition to a system that records the user's work information and generates a video based on it, an emotion engine for recognizing the user's emotions is incorporated.

[0310] 1. Recording of work information and emotion data

[0311] Terminal: When the user performs work, the tracking application on the terminal records the work information in the background. At the same time, emotion data is obtained from the user's expressions and voices via a camera or microphone with a built-in emotion engine.

[0312] 2. Transmission of work information and emotion data

[0313] Terminal: The emotion data and work information are sent to the server at regular time intervals and aggregated to the server in a state where the data is protected by security measures.

[0314] 3. Data preprocessing and analysis

[0315] Server: The transmitted sentiment data and work information are pre-processed on the server to improve data consistency and accuracy. Here, the sentiment data is used to evaluate the workload and user stress levels.

[0316] 4. Generating a handover video

[0317] Server: A generative model that takes emotional data into account creates transition video content tailored to the user's emotional state. For example, if the user is experiencing stress, the model will highlight the details of that scene and create a concise and easy-to-understand guide.

[0318] 5. Video distribution and feedback

[0319] Server: The completed handover video is delivered to the new person in charge's terminal. Users watch the video and learn a methodology for handing over tasks based on their own emotional data. The system is also continuously improved based on feedback after viewing.

[0320] Specific example

[0321] For example, during the handover of customer service duties, the emotion engine records user smiles and frustrations to determine which stages require particular attention. This information is reflected in the handover video, which specifically explains problem-solving methods and customer service techniques. Through the video, the new employee can acquire the skills to appropriately respond to customer needs.

[0322] This invention enables the handover of tasks in conjunction with the analysis of emotional data, making it possible to provide an efficient and high-quality handover process.

[0323] The following describes the processing flow.

[0324] Step 1:

[0325] Terminal: When a user starts work, the tracking app and emotion engine automatically activate. The tracking app records user activity information (software used, activity content, time), while the emotion engine uses the camera and microphone to analyze emotional data (e.g., joy, anger, surprise) from the user's facial expressions and voice in real time.

[0326] Step 2:

[0327] Terminal: Collected work information and sentiment data are securely packaged and periodically sent to the server via a specified protocol. Data encryption is performed during this process.

[0328] Step 3:

[0329] Server: The received data is stored in the database, and the cleansing process begins immediately. Specifically, missing values ​​are imputed, the data format is corrected, and noisy data is removed. This generates a high-quality dataset.

[0330] Step 4:

[0331] Server: Pre-processed data is analyzed using sentiment analysis tools to determine the user's workload and stress level. The analysis results indicate which parts of the work were difficult for the user and are reflected in the generation of handover videos.

[0332] Step 5:

[0333] Server: Based on the analysis results, the generative model creates a specific handover video script. The video incorporates the workflow, points to note, and emotionally-sensitive responses. Furthermore, voice narration and text explanations are added to facilitate visual and auditory understanding.

[0334] Step 6:

[0335] Server: The completed handover video will be delivered to the new person in charge's terminal via streaming or download. The delivery status and viewing history will be logged and used for quality control.

[0336] Step 7:

[0337] User: New employees will watch the handover video they receive to acquire knowledge about work procedures. In particular, the inclusion of personalized advice based on sentiment analysis has the potential to improve the quality of work. Furthermore, by providing feedback, the system will be improved for future use.

[0338] In this way, an effective process for handing over responsibilities has been established.

[0339] (Example 2)

[0340] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0341] Traditional business handover processes rely on individual experience and subjectivity, leading to inaccuracies and inefficiencies. Furthermore, they often fail to consider the user's emotional state or stress level, making them difficult for new employees to understand. Therefore, there is a need to improve the efficiency and quality of business handover processes.

[0342] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0343] In this invention, the server includes means for aggregating and preprocessing work information and emotional data; means for creating a summary that takes work information and emotional data into consideration using a generative model and automatically generating a handover video; and means for distributing the handover video to the new person in charge and collecting feedback from the user. This enables highly accurate work handover incorporating emotional data.

[0344] "Work information" refers to detailed information such as operation data, application usage history, and work time recorded when a user performs a task.

[0345] "Emotional data" refers to information about a user's emotional state and stress level, obtained by analyzing changes in their facial expressions and vocal characteristics.

[0346] An "information processing device" is a terminal device that records work information and emotional data, and inputs and collects user-related data.

[0347] A "computer" is a central processing unit, such as a server, that aggregates, preprocesses, and analyzes data transmitted from information processing devices.

[0348] A "generative model" is a program that uses machine learning algorithms to automatically generate handover videos based on input data.

[0349] A "handover video" is an explanatory video for new employees, generated based on work information and emotional data, designed to clearly convey work procedures and points to note.

[0350] "Feedback" refers to opinions and evaluations regarding the understanding of the content and areas for improvement, collected from new staff members and users after they have watched the video.

[0351] This invention relates to a system for improving user work efficiency and facilitating smooth handovers. This system collects work information and emotional data from the user's terminal, and a server analyzes this data to generate a handover video.

[0352] The terminal is an information processing device on which the user performs their work, and it has an application installed that collects work information. This application records operation data related to the user's work in the background. In addition, it analyzes the user's facial expressions and voice through input devices such as cameras and microphones using an emotion engine to acquire emotion data.

[0353] The server receives work information and emotional data transmitted from the terminal. This data is preprocessed on the server to improve its accuracy and consistency. The server uses a generative AI model to analyze this data and automatically generate a handover video that takes into account the user's emotional state. The generated video clearly summarizes the work content and points to note and is provided to the new person in charge.

[0354] Users can learn about job handover procedures, including emotional data, by watching newly distributed handover videos. Users are encouraged to provide feedback on the video content, and this feedback will be used to improve the system.

[0355] As a concrete example, in customer service operations, an emotion engine may record a user's smiles and frustrations, and highlight particularly critical stages in a handover video. This allows new staff members to improve their customer service skills through the video.

[0356] An example of a prompt would be: "Use the emotion engine to record user smiles and frustrations during customer service interactions and generate a handover video based on that data. The video should include situations requiring special attention and their solutions."

[0357] This system is expected to enable efficient and emotionally conscious handover of tasks, thereby improving the quality of the handover process.

[0358] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0359] Step 1:

[0360] The device collects work information based on user actions. In this process, a tracking application installed on the device records the applications used, actions performed, and work time in the background. Simultaneously, an emotion engine analyzes the user's facial expressions and voice via the camera and microphone connected to the device to acquire emotion data. The inputs to this step are user actions and emotional changes, and the outputs are work information and emotion data.

[0361] Step 2:

[0362] The terminal transmits collected work information and sentiment data to the server at regular intervals. To ensure data security, encryption technology is used for communication, and a checksum is added to guarantee data integrity. The input to this step is the work information and sentiment data obtained in the previous step, and the output is an encrypted data packet.

[0363] Step 3:

[0364] The server receives data sent from the terminal and performs preprocessing. Specifically, it performs formatting to improve data consistency, filters outliers, and removes duplicate data. The input for this step is encrypted data received from the terminal, and the output is the preprocessed dataset.

[0365] Step 4:

[0366] The server uses a generation AI model based on pre-processed data to automatically generate a handover video summarizing the work content. Here, in addition to work information, the video content and tone are adjusted based on emotional data. For example, particularly detailed explanations are added in high-stress situations. The input for this step is the pre-processed data obtained in the previous step, and the output is the generated handover video.

[0367] Step 5:

[0368] The server delivers the completed video to the new person in charge's terminal. Streaming technology is used for delivery to ensure smooth video playback for the recipient. Furthermore, user feedback is collected after viewing the video to help improve the system. The input for this step is the generated handover video, and the output is the new person in charge's viewing and feedback data.

[0369] (Application Example 2)

[0370] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0371] In the traditional method of handing over on-site work, simply conveying the job content without considering the emotional state of the on-site staff makes efficient and high-quality handover difficult. Therefore, a system is needed that can convey to the new staff member the situation and emotions the previous staff member was experiencing. In particular, in on-site work where stress and anxiety are likely to occur, a lack of proper handover can lead to a decline in the quality and efficiency of work.

[0372] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0373] In this invention, the server includes an information processing device for recording work information and user emotional information; a computer device for aggregating and pre-processing the work information and emotional information transmitted from the information processing device; and means for creating a summary of the work information and emotional information using a generative model in the computer device and automatically generating a handover video corresponding to the emotional state. This makes it possible to create a detailed and situational handover video using emotional information as a starting point.

[0374] "Work information" refers to the entirety of user actions, operating procedures, and related data recorded during the performance of a task.

[0375] "Emotional information" refers to data that indicates the emotional state of a user, obtained from their facial expressions, tone of voice, gestures, etc.

[0376] An "information processing device" refers to a terminal device used for recording and transmitting data, such as a smartphone or dedicated device.

[0377] A "computer device" is a computer system that aggregates and preprocesses received work information and emotional information, and uses it for video generation.

[0378] A "generative model" is an algorithm that automatically generates meaningful summaries and content from received information.

[0379] A "handover video" is a visual content created based on work information and emotional information, intended to convey the details of the job to the next person in charge.

[0380] The system that realizes this application utilizes information processing equipment, computer devices, and generative models to efficiently collect, analyze, and generate handover videos of work and emotional information.

[0381] The information processing device is used by users as a smartphone or dedicated device while they perform their work, recording work information and emotional information in the background. This device acquires emotional data in real time using Azure's facial recognition API and stores the data.

[0382] The computer receives work information and emotional information periodically transmitted from the information processing unit, and performs data aggregation and preprocessing. Here, processing is carried out to ensure data integrity and accuracy, and analysis is performed to evaluate workload and user stress levels. Furthermore, Google Cloud's video processing capabilities are used to visualize the data as a video.

[0383] The generative model automatically generates a handover video for the next person in charge, based on analyzed work and emotional information. This video highlights important and emotionally challenging scenes and includes detailed visual and audio explanations. The generated video is then distributed to an information processing device, enabling the new person in charge to learn effectively.

[0384] As a concrete example, surveillance cameras record a security guard handling a large number of visitors in a shopping mall. A computer system identifies the moments when the guard is calmly handling the situation and generates a handover video highlighting those parts. This video is later used by new security guards for training.

[0385] An example of a prompt message is: "Generate a video demonstrating best practices for crowd management based on the emotional data of security guards during their work."

[0386] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0387] Step 1:

[0388] The device records work information and emotional information in real time while the user is performing their tasks. Work information includes user operation logs and activity patterns, while emotional information is based on facial and voice data acquired using the camera and microphone. Input data includes user operation data and sensor data. This data is converted into a predetermined format within the device and stored for subsequent processing.

[0389] Step 2:

[0390] The terminal transmits recorded work information and emotional information to the server at regular time intervals. During transmission, a secure protocol is used to maintain data integrity and confidentiality. The input is recorded data stored on the terminal, and the output generates data that is transmitted to the server via the network.

[0391] Step 3:

[0392] The server preprocesses the received data into a format suitable for analysis. This process involves data cleaning, such as removing noise and imputing missing values. The input is raw data received via the network, and the output is a refined dataset. This refined data is then used as input for a generative AI model.

[0393] Step 4:

[0394] The server uses pre-processed data to activate a generative AI model, summarizing work and emotional information and automatically generating a handover video. The model integrates elements based on emotional data into the video, particularly highlighting scenes where the user experienced stress. The input is a well-organized dataset, and the output is video data containing visual and audio explanations for the new employee.

[0395] Step 5:

[0396] The server distributes the generated handover video to the new employee's terminal. This distribution process also employs methods that prioritize security and efficiency. The input is the generated video data, and the output is the secure transfer of this data to a specific terminal. Based on this video, the new employee can learn and practice their duties.

[0397] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0398] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0399] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0400] [Third Embodiment]

[0401] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0402] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0403] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0404] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0405] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0406] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0407] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0408] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0409] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0410] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0411] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0412] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0413] This system's implementation primarily involves three functional elements: terminals, servers, and users.

[0414] 1. Recording and collecting work information

[0415] Terminal: The terminals used by users for their work are equipped with a dedicated tracking application that records daily work activities in detail. The application collects data such as programs used, operations performed, related file information, and work time.

[0416] 2. Data aggregation and preprocessing

[0417] Server: Work information collected by terminals is periodically sent to the server. The server receives this data and performs preprocessing such as deduplication, data formatting, and data imputation to improve data quality.

[0418] 3. Generating a handover video

[0419] Server: Preprocessed data is analyzed by a generative model, and the workflow and key points are automatically summarized. Based on this summary, the server generates a handover video that visually shows the steps necessary for the handover. The video includes narration and diagrams to ensure that the new person in charge can understand it intuitively.

[0420] 4. Video distribution and use

[0421] Server: The completed handover video is automatically delivered to the new person in charge's terminal. By watching the received video and learning the workflow and necessary procedures, the user can smoothly transition to their duties.

[0422] Specific example

[0423] For example, during the handover of accounting duties, the current employee's daily bookkeeping and data analysis procedures are recorded. The tracking system accurately records these operations, and uses the aggregated data on the server to generate a handover video. The video provides detailed explanations of how to use specific software and how to deal with errors to watch out for, allowing the new employee to learn the tricks of the trade and specific processing methods simply by watching the video.

[0424] In this way, this system streamlines the business handover process by efficiently recording and analyzing user work and providing that information in an easy-to-understand format for new personnel.

[0425] The following describes the processing flow.

[0426] Step 1:

[0427] Device: When a user begins their daily work, a tracking application on the device automatically launches. The app monitors the user's actions in the background, recording specific tasks, software used, data entered, and operation time in real time.

[0428] Step 2:

[0429] Terminal: The collected data is packaged at predetermined time intervals or after specific operational events and sent to the server via the internet. Before transmission, the data is encrypted using a secure protocol.

[0430] Step 3:

[0431] Server: Receives data, saves it to the database, and performs cleansing. Specifically, it removes duplicate data, standardizes the format, and fills in missing parts to ensure data integrity and quality.

[0432] Step 4:

[0433] Server: Inputs pre-processed data into a generative model to analyze the workflow. The generative model uses machine learning algorithms to analyze the data, extract important operations and frequent patterns, and create a summary.

[0434] Step 5:

[0435] Server: Based on the summarized data, it automatically generates a script for the handover video. This video details specific operating procedures, necessary tools, and points to note. Voice narration and annotations are added to the video to create visually easy-to-understand content.

[0436] Step 6:

[0437] Server: Once video generation is complete, the server will distribute the video to the new person in charge's terminal. The server will monitor the distribution status and confirm that the video was delivered without any problems.

[0438] Step 7:

[0439] User: New employees watch the received video and learn the tasks by following the steps explained in the video. If necessary, they can pause the video or replay specific steps to fully understand the tasks.

[0440] This process makes it possible to smoothly transfer responsibilities.

[0441] (Example 1)

[0442] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0443] In modern work environments, the handover process is inefficient, making it difficult for new employees to quickly grasp the necessary knowledge and procedures. In particular, accurately understanding the workflow requires manually organizing vast amounts of work information, which increases the likelihood of important procedures being lost during the handover process. Solving this problem is crucial for improving operational efficiency.

[0444] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0445] In this invention, the server includes means for aggregating and pre-processing work information transmitted from an information processing device, means for creating a summary of the work information using an artificial intelligence model and automatically generating visual content, and means for delivering the visual content to a new user's information processing device. This makes it possible to visually and intuitively communicate the workflow to a new person in charge.

[0446] An "information processing device" is an electronic device used by users to perform their work, and is equipped with the function of collecting and recording work information.

[0447] An "information processing server" is a computing device that receives data transmitted from an information processing device, performs aggregation and preprocessing, and has the function of processing that data for other purposes.

[0448] An "artificial intelligence model" refers to a computational algorithm that can analyze patterns from vast amounts of data and automatically generate results for a specific task.

[0449] "Visual content" refers to digital media that visually represents specific information, and includes images, videos, diagrams, etc.

[0450] "Hidden state" refers to a state that the user is not explicitly aware of, and it represents the operating mode in which the system collects data in the background.

[0451] "Audio descriptions" are audio explanations provided to complement visual content, and include information designed to help viewers better understand the content.

[0452] This invention is an information processing system for achieving efficient handover of tasks. It mainly consists of three elements: a terminal, a server, and a user.

[0453] terminal

[0454] The terminal refers to the information processing device used by the user for their daily work. A dedicated tracking program is installed on it, and work information is automatically recorded in an undisclosed manner. Specifically, this includes operation logs of the software the user uses, work time, and information on accessed files. This data is automatically collected at specific time intervals and sent to the server in an encrypted state.

[0455] server

[0456] The server is the central component that receives and analyzes data transmitted from the information processing device. Data aggregation and preprocessing are performed here. Preprocessing includes removing duplicate data, adjusting data formats, and imputing missing information. The preprocessed data is then analyzed by a generative AI model to create a summary of business procedures. This analysis uses prompts such as, for example, "Please create a video demonstrating the procedures and points to note when using accounting software X for bookkeeping." Based on this prompt, the generative AI model automatically generates a handover video as visual content.

[0457] User

[0458] The user, as the new person in charge of the task, receives visual content delivered from the server. The received video is played and viewable on the user's information processing device. The video includes not only a visual explanation of the operating procedure but also related audio commentary, allowing the new person in charge to intuitively understand the work procedure.

[0459] Thus, the system of the present invention has the effect of streamlining the business handover process and reducing the time and costs associated with the handover.

[0460] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0461] Step 1:

[0462] The terminal activates a tracking program when a user begins work, recording work information. This program discreetly collects logs of user actions, software used, accessed file information, and work time. Specifically, when data entry begins in accounting software, the operation steps and time are recorded. By acquiring this data, foundational information is obtained to understand the workflow in detail.

[0463] Step 2:

[0464] The terminal encrypts the collected work information at regular intervals and sends it to the server using a secure communication protocol. End-to-end encryption technology is used to maintain data confidentiality during transmission. The input is each user's work information log, and the output is encrypted data.

[0465] Step 3:

[0466] The server receives work information sent from the terminal and performs data preprocessing. First, it removes duplicate data and detects and corrects outliers. Next, it processes the data to fill in missing data and prepare it for a standard format. The input is encrypted work information, and by analyzing this, a clean and high-quality dataset is output. Through these processing steps, data suitable for subsequent analysis is prepared.

[0467] Step 4:

[0468] The server inputs pre-processed data into a generating AI model, which automatically summarizes the workflow and key points. Prompts such as "Summarize the main steps of the accounting process" are used. Based on this, the generating AI model creates a handover video to visually represent the work procedure. Here, the input is formatted data, and the output is video content based on the request.

[0469] Step 5:

[0470] The server compresses the generated handover video and automatically distributes it to the new employee's device. The distribution process utilizes technology to optimize data size and enable rapid downloads. Users receive notifications and learn the overview and procedures of the work by watching the video on their device. The output includes visual content provided to the new employee, which ultimately deepens their understanding of the work and supports a smooth handover.

[0471] (Application Example 1)

[0472] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0473] In modern manufacturing lines, efficient knowledge transfer to new workers and operators is essential. In particular, practical understanding of procedures and precautions is required when operating specialized machinery and robots, but this training is time-consuming and labor-intensive. There is a need to solve this problem and create a method for quickly and accurately transferring work responsibilities.

[0474] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0475] In this invention, the server includes means for aggregating and pre-processing work information, means for creating a summary of the work information using a generation algorithm and automatically generating a handover video, and means for analyzing records made using a visual information processing device and extracting points to note. As a result, new employees can smoothly learn complex work procedures and points to note by receiving visual and auditory explanations.

[0476] "Work information" refers to data collected during work activities, including records of operating procedures, the status of equipment used, and working hours.

[0477] An "information processing device" refers to a terminal used by workers to collect and record data.

[0478] An "information processing system" is a network system, such as a server, that aggregates and preprocesses data from information processing devices.

[0479] A "generation algorithm" is a set of procedures and calculation formulas used to generate summaries and handover videos based on data.

[0480] A "handover video" is a video that visually demonstrates the workflow and key points of a task, and may include audio narration or annotations.

[0481] A "visual information processing device" is a device used to record and analyze information obtained from the work environment and the worker's perspective.

[0482] "Application software" refers to applications installed on information processing devices that collect work information in the background.

[0483] To implement this invention, an information processing device for collecting and recording work information, an information processing system for pre-processing and analyzing the data, and a display device for viewing the generated video are required. The information processing device consists of a portable information terminal or visual information processing device (e.g., smart glasses) used by the worker, and records the procedures and machine operations during work in real time.

[0484] The server receives recorded work information at regular intervals and stores it in a database. The data is organized to avoid duplication and fill in missing information, and after quality assurance is achieved, it is summarized using a generation algorithm. This generation algorithm uses machine learning models such as PyTorch and TensorFlow, and the handover video is automatically generated through data analysis.

[0485] The generated handover video includes visual explanations. This involves a process of incorporating visual information using OpenCV and FFmpeg, and it is also possible to add audio narration and annotations. The completed video is delivered to the new employee's information processing device using distribution technologies such as the Flask API.

[0486] As a concrete example, in a manufacturing plant, to teach new robot operation procedures, data recorded by experienced workers is analyzed to generate a video that visually and audibly explains the operation method to new workers. This video includes content designed to help viewers intuitively understand viewpoint control, operating procedures, and points to note. An example of a prompt message to be input into the generating AI model might be, "Use this program to generate a training video summarizing the necessary steps and points to note so that new factory workers can quickly learn specific robot operation procedures."

[0487] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0488] Step 1:

[0489] The terminal collects data in real time while the worker is performing their task. Specifically, it uses a visual information processing device to record video footage of the work and log information of operations. This data includes work procedures and machine operation history. Input is video and measurement data obtained from the worker's perspective, and output is the recorded raw data.

[0490] Step 2:

[0491] The server periodically receives work data sent from the terminal. After receiving the data, it performs preprocessing to remove duplicates, standardize the format, and fill in any missing data. The input is raw data sent from the terminal, and the output is normalized work data. Python or SQL can be used for this processing.

[0492] Step 3:

[0493] The server applies a generative AI model to preprocessed data to create a work summary. This model utilizes PyTorch and TensorFlow to extract and analyze important work steps and points to note. The input is normalized work data, and the output is a work summary based on the analysis results.

[0494] Step 4:

[0495] The server generates a handover video based on the summary data of the work. Using OpenCV and FFmpeg, it adds visual information and narration to the summary data to create easy-to-understand video content. The input is the summarized work data, and the output is the completed handover video.

[0496] Step 5:

[0497] The server distributes the generated handover video to the new employee's terminal. It uses the Flask API to send the video file, ensuring the user can access it at any time. The input is the handover video, and the output is the status of the video distribution to the terminal.

[0498] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0499] In this invention, in order to achieve efficient handover of tasks, a system is incorporated that records user work information and generates videos based on that information, as well as an emotion engine that recognizes the user's emotions.

[0500] 1. Recording of work information and emotional data

[0501] Device: As the user performs their tasks, a tracking application on the device records work information in the background. Simultaneously, an emotion engine acquires emotional data from the user's facial expressions and voice via the built-in camera and microphone.

[0502] 2. Sending work information and emotional data

[0503] Terminal: Emotional data and work information are sent to the server at regular intervals, and the data is aggregated on the server in a secure state.

[0504] 3. Data preprocessing and analysis

[0505] Server: The transmitted sentiment data and work information are pre-processed on the server to improve data consistency and accuracy. Here, the sentiment data is used to evaluate the workload and user stress levels.

[0506] 4. Generating a handover video

[0507] Server: A generative model that takes emotional data into account creates transition video content tailored to the user's emotional state. For example, if the user is experiencing stress, the model will highlight the details of that scene and create a concise and easy-to-understand guide.

[0508] 5. Video distribution and feedback

[0509] Server: The completed handover video is delivered to the new person in charge's terminal. Users watch the video and learn a methodology for handing over tasks based on their own emotional data. The system is also continuously improved based on feedback after viewing.

[0510] Specific example

[0511] For example, during the handover of customer service duties, the emotion engine records user smiles and frustrations to determine which stages require particular attention. This information is reflected in the handover video, which specifically explains problem-solving methods and customer service techniques. Through the video, the new employee can acquire the skills to appropriately respond to customer needs.

[0512] This invention enables the handover of tasks in conjunction with the analysis of emotional data, making it possible to provide an efficient and high-quality handover process.

[0513] The following describes the processing flow.

[0514] Step 1:

[0515] Terminal: When a user starts work, the tracking app and emotion engine automatically activate. The tracking app records user activity information (software used, activity content, time), while the emotion engine uses the camera and microphone to analyze emotional data (e.g., joy, anger, surprise) from the user's facial expressions and voice in real time.

[0516] Step 2:

[0517] Terminal: Collected work information and sentiment data are securely packaged and periodically sent to the server via a specified protocol. Data encryption is performed during this process.

[0518] Step 3:

[0519] Server: The received data is stored in the database, and the cleansing process begins immediately. Specifically, missing values ​​are imputed, the data format is corrected, and noisy data is removed. This generates a high-quality dataset.

[0520] Step 4:

[0521] Server: Pre-processed data is analyzed using sentiment analysis tools to determine the user's workload and stress level. The analysis results indicate which parts of the work were difficult for the user and are reflected in the generation of handover videos.

[0522] Step 5:

[0523] Server: Based on the analysis results, the generative model creates a specific handover video script. The video incorporates the workflow, points to note, and emotionally-sensitive responses. Furthermore, voice narration and text explanations are added to facilitate visual and auditory understanding.

[0524] Step 6:

[0525] Server: The completed handover video will be delivered to the new person in charge's terminal via streaming or download. The delivery status and viewing history will be logged and used for quality control.

[0526] Step 7:

[0527] User: New employees will watch the handover video they receive to acquire knowledge about work procedures. In particular, the inclusion of personalized advice based on sentiment analysis has the potential to improve the quality of work. Furthermore, by providing feedback, the system will be improved for future use.

[0528] In this way, an effective process for handing over responsibilities has been established.

[0529] (Example 2)

[0530] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0531] Traditional business handover processes rely on individual experience and subjectivity, leading to inaccuracies and inefficiencies. Furthermore, they often fail to consider the user's emotional state or stress level, making them difficult for new employees to understand. Therefore, there is a need to improve the efficiency and quality of business handover processes.

[0532] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0533] In this invention, the server includes means for aggregating and preprocessing work information and emotional data; means for creating a summary that takes work information and emotional data into consideration using a generative model and automatically generating a handover video; and means for distributing the handover video to the new person in charge and collecting feedback from the user. This enables highly accurate work handover incorporating emotional data.

[0534] "Work information" refers to detailed information such as operation data, application usage history, and work time recorded when a user performs a task.

[0535] "Emotional data" refers to information about a user's emotional state and stress level, obtained by analyzing changes in their facial expressions and vocal characteristics.

[0536] An "information processing device" is a terminal device that records work information and emotional data, and inputs and collects user-related data.

[0537] A "computer" is a central processing unit, such as a server, that aggregates, preprocesses, and analyzes data transmitted from information processing devices.

[0538] A "generative model" is a program that uses machine learning algorithms to automatically generate handover videos based on input data.

[0539] A "handover video" is an explanatory video for new employees, generated based on work information and emotional data, designed to clearly convey work procedures and points to note.

[0540] "Feedback" refers to opinions and evaluations regarding the understanding of the content and areas for improvement, collected from new staff members and users after they have watched the video.

[0541] This invention relates to a system for improving user work efficiency and facilitating smooth handovers. This system collects work information and emotional data from the user's terminal, and a server analyzes this data to generate a handover video.

[0542] The terminal is an information processing device on which the user performs their work, and it has an application installed that collects work information. This application records operation data related to the user's work in the background. In addition, it analyzes the user's facial expressions and voice through input devices such as cameras and microphones using an emotion engine to acquire emotion data.

[0543] The server receives work information and emotional data transmitted from the terminal. This data is preprocessed on the server to improve its accuracy and consistency. The server uses a generative AI model to analyze this data and automatically generate a handover video that takes into account the user's emotional state. The generated video clearly summarizes the work content and points to note and is provided to the new person in charge.

[0544] Users can learn about job handover procedures, including emotional data, by watching newly distributed handover videos. Users are encouraged to provide feedback on the video content, and this feedback will be used to improve the system.

[0545] As a concrete example, in customer service operations, an emotion engine may record a user's smiles and frustrations, and highlight particularly critical stages in a handover video. This allows new staff members to improve their customer service skills through the video.

[0546] An example of a prompt would be: "Use the emotion engine to record user smiles and frustrations during customer service interactions and generate a handover video based on that data. The video should include situations requiring special attention and their solutions."

[0547] This system is expected to enable efficient and emotionally conscious handover of tasks, thereby improving the quality of the handover process.

[0548] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0549] Step 1:

[0550] The device collects work information based on user actions. In this process, a tracking application installed on the device records the applications used, actions performed, and work time in the background. Simultaneously, an emotion engine analyzes the user's facial expressions and voice via the camera and microphone connected to the device to acquire emotion data. The inputs to this step are user actions and emotional changes, and the outputs are work information and emotion data.

[0551] Step 2:

[0552] The terminal transmits collected work information and sentiment data to the server at regular intervals. To ensure data security, encryption technology is used for communication, and a checksum is added to guarantee data integrity. The input to this step is the work information and sentiment data obtained in the previous step, and the output is an encrypted data packet.

[0553] Step 3:

[0554] The server receives data sent from the terminal and performs preprocessing. Specifically, it performs formatting to improve data consistency, filters outliers, and removes duplicate data. The input for this step is encrypted data received from the terminal, and the output is the preprocessed dataset.

[0555] Step 4:

[0556] The server uses a generation AI model based on pre-processed data to automatically generate a handover video summarizing the work content. Here, in addition to work information, the video content and tone are adjusted based on emotional data. For example, particularly detailed explanations are added in high-stress situations. The input for this step is the pre-processed data obtained in the previous step, and the output is the generated handover video.

[0557] Step 5:

[0558] The server delivers the completed video to the new person in charge's terminal. Streaming technology is used for delivery to ensure smooth video playback for the recipient. Furthermore, user feedback is collected after viewing the video to help improve the system. The input for this step is the generated handover video, and the output is the new person in charge's viewing and feedback data.

[0559] (Application Example 2)

[0560] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0561] In the traditional method of handing over on-site work, simply conveying the job content without considering the emotional state of the on-site staff makes efficient and high-quality handover difficult. Therefore, a system is needed that can convey to the new staff member the situation and emotions the previous staff member was experiencing. In particular, in on-site work where stress and anxiety are likely to occur, a lack of proper handover can lead to a decline in the quality and efficiency of work.

[0562] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0563] In this invention, the server includes an information processing device for recording work information and user emotional information; a computer device for aggregating and pre-processing the work information and emotional information transmitted from the information processing device; and means for creating a summary of the work information and emotional information using a generative model in the computer device and automatically generating a handover video corresponding to the emotional state. This makes it possible to create a detailed and situational handover video using emotional information as a starting point.

[0564] "Work information" refers to the entirety of user actions, operating procedures, and related data recorded during the performance of a task.

[0565] "Emotional information" refers to data that indicates the emotional state of a user, obtained from their facial expressions, tone of voice, gestures, etc.

[0566] An "information processing device" refers to a terminal device used for recording and transmitting data, such as a smartphone or dedicated device.

[0567] A "computer device" is a computer system that aggregates and preprocesses received work information and emotional information, and uses it for video generation.

[0568] A "generative model" is an algorithm that automatically generates meaningful summaries and content from received information.

[0569] A "handover video" is a visual content created based on work information and emotional information, intended to convey the details of the job to the next person in charge.

[0570] The system that realizes this application utilizes information processing equipment, computer devices, and generative models to efficiently collect, analyze, and generate handover videos of work and emotional information.

[0571] The information processing device is used by users as a smartphone or dedicated device while they perform their work, recording work information and emotional information in the background. This device acquires emotional data in real time using Azure's facial recognition API and stores the data.

[0572] The computer receives work information and emotional information periodically transmitted from the information processing unit, and performs data aggregation and preprocessing. Here, processing is carried out to ensure data integrity and accuracy, and analysis is performed to evaluate workload and user stress levels. Furthermore, Google Cloud's video processing capabilities are used to visualize the data as a video.

[0573] The generative model automatically generates a handover video for the next person in charge, based on analyzed work and emotional information. This video highlights important and emotionally challenging scenes and includes detailed visual and audio explanations. The generated video is then distributed to an information processing device, enabling the new person in charge to learn effectively.

[0574] As a concrete example, surveillance cameras record a security guard handling a large number of visitors in a shopping mall. A computer system identifies the moments when the guard is calmly handling the situation and generates a handover video highlighting those parts. This video is later used by new security guards for training.

[0575] An example of a prompt message is: "Generate a video demonstrating best practices for crowd management based on the emotional data of security guards during their work."

[0576] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0577] Step 1:

[0578] The device records work information and emotional information in real time while the user is performing their tasks. Work information includes user operation logs and activity patterns, while emotional information is based on facial and voice data acquired using the camera and microphone. Input data includes user operation data and sensor data. This data is converted into a predetermined format within the device and stored for subsequent processing.

[0579] Step 2:

[0580] The terminal transmits recorded work information and emotional information to the server at regular time intervals. During transmission, a secure protocol is used to maintain data integrity and confidentiality. The input is recorded data stored on the terminal, and the output generates data that is transmitted to the server via the network.

[0581] Step 3:

[0582] The server preprocesses the received data into a format suitable for analysis. This process involves data cleaning, such as removing noise and imputing missing values. The input is raw data received via the network, and the output is a refined dataset. This refined data is then used as input for a generative AI model.

[0583] Step 4:

[0584] The server uses pre-processed data to activate a generative AI model, summarizing work and emotional information and automatically generating a handover video. The model integrates elements based on emotional data into the video, particularly highlighting scenes where the user experienced stress. The input is a well-organized dataset, and the output is video data containing visual and audio explanations for the new employee.

[0585] Step 5:

[0586] The server distributes the generated handover video to the new employee's terminal. This distribution process also employs methods that prioritize security and efficiency. The input is the generated video data, and the output is the secure transfer of this data to a specific terminal. Based on this video, the new employee can learn and practice their duties.

[0587] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0588] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0589] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0590] [Fourth Embodiment]

[0591] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0592] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0593] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0594] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0595] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0596] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0597] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0598] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0599] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0600] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0601] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0602] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0603] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0604] This system's implementation primarily involves three functional elements: terminals, servers, and users.

[0605] 1. Recording and collecting work information

[0606] Terminal: The terminals used by users for their work are equipped with a dedicated tracking application that records daily work activities in detail. The application collects data such as programs used, operations performed, related file information, and work time.

[0607] 2. Data aggregation and preprocessing

[0608] Server: Work information collected by terminals is periodically sent to the server. The server receives this data and performs preprocessing such as deduplication, data formatting, and data imputation to improve data quality.

[0609] 3. Generating a handover video

[0610] Server: Preprocessed data is analyzed by a generative model, and the workflow and key points are automatically summarized. Based on this summary, the server generates a handover video that visually shows the steps necessary for the handover. The video includes narration and diagrams to ensure that the new person in charge can understand it intuitively.

[0611] 4. Video distribution and use

[0612] Server: The completed handover video is automatically delivered to the new person in charge's terminal. By watching the received video and learning the workflow and necessary procedures, the user can smoothly transition to their duties.

[0613] Specific example

[0614] For example, during the handover of accounting duties, the current employee's daily bookkeeping and data analysis procedures are recorded. The tracking system accurately records these operations, and uses the aggregated data on the server to generate a handover video. The video provides detailed explanations of how to use specific software and how to deal with errors to watch out for, allowing the new employee to learn the tricks of the trade and specific processing methods simply by watching the video.

[0615] In this way, this system streamlines the business handover process by efficiently recording and analyzing user work and providing that information in an easy-to-understand format for new personnel.

[0616] The following describes the processing flow.

[0617] Step 1:

[0618] Device: When a user begins their daily work, a tracking application on the device automatically launches. The app monitors the user's actions in the background, recording specific tasks, software used, data entered, and operation time in real time.

[0619] Step 2:

[0620] Terminal: The collected data is packaged at predetermined time intervals or after specific operational events and sent to the server via the internet. Before transmission, the data is encrypted using a secure protocol.

[0621] Step 3:

[0622] Server: Receives data, saves it to the database, and performs cleansing. Specifically, it removes duplicate data, standardizes the format, and fills in missing parts to ensure data integrity and quality.

[0623] Step 4:

[0624] Server: Inputs pre-processed data into a generative model to analyze the workflow. The generative model uses machine learning algorithms to analyze the data, extract important operations and frequent patterns, and create a summary.

[0625] Step 5:

[0626] Server: Based on the summarized data, it automatically generates a script for the handover video. This video details specific operating procedures, necessary tools, and points to note. Voice narration and annotations are added to the video to create visually easy-to-understand content.

[0627] Step 6:

[0628] Server: Once video generation is complete, the server will distribute the video to the new person in charge's terminal. The server will monitor the distribution status and confirm that the video was delivered without any problems.

[0629] Step 7:

[0630] User: New employees watch the received video and learn the tasks by following the steps explained in the video. If necessary, they can pause the video or replay specific steps to fully understand the tasks.

[0631] This process makes it possible to smoothly transfer responsibilities.

[0632] (Example 1)

[0633] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0634] In modern work environments, the handover process is inefficient, making it difficult for new employees to quickly grasp the necessary knowledge and procedures. In particular, accurately understanding the workflow requires manually organizing vast amounts of work information, which increases the likelihood of important procedures being lost during the handover process. Solving this problem is crucial for improving operational efficiency.

[0635] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0636] In this invention, the server includes means for aggregating and pre-processing work information transmitted from an information processing device, means for creating a summary of the work information using an artificial intelligence model and automatically generating visual content, and means for delivering the visual content to a new user's information processing device. This makes it possible to visually and intuitively communicate the workflow to a new person in charge.

[0637] An "information processing device" is an electronic device used by users to perform their work, and is equipped with the function of collecting and recording work information.

[0638] An "information processing server" is a computing device that receives data transmitted from an information processing device, performs aggregation and preprocessing, and has the function of processing that data for other purposes.

[0639] An "artificial intelligence model" refers to a computational algorithm that can analyze patterns from vast amounts of data and automatically generate results for a specific task.

[0640] "Visual content" refers to digital media that visually represents specific information, and includes images, videos, diagrams, etc.

[0641] "Hidden state" refers to a state that the user is not explicitly aware of, and it represents the operating mode in which the system collects data in the background.

[0642] "Audio descriptions" are audio explanations provided to complement visual content, and include information designed to help viewers better understand the content.

[0643] This invention is an information processing system for achieving efficient handover of tasks. It mainly consists of three elements: a terminal, a server, and a user.

[0644] terminal

[0645] The terminal refers to the information processing device used by the user for their daily work. A dedicated tracking program is installed on it, and work information is automatically recorded in an undisclosed manner. Specifically, this includes operation logs of the software the user uses, work time, and information on accessed files. This data is automatically collected at specific time intervals and sent to the server in an encrypted state.

[0646] server

[0647] The server is the central component that receives and analyzes data transmitted from the information processing device. Data aggregation and preprocessing are performed here. Preprocessing includes removing duplicate data, adjusting data formats, and imputing missing information. The preprocessed data is then analyzed by a generative AI model to create a summary of business procedures. This analysis uses prompts such as, for example, "Please create a video demonstrating the procedures and points to note when using accounting software X for bookkeeping." Based on this prompt, the generative AI model automatically generates a handover video as visual content.

[0648] User

[0649] The user, as the new person in charge of the task, receives visual content delivered from the server. The received video is played and viewable on the user's information processing device. The video includes not only a visual explanation of the operating procedure but also related audio commentary, allowing the new person in charge to intuitively understand the work procedure.

[0650] Thus, the system of the present invention has the effect of streamlining the business handover process and reducing the time and costs associated with the handover.

[0651] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0652] Step 1:

[0653] The terminal activates a tracking program when a user begins work, recording work information. This program discreetly collects logs of user actions, software used, accessed file information, and work time. Specifically, when data entry begins in accounting software, the operation steps and time are recorded. By acquiring this data, foundational information is obtained to understand the workflow in detail.

[0654] Step 2:

[0655] The terminal encrypts the collected work information at regular intervals and sends it to the server using a secure communication protocol. End-to-end encryption technology is used to maintain data confidentiality during transmission. The input is each user's work information log, and the output is encrypted data.

[0656] Step 3:

[0657] The server receives work information sent from the terminal and performs data preprocessing. First, it removes duplicate data and detects and corrects outliers. Next, it processes the data to fill in missing data and prepare it for a standard format. The input is encrypted work information, and by analyzing this, a clean and high-quality dataset is output. Through these processing steps, data suitable for subsequent analysis is prepared.

[0658] Step 4:

[0659] The server inputs pre-processed data into a generating AI model, which automatically summarizes the workflow and key points. Prompts such as "Summarize the main steps of the accounting process" are used. Based on this, the generating AI model creates a handover video to visually represent the work procedure. Here, the input is formatted data, and the output is video content based on the request.

[0660] Step 5:

[0661] The server compresses the generated handover video and automatically distributes it to the new employee's device. The distribution process utilizes technology to optimize data size and enable rapid downloads. Users receive notifications and learn the overview and procedures of the work by watching the video on their device. The output includes visual content provided to the new employee, which ultimately deepens their understanding of the work and supports a smooth handover.

[0662] (Application Example 1)

[0663] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0664] In modern manufacturing lines, efficient knowledge transfer to new workers and operators is essential. In particular, practical understanding of procedures and precautions is required when operating specialized machinery and robots, but this training is time-consuming and labor-intensive. There is a need to solve this problem and create a method for quickly and accurately transferring work responsibilities.

[0665] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0666] In this invention, the server includes means for aggregating and pre-processing work information, means for creating a summary of the work information using a generation algorithm and automatically generating a handover video, and means for analyzing records made using a visual information processing device and extracting points to note. As a result, new employees can smoothly learn complex work procedures and points to note by receiving visual and auditory explanations.

[0667] "Work information" refers to data collected during work activities, including records of operating procedures, the status of equipment used, and working hours.

[0668] An "information processing device" refers to a terminal used by workers to collect and record data.

[0669] An "information processing system" is a network system, such as a server, that aggregates and preprocesses data from information processing devices.

[0670] A "generation algorithm" is a set of procedures and calculation formulas used to generate summaries and handover videos based on data.

[0671] A "handover video" is a video that visually demonstrates the workflow and key points of a task, and may include audio narration or annotations.

[0672] A "visual information processing device" is a device used to record and analyze information obtained from the work environment and the worker's perspective.

[0673] "Application software" refers to applications installed on information processing devices that collect work information in the background.

[0674] To implement this invention, an information processing device for collecting and recording work information, an information processing system for pre-processing and analyzing the data, and a display device for viewing the generated video are required. The information processing device consists of a portable information terminal or visual information processing device (e.g., smart glasses) used by the worker, and records the procedures and machine operations during work in real time.

[0675] The server receives recorded work information at regular intervals and stores it in a database. The data is organized to avoid duplication and fill in missing information, and after quality assurance is achieved, it is summarized using a generation algorithm. This generation algorithm uses machine learning models such as PyTorch and TensorFlow, and the handover video is automatically generated through data analysis.

[0676] The generated handover video includes visual explanations. This involves a process of incorporating visual information using OpenCV and FFmpeg, and it is also possible to add audio narration and annotations. The completed video is delivered to the new employee's information processing device using distribution technologies such as the Flask API.

[0677] As a concrete example, in a manufacturing plant, to teach new robot operation procedures, data recorded by experienced workers is analyzed to generate a video that visually and audibly explains the operation method to new workers. This video includes content designed to help viewers intuitively understand viewpoint control, operating procedures, and points to note. An example of a prompt message to be input into the generating AI model might be, "Use this program to generate a training video summarizing the necessary steps and points to note so that new factory workers can quickly learn specific robot operation procedures."

[0678] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0679] Step 1:

[0680] The terminal collects data in real time while the worker is performing their task. Specifically, it uses a visual information processing device to record video footage of the work and log information of operations. This data includes work procedures and machine operation history. Input is video and measurement data obtained from the worker's perspective, and output is the recorded raw data.

[0681] Step 2:

[0682] The server periodically receives work data sent from the terminal. After receiving the data, it performs preprocessing to remove duplicates, standardize the format, and fill in any missing data. The input is raw data sent from the terminal, and the output is normalized work data. Python or SQL can be used for this processing.

[0683] Step 3:

[0684] The server applies a generative AI model to preprocessed data to create a work summary. This model utilizes PyTorch and TensorFlow to extract and analyze important work steps and points to note. The input is normalized work data, and the output is a work summary based on the analysis results.

[0685] Step 4:

[0686] The server generates a handover video based on the summary data of the work. Using OpenCV and FFmpeg, it adds visual information and narration to the summary data to create easy-to-understand video content. The input is the summarized work data, and the output is the completed handover video.

[0687] Step 5:

[0688] The server distributes the generated handover video to the new employee's terminal. It uses the Flask API to send the video file, ensuring the user can access it at any time. The input is the handover video, and the output is the status of the video distribution to the terminal.

[0689] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0690] In this invention, in order to achieve efficient handover of tasks, a system is incorporated that records user work information and generates videos based on that information, as well as an emotion engine that recognizes the user's emotions.

[0691] 1. Recording of work information and emotional data

[0692] Device: As the user performs their tasks, a tracking application on the device records work information in the background. Simultaneously, an emotion engine acquires emotional data from the user's facial expressions and voice via the built-in camera and microphone.

[0693] 2. Sending work information and emotional data

[0694] Terminal: Emotional data and work information are sent to the server at regular intervals, and the data is aggregated on the server in a secure state.

[0695] 3. Data preprocessing and analysis

[0696] Server: The transmitted sentiment data and work information are pre-processed on the server to improve data consistency and accuracy. Here, the sentiment data is used to evaluate the workload and user stress levels.

[0697] 4. Generating a handover video

[0698] Server: A generative model that takes emotional data into account creates transition video content tailored to the user's emotional state. For example, if the user is experiencing stress, the model will highlight the details of that scene and create a concise and easy-to-understand guide.

[0699] 5. Video distribution and feedback

[0700] Server: The completed handover video is delivered to the new person in charge's terminal. Users watch the video and learn a methodology for handing over tasks based on their own emotional data. The system is also continuously improved based on feedback after viewing.

[0701] Specific example

[0702] For example, during the handover of customer service duties, the emotion engine records user smiles and frustrations to determine which stages require particular attention. This information is reflected in the handover video, which specifically explains problem-solving methods and customer service techniques. Through the video, the new employee can acquire the skills to appropriately respond to customer needs.

[0703] This invention enables the handover of tasks in conjunction with the analysis of emotional data, making it possible to provide an efficient and high-quality handover process.

[0704] The following describes the processing flow.

[0705] Step 1:

[0706] Terminal: When a user starts work, the tracking app and emotion engine automatically activate. The tracking app records user activity information (software used, activity content, time), while the emotion engine uses the camera and microphone to analyze emotional data (e.g., joy, anger, surprise) from the user's facial expressions and voice in real time.

[0707] Step 2:

[0708] Terminal: Collected work information and sentiment data are securely packaged and periodically sent to the server via a specified protocol. Data encryption is performed during this process.

[0709] Step 3:

[0710] Server: The received data is stored in the database, and the cleansing process begins immediately. Specifically, missing values ​​are imputed, the data format is corrected, and noisy data is removed. This generates a high-quality dataset.

[0711] Step 4:

[0712] Server: Pre-processed data is analyzed using sentiment analysis tools to determine the user's workload and stress level. The analysis results indicate which parts of the work were difficult for the user and are reflected in the generation of handover videos.

[0713] Step 5:

[0714] Server: Based on the analysis results, the generative model creates a specific handover video script. The video incorporates the workflow, points to note, and emotionally-sensitive responses. Furthermore, voice narration and text explanations are added to facilitate visual and auditory understanding.

[0715] Step 6:

[0716] Server: The completed handover video will be delivered to the new person in charge's terminal via streaming or download. The delivery status and viewing history will be logged and used for quality control.

[0717] Step 7:

[0718] User: New employees will watch the handover video they receive to acquire knowledge about work procedures. In particular, the inclusion of personalized advice based on sentiment analysis has the potential to improve the quality of work. Furthermore, by providing feedback, the system will be improved for future use.

[0719] In this way, an effective process for handing over responsibilities has been established.

[0720] (Example 2)

[0721] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0722] Traditional business handover processes rely on individual experience and subjectivity, leading to inaccuracies and inefficiencies. Furthermore, they often fail to consider the user's emotional state or stress level, making them difficult for new employees to understand. Therefore, there is a need to improve the efficiency and quality of business handover processes.

[0723] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0724] In this invention, the server includes means for aggregating and preprocessing work information and emotional data; means for creating a summary that takes work information and emotional data into consideration using a generative model and automatically generating a handover video; and means for distributing the handover video to the new person in charge and collecting feedback from the user. This enables highly accurate work handover incorporating emotional data.

[0725] "Work information" refers to detailed information such as operation data, application usage history, and work time recorded when a user performs a task.

[0726] "Emotional data" refers to information about a user's emotional state and stress level, obtained by analyzing changes in their facial expressions and vocal characteristics.

[0727] An "information processing device" is a terminal device that records work information and emotional data, and inputs and collects user-related data.

[0728] A "computer" is a central processing unit, such as a server, that aggregates, preprocesses, and analyzes data transmitted from information processing devices.

[0729] A "generative model" is a program that uses machine learning algorithms to automatically generate handover videos based on input data.

[0730] A "handover video" is an explanatory video for new employees, generated based on work information and emotional data, designed to clearly convey work procedures and points to note.

[0731] "Feedback" refers to opinions and evaluations regarding the understanding of the content and areas for improvement, collected from new staff members and users after they have watched the video.

[0732] This invention relates to a system for improving user work efficiency and facilitating smooth handovers. This system collects work information and emotional data from the user's terminal, and a server analyzes this data to generate a handover video.

[0733] The terminal is an information processing device on which the user performs their work, and it has an application installed that collects work information. This application records operation data related to the user's work in the background. In addition, it analyzes the user's facial expressions and voice through input devices such as cameras and microphones using an emotion engine to acquire emotion data.

[0734] The server receives work information and emotional data transmitted from the terminal. This data is preprocessed on the server to improve its accuracy and consistency. The server uses a generative AI model to analyze this data and automatically generate a handover video that takes into account the user's emotional state. The generated video clearly summarizes the work content and points to note and is provided to the new person in charge.

[0735] Users can learn about job handover procedures, including emotional data, by watching newly distributed handover videos. Users are encouraged to provide feedback on the video content, and this feedback will be used to improve the system.

[0736] As a concrete example, in customer service operations, an emotion engine may record a user's smiles and frustrations, and highlight particularly critical stages in a handover video. This allows new staff members to improve their customer service skills through the video.

[0737] An example of a prompt would be: "Use the emotion engine to record user smiles and frustrations during customer service interactions and generate a handover video based on that data. The video should include situations requiring special attention and their solutions."

[0738] This system is expected to enable efficient and emotionally conscious handover of tasks, thereby improving the quality of the handover process.

[0739] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0740] Step 1:

[0741] The device collects work information based on user actions. In this process, a tracking application installed on the device records the applications used, actions performed, and work time in the background. Simultaneously, an emotion engine analyzes the user's facial expressions and voice via the camera and microphone connected to the device to acquire emotion data. The inputs to this step are user actions and emotional changes, and the outputs are work information and emotion data.

[0742] Step 2:

[0743] The terminal transmits collected work information and sentiment data to the server at regular intervals. To ensure data security, encryption technology is used for communication, and a checksum is added to guarantee data integrity. The input to this step is the work information and sentiment data obtained in the previous step, and the output is an encrypted data packet.

[0744] Step 3:

[0745] The server receives data sent from the terminal and performs preprocessing. Specifically, it performs formatting to improve data consistency, filters outliers, and removes duplicate data. The input for this step is encrypted data received from the terminal, and the output is the preprocessed dataset.

[0746] Step 4:

[0747] The server uses a generation AI model based on pre-processed data to automatically generate a handover video summarizing the work content. Here, in addition to work information, the video content and tone are adjusted based on emotional data. For example, particularly detailed explanations are added in high-stress situations. The input for this step is the pre-processed data obtained in the previous step, and the output is the generated handover video.

[0748] Step 5:

[0749] The server delivers the completed video to the new person in charge's terminal. Streaming technology is used for delivery to ensure smooth video playback for the recipient. Furthermore, user feedback is collected after viewing the video to help improve the system. The input for this step is the generated handover video, and the output is the new person in charge's viewing and feedback data.

[0750] (Application Example 2)

[0751] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0752] In the traditional method of handing over on-site work, simply conveying the job content without considering the emotional state of the on-site staff makes efficient and high-quality handover difficult. Therefore, a system is needed that can convey to the new staff member the situation and emotions the previous staff member was experiencing. In particular, in on-site work where stress and anxiety are likely to occur, a lack of proper handover can lead to a decline in the quality and efficiency of work.

[0753] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0754] In this invention, the server includes an information processing device for recording work information and user emotional information; a computer device for aggregating and pre-processing the work information and emotional information transmitted from the information processing device; and means for creating a summary of the work information and emotional information using a generative model in the computer device and automatically generating a handover video corresponding to the emotional state. This makes it possible to create a detailed and situational handover video using emotional information as a starting point.

[0755] "Work information" refers to the entirety of user actions, operating procedures, and related data recorded during the performance of a task.

[0756] "Emotional information" refers to data that indicates the emotional state of a user, obtained from their facial expressions, tone of voice, gestures, etc.

[0757] An "information processing device" refers to a terminal device used for recording and transmitting data, such as a smartphone or dedicated device.

[0758] A "computer device" is a computer system that aggregates and preprocesses received work information and emotional information, and uses it for video generation.

[0759] A "generative model" is an algorithm that automatically generates meaningful summaries and content from received information.

[0760] A "handover video" is a visual content created based on work information and emotional information, intended to convey the details of the job to the next person in charge.

[0761] The system that realizes this application utilizes information processing equipment, computer devices, and generative models to efficiently collect, analyze, and generate handover videos of work and emotional information.

[0762] The information processing device is used by users as a smartphone or dedicated device while they perform their work, recording work information and emotional information in the background. This device acquires emotional data in real time using Azure's facial recognition API and stores the data.

[0763] The computer receives work information and emotional information periodically transmitted from the information processing unit, and performs data aggregation and preprocessing. Here, processing is carried out to ensure data integrity and accuracy, and analysis is performed to evaluate workload and user stress levels. Furthermore, Google Cloud's video processing capabilities are used to visualize the data as a video.

[0764] The generative model automatically generates a handover video for the next person in charge, based on analyzed work and emotional information. This video highlights important and emotionally challenging scenes and includes detailed visual and audio explanations. The generated video is then distributed to an information processing device, enabling the new person in charge to learn effectively.

[0765] As a concrete example, surveillance cameras record a security guard handling a large number of visitors in a shopping mall. A computer system identifies the moments when the guard is calmly handling the situation and generates a handover video highlighting those parts. This video is later used by new security guards for training.

[0766] An example of a prompt message is: "Generate a video demonstrating best practices for crowd management based on the emotional data of security guards during their work."

[0767] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0768] Step 1:

[0769] The device records work information and emotional information in real time while the user is performing their tasks. Work information includes user operation logs and activity patterns, while emotional information is based on facial and voice data acquired using the camera and microphone. Input data includes user operation data and sensor data. This data is converted into a predetermined format within the device and stored for subsequent processing.

[0770] Step 2:

[0771] The terminal transmits recorded work information and emotional information to the server at regular time intervals. During transmission, a secure protocol is used to maintain data integrity and confidentiality. The input is recorded data stored on the terminal, and the output generates data that is transmitted to the server via the network.

[0772] Step 3:

[0773] The server preprocesses the received data into a format suitable for analysis. This process involves data cleaning, such as removing noise and imputing missing values. The input is raw data received via the network, and the output is a refined dataset. This refined data is then used as input for a generative AI model.

[0774] Step 4:

[0775] The server uses pre-processed data to activate a generative AI model, summarizing work and emotional information and automatically generating a handover video. The model integrates elements based on emotional data into the video, particularly highlighting scenes where the user experienced stress. The input is a well-organized dataset, and the output is video data containing visual and audio explanations for the new employee.

[0776] Step 5:

[0777] The server distributes the generated handover video to the new employee's terminal. This distribution process also employs methods that prioritize security and efficiency. The input is the generated video data, and the output is the secure transfer of this data to a specific terminal. Based on this video, the new employee can learn and practice their duties.

[0778] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0779] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0780] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0781] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0782] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0783] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0784] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0785] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0786] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0787] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0788] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0789] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0790] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0791] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0792] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0793] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0794] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0795] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0796] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0797] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0798] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0799] The following is further disclosed regarding the embodiments described above.

[0800] (Claim 1)

[0801] A terminal for recording work information,

[0802] A server that aggregates and preprocesses work information transmitted from the terminal,

[0803] The server provides a means for creating a summary of work information using a generative model and automatically generating a handover video,

[0804] A means of distributing the handover video to the new person in charge's terminal,

[0805] A system that includes this.

[0806] (Claim 2)

[0807] The system according to claim 1, wherein the collection of the aforementioned work information is performed in the background by an application installed on the terminal.

[0808] (Claim 3)

[0809] The system according to claim 1, wherein the server automatically generates content including audio narration in addition to the visual explanation generated in the handover video.

[0810] "Example 1"

[0811] (Claim 1)

[0812] An information processing device for recording work information,

[0813] An information processing server that aggregates and preprocesses work information transmitted from the information processing device,

[0814] The information processing server includes means for creating a summary of work information using an artificial intelligence model and automatically generating visual content,

[0815] A means for distributing the visual content to a new user's information processing device,

[0816] An information processing system that includes this.

[0817] (Claim 2)

[0818] The information processing system according to claim 1, wherein the collection of the aforementioned work information is performed in a hidden state by a program installed on the information processing device.

[0819] (Claim 3)

[0820] The information processing system according to claim 1, wherein the information processing server automatically generates information including audio commentary in addition to the visual explanation generated in the visual content.

[0821] "Application Example 1"

[0822] (Claim 1)

[0823] An information processing device for recording work information,

[0824] An information processing system that aggregates and preprocesses work information transmitted from the information processing device,

[0825] The information processing system includes a means for creating a summary of work information using a generation algorithm and automatically generating a handover video,

[0826] A means for distributing the handover video to the information processing device of the new person in charge,

[0827] The information processing device records information using a visual information processing device, and includes means for analyzing the visual information to extract points of interest.

[0828] ...

[0829] A system that includes this.

[0830] (Claim 2)

[0831] The system according to claim 1, wherein the collection of the aforementioned work information is performed in the background by application software installed on the information processing device.

[0832] (Claim 3)

[0833] The system according to claim 1, wherein the information processing system automatically generates content including audio narration in addition to the visual explanation generated in the handover video.

[0834] "Example 2 of combining an emotion engine"

[0835] (Claim 1)

[0836] An information processing device for recording work information and emotional data,

[0837] A computer that aggregates and preprocesses work information and emotional data transmitted from the information processing device,

[0838] The computer provides a means for creating a summary that takes into account work information and emotional data using a generative model, and for automatically generating a handover video.

[0839] A means of distributing the handover video to the new person in charge's information processing device and collecting user feedback,

[0840] A system that includes this.

[0841] (Claim 2)

[0842] The system according to claim 1, wherein the collection of the work information and emotion data is performed in the background by a program installed on the information processing device.

[0843] (Claim 3)

[0844] The system according to claim 1, wherein the computer automatically generates content that includes not only the visual explanation generated in the handover video, but also voice guidance based on emotion data.

[0845] "Application example 2 when combining with an emotional engine"

[0846] (Claim 1)

[0847] An information processing device that records work information and user emotional information,

[0848] A computer device that aggregates and preprocesses work information and emotional information transmitted from the information processing device,

[0849] The computer device includes means for creating summaries of work information and emotional information using a generative model, and for automatically generating handover videos according to the emotional state,

[0850] A means for distributing the handover video to the information processing device of the new person in charge,

[0851] A system that includes this.

[0852] (Claim 2)

[0853] The system according to claim 1, wherein the collection of work information and emotional information is performed in the background by a program installed on the information processing device.

[0854] (Claim 3)

[0855] The system according to claim 1, wherein the computer device automatically generates content including audio commentary in addition to the visual explanation generated in the handover video. [Explanation of Symbols]

[0856] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A terminal for recording work information, A server that aggregates and preprocesses work information transmitted from the terminal, The server provides a means for creating a summary of work information using a generative model and automatically generating a handover video, A means of distributing the handover video to the new person in charge's terminal, A system that includes this.

2. The system according to claim 1, wherein the collection of the aforementioned work information is performed in the background by an application installed on the terminal.

3. The system according to claim 1, wherein the server automatically generates content including audio narration in addition to the visual explanation generated in the handover video.

Images