system

Abstract

We provide the system. [Solution] A recording device for recording video of work processes, A receiving processing means that receives video data recorded by the recording means and performs preprocessing, An analysis means for analyzing pre-processed data by the aforementioned receiving processing means and learning patterns of business processes, A proposal generation means that identifies inefficient processes detected by the analysis means and generates proposals for efficiency improvements, A presentation means that provides the user with the analysis results and proposals generated by the proposal generation means, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In business processes, there are inefficiencies and wastes, but it is difficult to accurately grasp them and carry out unified improvement activities. Also, a significant reduction in the man-hours of business is required, but there is a lack of specific approaches. For this reason, a mechanism that can provide specific improvement proposals for improving business efficiency in real time is required.

Means for Solving the Problems

[0005] This invention relates to a system that learns patterns in business processes by recording video of business operations and receiving and processing the video data. The system includes recording means, receiving and processing means, analysis means, suggestion generation means, and presentation means. In particular, the analysis means can integrate and analyze video, audio, and text data using multimodal generation technology. This enables the generation of suggestions for improving efficiency for identified inefficient processes and provides real-time feedback and improvement suggestions.

[0006] "Recording means" refers to a device or function for recording video footage of work activities.

[0007] "Receiving processing means" refers to a device or function for receiving video data recorded by a recording means and performing preprocessing on it.

[0008] "Analysis means" refers to a device or function for analyzing pre-processed data and learning patterns in business processes.

[0009] "Proposal generation means" refers to a device or function for identifying inefficient processes detected by analysis means and generating proposals for improving efficiency.

[0010] "Presentation means" refers to a device or function for providing users with the analysis results and proposals generated by the proposal generation means.

[0011] "Multimodal generation technology" refers to a technology that combines and analyzes multiple data formats (video, audio, text, etc.).

[0012] "Real-time feedback" is a method of providing analyzed information to users in real time. [Brief explanation of the drawing]

[0013] [Figure 1]This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention relates to an embodiment of a system that automatically identifies and proposes improvements necessary to enhance operational efficiency. This system primarily aims to record and analyze video footage of work processes and provide concrete suggestions for automation and efficiency improvements. The program processing and specific examples of this system are described in detail below.

[0035] Explanation of program processing

[0036] The terminal is installed at the work site and records the actual work situation via camera and microphone. The recorded video and audio data is transmitted to a server via the internet.

[0037] The server first performs data preprocessing in order to analyze the received video and audio data. Preprocessing includes noise reduction and standardization of the data format.

[0038] The server uses analysis tools to analyze data using multimodal generation technology. In this process, motion recognition from video data and speech analysis from audio data are performed in an integrated manner, and patterns of business processes are learned.

[0039] Based on learned patterns, the server identifies inefficient processes and parts that can be automated, and uses a suggestion generation mechanism to create improvement suggestions for increased efficiency.

[0040] The server compiles the generated suggestions into a report and also creates a video replay. This makes it possible to present concrete improvement models.

[0041] Users can view these proposals and video replays on their devices. Based on the reported information, users can implement improvement activities in their actual work.

[0042] Specific example

[0043] Example 1: Improvement of the manufacturing line

[0044] The terminal takes pictures of the factory's production line and sends the data to the server.

[0045] The server identifies unnecessary actions in the product assembly process and suggests that manufacturing time can be reduced by 20% by omitting or automating these actions.

[0046] After reviewing the proposal, the user will implement the new work procedures to improve productivity.

[0047] Example 2: Improving the efficiency of meetings in the office

[0048] The device records and videos office meetings and sends the data to the server.

[0049] The server identifies redundant discussions in the meeting and proposes specific approaches to effectively manage the agenda.

[0050] The user will accept the suggestion and implement more efficient meeting procedures from the next meeting onwards.

[0051] As these examples demonstrate, by utilizing the system of the present invention, it is possible to promote operational efficiency and achieve significant cost reductions and productivity improvements.

[0052] The following describes the processing flow.

[0053] Step 1:

[0054] The terminal uses cameras and microphones installed at the work site to record video and audio of work activities in real time. This data is stored in a buffer at regular intervals, and when the stored data reaches a specified threshold, or when it is transmitted to the server via the network in real time, it is sent to the server.

[0055] Step 2:

[0056] The server performs initial processing on the received video and audio data. This initial processing includes data compression, trimming of unnecessary parts, and noise reduction. As a result of this processing, data suitable for analysis is generated.

[0057] Step 3:

[0058] The server analyzes the data after initial processing using analytical tools. The analysis utilizes multimodal generation AI to perform behavioral recognition from video data and extract discussion content and emotions from audio data. The analysis identifies patterns in business processes, providing information that can be used to determine efficiency and inefficiencies.

[0059] Step 4:

[0060] Based on identified business process patterns, the server generates improvement suggestions for efficiency improvements using a suggestion generation mechanism. These suggestions include time reductions, procedure restructuring, and equipment implementation. The suggestions are then listed as specific actions.

[0061] Step 5:

[0062] The server automatically generates a report of the suggestions that have been submitted. The report details the identified areas for improvement and suggestions, and may include data that can be played back as a visual video replay.

[0063] Step 6:

[0064] The server sends the completed report and video replay to the terminal via the network, displaying the results to the user. The user can review this information and gain insights to implement the identified improvements.

[0065] (Example 1)

[0066] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0067] To improve operational efficiency, it is necessary to accurately understand the work situation on-site, identify inefficient work processes, and improve them. However, in many current systems, the processes from data collection to analysis and improvement proposals are independent and not automated as a continuous flow, which slows down overall operational efficiency. Solving this problem is the challenge.

[0068] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0069] In this invention, the server includes a device for recording the status of work, a receiving means for receiving data recorded by the device and pre-processing the data, and a means for analyzing the pre-processed data by the receiving means and learning patterns of the work flow. This enables real-time data analysis and immediate generation of improvement suggestions at the work site.

[0070] A "device" is physical hardware installed at the work site to record the status of the work.

[0071] "Receiving means" refers to a procedure or function for receiving data recorded by a device and for performing preprocessing on that data.

[0072] "Means" refer to the methods, processes, or mechanisms used to achieve a specific objective.

[0073] "Analysis methods" refer to processes and techniques for learning and identifying workflows and work patterns by analyzing pre-processed data in detail.

[0074] "Means of delivery" refers to a system for presenting and providing the generated analysis results and suggestions to users.

[0075] This invention relates to a system designed to improve business efficiency. Specifically, it records and analyzes the workflow at the work site through a series of processes involving a server, terminal, and user, and generates suggestions for improvement.

[0076] First, the terminal is a device installed at the work site that uses a camera and microphone to record the work situation. This device has high-resolution shooting and audio recording capabilities, allowing for the collection of detailed data. This data is transmitted to a server using a secure communication protocol.

[0077] The server performs preprocessing on the received video and audio data. This preprocessing uses techniques such as FFT (Fast Fourier Transform) and H.264 encoding to remove noise and standardize data formats. Subsequently, a multimodal generative AI model is used to recognize human actions from the video and analyze spoken content from the audio. This allows the system to automatically analyze the workflow and learn patterns in business processes.

[0078] Based on the analysis results, the server identifies inefficient procedures and generates specific suggestions for improvement. The generative AI model used here automates data analysis and suggestion generation, and has the capability to create proposal documents and video replays. These suggestions include specific improvement actions directed at the user, promoting increased efficiency in operations.

[0079] Users can review these proposals and video replays via their devices and implement actual business improvements based on the presented content.

[0080] As a concrete example, consider improving a manufacturing line. A terminal photographs the factory's manufacturing line, and a server uses that data to identify inefficiencies in the product assembly process and generates suggestions to reduce man-hours by automating the process. This entire process is input into the AI ​​model using prompt messages like the following:

[0081] "Analyze the video data from the manufacturing line and create a proposal to increase efficiency by 20%."

[0082] This system enables efficient business operations by allowing for a detailed understanding of workflows and the immediate generation of improvement suggestions.

[0083] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0084] Step 1:

[0085] The terminal is installed at the work site and uses a camera and microphone to record the work situation. Specifically, a high-resolution camera captures the worker's movements as video, and a high-sensitivity microphone records audio of the work environment. This allows for the acquisition of detailed and accurate work data as input. This data is generated as time-series video and audio files.

[0086] Step 2:

[0087] The terminal transmits recorded video and audio data to the server via the internet. This transmission uses a secure and high-speed communication protocol to prevent data delays and loss. The output at this step is a data stream for the server to receive.

[0088] Step 3:

[0089] The server preprocesses the received data. Specifically, it uses FFT (Fast Fourier Transform) to remove noise from audio data and encodes video data in H.264 format. This results in a clean dataset that is easy to analyze. The preprocessed data is then used as input for subsequent analysis.

[0090] Step 4:

[0091] The server performs analysis using a multimodal generation AI model with pre-processed data. It uses open-pause technology to recognize motion from video data and speech recognition technology to convert spoken content into text from audio data. This allows the system to learn patterns in business processes and generate complex data analysis results as output.

[0092] Step 5:

[0093] Based on the analysis results obtained, the server identifies inefficient areas and uses a generative AI model to formulate improvement proposals. During this process, it compares and analyzes data from each process to specifically determine where and how improvements should be made. A proposal document and video replay are generated, which are output information provided to the user.

[0094] Step 6:

[0095] The user reviews the provided proposals and video replays via their terminal. They then examine the outputted improvement suggestions in detail and evaluate how they can be implemented in their actual work. The output of this step is the user's plan for specific improvement activities.

[0096] This completes a series of steps in which the entire system analyzes and optimizes business processes in detail.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] Efficient business operations are a critical challenge faced by many companies. In particular, traditional methods have made it difficult to identify areas for efficiency improvements in manufacturing and robot operation. Manual improvement suggestions are time-consuming and difficult to implement in real time. Therefore, there is a growing need for systems that can immediately propose and implement improvements to streamline production lines and business processes.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes: video recording means for recording video of work operations; data processing means for receiving and pre-processing video information recorded by the video recording means; pattern analysis means for analyzing the pre-processed information by the data processing means and learning patterns of work processes; proposal generation means for identifying inefficient processes detected by the pattern analysis means and generating suggestions for efficiency improvements; information presentation means for providing the user with the analysis results and suggestions generated by the proposal generation means; and real-time feedback generation means for generating real-time feedback to the robot. This makes it possible to immediately propose efficiency improvements in business and manufacturing processes and to achieve real-time improvements.

[0102] "Video recording means" refers to a device or system that records the status of work in real time using a camera or similar device.

[0103] "Data processing means" refers to a device or program that receives recorded video information and has the function of performing preprocessing such as noise reduction and format standardization.

[0104] A "pattern analysis system" is a system that analyzes patterns in business processes based on pre-processed information and extracts data for efficiency improvements.

[0105] A "proposal generation means" is a device or program that automatically detects inefficient business processes based on pattern analysis and generates proposals for improving efficiency.

[0106] "Information presentation means" refers to a device or interface for visually providing the user with the analysis results and improvement suggestions generated by the proposal generation means.

[0107] A "real-time feedback generation means" is a device or system that has the function of providing users with immediate feedback for process improvement and enabling immediate improvement.

[0108] The system for implementing this invention is configured to combine advanced data processing and analysis technologies in order to improve operational efficiency.

[0109] The server collects real-time video footage of the work environment from cameras mounted on factory robots. This video recording system consists of high-resolution cameras, enabling precise data acquisition. There is also a data processing system that receives the video information, performing noise reduction and standardizing the data format. Libraries such as OpenCV and TENSORFLOW® are primarily used for this purpose.

[0110] The pattern analysis method learns patterns in business processes based on pre-processed information. This utilizes multimodal generation technology, including Google® Cloud AI. This analysis identifies inefficient business processes, and the data is used to gather information for suggesting improvements.

[0111] The proposal generation mechanism generates specific proposals for automation and efficiency improvements based on information obtained from pattern analysis. This allows for the identification of specific methods for improving inefficient parts and enables the generation of improvement measures in real time.

[0112] The information presentation system plays the role of presenting the generated analysis results and suggestions to the user. This provides operators and users with visual improvement suggestions in real time via a head-mounted display. Therefore, users can immediately review the presented suggestions and quickly implement improvements to their work.

[0113] Furthermore, the real-time feedback generation system provides immediate feedback for improvements during operations, supporting immediate process adjustments. This enables continuous and dynamic optimization of the factory line.

[0114] A concrete example is when AI detects wasted time during assembly line work and suggests, "By shortening the movement of your right hand by 3 seconds, the next process will proceed more smoothly." This suggestion allows the operator to immediately correct their movements, improving overall production efficiency.

[0115] Examples of prompts for the generating AI model include "optimize assembly actions," "detect and suggest actions that can be improved," and "present streamlined assembly procedures." These prompts are used to quickly generate specific and feasible improvement suggestions.

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The server uses cameras installed on factory robots to acquire real-time video footage of the work processes. The input is raw data from the cameras, and the output is a video stream. This video is recorded in high resolution frame by frame to accurately capture changes in the work environment.

[0119] Step 2:

[0120] The server performs preprocessing on the acquired video stream to remove noise and standardize the format. The input is the video stream generated in step 1, and the output is the preprocessed, clear video data. Specifically, filtering is performed using the OpenCV library and data standardization is performed using TensorFlow.

[0121] Step 3:

[0122] The server analyzes patterns in business processes using pre-processed video data. The input is pre-processed video data, and the output is analytical information to identify inefficiencies. Google Cloud AI is used to identify actions in the video and perform pattern learning by comparing them with known processes.

[0123] Step 4:

[0124] The server generates automation targets and efficiency improvement proposals based on the analysis information. The input is the analysis information from step 3, and the output is an efficiency improvement proposal document. A generative AI model is used to generate the proposals, producing specific improvement measures in written form.

[0125] Step 5:

[0126] The server outputs the generated efficiency suggestions to the user via a head-mounted display. The input is the efficiency suggestion document, and the output is a display of the suggestions as visual feedback to the user. The user can then immediately adjust their work based on these suggestions.

[0127] Step 6:

[0128] The server generates real-time feedback during operations to support process improvement. Inputs are the current operational status and suggestions, while outputs are refined operational processes. This process includes guidance to make it easier for users to perform new actions.

[0129] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0130] This invention relates to a system that, in addition to streamlining business processes, provides more accurate improvement suggestions by recognizing user emotions. This system consists of recording means, receiving processing means, analysis means, suggestion generation means, presentation means, and an emotion engine.

[0131] Explanation of program processing

[0132] The terminal records video and audio from the work site in real time. The recorded data is transmitted to a server via the network. At the same time, the user's facial expressions and voice tone are also recorded and used for emotion analysis.

[0133] The server preprocesses the received data and analyzes the video and audio data. This includes analyzing the operation of business processes, as well as recognizing the user's emotional state using an emotion engine. The emotion engine extracts emotional data using facial expression analysis and voice analysis.

[0134] The server combines sentiment data with business process analysis results to identify areas that need improvement. The suggestion generation system identifies inefficient processes in business operations and creates acceptable improvement suggestions by taking user sentiment into consideration in the solutions.

[0135] The server automatically generates a detailed report outlining the analysis results and recommendations. A video replay is also created, including personalized feedback and suggestions based on emotions.

[0136] Users can view reports and video replays provided by the server through their terminals. This allows for improvement activities that not only streamline operations but also take into account the emotional state of employees.

[0137] Specific example

[0138] Example 1: Improving Call Center Operations

[0139] The terminal records the operator's conversation and facial expressions, and sends the data to the server.

[0140] The server analyzes emotional states such as stress levels during calls and suggests script revisions or training opportunities.

[0141] After receiving the suggestions, the user will implement improvements to operational methods and manage operator stress.

[0142] Example 2: Understanding and improving the flow of meetings

[0143] The terminal records the facial expressions and statements of meeting participants and sends that data to the server.

[0144] The server emotionally analyzes changes in participants' interest and the intensity of discussion during the meeting, and makes suggestions for refining the agenda and improving the meeting's progress.

[0145] Based on the feedback, users will implement better discussion strategies in future meetings.

[0146] By combining this with an emotional engine, it becomes possible to improve operational efficiency while simultaneously providing suggestions that take employees' emotions into consideration.

[0147] The following describes the processing flow.

[0148] Step 1:

[0149] The device begins recording video and audio of work activities using cameras and microphones installed at the work site. This includes employee facial expressions and tone of voice, providing data to recognize the user's emotions. Once a certain amount of data has been collected, it is transmitted to a server via the network.

[0150] Step 2:

[0151] The server performs initial processing on the received video and audio data. This initial processing includes trimming the video, denoising the audio, and standardizing the data format. This converts the data into a format suitable for analysis.

[0152] Step 3:

[0153] The server uses an emotion engine to recognize the user's emotional state from the processed data. Specifically, a facial recognition algorithm analyzes the video data, and an audio analysis tool extracts emotions from the tone and pitch of the voice.

[0154] Step 4:

[0155] The server learns patterns in business processes using analytical tools. This involves analyzing identified business actions and process flows to detect inefficiencies and areas that need improvement.

[0156] Step 5:

[0157] The server generates improvement suggestions by combining emotional states and business process analysis results. The suggestion generation mechanism formulates specific action plans for efficiency improvements and customizes the suggestions to be user-friendly, taking emotional data into consideration.

[0158] Step 6:

[0159] The server automatically generates a report of the improvement suggestions that have been created. Furthermore, a video replay visually illustrating the identified areas for improvement is also created, and this information is provided in a format that is easy for users to understand.

[0160] Step 7:

[0161] Users review reports and video replays sent from the server via their terminals. Based on the information provided, users plan improvement activities in their actual work and optimize operations in a way that is sensitive to emotions.

[0162] (Example 2)

[0163] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0164] Traditional business process improvement systems focused on streamlining operations but failed to consider employees' emotional states when making improvement suggestions. As a result, suggestions were sometimes ineffective for employees, and true business improvement was not achieved. Furthermore, integrated analysis of multimodal data and real-time feedback were not adequately implemented, resulting in a lack of constructive feedback based on user emotions.

[0165] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0166] In this invention, the server includes a device for collecting video footage of work processes, a processing device for receiving the video and audio data collected by the device and organizing the information, and an analysis device for analyzing the data organized by the processing device and identifying the workflow. This makes it possible to provide improvement suggestions that take into account the user's emotions by utilizing emotion recognition technology and multimodal data analysis technology.

[0167] "Work process video" refers to data that visually records the work procedures and actions performed during the execution of work.

[0168] The term "device" refers to equipment or systems used to achieve a specific purpose.

[0169] "Video and audio data" refers to visual and auditory information recorded and stored in digital format.

[0170] A "processing device" is a system of hardware or software used to receive, organize, and analyze data.

[0171] "Organizing information" means processing, filtering, and classifying acquired data according to its purpose.

[0172] An "analysis device" is a device or program used to verify processed data and identify specific patterns or trends.

[0173] "Identifying the workflow" means compiling and evaluating the progress and procedures of a business process.

[0174] An "emotion recognition device" is a device or system that incorporates technology to identify emotions from a user's facial expressions and voice.

[0175] A "proposal generation device" is a system that has the function of generating problem-solving methods and improvement proposals based on analyzed data.

[0176] An "output device" is a device or program that provides information or suggestions to a user visually or audibly.

[0177] This system aims to improve operational efficiency and is specifically designed to suggest improvement measures that take into account the emotional state of employees. Specifically, it is structured around three roles: terminal, server, and user.

[0178] The terminals are deployed at work sites and function as devices for acquiring video and audio of work activities. This utilizes high-resolution cameras and high-sensitivity microphones. The terminals record this data in real time and transmit it to a server via the network. This data also includes emotional elements such as the user's facial expressions and tone of voice, which are used for emotion recognition.

[0179] The server receives video and audio data from the terminal and performs preprocessing such as noise filtering and data formatting. Next, using the processed data, the analysis device analyzes the business process in detail and uses an emotion recognition device to determine the user's emotional state. Here, a generative AI model is used to identify inefficiencies in the business by comprehensively analyzing multiple data sources and to create improvement suggestions that take emotions into consideration.

[0180] Users receive and review reports and video replays provided by the server via their terminals. Based on this information, they can implement emotion-driven improvement measures and achieve continuous business improvement.

[0181] As a concrete example, in a call center, a terminal records the operator's conversation and facial expressions, while a server analyzes their stress level. Then, using a generative AI model, it presents improvement measures using a prompt message such as, "How should we analyze the operator's stress level and provide specific improvement suggestions?" In this way, the system enables the user to receive optimal improvement suggestions.

[0182] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0183] Step 1:

[0184] The terminal uses cameras and microphones installed at the work site to record video and audio of work activities in real time. Input consists of ambient sounds and video from the work site, which are processed as digital data. Output is the captured video and audio data. This data includes the user's facial expressions and voice tone, and is used for emotion analysis.

[0185] Step 2:

[0186] The terminal transmits recorded digital data to the server via the network. The input consists of video and audio digital data recorded by the terminal. The data is encrypted and transmitted securely. The output is the data stream delivered to the server.

[0187] Step 3:

[0188] The server acquires video and audio data received from the terminal. The input is a data stream transmitted over the network, which the server receives. Initial processing includes noise filtering and time synchronization to organize the data. The output is the pre-processed audio and video data.

[0189] Step 4:

[0190] The server analyzes pre-processed data and evaluates the workflow. The input is pre-processed video and audio data. The server identifies the business process and uses data analysis algorithms to pinpoint inefficient steps. The output is the analysis results highlighting the inefficient parts.

[0191] Step 5:

[0192] The server uses an emotion recognition device to analyze the user's emotional state. The input is pre-processed video and audio data, and the emotion engine extracts emotional data by performing facial expression and voice analysis. The output is analyzed data that includes the user's emotional state.

[0193] Step 6:

[0194] The server utilizes a generative AI model to integrate sentiment data and business process analysis results to generate improvement suggestions for increased efficiency. The input consists of sentiment data and analysis results. The generative AI model uses prompts to create specific improvement suggestions. The output is a detailed report containing the improvement suggestions.

[0195] Step 7:

[0196] Users receive and review reports and video replays provided by the server via their terminals. The input consists of reports and video replays from the server. Based on this, users implement business improvement measures. The output is sustainable process improvement that prioritizes operational efficiency and employee satisfaction.

[0197] (Application Example 2)

[0198] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0199] Traditional business efficiency systems focus on identifying inefficiencies and suggesting improvements, but they often fail to consider user emotions, potentially leading to reduced acceptance of suggestions. Furthermore, they haven't been able to achieve efficient task management that considers user emotions in home and office environments. Therefore, there's a need for a system that analyzes user emotions and individually optimizes suggestions.

[0200] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0201] In this invention, the server includes an acquisition means for recording work video, a reception control means for receiving and pre-processing the recorded data, an analysis control means for analyzing the pre-processed information and learning patterns of business processes, an emotion analysis means for analyzing the emotional state of the user, and an improvement means for optimizing the proposed content based on the emotion analysis results. This makes it possible to propose highly acceptable improvements that take into account the user's emotions in order to improve business efficiency.

[0202] "Means of acquisition" refers to devices or functions for recording video data of work or actions.

[0203] "Receiving control means" refers to a device or function for receiving recorded data and performing necessary preprocessing.

[0204] "Analysis and control means" refers to devices or functions that analyze pre-processed information and learn patterns of business processes and behaviors.

[0205] A "proposal generation means" refers to a device or function that identifies inefficient actions based on analysis results and generates improvement proposals.

[0206] "Presentation control means" refers to devices or functions for providing generated suggestions to users.

[0207] "Emotional analysis tools" refer to devices or functions used to analyze and understand a user's emotional state.

[0208] "Improvement measures" refer to devices or functions that individually optimize the content of suggestions based on the results of sentiment analysis.

[0209] To implement this invention, the system is composed of a combination of various hardware and software. The server, terminals, and users work together to perform data processing for efficient business improvement.

[0210] The server acquires video and audio data in real time from terminals equipped with cameras and audio sensors for recording work and daily activities. The data obtained by this acquisition means is transmitted to the server by a receiving control means and preprocessed. This process involves removing unwanted noise and converting the data format.

[0211] The server uses analysis control mechanisms to learn business processes and behavioral patterns in order to analyze pre-processed information. Similarly, emotion analysis mechanisms are used to detect emotional states from the user's facial expressions and tone of voice, and to store this data. For analysis, emotion analysis APIs such as Amazon Rekognition and Google Cloud Vision are utilized.

[0212] The data obtained through analysis is transformed into personalized improvement suggestions for each user by a suggestion generation system. For example, when a household robot determines the need for cleaning, it refers to emotion analysis data and suggests the optimal action at a time when the user is relaxed.

[0213] The terminal uses presentation control means to present generated suggestions and feedback to the user in real time. This feedback is optimized based on the user's emotional state, allowing the user to take appropriate action.

[0214] Specific examples include analyzing a user's emotional state while they are watching television to reduce stress and then suggesting the most appropriate household chores. An example of a prompt for the generative AI model would be, "Analyze the user's current facial expression and tone of voice to generate appropriate household chore suggestions."

[0215] This system will enable highly acceptable suggestions that take into account the user's emotional state in both work and daily life, leading to improved work efficiency and a more comfortable experience.

[0216] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0217] Step 1:

[0218] The device uses a camera and audio sensor to capture work and daily activities, acquiring video and audio data in real time. This data serves as input, which is then prepared for conversion into an appropriate format after removing noise and unwanted information. The output is appropriate video and audio data for pre-processing.

[0219] Step 2:

[0220] The terminal transmits the acquired data to the server. The server receives this data using a receiving control means and performs preprocessing such as noise reduction and format conversion. The input is the transmitted video and audio data, and the output is data in an analyzable format.

[0221] Step 3:

[0222] The server analyzes pre-processed data using analysis and control means to learn business processes and behavioral patterns. The input for this step is pre-processed data, and the output is learned pattern information. Specifically, data analysis is performed using machine learning algorithms.

[0223] Step 4:

[0224] The server performs data analysis to determine the user's emotional state using emotion analysis tools. Pre-processed data is treated as input, and data indicating the user's emotional state is output. Specifically, it utilizes facial expression analysis and voice analysis technologies to extract emotional data.

[0225] Step 5:

[0226] The server generates user-specific improvement suggestions based on analysis results and emotional state data using a suggestion generation mechanism. The input is the analysis results from steps 3 and 4, and the output is user-optimized improvement suggestions. The suggestions are generated using an algorithm that takes emotional state into consideration.

[0227] Step 6:

[0228] The server sends suggestions generated using the suggestion control means to the terminal and notifies the user. The input is the suggestions from the suggestion generation means, and the output is the content presented to the user terminal. The user can receive real-time feedback through the terminal and use it to improve their life and work.

[0229] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0230] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0231] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0232] [Second Embodiment]

[0233] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0234] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0235] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0236] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0237] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0238] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0239] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0240] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0241] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0242] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0243] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0244] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0245] This invention relates to an embodiment of a system that automatically identifies and proposes improvements necessary to enhance operational efficiency. This system primarily aims to record and analyze video footage of work processes and provide concrete suggestions for automation and efficiency improvements. The program processing and specific examples of this system are described in detail below.

[0246] Explanation of program processing

[0247] The terminal is installed at the work site and records the actual work situation via camera and microphone. The recorded video and audio data is transmitted to a server via the internet.

[0248] The server first performs data preprocessing in order to analyze the received video and audio data. Preprocessing includes noise reduction and standardization of the data format.

[0249] The server uses analysis tools to analyze data using multimodal generation technology. In this process, motion recognition from video data and speech analysis from audio data are performed in an integrated manner, and patterns of business processes are learned.

[0250] Based on learned patterns, the server identifies inefficient processes and parts that can be automated, and uses a suggestion generation mechanism to create improvement suggestions for increased efficiency.

[0251] The server compiles the generated suggestions into a report and also creates a video replay. This makes it possible to present concrete improvement models.

[0252] Users can view these proposals and video replays on their devices. Based on the reported information, users can implement improvement activities in their actual work.

[0253] Specific example

[0254] Example 1: Improvement of the manufacturing line

[0255] The terminal takes pictures of the factory's production line and sends the data to the server.

[0256] The server identifies unnecessary actions in the product assembly process and suggests that manufacturing time can be reduced by 20% by omitting or automating these actions.

[0257] After reviewing the proposal, the user will implement the new work procedures to improve productivity.

[0258] Example 2: Improving the efficiency of meetings in the office

[0259] The device records and videos office meetings and sends the data to the server.

[0260] The server identifies redundant discussions in the meeting and proposes specific approaches to effectively manage the agenda.

[0261] The user will accept the suggestion and implement more efficient meeting procedures from the next meeting onwards.

[0262] As these examples demonstrate, by utilizing the system of the present invention, it is possible to promote operational efficiency and achieve significant cost reductions and productivity improvements.

[0263] The following describes the processing flow.

[0264] Step 1:

[0265] The terminal uses cameras and microphones installed at the work site to record video and audio of work activities in real time. This data is stored in a buffer at regular intervals, and when the stored data reaches a specified threshold, or when it is transmitted to the server via the network in real time, it is sent to the server.

[0266] Step 2:

[0267] The server performs initial processing on the received video and audio data. This initial processing includes data compression, trimming of unnecessary parts, and noise reduction. As a result of this processing, data suitable for analysis is generated.

[0268] Step 3:

[0269] The server analyzes the data after initial processing using analytical tools. The analysis utilizes multimodal generation AI to perform behavioral recognition from video data and extract discussion content and emotions from audio data. The analysis identifies patterns in business processes, providing information that can be used to determine efficiency and inefficiencies.

[0270] Step 4:

[0271] Based on identified business process patterns, the server generates improvement suggestions for efficiency improvements using a suggestion generation mechanism. These suggestions include time reductions, procedure restructuring, and equipment implementation. The suggestions are then listed as specific actions.

[0272] Step 5:

[0273] The server automatically generates a report of the suggestions that have been submitted. The report details the identified areas for improvement and suggestions, and may include data that can be played back as a visual video replay.

[0274] Step 6:

[0275] The server sends the completed report and video replay to the terminal via the network, displaying the results to the user. The user can review this information and gain insights to implement the identified improvements.

[0276] (Example 1)

[0277] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0278] To improve operational efficiency, it is necessary to accurately understand the work situation on-site, identify inefficient work processes, and improve them. However, in many current systems, the processes from data collection to analysis and improvement proposals are independent and not automated as a continuous flow, which slows down overall operational efficiency. Solving this problem is the challenge.

[0279] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0280] In this invention, the server includes a device for recording the working status, a receiving means for receiving the data recorded by the device and performing preprocessing on the data, and a means for analyzing the preprocessed data by the receiving means and learning the pattern of the work flow. Thereby, real-time data analysis at the work site and generation of immediate improvement proposals become possible.

[0281] The "device" is physical hardware installed at the work site for recording the working status.

[0282] The "receiving means" is a procedure or function for receiving the data recorded by the device and further performing preprocessing on the data.

[0283] The "means" is a method, process, or mechanism used to achieve a specific purpose.

[0284] The "analysis means" is a process or technology for analyzing the preprocessed data in detail and learning / specifying the work flow and work pattern.

[0285] The "providing means" is a mechanism for presenting and providing the generated analysis results and proposals to the user.

[0286] This invention relates to a system designed to realize business efficiency. Specifically, through a series of processes involving the server, terminal, and user, it records and analyzes the work flow at the work site and generates proposals for improvement.

[0287] First, the terminal is a device installed at the work site, and it records the working status using a camera and a microphone. This device has a high-resolution shooting function and a voice recording function, and can collect detailed data. These data are transmitted to the server using a secure communication protocol.

[0288] The server performs preprocessing on the received video and audio data. This preprocessing uses techniques such as FFT (Fast Fourier Transform) and H.264 encoding to remove noise and standardize data formats. Subsequently, a multimodal generative AI model is used to recognize human actions from the video and analyze spoken content from the audio. This allows the system to automatically analyze the workflow and learn patterns in business processes.

[0289] Based on the analysis results, the server identifies inefficient procedures and generates specific suggestions for improvement. The generative AI model used here automates data analysis and suggestion generation, and has the capability to create proposal documents and video replays. These suggestions include specific improvement actions directed at the user, promoting increased efficiency in operations.

[0290] Users can review these proposals and video replays via their devices and implement actual business improvements based on the presented content.

[0291] As a concrete example, consider improving a manufacturing line. A terminal photographs the factory's manufacturing line, and a server uses that data to identify inefficiencies in the product assembly process and generates suggestions to reduce man-hours by automating the process. This entire process is input into the AI ​​model using prompt messages like the following:

[0292] "Analyze the video data from the manufacturing line and create a proposal to increase efficiency by 20%."

[0293] This system enables efficient business operations by allowing for a detailed understanding of workflows and the immediate generation of improvement suggestions.

[0294] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0295] Step 1:

[0296] The terminal is installed at the work site and uses a camera and microphone to record the work situation. Specifically, a high-resolution camera captures the worker's movements as video, and a high-sensitivity microphone records audio of the work environment. This allows for the acquisition of detailed and accurate work data as input. This data is generated as time-series video and audio files.

[0297] Step 2:

[0298] The terminal transmits recorded video and audio data to the server via the internet. This transmission uses a secure and high-speed communication protocol to prevent data delays and loss. The output at this step is a data stream for the server to receive.

[0299] Step 3:

[0300] The server preprocesses the received data. Specifically, it uses FFT (Fast Fourier Transform) to remove noise from audio data and encodes video data in H.264 format. This results in a clean dataset that is easy to analyze. The preprocessed data is then used as input for subsequent analysis.

[0301] Step 4:

[0302] The server performs analysis using a multimodal generation AI model with pre-processed data. It uses open-pause technology to recognize motion from video data and speech recognition technology to convert spoken content into text from audio data. This allows the system to learn patterns in business processes and generate complex data analysis results as output.

[0303] Step 5:

[0304] Based on the obtained analysis results, the server identifies the inefficient parts and formulates improvement proposals using the generative AI model. At this time, the data of each process is compared and analyzed to specifically determine where and how to improve. A proposal document and a video replay are generated, which are the output information provided to the user.

[0305] Step 6:

[0306] The user checks the provided proposal document and video replay through the terminal. While examining the output improvement proposals in detail, the user evaluates how they can be implemented in actual business operations. The output of this step is the plan for the user's specific improvement activities.

[0307] This completes a series of processes in which the entire system analyzes the business process in detail and improves its efficiency.

[0308] (Application Example 1)

[0309] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0310] Efficient business execution is an important issue faced by many enterprises. Especially in improving productivity at the manufacturing site and operating robots, it has been difficult to identify room for improvement with conventional methods. There are problems such as manual improvement proposals taking time and being difficult to adapt in real time. Therefore, there is an increasing need for a system that can immediately propose improvements to the production line and business processes and realize the improvements.

[0311] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0312] In this invention, the server includes: video recording means for recording video of work operations; data processing means for receiving and pre-processing video information recorded by the video recording means; pattern analysis means for analyzing the pre-processed information by the data processing means and learning patterns of work processes; proposal generation means for identifying inefficient processes detected by the pattern analysis means and generating suggestions for efficiency improvements; information presentation means for providing the user with the analysis results and suggestions generated by the proposal generation means; and real-time feedback generation means for generating real-time feedback to the robot. This makes it possible to immediately propose efficiency improvements in business and manufacturing processes and to achieve real-time improvements.

[0313] "Video recording means" refers to a device or system that records the status of work in real time using a camera or similar device.

[0314] "Data processing means" refers to a device or program that receives recorded video information and has the function of performing preprocessing such as noise reduction and format standardization.

[0315] A "pattern analysis system" is a system that analyzes patterns in business processes based on pre-processed information and extracts data for efficiency improvements.

[0316] A "proposal generation means" is a device or program that automatically detects inefficient business processes based on pattern analysis and generates proposals for improving efficiency.

[0317] "Information presentation means" refers to a device or interface for visually providing the user with the analysis results and improvement suggestions generated by the proposal generation means.

[0318] A "real-time feedback generation means" is a device or system that has the function of providing users with immediate feedback for process improvement and enabling immediate improvement.

[0319] The system for implementing this invention is configured to combine advanced data processing and analysis technologies in order to improve operational efficiency.

[0320] The server collects real-time video footage of the work environment from cameras mounted on factory robots. This video recording system utilizes high-resolution cameras, enabling precise data acquisition. Additionally, there is a data processing system that receives the video information, performing noise reduction and standardizing the data format. Libraries such as OpenCV and TensorFlow are primarily used for this purpose.

[0321] The pattern analysis method learns patterns in business processes based on pre-processed information. This utilizes multimodal generation technology, such as Google Cloud AI. This analysis identifies inefficient business processes, and the data is used to gather information for suggesting improvements.

[0322] The proposal generation mechanism generates specific proposals for automation and efficiency improvements based on information obtained from pattern analysis. This allows for the identification of specific methods for improving inefficient parts and enables the generation of improvement measures in real time.

[0323] The information presentation system plays the role of presenting the generated analysis results and suggestions to the user. This provides operators and users with visual improvement suggestions in real time via a head-mounted display. Therefore, users can immediately review the presented suggestions and quickly implement improvements to their work.

[0324] Furthermore, the real-time feedback generation system provides immediate feedback for improvements during operations, supporting immediate process adjustments. This enables continuous and dynamic optimization of the factory line.

[0325] A concrete example is when AI detects wasted time during assembly line work and suggests, "By shortening the movement of your right hand by 3 seconds, the next process will proceed more smoothly." This suggestion allows the operator to immediately correct their movements, improving overall production efficiency.

[0326] Examples of prompts for the generating AI model include "optimize assembly actions," "detect and suggest actions that can be improved," and "present streamlined assembly procedures." These prompts are used to quickly generate specific and feasible improvement suggestions.

[0327] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0328] Step 1:

[0329] The server uses cameras installed on factory robots to acquire real-time video footage of the work processes. The input is raw data from the cameras, and the output is a video stream. This video is recorded in high resolution frame by frame to accurately capture changes in the work environment.

[0330] Step 2:

[0331] The server performs preprocessing on the acquired video stream to remove noise and standardize the format. The input is the video stream generated in step 1, and the output is the preprocessed, clear video data. Specifically, filtering is performed using the OpenCV library and data standardization is performed using TensorFlow.

[0332] Step 3:

[0333] The server analyzes patterns in business processes using pre-processed video data. The input is pre-processed video data, and the output is analytical information to identify inefficiencies. Google Cloud AI is used to identify actions in the video and perform pattern learning by comparing them with known processes.

[0334] Step 4:

[0335] The server generates automation targets and efficiency improvement proposals based on the analysis information. The input is the analysis information from step 3, and the output is an efficiency improvement proposal document. A generative AI model is used to generate the proposals, producing specific improvement measures in written form.

[0336] Step 5:

[0337] The server outputs the generated efficiency suggestions to the user via a head-mounted display. The input is the efficiency suggestion document, and the output is a display of the suggestions as visual feedback to the user. The user can then immediately adjust their work based on these suggestions.

[0338] Step 6:

[0339] The server generates real-time feedback during operations to support process improvement. Inputs are the current operational status and suggestions, while outputs are refined operational processes. This process includes guidance to make it easier for users to perform new actions.

[0340] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0341] This invention relates to a system that, in addition to streamlining business processes, provides more accurate improvement suggestions by recognizing user emotions. This system consists of recording means, receiving processing means, analysis means, suggestion generation means, presentation means, and an emotion engine.

[0342] Explanation of program processing

[0343] The terminal records video and audio from the work site in real time. The recorded data is transmitted to a server via the network. At the same time, the user's facial expressions and voice tone are also recorded and used for emotion analysis.

[0344] The server preprocesses the received data and analyzes the video and audio data. This includes analyzing the operation of business processes, as well as recognizing the user's emotional state using an emotion engine. The emotion engine extracts emotional data using facial expression analysis and voice analysis.

[0345] The server combines sentiment data with business process analysis results to identify areas that need improvement. The suggestion generation system identifies inefficient processes in business operations and creates acceptable improvement suggestions by taking user sentiment into consideration in the solutions.

[0346] The server automatically generates a detailed report outlining the analysis results and recommendations. A video replay is also created, including personalized feedback and suggestions based on emotions.

[0347] Users can view reports and video replays provided by the server through their terminals. This allows for improvement activities that not only streamline operations but also take into account the emotional state of employees.

[0348] Specific example

[0349] Example 1: Improving Call Center Operations

[0350] The terminal records the operator's conversation and facial expressions, and sends the data to the server.

[0351] The server analyzes emotional states such as stress levels during calls and suggests script revisions or training opportunities.

[0352] After receiving the suggestions, the user will implement improvements to operational methods and manage operator stress.

[0353] Example 2: Understanding and improving the flow of meetings

[0354] The terminal records the facial expressions and statements of meeting participants and sends that data to the server.

[0355] The server emotionally analyzes changes in participants' interest and the intensity of discussion during the meeting, and makes suggestions for refining the agenda and improving the meeting's progress.

[0356] Based on the feedback, users will implement better discussion strategies in future meetings.

[0357] By combining this with an emotional engine, it becomes possible to improve operational efficiency while simultaneously providing suggestions that take employees' emotions into consideration.

[0358] The following describes the processing flow.

[0359] Step 1:

[0360] The device begins recording video and audio of work activities using cameras and microphones installed at the work site. This includes employee facial expressions and tone of voice, providing data to recognize the user's emotions. Once a certain amount of data has been collected, it is transmitted to a server via the network.

[0361] Step 2:

[0362] The server performs initial processing on the received video and audio data. This initial processing includes trimming the video, denoising the audio, and standardizing the data format. This converts the data into a format suitable for analysis.

[0363] Step 3:

[0364] The server uses an emotion engine to recognize the user's emotional state from the processed data. Specifically, a facial recognition algorithm analyzes the video data, and an audio analysis tool extracts emotions from the tone and pitch of the voice.

[0365] Step 4:

[0366] The server learns patterns in business processes using analytical tools. This involves analyzing identified business actions and process flows to detect inefficiencies and areas that need improvement.

[0367] Step 5:

[0368] The server generates improvement suggestions by combining emotional states and business process analysis results. The suggestion generation mechanism formulates specific action plans for efficiency improvements and customizes the suggestions to be user-friendly, taking emotional data into consideration.

[0369] Step 6:

[0370] The server automatically generates a report of the improvement suggestions that have been created. Furthermore, a video replay visually illustrating the identified areas for improvement is also created, and this information is provided in a format that is easy for users to understand.

[0371] Step 7:

[0372] Users review reports and video replays sent from the server via their terminals. Based on the information provided, users plan improvement activities in their actual work and optimize operations in a way that is sensitive to emotions.

[0373] (Example 2)

[0374] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0375] Traditional business process improvement systems focused on streamlining operations but failed to consider employees' emotional states when making improvement suggestions. As a result, suggestions were sometimes ineffective for employees, and true business improvement was not achieved. Furthermore, integrated analysis of multimodal data and real-time feedback were not adequately implemented, resulting in a lack of constructive feedback based on user emotions.

[0376] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0377] In this invention, the server includes a device for collecting video footage of work processes, a processing device for receiving the video and audio data collected by the device and organizing the information, and an analysis device for analyzing the data organized by the processing device and identifying the workflow. This makes it possible to provide improvement suggestions that take into account the user's emotions by utilizing emotion recognition technology and multimodal data analysis technology.

[0378] "Work process video" refers to data that visually records the work procedures and actions performed during the execution of work.

[0379] The term "device" refers to equipment or systems used to achieve a specific purpose.

[0380] "Video and audio data" refers to visual and auditory information recorded and stored in digital format.

[0381] A "processing device" is a system of hardware or software used to receive, organize, and analyze data.

[0382] "Organizing information" means processing, filtering, and classifying acquired data according to its purpose.

[0383] An "analysis device" is a device or program used to verify processed data and identify specific patterns or trends.

[0384] "Identifying the workflow" means compiling and evaluating the progress and procedures of a business process.

[0385] An "emotion recognition device" is a device or system that incorporates technology to identify emotions from a user's facial expressions and voice.

[0386] A "proposal generation device" is a system that has the function of generating problem-solving methods and improvement proposals based on analyzed data.

[0387] An "output device" is a device or program that provides information or suggestions to a user visually or audibly.

[0388] This system aims to improve operational efficiency and is specifically designed to suggest improvement measures that take into account the emotional state of employees. Specifically, it is structured around three roles: terminal, server, and user.

[0389] The terminals are deployed at work sites and function as devices for acquiring video and audio of work activities. This utilizes high-resolution cameras and high-sensitivity microphones. The terminals record this data in real time and transmit it to a server via the network. This data also includes emotional elements such as the user's facial expressions and tone of voice, which are used for emotion recognition.

[0390] The server receives video and audio data from the terminal and performs preprocessing such as noise filtering and data formatting. Next, using the processed data, the analysis device analyzes the business process in detail and uses an emotion recognition device to determine the user's emotional state. Here, a generative AI model is used to identify inefficiencies in the business by comprehensively analyzing multiple data sources and to create improvement suggestions that take emotions into consideration.

[0391] Users receive and review reports and video replays provided by the server via their terminals. Based on this information, they can implement emotion-driven improvement measures and achieve continuous business improvement.

[0392] As a concrete example, in a call center, a terminal records the operator's conversation and facial expressions, while a server analyzes their stress level. Then, using a generative AI model, it presents improvement measures using a prompt message such as, "How should we analyze the operator's stress level and provide specific improvement suggestions?" In this way, the system enables the user to receive optimal improvement suggestions.

[0393] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0394] Step 1:

[0395] The terminal uses cameras and microphones installed at the work site to record video and audio of work activities in real time. Input consists of ambient sounds and video from the work site, which are processed as digital data. Output is the captured video and audio data. This data includes the user's facial expressions and voice tone, and is used for emotion analysis.

[0396] Step 2:

[0397] The terminal transmits recorded digital data to the server via the network. The input consists of video and audio digital data recorded by the terminal. The data is encrypted and transmitted securely. The output is the data stream delivered to the server.

[0398] Step 3:

[0399] The server acquires video and audio data received from the terminal. The input is a data stream transmitted over the network, which the server receives. Initial processing includes noise filtering and time synchronization to organize the data. The output is the pre-processed audio and video data.

[0400] Step 4:

[0401] The server analyzes pre-processed data and evaluates the workflow. The input is pre-processed video and audio data. The server identifies the business process and uses data analysis algorithms to pinpoint inefficient steps. The output is the analysis results highlighting the inefficient parts.

[0402] Step 5:

[0403] The server uses an emotion recognition device to analyze the user's emotional state. The input is pre-processed video and audio data, and the emotion engine extracts emotional data by performing facial expression and voice analysis. The output is analyzed data that includes the user's emotional state.

[0404] Step 6:

[0405] The server utilizes a generative AI model to integrate sentiment data and business process analysis results to generate improvement suggestions for increased efficiency. The input consists of sentiment data and analysis results. The generative AI model uses prompts to create specific improvement suggestions. The output is a detailed report containing the improvement suggestions.

[0406] Step 7:

[0407] Users receive and review reports and video replays provided by the server via their terminals. The input consists of reports and video replays from the server. Based on this, users implement business improvement measures. The output is sustainable process improvement that prioritizes operational efficiency and employee satisfaction.

[0408] (Application Example 2)

[0409] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0410] Traditional business efficiency systems focus on identifying inefficiencies and suggesting improvements, but they often fail to consider user emotions, potentially leading to reduced acceptance of suggestions. Furthermore, they haven't been able to achieve efficient task management that considers user emotions in home and office environments. Therefore, there's a need for a system that analyzes user emotions and individually optimizes suggestions.

[0411] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0412] In this invention, the server includes an acquisition means for recording work video, a reception control means for receiving and pre-processing the recorded data, an analysis control means for analyzing the pre-processed information and learning patterns of business processes, an emotion analysis means for analyzing the emotional state of the user, and an improvement means for optimizing the proposed content based on the emotion analysis results. This makes it possible to propose highly acceptable improvements that take into account the user's emotions in order to improve business efficiency.

[0413] "Means of acquisition" refers to devices or functions for recording video data of work or actions.

[0414] "Receiving control means" refers to a device or function for receiving recorded data and performing necessary preprocessing.

[0415] "Analysis and control means" refers to devices or functions that analyze pre-processed information and learn patterns of business processes and behaviors.

[0416] A "proposal generation means" refers to a device or function that identifies inefficient actions based on analysis results and generates improvement proposals.

[0417] "Presentation control means" refers to devices or functions for providing generated suggestions to users.

[0418] "Emotional analysis tools" refer to devices or functions used to analyze and understand a user's emotional state.

[0419] "Improvement measures" refer to devices or functions that individually optimize the content of suggestions based on the results of sentiment analysis.

[0420] To implement this invention, the system is composed of a combination of various hardware and software. The server, terminals, and users work together to perform data processing for efficient business improvement.

[0421] The server acquires video and audio data in real time from terminals equipped with cameras and audio sensors for recording work and daily activities. The data obtained by this acquisition means is transmitted to the server by a receiving control means and preprocessed. This process involves removing unwanted noise and converting the data format.

[0422] The server uses analysis control mechanisms to learn business processes and behavioral patterns in order to analyze pre-processed information. Similarly, emotion analysis mechanisms are used to detect emotional states from the user's facial expressions and tone of voice, and to store this data. For analysis, emotion analysis APIs such as Amazon Rekognition and Google Cloud Vision are utilized.

[0423] The data obtained through analysis is transformed into personalized improvement suggestions for each user by a suggestion generation system. For example, when a household robot determines the need for cleaning, it refers to emotion analysis data and suggests the optimal action at a time when the user is relaxed.

[0424] The terminal uses presentation control means to present generated suggestions and feedback to the user in real time. This feedback is optimized based on the user's emotional state, allowing the user to take appropriate action.

[0425] Specific examples include analyzing a user's emotional state while they are watching television to reduce stress and then suggesting the most appropriate household chores. An example of a prompt for the generative AI model would be, "Analyze the user's current facial expression and tone of voice to generate appropriate household chore suggestions."

[0426] This system will enable highly acceptable suggestions that take into account the user's emotional state in both work and daily life, leading to improved work efficiency and a more comfortable experience.

[0427] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0428] Step 1:

[0429] The device uses a camera and audio sensor to capture work and daily activities, acquiring video and audio data in real time. This data serves as input, which is then prepared for conversion into an appropriate format after removing noise and unwanted information. The output is appropriate video and audio data for pre-processing.

[0430] Step 2:

[0431] The terminal transmits the acquired data to the server. The server receives this data using a receiving control means and performs preprocessing such as noise reduction and format conversion. The input is the transmitted video and audio data, and the output is data in an analyzable format.

[0432] Step 3:

[0433] The server analyzes pre-processed data using analysis and control means to learn business processes and behavioral patterns. The input for this step is pre-processed data, and the output is learned pattern information. Specifically, data analysis is performed using machine learning algorithms.

[0434] Step 4:

[0435] The server performs data analysis to determine the user's emotional state using emotion analysis tools. Pre-processed data is treated as input, and data indicating the user's emotional state is output. Specifically, it utilizes facial expression analysis and voice analysis technologies to extract emotional data.

[0436] Step 5:

[0437] The server generates user-specific improvement suggestions based on analysis results and emotional state data using a suggestion generation mechanism. The input is the analysis results from steps 3 and 4, and the output is user-optimized improvement suggestions. The suggestions are generated using an algorithm that takes emotional state into consideration.

[0438] Step 6:

[0439] The server sends suggestions generated using the suggestion control means to the terminal and notifies the user. The input is the suggestions from the suggestion generation means, and the output is the content presented to the user terminal. The user can receive real-time feedback through the terminal and use it to improve their life and work.

[0440] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0441] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0442] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0443] [Third Embodiment]

[0444] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0445] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0446] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0447] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0448] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0449] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0450] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0451] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0452] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0453] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0454] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0455] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0456] This invention relates to an embodiment of a system that automatically identifies and proposes improvements necessary to enhance operational efficiency. This system primarily aims to record and analyze video footage of work processes and provide concrete suggestions for automation and efficiency improvements. The program processing and specific examples of this system are described in detail below.

[0457] Explanation of program processing

[0458] The terminal is installed at the work site and records the actual work situation via camera and microphone. The recorded video and audio data is transmitted to a server via the internet.

[0459] The server first performs data preprocessing in order to analyze the received video and audio data. Preprocessing includes noise reduction and standardization of the data format.

[0460] The server uses analysis tools to analyze data using multimodal generation technology. In this process, motion recognition from video data and speech analysis from audio data are performed in an integrated manner, and patterns of business processes are learned.

[0461] Based on learned patterns, the server identifies inefficient processes and parts that can be automated, and uses a suggestion generation mechanism to create improvement suggestions for increased efficiency.

[0462] The server compiles the generated suggestions into a report and also creates a video replay. This makes it possible to present concrete improvement models.

[0463] Users can view these proposals and video replays on their devices. Based on the reported information, users can implement improvement activities in their actual work.

[0464] Specific example

[0465] Example 1: Improvement of the manufacturing line

[0466] The terminal takes pictures of the factory's production line and sends the data to the server.

[0467] The server identifies unnecessary actions in the product assembly process and suggests that manufacturing time can be reduced by 20% by omitting or automating these actions.

[0468] After reviewing the proposal, the user will implement the new work procedures to improve productivity.

[0469] Example 2: Improving the efficiency of meetings in the office

[0470] The device records and videos office meetings and sends the data to the server.

[0471] The server identifies redundant discussions in the meeting and proposes specific approaches to effectively manage the agenda.

[0472] The user will accept the suggestion and implement more efficient meeting procedures from the next meeting onwards.

[0473] As these examples demonstrate, by utilizing the system of the present invention, it is possible to promote operational efficiency and achieve significant cost reductions and productivity improvements.

[0474] The following describes the processing flow.

[0475] Step 1:

[0476] The terminal uses cameras and microphones installed at the work site to record video and audio of work activities in real time. This data is stored in a buffer at regular intervals, and when the stored data reaches a specified threshold, or when it is transmitted to the server via the network in real time, it is sent to the server.

[0477] Step 2:

[0478] The server performs initial processing on the received video and audio data. This initial processing includes data compression, trimming of unnecessary parts, and noise reduction. As a result of this processing, data suitable for analysis is generated.

[0479] Step 3:

[0480] The server analyzes the data after initial processing using analytical tools. The analysis utilizes multimodal generation AI to perform behavioral recognition from video data and extract discussion content and emotions from audio data. The analysis identifies patterns in business processes, providing information that can be used to determine efficiency and inefficiencies.

[0481] Step 4:

[0482] Based on identified business process patterns, the server generates improvement suggestions for efficiency improvements using a suggestion generation mechanism. These suggestions include time reductions, procedure restructuring, and equipment implementation. The suggestions are then listed as specific actions.

[0483] Step 5:

[0484] The server automatically generates a report of the suggestions that have been submitted. The report details the identified areas for improvement and suggestions, and may include data that can be played back as a visual video replay.

[0485] Step 6:

[0486] The server sends the completed report and video replay to the terminal via the network, displaying the results to the user. The user can review this information and gain insights to implement the identified improvements.

[0487] (Example 1)

[0488] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0489] To improve operational efficiency, it is necessary to accurately understand the work situation on-site, identify inefficient work processes, and improve them. However, in many current systems, the processes from data collection to analysis and improvement proposals are independent and not automated as a continuous flow, which slows down overall operational efficiency. Solving this problem is the challenge.

[0490] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0491] In this invention, the server includes a device for recording the status of work, a receiving means for receiving data recorded by the device and pre-processing the data, and a means for analyzing the pre-processed data by the receiving means and learning patterns of the work flow. This enables real-time data analysis and immediate generation of improvement suggestions at the work site.

[0492] A "device" is physical hardware installed at the work site to record the status of the work.

[0493] "Receiving means" refers to a procedure or function for receiving data recorded by a device and for performing preprocessing on that data.

[0494] "Means" refer to the methods, processes, or mechanisms used to achieve a specific objective.

[0495] "Analysis methods" refer to processes and techniques for learning and identifying workflows and work patterns by analyzing pre-processed data in detail.

[0496] "Means of delivery" refers to a system for presenting and providing the generated analysis results and suggestions to users.

[0497] This invention relates to a system designed to improve business efficiency. Specifically, it records and analyzes the workflow at the work site through a series of processes involving a server, terminal, and user, and generates suggestions for improvement.

[0498] First, the terminal is a device installed at the work site that uses a camera and microphone to record the work situation. This device has high-resolution shooting and audio recording capabilities, allowing for the collection of detailed data. This data is transmitted to a server using a secure communication protocol.

[0499] The server performs preprocessing on the received video and audio data. This preprocessing uses techniques such as FFT (Fast Fourier Transform) and H.264 encoding to remove noise and standardize data formats. Subsequently, a multimodal generative AI model is used to recognize human actions from the video and analyze spoken content from the audio. This allows the system to automatically analyze the workflow and learn patterns in business processes.

[0500] Based on the analysis results, the server identifies inefficient procedures and generates specific suggestions for improvement. The generative AI model used here automates data analysis and suggestion generation, and has the capability to create proposal documents and video replays. These suggestions include specific improvement actions directed at the user, promoting increased efficiency in operations.

[0501] Users can review these proposals and video replays via their devices and implement actual business improvements based on the presented content.

[0502] As a concrete example, consider improving a manufacturing line. A terminal photographs the factory's manufacturing line, and a server uses that data to identify inefficiencies in the product assembly process and generates suggestions to reduce man-hours by automating the process. This entire process is input into the AI ​​model using prompt messages like the following:

[0503] "Analyze the video data from the manufacturing line and create a proposal to increase efficiency by 20%."

[0504] This system enables efficient business operations by allowing for a detailed understanding of workflows and the immediate generation of improvement suggestions.

[0505] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0506] Step 1:

[0507] The terminal is installed at the work site and uses a camera and microphone to record the work situation. Specifically, a high-resolution camera captures the worker's movements as video, and a high-sensitivity microphone records audio of the work environment. This allows for the acquisition of detailed and accurate work data as input. This data is generated as time-series video and audio files.

[0508] Step 2:

[0509] The terminal transmits recorded video and audio data to the server via the internet. This transmission uses a secure and high-speed communication protocol to prevent data delays and loss. The output at this step is a data stream for the server to receive.

[0510] Step 3:

[0511] The server preprocesses the received data. Specifically, it uses FFT (Fast Fourier Transform) to remove noise from audio data and encodes video data in H.264 format. This results in a clean dataset that is easy to analyze. The preprocessed data is then used as input for subsequent analysis.

[0512] Step 4:

[0513] The server performs analysis using a multimodal generation AI model with pre-processed data. It uses open-pause technology to recognize motion from video data and speech recognition technology to convert spoken content into text from audio data. This allows the system to learn patterns in business processes and generate complex data analysis results as output.

[0514] Step 5:

[0515] Based on the analysis results obtained, the server identifies inefficient areas and uses a generative AI model to formulate improvement proposals. During this process, it compares and analyzes data from each process to specifically determine where and how improvements should be made. A proposal document and video replay are generated, which are output information provided to the user.

[0516] Step 6:

[0517] The user reviews the provided proposals and video replays via their terminal. They then examine the outputted improvement suggestions in detail and evaluate how they can be implemented in their actual work. The output of this step is the user's plan for specific improvement activities.

[0518] This completes a series of steps in which the entire system analyzes and optimizes business processes in detail.

[0519] (Application Example 1)

[0520] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0521] Efficient business operations are a critical challenge faced by many companies. In particular, traditional methods have made it difficult to identify areas for efficiency improvements in manufacturing and robot operation. Manual improvement suggestions are time-consuming and difficult to implement in real time. Therefore, there is a growing need for systems that can immediately propose and implement improvements to streamline production lines and business processes.

[0522] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0523] In this invention, the server includes: video recording means for recording video of work operations; data processing means for receiving and pre-processing video information recorded by the video recording means; pattern analysis means for analyzing the pre-processed information by the data processing means and learning patterns of work processes; proposal generation means for identifying inefficient processes detected by the pattern analysis means and generating suggestions for efficiency improvements; information presentation means for providing the user with the analysis results and suggestions generated by the proposal generation means; and real-time feedback generation means for generating real-time feedback to the robot. This makes it possible to immediately propose efficiency improvements in business and manufacturing processes and to achieve real-time improvements.

[0524] "Video recording means" refers to a device or system that records the status of work in real time using a camera or similar device.

[0525] "Data processing means" refers to a device or program that receives recorded video information and has the function of performing preprocessing such as noise reduction and format standardization.

[0526] A "pattern analysis system" is a system that analyzes patterns in business processes based on pre-processed information and extracts data for efficiency improvements.

[0527] A "proposal generation means" is a device or program that automatically detects inefficient business processes based on pattern analysis and generates proposals for improving efficiency.

[0528] "Information presentation means" refers to a device or interface for visually providing the user with the analysis results and improvement suggestions generated by the proposal generation means.

[0529] A "real-time feedback generation means" is a device or system that has the function of providing users with immediate feedback for process improvement and enabling immediate improvement.

[0530] The system for implementing this invention is configured to combine advanced data processing and analysis technologies in order to improve operational efficiency.

[0531] The server collects real-time video footage of the work environment from cameras mounted on factory robots. This video recording system utilizes high-resolution cameras, enabling precise data acquisition. Additionally, there is a data processing system that receives the video information, performing noise reduction and standardizing the data format. Libraries such as OpenCV and TensorFlow are primarily used for this purpose.

[0532] The pattern analysis method learns patterns in business processes based on pre-processed information. This utilizes multimodal generation technology, such as Google Cloud AI. This analysis identifies inefficient business processes, and the data is used to gather information for suggesting improvements.

[0533] The proposal generation mechanism generates specific proposals for automation and efficiency improvements based on information obtained from pattern analysis. This allows for the identification of specific methods for improving inefficient parts and enables the generation of improvement measures in real time.

[0534] The information presentation system plays the role of presenting the generated analysis results and suggestions to the user. This provides operators and users with visual improvement suggestions in real time via a head-mounted display. Therefore, users can immediately review the presented suggestions and quickly implement improvements to their work.

[0535] Furthermore, the real-time feedback generation system provides immediate feedback for improvements during operations, supporting immediate process adjustments. This enables continuous and dynamic optimization of the factory line.

[0536] A concrete example is when AI detects wasted time during assembly line work and suggests, "By shortening the movement of your right hand by 3 seconds, the next process will proceed more smoothly." This suggestion allows the operator to immediately correct their movements, improving overall production efficiency.

[0537] Examples of prompts for the generating AI model include "optimize assembly actions," "detect and suggest actions that can be improved," and "present streamlined assembly procedures." These prompts are used to quickly generate specific and feasible improvement suggestions.

[0538] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0539] Step 1:

[0540] The server uses cameras installed on factory robots to acquire real-time video footage of the work processes. The input is raw data from the cameras, and the output is a video stream. This video is recorded in high resolution frame by frame to accurately capture changes in the work environment.

[0541] Step 2:

[0542] The server performs preprocessing on the acquired video stream to remove noise and standardize the format. The input is the video stream generated in step 1, and the output is the preprocessed, clear video data. Specifically, filtering is performed using the OpenCV library and data standardization is performed using TensorFlow.

[0543] Step 3:

[0544] The server analyzes patterns in business processes using pre-processed video data. The input is pre-processed video data, and the output is analytical information to identify inefficiencies. Google Cloud AI is used to identify actions in the video and perform pattern learning by comparing them with known processes.

[0545] Step 4:

[0546] The server generates automation targets and efficiency improvement proposals based on the analysis information. The input is the analysis information from step 3, and the output is an efficiency improvement proposal document. A generative AI model is used to generate the proposals, producing specific improvement measures in written form.

[0547] Step 5:

[0548] The server outputs the generated efficiency suggestions to the user via a head-mounted display. The input is the efficiency suggestion document, and the output is a display of the suggestions as visual feedback to the user. The user can then immediately adjust their work based on these suggestions.

[0549] Step 6:

[0550] The server generates real-time feedback during operations to support process improvement. Inputs are the current operational status and suggestions, while outputs are refined operational processes. This process includes guidance to make it easier for users to perform new actions.

[0551] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0552] This invention relates to a system that, in addition to streamlining business processes, provides more accurate improvement suggestions by recognizing user emotions. This system consists of recording means, receiving processing means, analysis means, suggestion generation means, presentation means, and an emotion engine.

[0553] Explanation of program processing

[0554] The terminal records video and audio from the work site in real time. The recorded data is transmitted to a server via the network. At the same time, the user's facial expressions and voice tone are also recorded and used for emotion analysis.

[0555] The server preprocesses the received data and analyzes the video and audio data. This includes analyzing the operation of business processes, as well as recognizing the user's emotional state using an emotion engine. The emotion engine extracts emotional data using facial expression analysis and voice analysis.

[0556] The server combines sentiment data with business process analysis results to identify areas that need improvement. The suggestion generation system identifies inefficient processes in business operations and creates acceptable improvement suggestions by taking user sentiment into consideration in the solutions.

[0557] The server automatically generates a detailed report outlining the analysis results and recommendations. A video replay is also created, including personalized feedback and suggestions based on emotions.

[0558] Users can view reports and video replays provided by the server through their terminals. This allows for improvement activities that not only streamline operations but also take into account the emotional state of employees.

[0559] Specific example

[0560] Example 1: Improving Call Center Operations

[0561] The terminal records the operator's conversation and facial expressions, and sends the data to the server.

[0562] The server analyzes emotional states such as stress levels during calls and suggests script revisions or training opportunities.

[0563] After receiving the suggestions, the user will implement improvements to operational methods and manage operator stress.

[0564] Example 2: Understanding and improving the flow of meetings

[0565] The terminal records the facial expressions and statements of meeting participants and sends that data to the server.

[0566] The server emotionally analyzes changes in participants' interest and the intensity of discussion during the meeting, and makes suggestions for refining the agenda and improving the meeting's progress.

[0567] Based on the feedback, users will implement better discussion strategies in future meetings.

[0568] By combining this with an emotional engine, it becomes possible to improve operational efficiency while simultaneously providing suggestions that take employees' emotions into consideration.

[0569] The following describes the processing flow.

[0570] Step 1:

[0571] The device begins recording video and audio of work activities using cameras and microphones installed at the work site. This includes employee facial expressions and tone of voice, providing data to recognize the user's emotions. Once a certain amount of data has been collected, it is transmitted to a server via the network.

[0572] Step 2:

[0573] The server performs initial processing on the received video and audio data. This initial processing includes trimming the video, denoising the audio, and standardizing the data format. This converts the data into a format suitable for analysis.

[0574] Step 3:

[0575] The server uses an emotion engine to recognize the user's emotional state from the processed data. Specifically, a facial recognition algorithm analyzes the video data, and an audio analysis tool extracts emotions from the tone and pitch of the voice.

[0576] Step 4:

[0577] The server learns patterns in business processes using analytical tools. This involves analyzing identified business actions and process flows to detect inefficiencies and areas that need improvement.

[0578] Step 5:

[0579] The server generates improvement suggestions by combining emotional states and business process analysis results. The suggestion generation mechanism formulates specific action plans for efficiency improvements and customizes the suggestions to be user-friendly, taking emotional data into consideration.

[0580] Step 6:

[0581] The server automatically generates a report of the improvement suggestions that have been created. Furthermore, a video replay visually illustrating the identified areas for improvement is also created, and this information is provided in a format that is easy for users to understand.

[0582] Step 7:

[0583] Users review reports and video replays sent from the server via their terminals. Based on the information provided, users plan improvement activities in their actual work and optimize operations in a way that is sensitive to emotions.

[0584] (Example 2)

[0585] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0586] Traditional business process improvement systems focused on streamlining operations but failed to consider employees' emotional states when making improvement suggestions. As a result, suggestions were sometimes ineffective for employees, and true business improvement was not achieved. Furthermore, integrated analysis of multimodal data and real-time feedback were not adequately implemented, resulting in a lack of constructive feedback based on user emotions.

[0587] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0588] In this invention, the server includes a device for collecting video footage of work processes, a processing device for receiving the video and audio data collected by the device and organizing the information, and an analysis device for analyzing the data organized by the processing device and identifying the workflow. This makes it possible to provide improvement suggestions that take into account the user's emotions by utilizing emotion recognition technology and multimodal data analysis technology.

[0589] "Work process video" refers to data that visually records the work procedures and actions performed during the execution of work.

[0590] The term "device" refers to equipment or systems used to achieve a specific purpose.

[0591] "Video and audio data" refers to visual and auditory information recorded and stored in digital format.

[0592] A "processing device" is a system of hardware or software used to receive, organize, and analyze data.

[0593] "Organizing information" means processing, filtering, and classifying acquired data according to its purpose.

[0594] An "analysis device" is a device or program used to verify processed data and identify specific patterns or trends.

[0595] "Identifying the workflow" means compiling and evaluating the progress and procedures of a business process.

[0596] An "emotion recognition device" is a device or system that incorporates technology to identify emotions from a user's facial expressions and voice.

[0597] A "proposal generation device" is a system that has the function of generating problem-solving methods and improvement proposals based on analyzed data.

[0598] An "output device" is a device or program that provides information or suggestions to a user visually or audibly.

[0599] This system aims to improve operational efficiency and is specifically designed to suggest improvement measures that take into account the emotional state of employees. Specifically, it is structured around three roles: terminal, server, and user.

[0600] The terminals are deployed at work sites and function as devices for acquiring video and audio of work activities. This utilizes high-resolution cameras and high-sensitivity microphones. The terminals record this data in real time and transmit it to a server via the network. This data also includes emotional elements such as the user's facial expressions and tone of voice, which are used for emotion recognition.

[0601] The server receives video and audio data from the terminal and performs preprocessing such as noise filtering and data formatting. Next, using the processed data, the analysis device analyzes the business process in detail and uses an emotion recognition device to determine the user's emotional state. Here, a generative AI model is used to identify inefficiencies in the business by comprehensively analyzing multiple data sources and to create improvement suggestions that take emotions into consideration.

[0602] Users receive and review reports and video replays provided by the server via their terminals. Based on this information, they can implement emotion-driven improvement measures and achieve continuous business improvement.

[0603] As a concrete example, in a call center, a terminal records the operator's conversation and facial expressions, while a server analyzes their stress level. Then, using a generative AI model, it presents improvement measures using a prompt message such as, "How should we analyze the operator's stress level and provide specific improvement suggestions?" In this way, the system enables the user to receive optimal improvement suggestions.

[0604] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0605] Step 1:

[0606] The terminal uses cameras and microphones installed at the work site to record video and audio of work activities in real time. Input consists of ambient sounds and video from the work site, which are processed as digital data. Output is the captured video and audio data. This data includes the user's facial expressions and voice tone, and is used for emotion analysis.

[0607] Step 2:

[0608] The terminal transmits recorded digital data to the server via the network. The input consists of video and audio digital data recorded by the terminal. The data is encrypted and transmitted securely. The output is the data stream delivered to the server.

[0609] Step 3:

[0610] The server acquires video and audio data received from the terminal. The input is a data stream transmitted over the network, which the server receives. Initial processing includes noise filtering and time synchronization to organize the data. The output is the pre-processed audio and video data.

[0611] Step 4:

[0612] The server analyzes pre-processed data and evaluates the workflow. The input is pre-processed video and audio data. The server identifies the business process and uses data analysis algorithms to pinpoint inefficient steps. The output is the analysis results highlighting the inefficient parts.

[0613] Step 5:

[0614] The server uses an emotion recognition device to analyze the user's emotional state. The input is pre-processed video and audio data, and the emotion engine extracts emotional data by performing facial expression and voice analysis. The output is analyzed data that includes the user's emotional state.

[0615] Step 6:

[0616] The server utilizes a generative AI model to integrate sentiment data and business process analysis results to generate improvement suggestions for increased efficiency. The input consists of sentiment data and analysis results. The generative AI model uses prompts to create specific improvement suggestions. The output is a detailed report containing the improvement suggestions.

[0617] Step 7:

[0618] Users receive and review reports and video replays provided by the server via their terminals. The input consists of reports and video replays from the server. Based on this, users implement business improvement measures. The output is sustainable process improvement that prioritizes operational efficiency and employee satisfaction.

[0619] (Application Example 2)

[0620] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0621] Traditional business efficiency systems focus on identifying inefficiencies and suggesting improvements, but they often fail to consider user emotions, potentially leading to reduced acceptance of suggestions. Furthermore, they haven't been able to achieve efficient task management that considers user emotions in home and office environments. Therefore, there's a need for a system that analyzes user emotions and individually optimizes suggestions.

[0622] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0623] In this invention, the server includes an acquisition means for recording work video, a reception control means for receiving and pre-processing the recorded data, an analysis control means for analyzing the pre-processed information and learning patterns of business processes, an emotion analysis means for analyzing the emotional state of the user, and an improvement means for optimizing the proposed content based on the emotion analysis results. This makes it possible to propose highly acceptable improvements that take into account the user's emotions in order to improve business efficiency.

[0624] "Means of acquisition" refers to devices or functions for recording video data of work or actions.

[0625] "Receiving control means" refers to a device or function for receiving recorded data and performing necessary preprocessing.

[0626] "Analysis and control means" refers to devices or functions that analyze pre-processed information and learn patterns of business processes and behaviors.

[0627] A "proposal generation means" refers to a device or function that identifies inefficient actions based on analysis results and generates improvement proposals.

[0628] "Presentation control means" refers to devices or functions for providing generated suggestions to users.

[0629] "Emotional analysis tools" refer to devices or functions used to analyze and understand a user's emotional state.

[0630] "Improvement measures" refer to devices or functions that individually optimize the content of suggestions based on the results of sentiment analysis.

[0631] To implement this invention, the system is composed of a combination of various hardware and software. The server, terminals, and users work together to perform data processing for efficient business improvement.

[0632] The server acquires video and audio data in real time from terminals equipped with cameras and audio sensors for recording work and daily activities. The data obtained by this acquisition means is transmitted to the server by a receiving control means and preprocessed. This process involves removing unwanted noise and converting the data format.

[0633] The server uses analysis control mechanisms to learn business processes and behavioral patterns in order to analyze pre-processed information. Similarly, emotion analysis mechanisms are used to detect emotional states from the user's facial expressions and tone of voice, and to store this data. For analysis, emotion analysis APIs such as Amazon Rekognition and Google Cloud Vision are utilized.

[0634] The data obtained through analysis is transformed into personalized improvement suggestions for each user by a suggestion generation system. For example, when a household robot determines the need for cleaning, it refers to emotion analysis data and suggests the optimal action at a time when the user is relaxed.

[0635] The terminal uses presentation control means to present generated suggestions and feedback to the user in real time. This feedback is optimized based on the user's emotional state, allowing the user to take appropriate action.

[0636] Specific examples include analyzing a user's emotional state while they are watching television to reduce stress and then suggesting the most appropriate household chores. An example of a prompt for the generative AI model would be, "Analyze the user's current facial expression and tone of voice to generate appropriate household chore suggestions."

[0637] This system will enable highly acceptable suggestions that take into account the user's emotional state in both work and daily life, leading to improved work efficiency and a more comfortable experience.

[0638] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0639] Step 1:

[0640] The device uses a camera and audio sensor to capture work and daily activities, acquiring video and audio data in real time. This data serves as input, which is then prepared for conversion into an appropriate format after removing noise and unwanted information. The output is appropriate video and audio data for pre-processing.

[0641] Step 2:

[0642] The terminal transmits the acquired data to the server. The server receives this data using a receiving control means and performs preprocessing such as noise reduction and format conversion. The input is the transmitted video and audio data, and the output is data in an analyzable format.

[0643] Step 3:

[0644] The server analyzes pre-processed data using analysis and control means to learn business processes and behavioral patterns. The input for this step is pre-processed data, and the output is learned pattern information. Specifically, data analysis is performed using machine learning algorithms.

[0645] Step 4:

[0646] The server performs data analysis to determine the user's emotional state using emotion analysis tools. Pre-processed data is treated as input, and data indicating the user's emotional state is output. Specifically, it utilizes facial expression analysis and voice analysis technologies to extract emotional data.

[0647] Step 5:

[0648] The server generates user-specific improvement suggestions based on analysis results and emotional state data using a suggestion generation mechanism. The input is the analysis results from steps 3 and 4, and the output is user-optimized improvement suggestions. The suggestions are generated using an algorithm that takes emotional state into consideration.

[0649] Step 6:

[0650] The server sends suggestions generated using the suggestion control means to the terminal and notifies the user. The input is the suggestions from the suggestion generation means, and the output is the content presented to the user terminal. The user can receive real-time feedback through the terminal and use it to improve their life and work.

[0651] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0652] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0653] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0654] [Fourth Embodiment]

[0655] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0656] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0657] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0658] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0659] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0660] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0661] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0662] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0663] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0664] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0665] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0666] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0667] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] This invention relates to an embodiment of a system that automatically identifies and proposes improvements necessary to enhance operational efficiency. This system primarily aims to record and analyze video footage of work processes and provide concrete suggestions for automation and efficiency improvements. The program processing and specific examples of this system are described in detail below.

[0669] Explanation of program processing

[0670] The terminal is installed at the work site and records the actual work situation via camera and microphone. The recorded video and audio data is transmitted to a server via the internet.

[0671] The server first performs data preprocessing in order to analyze the received video and audio data. Preprocessing includes noise reduction and standardization of the data format.

[0672] The server uses analysis tools to analyze data using multimodal generation technology. In this process, motion recognition from video data and speech analysis from audio data are performed in an integrated manner, and patterns of business processes are learned.

[0673] Based on learned patterns, the server identifies inefficient processes and parts that can be automated, and uses a suggestion generation mechanism to create improvement suggestions for increased efficiency.

[0674] The server compiles the generated suggestions into a report and also creates a video replay. This makes it possible to present concrete improvement models.

[0675] Users can view these proposals and video replays on their devices. Based on the reported information, users can implement improvement activities in their actual work.

[0676] Specific example

[0677] Example 1: Improvement of the manufacturing line

[0678] The terminal takes pictures of the factory's production line and sends the data to the server.

[0679] The server identifies unnecessary actions in the product assembly process and suggests that manufacturing time can be reduced by 20% by omitting or automating these actions.

[0680] After reviewing the proposal, the user will implement the new work procedures to improve productivity.

[0681] Example 2: Improving the efficiency of meetings in the office

[0682] The device records and videos office meetings and sends the data to the server.

[0683] The server identifies redundant discussions in the meeting and proposes specific approaches to effectively manage the agenda.

[0684] The user will accept the suggestion and implement more efficient meeting procedures from the next meeting onwards.

[0685] As these examples demonstrate, by utilizing the system of the present invention, it is possible to promote operational efficiency and achieve significant cost reductions and productivity improvements.

[0686] The following describes the processing flow.

[0687] Step 1:

[0688] The terminal uses cameras and microphones installed at the work site to record video and audio of work activities in real time. This data is stored in a buffer at regular intervals, and when the stored data reaches a specified threshold, or when it is transmitted to the server via the network in real time, it is sent to the server.

[0689] Step 2:

[0690] The server performs initial processing on the received video and audio data. This initial processing includes data compression, trimming of unnecessary parts, and noise reduction. As a result of this processing, data suitable for analysis is generated.

[0691] Step 3:

[0692] The server analyzes the data after initial processing using analytical tools. The analysis utilizes multimodal generation AI to perform behavioral recognition from video data and extract discussion content and emotions from audio data. The analysis identifies patterns in business processes, providing information that can be used to determine efficiency and inefficiencies.

[0693] Step 4:

[0694] Based on identified business process patterns, the server generates improvement suggestions for efficiency improvements using a suggestion generation mechanism. These suggestions include time reductions, procedure restructuring, and equipment implementation. The suggestions are then listed as specific actions.

[0695] Step 5:

[0696] The server automatically generates a report of the suggestions that have been submitted. The report details the identified areas for improvement and suggestions, and may include data that can be played back as a visual video replay.

[0697] Step 6:

[0698] The server sends the completed report and video replay to the terminal via the network, displaying the results to the user. The user can review this information and gain insights to implement the identified improvements.

[0699] (Example 1)

[0700] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0701] To improve operational efficiency, it is necessary to accurately understand the work situation on-site, identify inefficient work processes, and improve them. However, in many current systems, the processes from data collection to analysis and improvement proposals are independent and not automated as a continuous flow, which slows down overall operational efficiency. Solving this problem is the challenge.

[0702] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0703] In this invention, the server includes a device for recording the status of work, a receiving means for receiving data recorded by the device and pre-processing the data, and a means for analyzing the pre-processed data by the receiving means and learning patterns of the work flow. This enables real-time data analysis and immediate generation of improvement suggestions at the work site.

[0704] A "device" is physical hardware installed at the work site to record the status of the work.

[0705] "Receiving means" refers to a procedure or function for receiving data recorded by a device and for performing preprocessing on that data.

[0706] "Means" refer to the methods, processes, or mechanisms used to achieve a specific objective.

[0707] "Analysis methods" refer to processes and techniques for learning and identifying workflows and work patterns by analyzing pre-processed data in detail.

[0708] "Means of delivery" refers to a system for presenting and providing the generated analysis results and suggestions to users.

[0709] This invention relates to a system designed to improve business efficiency. Specifically, it records and analyzes the workflow at the work site through a series of processes involving a server, terminal, and user, and generates suggestions for improvement.

[0710] First, the terminal is a device installed at the work site that uses a camera and microphone to record the work situation. This device has high-resolution shooting and audio recording capabilities, allowing for the collection of detailed data. This data is transmitted to a server using a secure communication protocol.

[0711] The server performs preprocessing on the received video and audio data. This preprocessing uses techniques such as FFT (Fast Fourier Transform) and H.264 encoding to remove noise and standardize data formats. Subsequently, a multimodal generative AI model is used to recognize human actions from the video and analyze spoken content from the audio. This allows the system to automatically analyze the workflow and learn patterns in business processes.

[0712] Based on the analysis results, the server identifies inefficient procedures and generates specific suggestions for improvement. The generative AI model used here automates data analysis and suggestion generation, and has the capability to create proposal documents and video replays. These suggestions include specific improvement actions directed at the user, promoting increased efficiency in operations.

[0713] Users can review these proposals and video replays via their devices and implement actual business improvements based on the presented content.

[0714] As a concrete example, consider improving a manufacturing line. A terminal photographs the factory's manufacturing line, and a server uses that data to identify inefficiencies in the product assembly process and generates suggestions to reduce man-hours by automating the process. This entire process is input into the AI ​​model using prompt messages like the following:

[0715] "Analyze the video data from the manufacturing line and create a proposal to increase efficiency by 20%."

[0716] This system enables efficient business operations by allowing for a detailed understanding of workflows and the immediate generation of improvement suggestions.

[0717] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0718] Step 1:

[0719] The terminal is installed at the work site and uses a camera and microphone to record the work situation. Specifically, a high-resolution camera captures the worker's movements as video, and a high-sensitivity microphone records audio of the work environment. This allows for the acquisition of detailed and accurate work data as input. This data is generated as time-series video and audio files.

[0720] Step 2:

[0721] The terminal transmits recorded video and audio data to the server via the internet. This transmission uses a secure and high-speed communication protocol to prevent data delays and loss. The output at this step is a data stream for the server to receive.

[0722] Step 3:

[0723] The server preprocesses the received data. Specifically, it uses FFT (Fast Fourier Transform) to remove noise from audio data and encodes video data in H.264 format. This results in a clean dataset that is easy to analyze. The preprocessed data is then used as input for subsequent analysis.

[0724] Step 4:

[0725] The server performs analysis using a multimodal generation AI model with pre-processed data. It uses open-pause technology to recognize motion from video data and speech recognition technology to convert spoken content into text from audio data. This allows the system to learn patterns in business processes and generate complex data analysis results as output.

[0726] Step 5:

[0727] Based on the analysis results obtained, the server identifies inefficient areas and uses a generative AI model to formulate improvement proposals. During this process, it compares and analyzes data from each process to specifically determine where and how improvements should be made. A proposal document and video replay are generated, which are output information provided to the user.

[0728] Step 6:

[0729] The user reviews the provided proposals and video replays via their terminal. They then examine the outputted improvement suggestions in detail and evaluate how they can be implemented in their actual work. The output of this step is the user's plan for specific improvement activities.

[0730] This completes a series of steps in which the entire system analyzes and optimizes business processes in detail.

[0731] (Application Example 1)

[0732] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0733] Efficient business operations are a critical challenge faced by many companies. In particular, traditional methods have made it difficult to identify areas for efficiency improvements in manufacturing and robot operation. Manual improvement suggestions are time-consuming and difficult to implement in real time. Therefore, there is a growing need for systems that can immediately propose and implement improvements to streamline production lines and business processes.

[0734] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0735] In this invention, the server includes: video recording means for recording video of work operations; data processing means for receiving and pre-processing video information recorded by the video recording means; pattern analysis means for analyzing the pre-processed information by the data processing means and learning patterns of work processes; proposal generation means for identifying inefficient processes detected by the pattern analysis means and generating suggestions for efficiency improvements; information presentation means for providing the user with the analysis results and suggestions generated by the proposal generation means; and real-time feedback generation means for generating real-time feedback to the robot. This makes it possible to immediately propose efficiency improvements in business and manufacturing processes and to achieve real-time improvements.

[0736] "Video recording means" refers to a device or system that records the status of work in real time using a camera or similar device.

[0737] "Data processing means" refers to a device or program that receives recorded video information and has the function of performing preprocessing such as noise reduction and format standardization.

[0738] A "pattern analysis system" is a system that analyzes patterns in business processes based on pre-processed information and extracts data for efficiency improvements.

[0739] A "proposal generation means" is a device or program that automatically detects inefficient business processes based on pattern analysis and generates proposals for improving efficiency.

[0740] "Information presentation means" refers to a device or interface for visually providing the user with the analysis results and improvement suggestions generated by the proposal generation means.

[0741] A "real-time feedback generation means" is a device or system that has the function of providing users with immediate feedback for process improvement and enabling immediate improvement.

[0742] The system for implementing this invention is configured to combine advanced data processing and analysis technologies in order to improve operational efficiency.

[0743] The server collects real-time video footage of the work environment from cameras mounted on factory robots. This video recording system utilizes high-resolution cameras, enabling precise data acquisition. Additionally, there is a data processing system that receives the video information, performing noise reduction and standardizing the data format. Libraries such as OpenCV and TensorFlow are primarily used for this purpose.

[0744] The pattern analysis method learns patterns in business processes based on pre-processed information. This utilizes multimodal generation technology, such as Google Cloud AI. This analysis identifies inefficient business processes, and the data is used to gather information for suggesting improvements.

[0745] The proposal generation mechanism generates specific proposals for automation and efficiency improvements based on information obtained from pattern analysis. This allows for the identification of specific methods for improving inefficient parts and enables the generation of improvement measures in real time.

[0746] The information presentation system plays the role of presenting the generated analysis results and suggestions to the user. This provides operators and users with visual improvement suggestions in real time via a head-mounted display. Therefore, users can immediately review the presented suggestions and quickly implement improvements to their work.

[0747] Furthermore, the real-time feedback generation system provides immediate feedback for improvements during operations, supporting immediate process adjustments. This enables continuous and dynamic optimization of the factory line.

[0748] A concrete example is when AI detects wasted time during assembly line work and suggests, "By shortening the movement of your right hand by 3 seconds, the next process will proceed more smoothly." This suggestion allows the operator to immediately correct their movements, improving overall production efficiency.

[0749] Examples of prompts for the generating AI model include "optimize assembly actions," "detect and suggest actions that can be improved," and "present streamlined assembly procedures." These prompts are used to quickly generate specific and feasible improvement suggestions.

[0750] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0751] Step 1:

[0752] The server uses cameras installed on factory robots to acquire real-time video footage of the work processes. The input is raw data from the cameras, and the output is a video stream. This video is recorded in high resolution frame by frame to accurately capture changes in the work environment.

[0753] Step 2:

[0754] The server performs preprocessing on the acquired video stream to remove noise and standardize the format. The input is the video stream generated in step 1, and the output is the preprocessed, clear video data. Specifically, filtering is performed using the OpenCV library and data standardization is performed using TensorFlow.

[0755] Step 3:

[0756] The server analyzes patterns in business processes using pre-processed video data. The input is pre-processed video data, and the output is analytical information to identify inefficiencies. Google Cloud AI is used to identify actions in the video and perform pattern learning by comparing them with known processes.

[0757] Step 4:

[0758] The server generates automation targets and efficiency improvement proposals based on the analysis information. The input is the analysis information from step 3, and the output is an efficiency improvement proposal document. A generative AI model is used to generate the proposals, producing specific improvement measures in written form.

[0759] Step 5:

[0760] The server outputs the generated efficiency suggestions to the user via a head-mounted display. The input is the efficiency suggestion document, and the output is a display of the suggestions as visual feedback to the user. The user can then immediately adjust their work based on these suggestions.

[0761] Step 6:

[0762] The server generates real-time feedback during operations to support process improvement. Inputs are the current operational status and suggestions, while outputs are refined operational processes. This process includes guidance to make it easier for users to perform new actions.

[0763] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0764] This invention relates to a system that, in addition to streamlining business processes, provides more accurate improvement suggestions by recognizing user emotions. This system consists of recording means, receiving processing means, analysis means, suggestion generation means, presentation means, and an emotion engine.

[0765] Explanation of program processing

[0766] The terminal records video and audio from the work site in real time. The recorded data is transmitted to a server via the network. At the same time, the user's facial expressions and voice tone are also recorded and used for emotion analysis.

[0767] The server preprocesses the received data and analyzes the video and audio data. This includes analyzing the operation of business processes, as well as recognizing the user's emotional state using an emotion engine. The emotion engine extracts emotional data using facial expression analysis and voice analysis.

[0768] The server combines sentiment data with business process analysis results to identify areas that need improvement. The suggestion generation system identifies inefficient processes in business operations and creates acceptable improvement suggestions by taking user sentiment into consideration in the solutions.

[0769] The server automatically generates a detailed report outlining the analysis results and recommendations. A video replay is also created, including personalized feedback and suggestions based on emotions.

[0770] Users can view reports and video replays provided by the server through their terminals. This allows for improvement activities that not only streamline operations but also take into account the emotional state of employees.

[0771] Specific example

[0772] Example 1: Improving Call Center Operations

[0773] The terminal records the operator's conversation and facial expressions, and sends the data to the server.

[0774] The server analyzes emotional states such as stress levels during calls and suggests script revisions or training opportunities.

[0775] After receiving the suggestions, the user will implement improvements to operational methods and manage operator stress.

[0776] Example 2: Understanding and improving the flow of meetings

[0777] The terminal records the facial expressions and statements of meeting participants and sends that data to the server.

[0778] The server emotionally analyzes changes in participants' interest and the intensity of discussion during the meeting, and makes suggestions for refining the agenda and improving the meeting's progress.

[0779] Based on the feedback, users will implement better discussion strategies in future meetings.

[0780] By combining this with an emotional engine, it becomes possible to improve operational efficiency while simultaneously providing suggestions that take employees' emotions into consideration.

[0781] The following describes the processing flow.

[0782] Step 1:

[0783] The device begins recording video and audio of work activities using cameras and microphones installed at the work site. This includes employee facial expressions and tone of voice, providing data to recognize the user's emotions. Once a certain amount of data has been collected, it is transmitted to a server via the network.

[0784] Step 2:

[0785] The server performs initial processing on the received video and audio data. This initial processing includes trimming the video, denoising the audio, and standardizing the data format. This converts the data into a format suitable for analysis.

[0786] Step 3:

[0787] The server uses an emotion engine to recognize the user's emotional state from the processed data. Specifically, a facial recognition algorithm analyzes the video data, and an audio analysis tool extracts emotions from the tone and pitch of the voice.

[0788] Step 4:

[0789] The server learns patterns in business processes using analytical tools. This involves analyzing identified business actions and process flows to detect inefficiencies and areas that need improvement.

[0790] Step 5:

[0791] The server generates improvement suggestions by combining emotional states and business process analysis results. The suggestion generation mechanism formulates specific action plans for efficiency improvements and customizes the suggestions to be user-friendly, taking emotional data into consideration.

[0792] Step 6:

[0793] The server automatically generates a report of the improvement suggestions that have been created. Furthermore, a video replay visually illustrating the identified areas for improvement is also created, and this information is provided in a format that is easy for users to understand.

[0794] Step 7:

[0795] Users review reports and video replays sent from the server via their terminals. Based on the information provided, users plan improvement activities in their actual work and optimize operations in a way that is sensitive to emotions.

[0796] (Example 2)

[0797] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0798] Traditional business process improvement systems focused on streamlining operations but failed to consider employees' emotional states when making improvement suggestions. As a result, suggestions were sometimes ineffective for employees, and true business improvement was not achieved. Furthermore, integrated analysis of multimodal data and real-time feedback were not adequately implemented, resulting in a lack of constructive feedback based on user emotions.

[0799] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0800] In this invention, the server includes a device for collecting video footage of work processes, a processing device for receiving the video and audio data collected by the device and organizing the information, and an analysis device for analyzing the data organized by the processing device and identifying the workflow. This makes it possible to provide improvement suggestions that take into account the user's emotions by utilizing emotion recognition technology and multimodal data analysis technology.

[0801] "Work process video" refers to data that visually records the work procedures and actions performed during the execution of work.

[0802] The term "device" refers to equipment or systems used to achieve a specific purpose.

[0803] "Video and audio data" refers to visual and auditory information recorded and stored in digital format.

[0804] A "processing device" is a system of hardware or software used to receive, organize, and analyze data.

[0805] "Organizing information" means processing, filtering, and classifying acquired data according to its purpose.

[0806] An "analysis device" is a device or program used to verify processed data and identify specific patterns or trends.

[0807] "Identifying the workflow" means compiling and evaluating the progress and procedures of a business process.

[0808] An "emotion recognition device" is a device or system that incorporates technology to identify emotions from a user's facial expressions and voice.

[0809] A "proposal generation device" is a system that has the function of generating problem-solving methods and improvement proposals based on analyzed data.

[0810] An "output device" is a device or program that provides information or suggestions to a user visually or audibly.

[0811] This system aims to improve operational efficiency and is specifically designed to suggest improvement measures that take into account the emotional state of employees. Specifically, it is structured around three roles: terminal, server, and user.

[0812] The terminals are deployed at work sites and function as devices for acquiring video and audio of work activities. This utilizes high-resolution cameras and high-sensitivity microphones. The terminals record this data in real time and transmit it to a server via the network. This data also includes emotional elements such as the user's facial expressions and tone of voice, which are used for emotion recognition.

[0813] The server receives video and audio data from the terminal and performs preprocessing such as noise filtering and data formatting. Next, using the processed data, the analysis device analyzes the business process in detail and uses an emotion recognition device to determine the user's emotional state. Here, a generative AI model is used to identify inefficiencies in the business by comprehensively analyzing multiple data sources and to create improvement suggestions that take emotions into consideration.

[0814] Users receive and review reports and video replays provided by the server via their terminals. Based on this information, they can implement emotion-driven improvement measures and achieve continuous business improvement.

[0815] As a concrete example, in a call center, a terminal records the operator's conversation and facial expressions, while a server analyzes their stress level. Then, using a generative AI model, it presents improvement measures using a prompt message such as, "How should we analyze the operator's stress level and provide specific improvement suggestions?" In this way, the system enables the user to receive optimal improvement suggestions.

[0816] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0817] Step 1:

[0818] The terminal uses cameras and microphones installed at the work site to record video and audio of work activities in real time. Input consists of ambient sounds and video from the work site, which are processed as digital data. Output is the captured video and audio data. This data includes the user's facial expressions and voice tone, and is used for emotion analysis.

[0819] Step 2:

[0820] The terminal transmits recorded digital data to the server via the network. The input consists of video and audio digital data recorded by the terminal. The data is encrypted and transmitted securely. The output is the data stream delivered to the server.

[0821] Step 3:

[0822] The server acquires video and audio data received from the terminal. The input is a data stream transmitted over the network, which the server receives. Initial processing includes noise filtering and time synchronization to organize the data. The output is the pre-processed audio and video data.

[0823] Step 4:

[0824] The server analyzes pre-processed data and evaluates the workflow. The input is pre-processed video and audio data. The server identifies the business process and uses data analysis algorithms to pinpoint inefficient steps. The output is the analysis results highlighting the inefficient parts.

[0825] Step 5:

[0826] The server uses an emotion recognition device to analyze the user's emotional state. The input is pre-processed video and audio data, and the emotion engine extracts emotional data by performing facial expression and voice analysis. The output is analyzed data that includes the user's emotional state.

[0827] Step 6:

[0828] The server utilizes a generative AI model to integrate sentiment data and business process analysis results to generate improvement suggestions for increased efficiency. The input consists of sentiment data and analysis results. The generative AI model uses prompts to create specific improvement suggestions. The output is a detailed report containing the improvement suggestions.

[0829] Step 7:

[0830] Users receive and review reports and video replays provided by the server via their terminals. The input consists of reports and video replays from the server. Based on this, users implement business improvement measures. The output is sustainable process improvement that prioritizes operational efficiency and employee satisfaction.

[0831] (Application Example 2)

[0832] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0833] Traditional business efficiency systems focus on identifying inefficiencies and suggesting improvements, but they often fail to consider user emotions, potentially leading to reduced acceptance of suggestions. Furthermore, they haven't been able to achieve efficient task management that considers user emotions in home and office environments. Therefore, there's a need for a system that analyzes user emotions and individually optimizes suggestions.

[0834] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0835] In this invention, the server includes an acquisition means for recording work video, a reception control means for receiving and pre-processing the recorded data, an analysis control means for analyzing the pre-processed information and learning patterns of business processes, an emotion analysis means for analyzing the emotional state of the user, and an improvement means for optimizing the proposed content based on the emotion analysis results. This makes it possible to propose highly acceptable improvements that take into account the user's emotions in order to improve business efficiency.

[0836] "Means of acquisition" refers to devices or functions for recording video data of work or actions.

[0837] "Receiving control means" refers to a device or function for receiving recorded data and performing necessary preprocessing.

[0838] "Analysis and control means" refers to devices or functions that analyze pre-processed information and learn patterns of business processes and behaviors.

[0839] A "proposal generation means" refers to a device or function that identifies inefficient actions based on analysis results and generates improvement proposals.

[0840] "Presentation control means" refers to devices or functions for providing generated suggestions to users.

[0841] "Emotional analysis tools" refer to devices or functions used to analyze and understand a user's emotional state.

[0842] "Improvement measures" refer to devices or functions that individually optimize the content of suggestions based on the results of sentiment analysis.

[0843] To implement this invention, the system is composed of a combination of various hardware and software. The server, terminals, and users work together to perform data processing for efficient business improvement.

[0844] The server acquires video and audio data in real time from terminals equipped with cameras and audio sensors for recording work and daily activities. The data obtained by this acquisition means is transmitted to the server by a receiving control means and preprocessed. This process involves removing unwanted noise and converting the data format.

[0845] The server uses analysis control mechanisms to learn business processes and behavioral patterns in order to analyze pre-processed information. Similarly, emotion analysis mechanisms are used to detect emotional states from the user's facial expressions and tone of voice, and to store this data. For analysis, emotion analysis APIs such as Amazon Rekognition and Google Cloud Vision are utilized.

[0846] The data obtained through analysis is transformed into personalized improvement suggestions for each user by a suggestion generation system. For example, when a household robot determines the need for cleaning, it refers to emotion analysis data and suggests the optimal action at a time when the user is relaxed.

[0847] The terminal uses presentation control means to present generated suggestions and feedback to the user in real time. This feedback is optimized based on the user's emotional state, allowing the user to take appropriate action.

[0848] Specific examples include analyzing a user's emotional state while they are watching television to reduce stress and then suggesting the most appropriate household chores. An example of a prompt for the generative AI model would be, "Analyze the user's current facial expression and tone of voice to generate appropriate household chore suggestions."

[0849] This system will enable highly acceptable suggestions that take into account the user's emotional state in both work and daily life, leading to improved work efficiency and a more comfortable experience.

[0850] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0851] Step 1:

[0852] The device uses a camera and audio sensor to capture work and daily activities, acquiring video and audio data in real time. This data serves as input, which is then prepared for conversion into an appropriate format after removing noise and unwanted information. The output is appropriate video and audio data for pre-processing.

[0853] Step 2:

[0854] The terminal transmits the acquired data to the server. The server receives this data using a receiving control means and performs preprocessing such as noise reduction and format conversion. The input is the transmitted video and audio data, and the output is data in an analyzable format.

[0855] Step 3:

[0856] The server analyzes pre-processed data using analysis and control means to learn business processes and behavioral patterns. The input for this step is pre-processed data, and the output is learned pattern information. Specifically, data analysis is performed using machine learning algorithms.

[0857] Step 4:

[0858] The server performs data analysis to determine the user's emotional state using emotion analysis tools. Pre-processed data is treated as input, and data indicating the user's emotional state is output. Specifically, it utilizes facial expression analysis and voice analysis technologies to extract emotional data.

[0859] Step 5:

[0860] The server generates user-specific improvement suggestions based on analysis results and emotional state data using a suggestion generation mechanism. The input is the analysis results from steps 3 and 4, and the output is user-optimized improvement suggestions. The suggestions are generated using an algorithm that takes emotional state into consideration.

[0861] Step 6:

[0862] The server sends suggestions generated using the suggestion control means to the terminal and notifies the user. The input is the suggestions from the suggestion generation means, and the output is the content presented to the user terminal. The user can receive real-time feedback through the terminal and use it to improve their life and work.

[0863] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0864] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0865] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0866] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0867] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0868] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0869] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0870] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0871] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0872] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0873] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0874] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0875] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0876] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0877] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0878] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0879] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0880] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0881] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0882] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0883] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0884] The following is further disclosed regarding the embodiments described above.

[0885] (Claim 1)

[0886] A recording device for recording video of work processes,

[0887] A receiving processing means that receives video data recorded by the recording means and performs preprocessing,

[0888] An analysis means for analyzing pre-processed data by the aforementioned receiving processing means and learning patterns of business processes,

[0889] A proposal generation means that identifies inefficient processes detected by the analysis means and generates proposals for efficiency improvements,

[0890] A presentation means that provides the user with the analysis results and proposals generated by the proposal generation means,

[0891] A system that includes this.

[0892] (Claim 2)

[0893] The system according to claim 1, wherein the analysis means integrates and analyzes video, audio, and text data using multimodal generation technology.

[0894] (Claim 3)

[0895] The system according to claim 1, wherein the presentation means generates and provides to the user real time feedback, proposals, and video replays.

[0896] "Example 1"

[0897] (Claim 1)

[0898] A device for recording the status of work,

[0899] Receiving means for receiving data recorded by the aforementioned device and performing data preprocessing,

[0900] A means for analyzing pre-processed data received by the aforementioned receiving means and learning patterns in the workflow,

[0901] A means for identifying inefficient procedures detected by the aforementioned analysis means and generating suggestions for improvement,

[0902] Means for providing the user with the results and proposals generated by the proposed means,

[0903] A system that includes this.

[0904] (Claim 2)

[0905] The system according to claim 1, wherein the analysis means comprehensively analyzes data using a variety of data formats.

[0906] (Claim 3)

[0907] The system according to claim 1, wherein the providing means generates and provides real-time feedback and suggestions to the user.

[0908] "Application Example 1"

[0909] (Claim 1)

[0910] A video recording device for recording work footage,

[0911] A data processing means that receives video information recorded by the aforementioned video recording means and performs preprocessing,

[0912] A pattern analysis means analyzes the pre-processed information by the aforementioned data processing means and learns patterns in the business process,

[0913] A proposal generation means that identifies inefficient processes detected by the pattern analysis means and generates proposals for efficiency improvements,

[0914] Information presentation means for providing users with analysis results and proposals generated by the proposal generation means,

[0915] A real-time feedback generation means for generating real-time feedback to a robot,

[0916] A system that includes this.

[0917] (Claim 2)

[0918] The system according to claim 1, wherein the pattern analysis means integrates and analyzes video, audio, and text information using multimodal generation technology, and further provides visual improvement suggestions to the robot operator.

[0919] (Claim 3)

[0920] The system according to claim 1, wherein the information presentation means generates and provides to the user real-time feedback, proposals, and specific improvement measures.

[0921] "Example 2 of combining an emotion engine"

[0922] (Claim 1)

[0923] A device for collecting video footage of work processes,

[0924] A processing device that receives video and audio data collected by the aforementioned device and organizes the information,

[0925] An analysis device that analyzes the data organized by the aforementioned processing device and identifies the workflow,

[0926] A proposal generation device that identifies inefficient procedures detected by the aforementioned analysis device and generates proposals for improving efficiency,

[0927] An output device that provides the generated proposal and analysis results,

[0928] An emotion recognition device that identifies the user's emotional state,

[0929] An emotion response suggestion device that generates improvement suggestions tailored to the user based on the results of the emotion recognition device,

[0930] A system that includes this.

[0931] (Claim 2)

[0932] The system according to claim 1, wherein the analysis device uses a technique for integrating and analyzing multiple types of data.

[0933] (Claim 3)

[0934] The system according to claim 1, wherein the output device generates and provides to the user real-time feedback and a reproduction of proposals and visual data.

[0935] "Application example 2 when combining with an emotional engine"

[0936] (Claim 1)

[0937] A means of acquiring video footage of work processes,

[0938] A receiving control means that receives data recorded by the acquisition means and performs preprocessing,

[0939] An analysis control means analyzes the pre-processed information by the receiving control means and learns patterns of business processes,

[0940] The aforementioned analysis and control means identifies inefficient behaviors and generates suggestions for improving efficiency,

[0941] Presentation control means for providing the user with the analysis results and proposals generated by the proposal generation means,

[0942] An emotion analysis tool for analyzing the emotional state of users,

[0943] An improvement means for optimizing the proposed content based on the emotional analysis results from the aforementioned emotional analysis means,

[0944] A system that includes this.

[0945] (Claim 2)

[0946] The system according to claim 1, wherein the analysis control means integrates and analyzes video, audio, and text data using multimodal generation technology and takes emotion analysis results into consideration.

[0947] (Claim 3)

[0948] The system according to claim 1, wherein the presentation control means generates and provides to the user a proposal document or video replay based on real-time feedback and emotional state. [Explanation of symbols]

[0949] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A recording device for recording video of work processes, A receiving processing means that receives video data recorded by the recording means and performs preprocessing, An analysis means for analyzing pre-processed data by the aforementioned receiving processing means and learning patterns of business processes, A proposal generation means that identifies inefficient processes detected by the analysis means and generates proposals for efficiency improvements, A presentation means that provides the user with the analysis results and proposals generated by the proposal generation means, A system that includes this.

2. The system according to claim 1, wherein the analysis means integrates and analyzes video, audio, and text data using multimodal generation technology.

3. The system according to claim 1, wherein the presentation means generates and provides to the user real time feedback, proposals, and video replays.

Images