system

Abstract

We provide the system. [Solution] A device for acquiring visual data, A means of identifying equipment abnormalities by analyzing acquired visual data, A means for generating repair procedures based on analysis results, A means for presenting the generated repair procedure to the worker via a display device, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to quickly and accurately repair defects in facilities in social infrastructure such as mobile base stations, there is a need to provide a repair support system that allows even workers without advanced specialized knowledge to efficiently respond. Conventional methods have problems in that insufficient skills and differences in experience among technicians affect the quality and speed of repairs, leading to prolonged defects.

Means for Solving the Problems

[0005] This invention provides a system that directly captures on-site conditions using a device that acquires visual data, analyzes that data to identify equipment malfunctions, generates an optimal repair procedure based on the analysis results, and visually presents that procedure on a display device worn by the worker. This system has the ability to receive additional data in real time and update the repair procedure, and can also analyze audio data to identify abnormal sounds. As a result, workers can efficiently perform repairs by following the drawn guidelines.

[0006] "Visual data" refers to visual information such as images and videos acquired using cameras and sensors.

[0007] "Device" refers to equipment or machinery used to perform a specific purpose.

[0008] "Analysis" is the process of analyzing acquired data to derive specific information or patterns.

[0009] "Equipment malfunction" refers to a state in which equipment deviates from normal operation or condition, resulting in a failure or malfunction.

[0010] A "repair procedure" is a guideline or instruction manual that describes the necessary work and sequence to resolve a malfunction.

[0011] A "display device" is a device that provides visual information to a user, and includes displays and the like.

[0012] A "system" refers to a combination of multiple elements or components designed to achieve a specific function.

[0013] A "worker" refers to an individual trained to perform the repair and maintenance of equipment.

[0014] "Real-time" refers to a state where data processing and information provision occur instantly without delay.

[0015] "Voice data" refers to what records or relays voices, and is mainly the sound signal acquired through a microphone or the like.

Brief Description of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiment for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] As an embodiment of the present invention, the behavior of a system that acquires visual data, identifies and analyzes equipment abnormalities at the site, and presents appropriate repair procedures to the user will be specifically described below.

[0038] First, this system uses an AR device to collect visual data. As a terminal, this device captures the on-site situation with a camera and acquires supplementary data with sensors. This ensures that the data necessary for the system is available in real time. The visual data includes video information that reflects the appearance and condition of the equipment.

[0039] Next, this data is securely transmitted to a server. The server analyzes the received data, using computer vision and voice analysis technologies to identify faulty areas and diagnose the cause of the anomaly. Based on this analysis, an AI agent generates the optimal repair procedure. The repair procedure includes specific instructions on which parts to handle and how, and is presented in a format that is easy for the worker to understand.

[0040] The server returns the repair procedure to the terminal, which then visually provides the repair guide to the user's AR device. For example, the AR device's display might highlight the damaged fan bolts in green and show the order in which to remove them. It also provides instructions on precautions to take while handling the parts and the tools to use.

[0041] Users can proceed with the repair by following the displayed visual guidelines. As the repair process progresses, instructions based on newly analyzed data from the server are updated via the terminal as needed. In this way, repairs can be completed efficiently and accurately, regardless of the user's technical level.

[0042] As a concrete example of this system, consider a case where an abnormal noise is occurring in the equipment. In this case, the terminal acquires audio data, and the server analyzes it to identify the source of the abnormal noise. For example, if it is diagnosed that the cause is a malfunction in the shaft of a rotating fan, the system generates a corresponding repair procedure and presents the specific workflow to the user through the terminal.

[0043] As described above, the system of the present invention provides users with an environment in which they can efficiently repair equipment malfunctions on-site.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The terminal uses an AR device to acquire visual and audio data from the site. It utilizes cameras and microphones to capture detailed information about the base station's structure and status.

[0047] Step 2:

[0048] The terminal compresses the acquired visual and audio data into a predetermined format and sends it to the server using a secure communication protocol.

[0049] Step 3:

[0050] The server begins analyzing the transmitted data. Using computer vision technology, it detects anomalies in the video data and identifies abnormal sounds in the audio data.

[0051] Step 4:

[0052] The server uses an AI agent to generate the optimal repair procedure based on the analysis results. The procedure includes information such as the order in which parts should be removed and installed, and recommended tools.

[0053] Step 5:

[0054] The server sends the generated repair instructions to the device. The data is converted into a format optimized for display on AR devices.

[0055] Step 6:

[0056] The device visually presents the repair procedure to the user. The AR device screen uses highlights and animations to display the location and handling procedures for important parts.

[0057] Step 7:

[0058] The user performs repair work by following the instructions of the AR device. Based on the displayed information, they use tools to replace or adjust parts.

[0059] Step 8:

[0060] The terminal sends newly acquired data to the server in real time during the operation, and instructions are updated as needed.

[0061] Step 9:

[0062] The user performs a final check of the repair and verifies the proper functioning of the equipment using an AR device. The server records the verification results and stores a log of the entire process.

[0063] (Example 1)

[0064] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0065] In recent years, there has been a growing need to quickly and efficiently identify abnormalities in increasingly complex equipment and provide appropriate repair procedures. However, conventional technologies often suffer from insufficient analysis of visual data and supplementary information, leading to delays in diagnosing malfunctions and increasing the risk of incorrect repairs due to erroneous judgments by operators. An information system capable of solving these problems is necessary.

[0066] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0067] In this invention, the server includes an information acquisition device that acquires visual data and additional information, an analysis means that analyzes the acquired information to identify equipment abnormalities, and a means that derives repair procedures based on the analysis results. This enables accurate diagnosis of abnormalities and efficient provision of repair procedures.

[0068] "Visual data" refers to video information and related information that shows the appearance and condition of equipment.

[0069] "Additional information" refers to information about the operating environment of the equipment, acquired along with visual data, and includes temperature, vibration, and other factors.

[0070] An "information acquisition device" is a device for collecting visual data and additional information, and includes devices such as cameras and various sensors.

[0071] "Analysis means" refers to methods for processing acquired data and utilizing various technologies to identify equipment malfunctions.

[0072] A "repair procedure" is a set of steps that outlines the specific work process for repairing a malfunctioning part of the equipment.

[0073] A "presentation device" is a device used to provide repair procedures or other information to users, and includes devices such as displays and AR devices.

[0074] This invention is an information system that combines visual data and additional information to enable precise equipment diagnosis and repair guidance. The following hardware and software will be used to implement the system.

[0075] The system uses an AR (Augmented Reality) device as the terminal, acquiring visual data and additional information through its camera and various sensors. This data includes the device's appearance, status, temperature, vibration, etc. The AR device is used to visually present this information to the user.

[0076] The server performs analysis based on the received data. Here, "OpenCV" is used for computer vision technology, "Librosa" for sound wave data analysis, and "TENSORFLOW®" and "PyTorch" are used as deep learning models to perform highly accurate anomaly identification. This identifies the location of the equipment malfunction and analyzes its cause.

[0077] Based on the analysis results, the server uses a generated AI model to create repair procedures. The procedure generation is customized according to the user's technical level, providing instructions that are easy for non-technical users to understand.

[0078] As a concrete example, consider a situation where abnormal noises are emitted from rotating machinery. In this case, the terminal collects sound wave data, and through analysis, it can be identified that, for example, a malfunction in the rotating shaft is the cause. In that case, an appropriate repair procedure is presented to the user along with a visual guide.

[0079] An example of a prompt message is: "Use the visual and sensor data captured by the AR device to identify the location and cause of the equipment malfunction, and generate a repair procedure."

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The terminal uses an AR device to acquire visual data and additional information. Inputs include video from a camera and environmental information (temperature, vibration, etc.) from sensors. This data is collected in real time and processed to accurately reflect the equipment status at the site. The output is a dataset that can be analyzed by the system.

[0083] Step 2:

[0084] The terminal securely transmits the acquired dataset to the server. This process uses encryption technology (e.g., TLS / SSL) to ensure data security. Input includes visual data and additional information, and output is the storage of this information in a database.

[0085] Step 3:

[0086] The server performs preprocessing on the received data for analysis. The input is the dataset obtained in step 2. The data attributes are standardized and formatted for analysis. The output is data in an analyzable format.

[0087] Step 4:

[0088] The server analyzes data using computer vision and acoustic analysis techniques. It analyzes video data using "OpenCV" and acoustic data using "Librosa". Pre-processed data is the input, and the output identifies the location and cause of equipment malfunctions.

[0089] Step 5:

[0090] The server generates repair procedures using an AI model based on the analysis results. The input includes information on the location and cause of the malfunction, which is used to subdivide the repair steps. The output is a repair manual containing specific operating procedures.

[0091] Step 6:

[0092] The terminal displays repair instructions retrieved from the server on the user's AR device. A generated repair manual is provided as input. Work areas and procedures are visualized using highlighting. As output, an interactive repair guide is presented on the user's device.

[0093] Step 7:

[0094] The user proceeds with the repair work by following the displayed guide. The server updates instructions in real time as needed and provides them to the user via the terminal. Inputs include work progress and new environmental data, while output is the smooth completion of the repair process.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] In industrial machinery maintenance, there is a challenge in immediately identifying problems and providing appropriate repair procedures when abnormalities occur. In particular, real-time information provision is crucial because visibility and accuracy of procedures are required for on-site workers to solve problems quickly and efficiently.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes means for collecting visual information, means for analyzing the acquired visual information to identify equipment abnormalities, and means for generating repair procedures based on the analysis results. This enables on-site workers to visually confirm the location of the abnormality in real time and efficiently implement the optimal repair procedure.

[0100] "Means for collecting visual information" refers to devices that use cameras or sensors to photograph or record the operating status of machinery or equipment, thereby acquiring visual data.

[0101] "Means for analyzing acquired visual information to identify equipment malfunctions" refers to a system that processes collected visual data to identify the location of the malfunction and its cause.

[0102] "A means of generating repair procedures based on analysis results" refers to a process that automatically creates and presents procedures and methods for correcting abnormalities based on the results of identifying the abnormal areas.

[0103] "Means of presenting to the operator through a visual device" refers to devices or interfaces that visually inform the worker of the generated repair procedure.

[0104] "A means of checking the maintenance status of industrial machinery in real time and visually presenting repair guidelines in the event of an abnormality" refers to a system that detects abnormalities during machine operation and immediately communicates appropriate repair procedures to the operator visually.

[0105] The system for implementing this invention mainly consists of a server, a terminal, and a user. The server is responsible for analyzing visual data and generating repair procedures. Specifically, a terminal such as smart glasses or a smartphone first collects visual information of industrial machinery using a camera and sensors (e.g., Microsoft® HoloLens®). This visual data is transmitted to the server in real time.

[0106] On the server, this data is analyzed using computer vision technologies (e.g., OpenCV, TensorFlow) to identify anomalies. Based on the analysis results, a generated AI model creates the optimal repair procedure. This repair procedure is fed back to the user's visual device via the terminal and displayed in an easy-to-understand format. For example, the anomaly and the tools to be used are visually highlighted.

[0107] Users can perform maintenance and repairs on their devices by following the instructions displayed on the smart glasses' screen. If additional data is sent from the device during the repair process, the server will update the instructions accordingly in real time.

[0108] A concrete example is when a malfunction occurs in the joint of a factory robot. The smart glasses scan the joint, the server identifies the faulty part, and generates appropriate repair instructions. The user can then follow the visual instructions through the glasses to perform tasks such as replacing lubricant or adjusting bolts.

[0109] An example of a prompt message is: "There is a problem with the factory robot. Scan the robot with your smart glasses and highlight the problem area. Identify the cause and provide the optimal repair procedure in real time."

[0110] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0111] Step 1:

[0112] The terminal uses a camera and sensors to collect images and data from industrial machinery. The input is real-time visual data, and the output is high-resolution image data and sensor data. This data is sent directly to the server.

[0113] Step 2:

[0114] The server analyzes the visual data received from the terminal. In this step, computer vision technologies such as OpenCV and TensorFlow are used to identify anomalies from the input image data. The output is information indicating the location and type of the anomaly.

[0115] Step 3:

[0116] The server generates repair procedures using an AI model based on the analysis results. The output from step 2 is used as input, and data processing is performed to generate the optimal repair procedure. The output includes information such as repair procedure manuals and specific repair steps.

[0117] Step 4:

[0118] The server generates repair instructions and sends them to the terminal, which then visually presents them to the user. The input is the generated repair instructions, and the output is a repair guide presented through a visual device. Specifically, it highlights the faulty area and displays each step of the procedure sequentially on the screen.

[0119] Step 5:

[0120] The user performs the actual repair work following the repair procedures presented through the device. The input is the repair procedure presented in step 4, and the output is the result of the repair work performed. Specifically, the user replaces parts or makes adjustments using tools.

[0121] Step 6:

[0122] The device collects additional data during repair and sends it to the server. The input is real-time additional data, and the output is updated visual data. The server uses this data to re-evaluate the procedure and update the repair procedure as needed.

[0123] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0124] As an embodiment of the present invention, the behavior of a system equipped with an emotion engine that recognizes and responds to the user's emotional state, in addition to acquiring and analyzing visual and audio data, will be specifically described below.

[0125] First, the device uses an AR device to collect visual and audio data from the site. By capturing detailed images of the environment and equipment with a camera and collecting surrounding sounds with a microphone, it identifies various data, including any anomalies in the equipment. At the same time, an emotion engine analyzes the user's facial expressions and tone of voice, monitoring the user's emotional state in real time.

[0126] The server analyzes the received visual and audio data. This uses AI-powered computer vision and audio analysis algorithms to detect specific anomalies and diagnose their causes. Based on the analysis results, the AI ​​agent generates appropriate repair procedures. Furthermore, if the emotion engine's analysis indicates that the user is experiencing stress, the server simplifies the repair procedures or adjusts them to provide supplementary information to help the user understand.

[0127] Next, the server sends the generated repair instructions to the device in a format that is appropriate to the user's emotional state. The device then displays a visual repair guide through an AR device. This includes highlighting the location of parts and animations that show the sequence of steps. It can also provide voice assistance in a more user-friendly tone depending on the user's emotional state.

[0128] The user proceeds with the repair by following the displayed guidelines. Based on feedback from the emotion engine, the instructions are further adjusted based on the user's emotional changes, reducing stress and improving work efficiency.

[0129] For example, if the equipment malfunction is complex and the user shows signs of confusion, the emotion engine will detect this. The terminal will then break down each repair step by step, along with easy-to-understand instructions resent from the server. This allows the user to easily grasp the task and perform the work calmly and confidently.

[0130] In this way, by integrating an emotion engine, a system can be realized that efficiently supports equipment repair while taking into account the user's emotional state.

[0131] The following describes the processing flow.

[0132] Step 1:

[0133] The device uses an AR device to acquire visual and audio data from the site. It captures video of the equipment in real time with a camera and records ambient sounds and abnormal sounds with a microphone. Furthermore, an emotion engine analyzes the user's facial expressions and voice tone to recognize their emotional state in real time.

[0134] Step 2:

[0135] The device compresses the acquired visual, audio, and emotional state data and sends it to the server. Data transmission is performed using a secure protocol, allowing for rapid progression to the analysis process.

[0136] Step 3:

[0137] The server analyzes the transmitted data. Image recognition technology identifies abnormal areas from the video data, and audio analysis technology determines the cause of abnormal sounds. Based on these analysis results, an AI agent creates a repair procedure.

[0138] Step 4:

[0139] The server uses an emotion engine to assess the user's emotional state. If the user is feeling confused or stressed, the server makes adjustments, such as simplifying the repair procedure or adding supplementary information.

[0140] Step 5:

[0141] The server sends optimized repair instructions to the device. These instructions are designed to be visually easy to understand and tailored to the user's emotional state.

[0142] Step 6:

[0143] The device visually presents repair procedures to the user through an augmented reality (AR) device. It highlights the location of parts, displays work instructions with animations, and provides clear guidance to the user.

[0144] Step 7:

[0145] Users perform repair work by following instructions received through an AR device. They can proceed with the work without anxiety while receiving feedback based on analysis by an emotion engine.

[0146] Step 8:

[0147] The device continuously records additional visual and emotional data obtained during the work and sends it to the server, updating the repair procedure as needed. Optimal information is provided in real time to support the work process.

[0148] (Example 2)

[0149] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0150] In on-site equipment repair, workers need to quickly and accurately identify equipment malfunctions and understand and execute appropriate repair procedures. However, current systems do not fully utilize visual and auditory information, and lack mechanisms to optimize repair procedures by considering the worker's emotional state. As a result, workers may experience stress and confusion, which can hinder the efficiency of repair work.

[0151] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0152] In this invention, the server includes means for analyzing digital video to identify equipment abnormalities, means for analyzing acoustic data to identify abnormal sounds, and means for analyzing the emotional state of the worker. This makes it possible to identify equipment abnormalities early and to present appropriate repair procedures that reflect the emotional state of the worker.

[0153] "Digital images" are visual information acquired by cameras and other imaging devices that represent the device and its surrounding environment.

[0154] "Analysis" is a process performed to identify anomalies or abnormal sounds based on acquired digital video and audio data, and to pinpoint the cause of a problem.

[0155] A "repair procedure" is a set of instructions outlining the specific steps necessary to correct equipment malfunctions, and by following it, workers can perform effective repairs.

[0156] A "visual display device" is a display device used by workers to confirm repair procedures, and includes AR devices and monitors.

[0157] "Acoustic data" refers to sound information collected through audio input devices such as microphones, and is used to detect abnormal sounds.

[0158] "Emotional state" refers to the internal psychological state of a worker, as judged from their facial expressions, tone of voice, etc., and includes stress and confusion.

[0159] "Adjustment" refers to appropriately modifying repair procedures and supplementary information according to the emotional state of the worker.

[0160] In this embodiment, the system uses digital video and audio data to identify equipment malfunctions and implements a process to support workers. Specifically, the terminal collects digital video and audio data in combination with an AR device. This allows for detailed acquisition of visual information of the site using a camera and recording of audio information, including background sounds, using a microphone.

[0161] The server uses advanced AI algorithms to analyze received digital video and detect anomalies. During this process, it also analyzes acoustic data, utilizing voice analysis technology to identify abnormal sounds. In addition, an emotion engine evaluates the worker's facial expressions and voice tone, analyzing their emotional state in real time.

[0162] Based on the analysis, the server uses a generative AI model to generate appropriate repair procedures. These procedures include the repair process for the faulty area and are customized to be easily understood by the worker. Furthermore, the procedures are adjusted and supplementary information is provided, taking into account the worker's emotional state. The emotion engine optimizes the guidance to reduce stress and confusion.

[0163] For example, if a complex equipment malfunction is detected, the terminal receives easy-to-follow instructions from the server and displays them visually on an AR device. This may include highlighting part locations and animations showing the steps. In addition, voice guidance may be provided, with instructions delivered in a gentle tone that adapts to the user's emotional state.

[0164] An example of input to the generating AI model is a prompt such as, "Use data acquired from an AR device to create a repair guide that takes into account the user's emotional state." Through this prompt, the model can generate detailed repair procedures and provide quick and appropriate assistance to the worker.

[0165] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0166] Step 1:

[0167] The terminal uses an AR device to collect digital video and audio data from the site. Inputs include video captured by a camera and audio recorded by a microphone. This data is used to understand the specific situation and equipment status at the site. Specifically, the terminal continuously acquires video and records audio in real time. The collected digital video and audio data are sent to a server as output.

[0168] Step 2:

[0169] The server analyzes the received digital video to identify equipment malfunctions. The input is digital video transmitted from a terminal. It uses computer vision technology to perform data processing to detect abnormal areas in the video. Specifically, the server uses an image recognition algorithm to identify the abnormal points and outputs the results as vector data.

[0170] Step 3:

[0171] In parallel, the server analyzes the acoustic data. The input is acoustic data transmitted from the terminal. Using an audio analysis algorithm, it removes background noise and then performs data calculations to identify abnormal sounds. Specifically, the server performs frequency analysis of the sound and identifies abnormal patterns. The output is the result of detecting abnormal sounds.

[0172] Step 4:

[0173] The server analyzes the worker's emotional state. Input includes facial expression data and voice tone information received from the terminal. The emotion analysis engine uses this data to perform calculations that evaluate the worker's stress level and level of confusion. Specifically, the server uses a machine learning model to output an index that quantifies the emotional state.

[0174] Step 5:

[0175] The server generates repair procedures based on anomaly detection results and emotion evaluations. Inputs include data on the location of the anomaly, information on abnormal sounds, and evaluation results of the emotion state. A generation AI model is used to perform data processing to adaptively create the repair procedures. Specifically, the server inputs prompt statements into the model and outputs customized repair procedures in text format.

[0176] Step 6:

[0177] The terminal presents repair procedures to the worker. The input is the repair procedure transmitted from the server. Using an AR device, it performs specific actions such as highlighting parts and displaying animations illustrating the procedure. It also provides user-friendly voice guidance that adapts to the worker's emotional state. The output is the repair procedure communicated to the worker both visually and audibly.

[0178] (Application Example 2)

[0179] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0180] Conventional equipment maintenance systems often provide uniform maintenance procedures without considering the emotional state of users, which can lead to decreased user understanding and work efficiency. Furthermore, information used to identify anomalies is often reliant on either visual or auditory cues, resulting in insufficient accuracy in anomaly detection. This can lead to stressful situations for workers and hinder the efficiency of equipment maintenance.

[0181] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0182] In this invention, the server includes a medium for acquiring visual information, means for analyzing the acquired visual information to identify abnormalities in the device, means for analyzing audio information to recognize the user's emotions, and means for generating maintenance procedures based on the analysis results. This enables optimal guidance tailored to the user's emotional state.

[0183] "Visual information" refers to image and video data acquired through cameras and sensors.

[0184] A "medium" refers to a device or apparatus used to acquire or transmit information.

[0185] "Analysis" is a method for processing data and understanding its content and meaning.

[0186] The term "device" is a general term for machines or systems designed to perform a specific function.

[0187] "Abnormal" refers to a state that deviates from the normal condition or function.

[0188] "Audio information" refers to sound data collected through audio input devices such as microphones.

[0189] "User" refers to an individual or group that uses the system.

[0190] "Recognizing emotions" means judging a user's emotional state from their facial expressions and tone of voice.

[0191] "Maintenance procedures" are specific work procedures necessary to keep equipment and devices in a normal state.

[0192] "Guidance" refers to instructions or information provided to encourage users to take action or perform tasks.

[0193] In the system implementing this invention, a robot equipped with an AR device is used to efficiently maintain and manage equipment and machinery within a factory. The AR device is equipped with a camera and a microphone, and acquires visual and audio information in real time. This allows for the collection of detailed data to detect abnormalities in the equipment.

[0194] The server receives this data and uses computer vision technology to analyze the visual information and recognize specific anomalies. Computer vision technologies used include OpenCV. Audio information is analyzed using audio analysis software such as Affectiva. Through this audio analysis, the system grasps the user's emotional state in real time.

[0195] Based on the emotion recognition results, the server generates maintenance procedures, adjusts and optimizes them, and sends them to the robot's guidance system. This uses a generative AI model, which leverages prompts to design optimal guidance. Specific guidance is presented to the user as 3D models and visually clear instructions. A user-friendly voice interface also provides intuitive support.

[0196] For example, if an abnormal operation is detected in a machine on the production line, the server analyzes the abnormality with high accuracy and provides step-by-step guidance so that the operator can easily understand it. If the user's emotions are unstable, the AI ​​will provide further supplementary explanations using prompt messages.

[0197] As an example, here is an example of a prompt statement:

[0198] "Design an AI plan to create machine maintenance procedures within a factory that are adjusted based on the emotional state of the workers."

[0199] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0200] Step 1:

[0201] The terminal uses cameras and microphones mounted on robots patrolling the factory to acquire visual and auditory information in real time. The input data consists of surrounding video and audio, and the output is this digital data. The information acquired by the camera and microphone is detailed and wide-ranging.

[0202] Step 2:

[0203] The server receives visual information transmitted from the terminal and begins analysis using computer vision technology (such as OpenCV). The input is video data received from the terminal, and the output is information about the recognized anomalies. Specifically, it detects specific patterns or changes within the image and identifies equipment malfunctions.

[0204] Step 3:

[0205] The server uses speech analysis software (such as Affectiva) to analyze speech information. The input is speech data received from the terminal, and the output is the result of identifying the user's emotional state. It analyzes emotions from speech tone and manner of speaking and performs specific actions to determine whether the user is feeling tense or stressed.

[0206] Step 4:

[0207] The server generates maintenance procedures based on the analysis results and adjusts the procedures according to the user's emotional state. The input is information about the abnormal location and the emotional state, and the output is an optimized maintenance procedure. A generation AI model is used to design supplementary guidance with prompt messages, specifically to make the step order easier to understand and to add additional explanations.

[0208] Step 5:

[0209] The terminal receives optimized maintenance procedures generated by the server and presents them visually to the worker via an AR display. The input is the optimized maintenance procedure, and the output is a guide that the user can understand visually and audibly. Specifically, it uses 3D models and visual highlighting to show equipment repair procedures in animation.

[0210] Step 6:

[0211] The user performs the actual maintenance work by referring to the provided guide. The input is the guide from the terminal, and the final output is the equipment maintained in a normal state. If the user requires further assistance during the work, the terminal will request additional advice or further instructions as needed.

[0212] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0213] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0214] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0215] [Second Embodiment]

[0216] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0217] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0218] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0219] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0220] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0221] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0222] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0223] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0224] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0225] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0226] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0227] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0228] As an embodiment of the present invention, the behavior of a system that acquires visual data, identifies and analyzes equipment abnormalities at the site, and presents appropriate repair procedures to the user will be specifically described below.

[0229] First, this system uses an AR device to collect visual data. As a terminal, this device captures the on-site situation with a camera and acquires supplementary data with sensors. This ensures that the data necessary for the system is available in real time. The visual data includes video information that reflects the appearance and condition of the equipment.

[0230] Next, this data is securely transmitted to a server. The server analyzes the received data, using computer vision and voice analysis technologies to identify faulty areas and diagnose the cause of the anomaly. Based on this analysis, an AI agent generates the optimal repair procedure. The repair procedure includes specific instructions on which parts to handle and how, and is presented in a format that is easy for the worker to understand.

[0231] The server returns the repair procedure to the terminal, which then visually provides the repair guide to the user's AR device. For example, the AR device's display might highlight the damaged fan bolts in green and show the order in which to remove them. It also provides instructions on precautions to take while handling the parts and the tools to use.

[0232] Users can proceed with the repair by following the displayed visual guidelines. As the repair process progresses, instructions based on newly analyzed data from the server are updated via the terminal as needed. In this way, repairs can be completed efficiently and accurately, regardless of the user's technical level.

[0233] As a concrete example of this system, consider a case where an abnormal noise is occurring in the equipment. In this case, the terminal acquires audio data, and the server analyzes it to identify the source of the abnormal noise. For example, if it is diagnosed that the cause is a malfunction in the shaft of a rotating fan, the system generates a corresponding repair procedure and presents the specific workflow to the user through the terminal.

[0234] As described above, the system of the present invention provides users with an environment in which they can efficiently repair equipment malfunctions on-site.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] The terminal uses an AR device to acquire visual and audio data from the site. It utilizes cameras and microphones to capture detailed information about the base station's structure and status.

[0238] Step 2:

[0239] The terminal compresses the acquired visual and audio data into a predetermined format and sends it to the server using a secure communication protocol.

[0240] Step 3:

[0241] The server begins analyzing the transmitted data. Using computer vision technology, it detects anomalies in the video data and identifies abnormal sounds in the audio data.

[0242] Step 4:

[0243] The server uses an AI agent to generate the optimal repair procedure based on the analysis results. The procedure includes information such as the order in which parts should be removed and installed, and recommended tools.

[0244] Step 5:

[0245] The server sends the generated repair instructions to the device. The data is converted into a format optimized for display on AR devices.

[0246] Step 6:

[0247] The device visually presents the repair procedure to the user. The AR device screen uses highlights and animations to display the location and handling procedures for important parts.

[0248] Step 7:

[0249] The user performs repair work by following the instructions of the AR device. Based on the displayed information, they use tools to replace or adjust parts.

[0250] Step 8:

[0251] The terminal sends newly acquired data to the server in real time during the operation, and instructions are updated as needed.

[0252] Step 9:

[0253] The user performs a final check of the repair and verifies the proper functioning of the equipment using an AR device. The server records the verification results and stores a log of the entire process.

[0254] (Example 1)

[0255] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0256] In recent years, there has been a growing need to quickly and efficiently identify abnormalities in increasingly complex equipment and provide appropriate repair procedures. However, conventional technologies often suffer from insufficient analysis of visual data and supplementary information, leading to delays in diagnosing malfunctions and increasing the risk of incorrect repairs due to erroneous judgments by operators. An information system capable of solving these problems is necessary.

[0257] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0258] In this invention, the server includes an information acquisition device that acquires visual data and additional information, an analysis means that analyzes the acquired information to identify equipment abnormalities, and a means that derives repair procedures based on the analysis results. This enables accurate diagnosis of abnormalities and efficient provision of repair procedures.

[0259] "Visual data" refers to video information and related information that shows the appearance and condition of equipment.

[0260] "Additional information" refers to information about the operating environment of the equipment, acquired along with visual data, and includes temperature, vibration, and other factors.

[0261] An "information acquisition device" is a device for collecting visual data and additional information, and includes devices such as cameras and various sensors.

[0262] "Analysis means" refers to methods for processing acquired data and utilizing various technologies to identify equipment malfunctions.

[0263] A "repair procedure" is a set of steps that outlines the specific work process for repairing a malfunctioning part of the equipment.

[0264] A "presentation device" is a device used to provide repair procedures or other information to users, and includes devices such as displays and AR devices.

[0265] This invention is an information system that combines visual data and additional information to enable precise equipment diagnosis and repair guidance. The following hardware and software will be used to implement the system.

[0266] The system uses an AR (Augmented Reality) device as the terminal, acquiring visual data and additional information through its camera and various sensors. This data includes the device's appearance, status, temperature, vibration, etc. The AR device is used to visually present this information to the user.

[0267] The server performs analysis based on the received data. Here, it utilizes "OpenCV" for computer vision technology, "Librosa" for sound wave data analysis, and "TensorFlow" and "PyTorch" as deep learning models to perform highly accurate anomaly identification. This allows it to identify the location of equipment malfunctions and analyze their causes.

[0268] Based on the analysis results, the server uses a generated AI model to create repair procedures. The procedure generation is customized according to the user's technical level, providing instructions that are easy for non-technical users to understand.

[0269] As a concrete example, consider a situation where abnormal noises are emitted from rotating machinery. In this case, the terminal collects sound wave data, and through analysis, it can be identified that, for example, a malfunction in the rotating shaft is the cause. In that case, an appropriate repair procedure is presented to the user along with a visual guide.

[0270] An example of a prompt message is: "Use the visual and sensor data captured by the AR device to identify the location and cause of the equipment malfunction, and generate a repair procedure."

[0271] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0272] Step 1:

[0273] The terminal uses an AR device to acquire visual data and additional information. Inputs include video from a camera and environmental information (temperature, vibration, etc.) from sensors. This data is collected in real time and processed to accurately reflect the equipment status at the site. The output is a dataset that can be analyzed by the system.

[0274] Step 2:

[0275] The terminal securely transmits the acquired dataset to the server. This process uses encryption technology (e.g., TLS / SSL) to ensure data security. Input includes visual data and additional information, and output is the storage of this information in a database.

[0276] Step 3:

[0277] The server performs preprocessing to analyze the received data. As input, there is the dataset obtained in step 2. It normalizes the attributes of the data and formats it into a form suitable for analysis. As output, data in an analyzable form is prepared.

[0278] Step 4:

[0279] The server analyzes the data using computer vision and sound wave analysis techniques. It uses "OpenCV" to analyze video data and "Librosa" to analyze sound wave data. As input, there is the preprocessed data, and as output, the abnormal parts and causes of the device are identified.

[0280] Step 5:

[0281] Based on the analysis results, the server utilizes a generative AI model to generate repair procedures. As input, there is information about the abnormal parts and causes, and based on this, the repair steps are refined. As output, a repair manual including specific operation procedures is created.

[0282] Step 6:

[0283] The terminal displays the repair procedures obtained from the server on the user's AR device. As input, the generated repair manual is provided. Using highlight display, the working locations and procedures are visualized. As output, an interactive repair guide is presented on the device worn by the user.

[0284] Step 7:

[0285] The user proceeds with the repair work according to the displayed guide. If necessary, the server updates the instructions in real time and provides them to the user via the terminal. As input, there is the work progress and new environmental data, and as output, the repair process is smoothly completed.

[0286] (Application Example 1)

[0287] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0288] In industrial machinery maintenance, there is a challenge in immediately identifying problems and providing appropriate repair procedures when abnormalities occur. In particular, real-time information provision is crucial because visibility and accuracy of procedures are required for on-site workers to solve problems quickly and efficiently.

[0289] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0290] In this invention, the server includes means for collecting visual information, means for analyzing the acquired visual information to identify equipment abnormalities, and means for generating repair procedures based on the analysis results. This enables on-site workers to visually confirm the location of the abnormality in real time and efficiently implement the optimal repair procedure.

[0291] "Means for collecting visual information" refers to devices that use cameras or sensors to photograph or record the operating status of machinery or equipment, thereby acquiring visual data.

[0292] "Means for analyzing acquired visual information to identify equipment malfunctions" refers to a system that processes collected visual data to identify the location of the malfunction and its cause.

[0293] "A means of generating repair procedures based on analysis results" refers to a process that automatically creates and presents procedures and methods for correcting abnormalities based on the results of identifying the abnormal areas.

[0294] "Means of presenting to the operator through a visual device" refers to devices or interfaces that visually inform the worker of the generated repair procedure.

[0295] "A means of checking the maintenance status of industrial machinery in real time and visually presenting repair guidelines in the event of an abnormality" refers to a system that detects abnormalities during machine operation and immediately communicates appropriate repair procedures to the operator visually.

[0296] The system for implementing this invention mainly consists of a server, terminals, and users. The server is responsible for analyzing visual data and generating repair procedures. Specifically, terminals such as smart glasses or smartphones first collect visual information of industrial machinery using cameras and sensors (e.g., Microsoft HoloLens). This visual data is transmitted to the server in real time.

[0297] On the server, this data is analyzed using computer vision technologies (e.g., OpenCV, TensorFlow) to identify anomalies. Based on the analysis results, a generated AI model creates the optimal repair procedure. This repair procedure is fed back to the user's visual device via the terminal and displayed in an easy-to-understand format. For example, the anomaly and the tools to be used are visually highlighted.

[0298] Users can perform maintenance and repairs on their devices by following the instructions displayed on the smart glasses' screen. If additional data is sent from the device during the repair process, the server will update the instructions accordingly in real time.

[0299] A concrete example is when a malfunction occurs in the joint of a factory robot. The smart glasses scan the joint, the server identifies the faulty part, and generates appropriate repair instructions. The user can then follow the visual instructions through the glasses to perform tasks such as replacing lubricant or adjusting bolts.

[0300] An example of a prompt message is: "There is a problem with the factory robot. Scan the robot with your smart glasses and highlight the problem area. Identify the cause and provide the optimal repair procedure in real time."

[0301] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0302] Step 1:

[0303] The terminal uses the camera and sensors to collect images and data of industrial machines. The input is real-time visual data, and high-resolution image data and sensor data are obtained as outputs. This data is directly sent to the server.

[0304] Step 2:

[0305] The server analyzes the visual data received from the terminal. In this step, computer vision technologies such as OpenCV and TensorFlow are used to identify abnormal locations from the input image data. The output is information indicating the positions and types of abnormal locations.

[0306] Step 3:

[0307] The server generates repair procedures using the AI model generated based on the analysis results. The output of Step 2 is used as the input, and data processing is performed to generate optimal repair procedures. The output is information including repair procedure manuals and specific repair procedures.

[0308] Step 4:

[0309] The server sends the generated repair procedures to the terminal, and the terminal visually presents them to the user. The input is the generated repair procedures, and the output is a repair guide presented through a visual device. Specifically, the abnormal locations are highlighted, and each operation procedure is sequentially displayed on the screen.

[0310] Step 5:

[0311] The user performs the actual repair work following the repair procedures presented through the device. The input is the repair procedure presented in step 4, and the output is the result of the repair work performed. Specifically, the user replaces parts or makes adjustments using tools.

[0312] Step 6:

[0313] The device collects additional data during repair and sends it to the server. The input is real-time additional data, and the output is updated visual data. The server uses this data to re-evaluate the procedure and update the repair procedure as needed.

[0314] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0315] As an embodiment of the present invention, the behavior of a system equipped with an emotion engine that recognizes and responds to the user's emotional state, in addition to acquiring and analyzing visual and audio data, will be specifically described below.

[0316] First, the device uses an AR device to collect visual and audio data from the site. By capturing detailed images of the environment and equipment with a camera and collecting surrounding sounds with a microphone, it identifies various data, including any anomalies in the equipment. At the same time, an emotion engine analyzes the user's facial expressions and tone of voice, monitoring the user's emotional state in real time.

[0317] The server analyzes the received visual and audio data. This uses AI-powered computer vision and audio analysis algorithms to detect specific anomalies and diagnose their causes. Based on the analysis results, the AI ​​agent generates appropriate repair procedures. Furthermore, if the emotion engine's analysis indicates that the user is experiencing stress, the server simplifies the repair procedures or adjusts them to provide supplementary information to help the user understand.

[0318] Next, the server sends the generated repair instructions to the device in a format that is appropriate to the user's emotional state. The device then displays a visual repair guide through an AR device. This includes highlighting the location of parts and animations that show the sequence of steps. It can also provide voice assistance in a more user-friendly tone depending on the user's emotional state.

[0319] The user proceeds with the repair by following the displayed guidelines. Based on feedback from the emotion engine, the instructions are further adjusted based on the user's emotional changes, reducing stress and improving work efficiency.

[0320] For example, if the equipment malfunction is complex and the user shows signs of confusion, the emotion engine will detect this. The terminal will then break down each repair step by step, along with easy-to-understand instructions resent from the server. This allows the user to easily grasp the task and perform the work calmly and confidently.

[0321] In this way, by integrating an emotion engine, a system can be realized that efficiently supports equipment repair while taking into account the user's emotional state.

[0322] The following describes the processing flow.

[0323] Step 1:

[0324] The device uses an AR device to acquire visual and audio data from the site. It captures video of the equipment in real time with a camera and records ambient sounds and abnormal sounds with a microphone. Furthermore, an emotion engine analyzes the user's facial expressions and voice tone to recognize their emotional state in real time.

[0325] Step 2:

[0326] The device compresses the acquired visual, audio, and emotional state data and sends it to the server. Data transmission is performed using a secure protocol, allowing for rapid progression to the analysis process.

[0327] Step 3:

[0328] The server analyzes the transmitted data. Image recognition technology identifies abnormal areas from the video data, and audio analysis technology determines the cause of abnormal sounds. Based on these analysis results, an AI agent creates a repair procedure.

[0329] Step 4:

[0330] The server uses an emotion engine to assess the user's emotional state. If the user is feeling confused or stressed, the server makes adjustments, such as simplifying the repair procedure or adding supplementary information.

[0331] Step 5:

[0332] The server sends optimized repair instructions to the device. These instructions are designed to be visually easy to understand and tailored to the user's emotional state.

[0333] Step 6:

[0334] The device visually presents repair procedures to the user through an augmented reality (AR) device. It highlights the location of parts, displays work instructions with animations, and provides clear guidance to the user.

[0335] Step 7:

[0336] Users perform repair work by following instructions received through an AR device. They can proceed with the work without anxiety while receiving feedback based on analysis by an emotion engine.

[0337] Step 8:

[0338] The device continuously records additional visual and emotional data obtained during the work and sends it to the server, updating the repair procedure as needed. Optimal information is provided in real time to support the work process.

[0339] (Example 2)

[0340] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0341] In on-site equipment repair, workers need to quickly and accurately identify equipment malfunctions and understand and execute appropriate repair procedures. However, current systems do not fully utilize visual and auditory information, and lack mechanisms to optimize repair procedures by considering the worker's emotional state. As a result, workers may experience stress and confusion, which can hinder the efficiency of repair work.

[0342] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0343] In this invention, the server includes means for analyzing digital video to identify equipment abnormalities, means for analyzing acoustic data to identify abnormal sounds, and means for analyzing the emotional state of the worker. This makes it possible to identify equipment abnormalities early and to present appropriate repair procedures that reflect the emotional state of the worker.

[0344] "Digital images" are visual information acquired by cameras and other imaging devices that represent the device and its surrounding environment.

[0345] "Analysis" is a process performed to identify anomalies or abnormal sounds based on acquired digital video and audio data, and to pinpoint the cause of a problem.

[0346] A "repair procedure" is a set of instructions outlining the specific steps necessary to correct equipment malfunctions, and by following it, workers can perform effective repairs.

[0347] A "visual display device" is a display device used by workers to confirm repair procedures, and includes AR devices and monitors.

[0348] "Acoustic data" refers to sound information collected through audio input devices such as microphones, and is used to detect abnormal sounds.

[0349] "Emotional state" refers to the internal psychological state of a worker, as judged from their facial expressions, tone of voice, etc., and includes stress and confusion.

[0350] "Adjustment" refers to appropriately modifying repair procedures and supplementary information according to the emotional state of the worker.

[0351] In this embodiment, the system uses digital video and audio data to identify equipment malfunctions and implements a process to support workers. Specifically, the terminal collects digital video and audio data in combination with an AR device. This allows for detailed acquisition of visual information of the site using a camera and recording of audio information, including background sounds, using a microphone.

[0352] The server uses advanced AI algorithms to analyze received digital video and detect anomalies. During this process, it also analyzes acoustic data, utilizing voice analysis technology to identify abnormal sounds. In addition, an emotion engine evaluates the worker's facial expressions and voice tone, analyzing their emotional state in real time.

[0353] Based on the analysis, the server uses a generative AI model to generate appropriate repair procedures. These procedures include the repair process for the faulty area and are customized to be easily understood by the worker. Furthermore, the procedures are adjusted and supplementary information is provided, taking into account the worker's emotional state. The emotion engine optimizes the guidance to reduce stress and confusion.

[0354] For example, if a complex equipment malfunction is detected, the terminal receives easy-to-follow instructions from the server and displays them visually on an AR device. This may include highlighting part locations and animations showing the steps. In addition, voice guidance may be provided, with instructions delivered in a gentle tone that adapts to the user's emotional state.

[0355] An example of input to the generating AI model is a prompt such as, "Use data acquired from an AR device to create a repair guide that takes into account the user's emotional state." Through this prompt, the model can generate detailed repair procedures and provide quick and appropriate assistance to the worker.

[0356] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0357] Step 1:

[0358] The terminal uses an AR device to collect digital video and audio data from the site. Inputs include video captured by a camera and audio recorded by a microphone. This data is used to understand the specific situation and equipment status at the site. Specifically, the terminal continuously acquires video and records audio in real time. The collected digital video and audio data are sent to a server as output.

[0359] Step 2:

[0360] The server analyzes the received digital video to identify equipment malfunctions. The input is digital video transmitted from a terminal. It uses computer vision technology to perform data processing to detect abnormal areas in the video. Specifically, the server uses an image recognition algorithm to identify the abnormal points and outputs the results as vector data.

[0361] Step 3:

[0362] In parallel, the server analyzes the acoustic data. The input is acoustic data transmitted from the terminal. Using an audio analysis algorithm, it removes background noise and then performs data calculations to identify abnormal sounds. Specifically, the server performs frequency analysis of the sound and identifies abnormal patterns. The output is the result of detecting abnormal sounds.

[0363] Step 4:

[0364] The server analyzes the worker's emotional state. Input includes facial expression data and voice tone information received from the terminal. The emotion analysis engine uses this data to perform calculations that evaluate the worker's stress level and level of confusion. Specifically, the server uses a machine learning model to output an index that quantifies the emotional state.

[0365] Step 5:

[0366] The server generates repair procedures based on anomaly detection results and emotion evaluations. Inputs include data on the location of the anomaly, information on abnormal sounds, and evaluation results of the emotion state. A generation AI model is used to perform data processing to adaptively create the repair procedures. Specifically, the server inputs prompt statements into the model and outputs customized repair procedures in text format.

[0367] Step 6:

[0368] The terminal presents repair procedures to the worker. The input is the repair procedure transmitted from the server. Using an AR device, it performs specific actions such as highlighting parts and displaying animations illustrating the procedure. It also provides user-friendly voice guidance that adapts to the worker's emotional state. The output is the repair procedure communicated to the worker both visually and audibly.

[0369] (Application Example 2)

[0370] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0371] Conventional equipment maintenance systems often provide uniform maintenance procedures without considering the emotional state of users, which can lead to decreased user understanding and work efficiency. Furthermore, information used to identify anomalies is often reliant on either visual or auditory cues, resulting in insufficient accuracy in anomaly detection. This can lead to stressful situations for workers and hinder the efficiency of equipment maintenance.

[0372] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0373] In this invention, the server includes a medium for acquiring visual information, means for analyzing the acquired visual information to identify abnormalities in the device, means for analyzing audio information to recognize the user's emotions, and means for generating maintenance procedures based on the analysis results. This enables optimal guidance tailored to the user's emotional state.

[0374] "Visual information" refers to image and video data acquired through cameras and sensors.

[0375] A "medium" refers to a device or apparatus used to acquire or transmit information.

[0376] "Analysis" is a method for processing data and understanding its content and meaning.

[0377] The term "device" is a general term for machines or systems designed to perform a specific function.

[0378] "Abnormal" refers to a state that deviates from the normal condition or function.

[0379] "Audio information" refers to sound data collected through audio input devices such as microphones.

[0380] "User" refers to an individual or group that uses the system.

[0381] "Recognizing emotions" means judging a user's emotional state from their facial expressions and tone of voice.

[0382] "Maintenance procedures" are specific work procedures necessary to keep equipment and devices in a normal state.

[0383] "Guidance" refers to instructions or information provided to encourage users to take action or perform tasks.

[0384] In the system implementing this invention, a robot equipped with an AR device is used to efficiently maintain and manage equipment and machinery within a factory. The AR device is equipped with a camera and a microphone, and acquires visual and audio information in real time. This allows for the collection of detailed data to detect abnormalities in the equipment.

[0385] The server receives this data and uses computer vision technology to analyze the visual information and recognize specific anomalies. Computer vision technologies used include OpenCV. Audio information is analyzed using audio analysis software such as Affectiva. Through this audio analysis, the system grasps the user's emotional state in real time.

[0386] Based on the emotion recognition results, the server generates maintenance procedures, adjusts and optimizes them, and sends them to the robot's guidance system. This uses a generative AI model, which leverages prompts to design optimal guidance. Specific guidance is presented to the user as 3D models and visually clear instructions. A user-friendly voice interface also provides intuitive support.

[0387] For example, if an abnormal operation is detected in a machine on the production line, the server analyzes the abnormality with high accuracy and provides step-by-step guidance so that the operator can easily understand it. If the user's emotions are unstable, the AI ​​will provide further supplementary explanations using prompt messages.

[0388] As an example, here is an example of a prompt statement:

[0389] "Design an AI plan to create machine maintenance procedures within a factory that are adjusted based on the emotional state of the workers."

[0390] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0391] Step 1:

[0392] The terminal uses cameras and microphones mounted on robots patrolling the factory to acquire visual and auditory information in real time. The input data consists of surrounding video and audio, and the output is this digital data. The information acquired by the camera and microphone is detailed and wide-ranging.

[0393] Step 2:

[0394] The server receives visual information transmitted from the terminal and begins analysis using computer vision technology (such as OpenCV). The input is video data received from the terminal, and the output is information about the recognized anomalies. Specifically, it detects specific patterns or changes within the image and identifies equipment malfunctions.

[0395] Step 3:

[0396] The server uses speech analysis software (such as Affectiva) to analyze speech information. The input is speech data received from the terminal, and the output is the result of identifying the user's emotional state. It analyzes emotions from speech tone and manner of speaking and performs specific actions to determine whether the user is feeling tense or stressed.

[0397] Step 4:

[0398] The server generates maintenance procedures based on the analysis results and adjusts the procedures according to the user's emotional state. The input is information about the abnormal location and the emotional state, and the output is an optimized maintenance procedure. A generation AI model is used to design supplementary guidance with prompt messages, specifically to make the step order easier to understand and to add additional explanations.

[0399] Step 5:

[0400] The terminal receives optimized maintenance procedures generated by the server and presents them visually to the worker via an AR display. The input is the optimized maintenance procedure, and the output is a guide that the user can understand visually and audibly. Specifically, it uses 3D models and visual highlighting to show equipment repair procedures in animation.

[0401] Step 6:

[0402] The user performs the actual maintenance work by referring to the provided guide. The input is the guide from the terminal, and the final output is the equipment maintained in a normal state. If the user requires further assistance during the work, the terminal will request additional advice or further instructions as needed.

[0403] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0404] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0405] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0406] [Third Embodiment]

[0407] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0408] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0409] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0410] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0411] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0412] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0413] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0414] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0415] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0416] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0417] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0418] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0419] As an embodiment of the present invention, the behavior of a system that acquires visual data, identifies and analyzes equipment abnormalities at the site, and presents appropriate repair procedures to the user will be specifically described below.

[0420] First, this system uses an AR device to collect visual data. As a terminal, this device captures the on-site situation with a camera and acquires supplementary data with sensors. This ensures that the data necessary for the system is available in real time. The visual data includes video information that reflects the appearance and condition of the equipment.

[0421] Next, this data is securely transmitted to a server. The server analyzes the received data, using computer vision and voice analysis technologies to identify faulty areas and diagnose the cause of the anomaly. Based on this analysis, an AI agent generates the optimal repair procedure. The repair procedure includes specific instructions on which parts to handle and how, and is presented in a format that is easy for the worker to understand.

[0422] The server returns the repair procedure to the terminal, which then visually provides the repair guide to the user's AR device. For example, the AR device's display might highlight the damaged fan bolts in green and show the order in which to remove them. It also provides instructions on precautions to take while handling the parts and the tools to use.

[0423] Users can proceed with the repair by following the displayed visual guidelines. As the repair process progresses, instructions based on newly analyzed data from the server are updated via the terminal as needed. In this way, repairs can be completed efficiently and accurately, regardless of the user's technical level.

[0424] As a concrete example of this system, consider a case where an abnormal noise is occurring in the equipment. In this case, the terminal acquires audio data, and the server analyzes it to identify the source of the abnormal noise. For example, if it is diagnosed that the cause is a malfunction in the shaft of a rotating fan, the system generates a corresponding repair procedure and presents the specific workflow to the user through the terminal.

[0425] As described above, the system of the present invention provides users with an environment in which they can efficiently repair equipment malfunctions on-site.

[0426] The following describes the processing flow.

[0427] Step 1:

[0428] The terminal uses an AR device to acquire visual and audio data from the site. It utilizes cameras and microphones to capture detailed information about the base station's structure and status.

[0429] Step 2:

[0430] The terminal compresses the acquired visual and audio data into a predetermined format and sends it to the server using a secure communication protocol.

[0431] Step 3:

[0432] The server begins analyzing the transmitted data. Using computer vision technology, it detects anomalies in the video data and identifies abnormal sounds in the audio data.

[0433] Step 4:

[0434] The server uses an AI agent to generate the optimal repair procedure based on the analysis results. The procedure includes information such as the order in which parts should be removed and installed, and recommended tools.

[0435] Step 5:

[0436] The server sends the generated repair instructions to the device. The data is converted into a format optimized for display on AR devices.

[0437] Step 6:

[0438] The device visually presents the repair procedure to the user. The AR device screen uses highlights and animations to display the location and handling procedures for important parts.

[0439] Step 7:

[0440] The user performs repair work by following the instructions of the AR device. Based on the displayed information, they use tools to replace or adjust parts.

[0441] Step 8:

[0442] The terminal sends newly acquired data to the server in real time during the operation, and instructions are updated as needed.

[0443] Step 9:

[0444] The user performs a final check of the repair and verifies the proper functioning of the equipment using an AR device. The server records the verification results and stores a log of the entire process.

[0445] (Example 1)

[0446] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0447] In recent years, there has been a growing need to quickly and efficiently identify abnormalities in increasingly complex equipment and provide appropriate repair procedures. However, conventional technologies often suffer from insufficient analysis of visual data and supplementary information, leading to delays in diagnosing malfunctions and increasing the risk of incorrect repairs due to erroneous judgments by operators. An information system capable of solving these problems is necessary.

[0448] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0449] In this invention, the server includes an information acquisition device that acquires visual data and additional information, an analysis means that analyzes the acquired information to identify equipment abnormalities, and a means that derives repair procedures based on the analysis results. This enables accurate diagnosis of abnormalities and efficient provision of repair procedures.

[0450] "Visual data" refers to video information and related information that shows the appearance and condition of equipment.

[0451] "Additional information" refers to information about the operating environment of the equipment, acquired along with visual data, and includes temperature, vibration, and other factors.

[0452] An "information acquisition device" is a device for collecting visual data and additional information, and includes devices such as cameras and various sensors.

[0453] "Analysis means" refers to methods for processing acquired data and utilizing various technologies to identify equipment malfunctions.

[0454] A "repair procedure" is a set of steps that outlines the specific work process for repairing a malfunctioning part of the equipment.

[0455] A "presentation device" is a device used to provide repair procedures or other information to users, and includes devices such as displays and AR devices.

[0456] This invention is an information system that combines visual data and additional information to enable precise equipment diagnosis and repair guidance. The following hardware and software will be used to implement the system.

[0457] The system uses an AR (Augmented Reality) device as the terminal, acquiring visual data and additional information through its camera and various sensors. This data includes the device's appearance, status, temperature, vibration, etc. The AR device is used to visually present this information to the user.

[0458] The server performs analysis based on the received data. Here, it utilizes "OpenCV" for computer vision technology, "Librosa" for sound wave data analysis, and "TensorFlow" and "PyTorch" as deep learning models to perform highly accurate anomaly identification. This allows it to identify the location of equipment malfunctions and analyze their causes.

[0459] Based on the analysis results, the server uses a generated AI model to create repair procedures. The procedure generation is customized according to the user's technical level, providing instructions that are easy for non-technical users to understand.

[0460] As a concrete example, consider a situation where abnormal noises are emitted from rotating machinery. In this case, the terminal collects sound wave data, and through analysis, it can be identified that, for example, a malfunction in the rotating shaft is the cause. In that case, an appropriate repair procedure is presented to the user along with a visual guide.

[0461] An example of a prompt message is: "Use the visual and sensor data captured by the AR device to identify the location and cause of the equipment malfunction, and generate a repair procedure."

[0462] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0463] Step 1:

[0464] The terminal uses an AR device to acquire visual data and additional information. Inputs include video from a camera and environmental information (temperature, vibration, etc.) from sensors. This data is collected in real time and processed to accurately reflect the equipment status at the site. The output is a dataset that can be analyzed by the system.

[0465] Step 2:

[0466] The terminal securely transmits the acquired dataset to the server. This process uses encryption technology (e.g., TLS / SSL) to ensure data security. Input includes visual data and additional information, and output is the storage of this information in a database.

[0467] Step 3:

[0468] The server performs preprocessing on the received data for analysis. The input is the dataset obtained in step 2. The data attributes are standardized and formatted for analysis. The output is data in an analyzable format.

[0469] Step 4:

[0470] The server analyzes data using computer vision and acoustic analysis techniques. It analyzes video data using "OpenCV" and acoustic data using "Librosa". Pre-processed data is the input, and the output identifies the location and cause of equipment malfunctions.

[0471] Step 5:

[0472] The server generates repair procedures using an AI model based on the analysis results. The input includes information on the location and cause of the malfunction, which is used to subdivide the repair steps. The output is a repair manual containing specific operating procedures.

[0473] Step 6:

[0474] The terminal displays repair instructions retrieved from the server on the user's AR device. A generated repair manual is provided as input. Work areas and procedures are visualized using highlighting. As output, an interactive repair guide is presented on the user's device.

[0475] Step 7:

[0476] The user proceeds with the repair work by following the displayed guide. The server updates instructions in real time as needed and provides them to the user via the terminal. Inputs include work progress and new environmental data, while output is the smooth completion of the repair process.

[0477] (Application Example 1)

[0478] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0479] In industrial machinery maintenance, there is a challenge in immediately identifying problems and providing appropriate repair procedures when abnormalities occur. In particular, real-time information provision is crucial because visibility and accuracy of procedures are required for on-site workers to solve problems quickly and efficiently.

[0480] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0481] In this invention, the server includes means for collecting visual information, means for analyzing the acquired visual information to identify equipment abnormalities, and means for generating repair procedures based on the analysis results. This enables on-site workers to visually confirm the location of the abnormality in real time and efficiently implement the optimal repair procedure.

[0482] "Means for collecting visual information" refers to devices that use cameras or sensors to photograph or record the operating status of machinery or equipment, thereby acquiring visual data.

[0483] "Means for analyzing acquired visual information to identify equipment malfunctions" refers to a system that processes collected visual data to identify the location of the malfunction and its cause.

[0484] "A means of generating repair procedures based on analysis results" refers to a process that automatically creates and presents procedures and methods for correcting abnormalities based on the results of identifying the abnormal areas.

[0485] "Means of presenting to the operator through a visual device" refers to devices or interfaces that visually inform the worker of the generated repair procedure.

[0486] "A means of checking the maintenance status of industrial machinery in real time and visually presenting repair guidelines in the event of an abnormality" refers to a system that detects abnormalities during machine operation and immediately communicates appropriate repair procedures to the operator visually.

[0487] The system for implementing this invention mainly consists of a server, terminals, and users. The server is responsible for analyzing visual data and generating repair procedures. Specifically, terminals such as smart glasses or smartphones first collect visual information of industrial machinery using cameras and sensors (e.g., Microsoft HoloLens). This visual data is transmitted to the server in real time.

[0488] On the server, this data is analyzed using computer vision technologies (e.g., OpenCV, TensorFlow) to identify anomalies. Based on the analysis results, a generated AI model creates the optimal repair procedure. This repair procedure is fed back to the user's visual device via the terminal and displayed in an easy-to-understand format. For example, the anomaly and the tools to be used are visually highlighted.

[0489] Users can perform maintenance and repairs on their devices by following the instructions displayed on the smart glasses' screen. If additional data is sent from the device during the repair process, the server will update the instructions accordingly in real time.

[0490] A concrete example is when a malfunction occurs in the joint of a factory robot. The smart glasses scan the joint, the server identifies the faulty part, and generates appropriate repair instructions. The user can then follow the visual instructions through the glasses to perform tasks such as replacing lubricant or adjusting bolts.

[0491] An example of a prompt message is: "There is a problem with the factory robot. Scan the robot with your smart glasses and highlight the problem area. Identify the cause and provide the optimal repair procedure in real time."

[0492] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0493] Step 1:

[0494] The terminal uses a camera and sensors to collect images and data from industrial machinery. The input is real-time visual data, and the output is high-resolution image data and sensor data. This data is sent directly to the server.

[0495] Step 2:

[0496] The server analyzes the visual data received from the terminal. In this step, computer vision technologies such as OpenCV and TensorFlow are used to identify anomalies from the input image data. The output is information indicating the location and type of the anomaly.

[0497] Step 3:

[0498] The server generates repair procedures using an AI model based on the analysis results. The output from step 2 is used as input, and data processing is performed to generate the optimal repair procedure. The output includes information such as repair procedure manuals and specific repair steps.

[0499] Step 4:

[0500] The server generates repair instructions and sends them to the terminal, which then visually presents them to the user. The input is the generated repair instructions, and the output is a repair guide presented through a visual device. Specifically, it highlights the faulty area and displays each step of the procedure sequentially on the screen.

[0501] Step 5:

[0502] The user performs the actual repair work following the repair procedures presented through the device. The input is the repair procedure presented in step 4, and the output is the result of the repair work performed. Specifically, the user replaces parts or makes adjustments using tools.

[0503] Step 6:

[0504] The device collects additional data during repair and sends it to the server. The input is real-time additional data, and the output is updated visual data. The server uses this data to re-evaluate the procedure and update the repair procedure as needed.

[0505] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0506] As an embodiment of the present invention, the behavior of a system equipped with an emotion engine that recognizes and responds to the user's emotional state, in addition to acquiring and analyzing visual and audio data, will be specifically described below.

[0507] First, the device uses an AR device to collect visual and audio data from the site. By capturing detailed images of the environment and equipment with a camera and collecting surrounding sounds with a microphone, it identifies various data, including any anomalies in the equipment. At the same time, an emotion engine analyzes the user's facial expressions and tone of voice, monitoring the user's emotional state in real time.

[0508] The server analyzes the received visual and audio data. This uses AI-powered computer vision and audio analysis algorithms to detect specific anomalies and diagnose their causes. Based on the analysis results, the AI ​​agent generates appropriate repair procedures. Furthermore, if the emotion engine's analysis indicates that the user is experiencing stress, the server simplifies the repair procedures or adjusts them to provide supplementary information to help the user understand.

[0509] Next, the server sends the generated repair instructions to the device in a format that is appropriate to the user's emotional state. The device then displays a visual repair guide through an AR device. This includes highlighting the location of parts and animations that show the sequence of steps. It can also provide voice assistance in a more user-friendly tone depending on the user's emotional state.

[0510] The user proceeds with the repair by following the displayed guidelines. Based on feedback from the emotion engine, the instructions are further adjusted based on the user's emotional changes, reducing stress and improving work efficiency.

[0511] For example, if the equipment malfunction is complex and the user shows signs of confusion, the emotion engine will detect this. The terminal will then break down each repair step by step, along with easy-to-understand instructions resent from the server. This allows the user to easily grasp the task and perform the work calmly and confidently.

[0512] In this way, by integrating an emotion engine, a system can be realized that efficiently supports equipment repair while taking into account the user's emotional state.

[0513] The following describes the processing flow.

[0514] Step 1:

[0515] The device uses an AR device to acquire visual and audio data from the site. It captures video of the equipment in real time with a camera and records ambient sounds and abnormal sounds with a microphone. Furthermore, an emotion engine analyzes the user's facial expressions and voice tone to recognize their emotional state in real time.

[0516] Step 2:

[0517] The device compresses the acquired visual, audio, and emotional state data and sends it to the server. Data transmission is performed using a secure protocol, allowing for rapid progression to the analysis process.

[0518] Step 3:

[0519] The server analyzes the transmitted data. Image recognition technology identifies abnormal areas from the video data, and audio analysis technology determines the cause of abnormal sounds. Based on these analysis results, an AI agent creates a repair procedure.

[0520] Step 4:

[0521] The server uses an emotion engine to assess the user's emotional state. If the user is feeling confused or stressed, the server makes adjustments, such as simplifying the repair procedure or adding supplementary information.

[0522] Step 5:

[0523] The server sends optimized repair instructions to the device. These instructions are designed to be visually easy to understand and tailored to the user's emotional state.

[0524] Step 6:

[0525] The device visually presents repair procedures to the user through an augmented reality (AR) device. It highlights the location of parts, displays work instructions with animations, and provides clear guidance to the user.

[0526] Step 7:

[0527] Users perform repair work by following instructions received through an AR device. They can proceed with the work without anxiety while receiving feedback based on analysis by an emotion engine.

[0528] Step 8:

[0529] The device continuously records additional visual and emotional data obtained during the work and sends it to the server, updating the repair procedure as needed. Optimal information is provided in real time to support the work process.

[0530] (Example 2)

[0531] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0532] In on-site equipment repair, workers need to quickly and accurately identify equipment malfunctions and understand and execute appropriate repair procedures. However, current systems do not fully utilize visual and auditory information, and lack mechanisms to optimize repair procedures by considering the worker's emotional state. As a result, workers may experience stress and confusion, which can hinder the efficiency of repair work.

[0533] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0534] In this invention, the server includes means for analyzing digital video to identify equipment abnormalities, means for analyzing acoustic data to identify abnormal sounds, and means for analyzing the emotional state of the worker. This makes it possible to identify equipment abnormalities early and to present appropriate repair procedures that reflect the emotional state of the worker.

[0535] "Digital images" are visual information acquired by cameras and other imaging devices that represent the device and its surrounding environment.

[0536] "Analysis" is a process performed to identify anomalies or abnormal sounds based on acquired digital video and audio data, and to pinpoint the cause of a problem.

[0537] A "repair procedure" is a set of instructions outlining the specific steps necessary to correct equipment malfunctions, and by following it, workers can perform effective repairs.

[0538] A "visual display device" is a display device used by workers to confirm repair procedures, and includes AR devices and monitors.

[0539] "Acoustic data" refers to sound information collected through audio input devices such as microphones, and is used to detect abnormal sounds.

[0540] "Emotional state" refers to the internal psychological state of a worker, as judged from their facial expressions, tone of voice, etc., and includes stress and confusion.

[0541] "Adjustment" refers to appropriately modifying repair procedures and supplementary information according to the emotional state of the worker.

[0542] In this embodiment, the system uses digital video and audio data to identify equipment malfunctions and implements a process to support workers. Specifically, the terminal collects digital video and audio data in combination with an AR device. This allows for detailed acquisition of visual information of the site using a camera and recording of audio information, including background sounds, using a microphone.

[0543] The server uses advanced AI algorithms to analyze received digital video and detect anomalies. During this process, it also analyzes acoustic data, utilizing voice analysis technology to identify abnormal sounds. In addition, an emotion engine evaluates the worker's facial expressions and voice tone, analyzing their emotional state in real time.

[0544] Based on the analysis, the server uses a generative AI model to generate appropriate repair procedures. These procedures include the repair process for the faulty area and are customized to be easily understood by the worker. Furthermore, the procedures are adjusted and supplementary information is provided, taking into account the worker's emotional state. The emotion engine optimizes the guidance to reduce stress and confusion.

[0545] For example, if a complex equipment malfunction is detected, the terminal receives easy-to-follow instructions from the server and displays them visually on an AR device. This may include highlighting part locations and animations showing the steps. In addition, voice guidance may be provided, with instructions delivered in a gentle tone that adapts to the user's emotional state.

[0546] An example of input to the generating AI model is a prompt such as, "Use data acquired from an AR device to create a repair guide that takes into account the user's emotional state." Through this prompt, the model can generate detailed repair procedures and provide quick and appropriate assistance to the worker.

[0547] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0548] Step 1:

[0549] The terminal uses an AR device to collect digital video and audio data from the site. Inputs include video captured by a camera and audio recorded by a microphone. This data is used to understand the specific situation and equipment status at the site. Specifically, the terminal continuously acquires video and records audio in real time. The collected digital video and audio data are sent to a server as output.

[0550] Step 2:

[0551] The server analyzes the received digital video to identify equipment malfunctions. The input is digital video transmitted from a terminal. It uses computer vision technology to perform data processing to detect abnormal areas in the video. Specifically, the server uses an image recognition algorithm to identify the abnormal points and outputs the results as vector data.

[0552] Step 3:

[0553] In parallel, the server analyzes the acoustic data. The input is acoustic data transmitted from the terminal. Using an audio analysis algorithm, it removes background noise and then performs data calculations to identify abnormal sounds. Specifically, the server performs frequency analysis of the sound and identifies abnormal patterns. The output is the result of detecting abnormal sounds.

[0554] Step 4:

[0555] The server analyzes the worker's emotional state. Input includes facial expression data and voice tone information received from the terminal. The emotion analysis engine uses this data to perform calculations that evaluate the worker's stress level and level of confusion. Specifically, the server uses a machine learning model to output an index that quantifies the emotional state.

[0556] Step 5:

[0557] The server generates repair procedures based on anomaly detection results and emotion evaluations. Inputs include data on the location of the anomaly, information on abnormal sounds, and evaluation results of the emotion state. A generation AI model is used to perform data processing to adaptively create the repair procedures. Specifically, the server inputs prompt statements into the model and outputs customized repair procedures in text format.

[0558] Step 6:

[0559] The terminal presents repair procedures to the worker. The input is the repair procedure transmitted from the server. Using an AR device, it performs specific actions such as highlighting parts and displaying animations illustrating the procedure. It also provides user-friendly voice guidance that adapts to the worker's emotional state. The output is the repair procedure communicated to the worker both visually and audibly.

[0560] (Application Example 2)

[0561] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0562] Conventional equipment maintenance systems often provide uniform maintenance procedures without considering the emotional state of users, which can lead to decreased user understanding and work efficiency. Furthermore, information used to identify anomalies is often reliant on either visual or auditory cues, resulting in insufficient accuracy in anomaly detection. This can lead to stressful situations for workers and hinder the efficiency of equipment maintenance.

[0563] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0564] In this invention, the server includes a medium for acquiring visual information, means for analyzing the acquired visual information to identify abnormalities in the device, means for analyzing audio information to recognize the user's emotions, and means for generating maintenance procedures based on the analysis results. This enables optimal guidance tailored to the user's emotional state.

[0565] "Visual information" refers to image and video data acquired through cameras and sensors.

[0566] A "medium" refers to a device or apparatus used to acquire or transmit information.

[0567] "Analysis" is a method for processing data and understanding its content and meaning.

[0568] The term "device" is a general term for machines or systems designed to perform a specific function.

[0569] "Abnormal" refers to a state that deviates from the normal condition or function.

[0570] "Audio information" refers to sound data collected through audio input devices such as microphones.

[0571] "User" refers to an individual or group that uses the system.

[0572] "Recognizing emotions" means judging a user's emotional state from their facial expressions and tone of voice.

[0573] "Maintenance procedures" are specific work procedures necessary to keep equipment and devices in a normal state.

[0574] "Guidance" refers to instructions or information provided to encourage users to take action or perform tasks.

[0575] In the system implementing this invention, a robot equipped with an AR device is used to efficiently maintain and manage equipment and machinery within a factory. The AR device is equipped with a camera and a microphone, and acquires visual and audio information in real time. This allows for the collection of detailed data to detect abnormalities in the equipment.

[0576] The server receives this data and uses computer vision technology to analyze the visual information and recognize specific anomalies. Computer vision technologies used include OpenCV. Audio information is analyzed using audio analysis software such as Affectiva. Through this audio analysis, the system grasps the user's emotional state in real time.

[0577] Based on the emotion recognition results, the server generates maintenance procedures, adjusts and optimizes them, and sends them to the robot's guidance system. This uses a generative AI model, which leverages prompts to design optimal guidance. Specific guidance is presented to the user as 3D models and visually clear instructions. A user-friendly voice interface also provides intuitive support.

[0578] For example, if an abnormal operation is detected in a machine on the production line, the server analyzes the abnormality with high accuracy and provides step-by-step guidance so that the operator can easily understand it. If the user's emotions are unstable, the AI ​​will provide further supplementary explanations using prompt messages.

[0579] As an example, here is an example of a prompt statement:

[0580] "Design an AI plan to create machine maintenance procedures within a factory that are adjusted based on the emotional state of the workers."

[0581] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0582] Step 1:

[0583] The terminal uses cameras and microphones mounted on robots patrolling the factory to acquire visual and auditory information in real time. The input data consists of surrounding video and audio, and the output is this digital data. The information acquired by the camera and microphone is detailed and wide-ranging.

[0584] Step 2:

[0585] The server receives visual information transmitted from the terminal and begins analysis using computer vision technology (such as OpenCV). The input is video data received from the terminal, and the output is information about the recognized anomalies. Specifically, it detects specific patterns or changes within the image and identifies equipment malfunctions.

[0586] Step 3:

[0587] The server uses speech analysis software (such as Affectiva) to analyze speech information. The input is speech data received from the terminal, and the output is the result of identifying the user's emotional state. It analyzes emotions from speech tone and manner of speaking and performs specific actions to determine whether the user is feeling tense or stressed.

[0588] Step 4:

[0589] The server generates maintenance procedures based on the analysis results and adjusts the procedures according to the user's emotional state. The input is information about the abnormal location and the emotional state, and the output is an optimized maintenance procedure. A generation AI model is used to design supplementary guidance with prompt messages, specifically to make the step order easier to understand and to add additional explanations.

[0590] Step 5:

[0591] The terminal receives optimized maintenance procedures generated by the server and presents them visually to the worker via an AR display. The input is the optimized maintenance procedure, and the output is a guide that the user can understand visually and audibly. Specifically, it uses 3D models and visual highlighting to show equipment repair procedures in animation.

[0592] Step 6:

[0593] The user performs the actual maintenance work by referring to the provided guide. The input is the guide from the terminal, and the final output is the equipment maintained in a normal state. If the user requires further assistance during the work, the terminal will request additional advice or further instructions as needed.

[0594] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0595] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0596] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0597] [Fourth Embodiment]

[0598] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0599] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0600] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0601] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0602] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0603] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0604] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0605] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0606] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0607] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0608] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0609] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0610] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0611] As an embodiment of the present invention, the behavior of a system that acquires visual data, identifies and analyzes equipment abnormalities at the site, and presents appropriate repair procedures to the user will be specifically described below.

[0612] First, this system uses an AR device to collect visual data. As a terminal, this device captures the on-site situation with a camera and acquires supplementary data with sensors. This ensures that the data necessary for the system is available in real time. The visual data includes video information that reflects the appearance and condition of the equipment.

[0613] Next, this data is securely transmitted to a server. The server analyzes the received data, using computer vision and voice analysis technologies to identify faulty areas and diagnose the cause of the anomaly. Based on this analysis, an AI agent generates the optimal repair procedure. The repair procedure includes specific instructions on which parts to handle and how, and is presented in a format that is easy for the worker to understand.

[0614] The server returns the repair procedure to the terminal, which then visually provides the repair guide to the user's AR device. For example, the AR device's display might highlight the damaged fan bolts in green and show the order in which to remove them. It also provides instructions on precautions to take while handling the parts and the tools to use.

[0615] Users can proceed with the repair by following the displayed visual guidelines. As the repair process progresses, instructions based on newly analyzed data from the server are updated via the terminal as needed. In this way, repairs can be completed efficiently and accurately, regardless of the user's technical level.

[0616] As a concrete example of this system, consider a case where an abnormal noise is occurring in the equipment. In this case, the terminal acquires audio data, and the server analyzes it to identify the source of the abnormal noise. For example, if it is diagnosed that the cause is a malfunction in the shaft of a rotating fan, the system generates a corresponding repair procedure and presents the specific workflow to the user through the terminal.

[0617] As described above, the system of the present invention provides users with an environment in which they can efficiently repair equipment malfunctions on-site.

[0618] The following describes the processing flow.

[0619] Step 1:

[0620] The terminal uses an AR device to acquire visual and audio data from the site. It utilizes cameras and microphones to capture detailed information about the base station's structure and status.

[0621] Step 2:

[0622] The terminal compresses the acquired visual and audio data into a predetermined format and sends it to the server using a secure communication protocol.

[0623] Step 3:

[0624] The server begins analyzing the transmitted data. Using computer vision technology, it detects anomalies in the video data and identifies abnormal sounds in the audio data.

[0625] Step 4:

[0626] The server uses an AI agent to generate the optimal repair procedure based on the analysis results. The procedure includes information such as the order in which parts should be removed and installed, and recommended tools.

[0627] Step 5:

[0628] The server sends the generated repair instructions to the device. The data is converted into a format optimized for display on AR devices.

[0629] Step 6:

[0630] The device visually presents the repair procedure to the user. The AR device screen uses highlights and animations to display the location and handling procedures for important parts.

[0631] Step 7:

[0632] The user performs repair work by following the instructions of the AR device. Based on the displayed information, they use tools to replace or adjust parts.

[0633] Step 8:

[0634] The terminal sends newly acquired data to the server in real time during the operation, and instructions are updated as needed.

[0635] Step 9:

[0636] The user performs a final check of the repair and verifies the proper functioning of the equipment using an AR device. The server records the verification results and stores a log of the entire process.

[0637] (Example 1)

[0638] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] In recent years, there has been a growing need to quickly and efficiently identify abnormalities in increasingly complex equipment and provide appropriate repair procedures. However, conventional technologies often suffer from insufficient analysis of visual data and supplementary information, leading to delays in diagnosing malfunctions and increasing the risk of incorrect repairs due to erroneous judgments by operators. An information system capable of solving these problems is necessary.

[0640] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0641] In this invention, the server includes an information acquisition device that acquires visual data and additional information, an analysis means that analyzes the acquired information to identify equipment abnormalities, and a means that derives repair procedures based on the analysis results. This enables accurate diagnosis of abnormalities and efficient provision of repair procedures.

[0642] "Visual data" refers to video information and related information that shows the appearance and condition of equipment.

[0643] "Additional information" refers to information about the operating environment of the equipment, acquired along with visual data, and includes temperature, vibration, and other factors.

[0644] An "information acquisition device" is a device for collecting visual data and additional information, and includes devices such as cameras and various sensors.

[0645] "Analysis means" refers to methods for processing acquired data and utilizing various technologies to identify equipment malfunctions.

[0646] A "repair procedure" is a set of steps that outlines the specific work process for repairing a malfunctioning part of the equipment.

[0647] A "presentation device" is a device used to provide repair procedures or other information to users, and includes devices such as displays and AR devices.

[0648] This invention is an information system that combines visual data and additional information to enable precise equipment diagnosis and repair guidance. The following hardware and software will be used to implement the system.

[0649] The system uses an AR (Augmented Reality) device as the terminal, acquiring visual data and additional information through its camera and various sensors. This data includes the device's appearance, status, temperature, vibration, etc. The AR device is used to visually present this information to the user.

[0650] The server performs analysis based on the received data. Here, it utilizes "OpenCV" for computer vision technology, "Librosa" for sound wave data analysis, and "TensorFlow" and "PyTorch" as deep learning models to perform highly accurate anomaly identification. This allows it to identify the location of equipment malfunctions and analyze their causes.

[0651] Based on the analysis results, the server uses a generated AI model to create repair procedures. The procedure generation is customized according to the user's technical level, providing instructions that are easy for non-technical users to understand.

[0652] As a concrete example, consider a situation where abnormal noises are emitted from rotating machinery. In this case, the terminal collects sound wave data, and through analysis, it can be identified that, for example, a malfunction in the rotating shaft is the cause. In that case, an appropriate repair procedure is presented to the user along with a visual guide.

[0653] An example of a prompt message is: "Use the visual and sensor data captured by the AR device to identify the location and cause of the equipment malfunction, and generate a repair procedure."

[0654] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0655] Step 1:

[0656] The terminal uses an AR device to acquire visual data and additional information. Inputs include video from a camera and environmental information (temperature, vibration, etc.) from sensors. This data is collected in real time and processed to accurately reflect the equipment status at the site. The output is a dataset that can be analyzed by the system.

[0657] Step 2:

[0658] The terminal securely transmits the acquired dataset to the server. This process uses encryption technology (e.g., TLS / SSL) to ensure data security. Input includes visual data and additional information, and output is the storage of this information in a database.

[0659] Step 3:

[0660] The server performs preprocessing on the received data for analysis. The input is the dataset obtained in step 2. The data attributes are standardized and formatted for analysis. The output is data in an analyzable format.

[0661] Step 4:

[0662] The server analyzes data using computer vision and acoustic analysis techniques. It analyzes video data using "OpenCV" and acoustic data using "Librosa". Pre-processed data is the input, and the output identifies the location and cause of equipment malfunctions.

[0663] Step 5:

[0664] The server generates repair procedures using an AI model based on the analysis results. The input includes information on the location and cause of the malfunction, which is used to subdivide the repair steps. The output is a repair manual containing specific operating procedures.

[0665] Step 6:

[0666] The terminal displays repair instructions retrieved from the server on the user's AR device. A generated repair manual is provided as input. Work areas and procedures are visualized using highlighting. As output, an interactive repair guide is presented on the user's device.

[0667] Step 7:

[0668] The user proceeds with the repair work by following the displayed guide. The server updates instructions in real time as needed and provides them to the user via the terminal. Inputs include work progress and new environmental data, while output is the smooth completion of the repair process.

[0669] (Application Example 1)

[0670] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0671] In industrial machinery maintenance, there is a challenge in immediately identifying problems and providing appropriate repair procedures when abnormalities occur. In particular, real-time information provision is crucial because visibility and accuracy of procedures are required for on-site workers to solve problems quickly and efficiently.

[0672] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0673] In this invention, the server includes means for collecting visual information, means for analyzing the acquired visual information to identify equipment abnormalities, and means for generating repair procedures based on the analysis results. This enables on-site workers to visually confirm the location of the abnormality in real time and efficiently implement the optimal repair procedure.

[0674] "Means for collecting visual information" refers to devices that use cameras or sensors to photograph or record the operating status of machinery or equipment, thereby acquiring visual data.

[0675] "Means for analyzing acquired visual information to identify equipment malfunctions" refers to a system that processes collected visual data to identify the location of the malfunction and its cause.

[0676] "A means of generating repair procedures based on analysis results" refers to a process that automatically creates and presents procedures and methods for correcting abnormalities based on the results of identifying the abnormal areas.

[0677] "Means of presenting to the operator through a visual device" refers to devices or interfaces that visually inform the worker of the generated repair procedure.

[0678] "A means of checking the maintenance status of industrial machinery in real time and visually presenting repair guidelines in the event of an abnormality" refers to a system that detects abnormalities during machine operation and immediately communicates appropriate repair procedures to the operator visually.

[0679] The system for implementing this invention mainly consists of a server, terminals, and users. The server is responsible for analyzing visual data and generating repair procedures. Specifically, terminals such as smart glasses or smartphones first collect visual information of industrial machinery using cameras and sensors (e.g., Microsoft HoloLens). This visual data is transmitted to the server in real time.

[0680] On the server, this data is analyzed using computer vision technologies (e.g., OpenCV, TensorFlow) to identify anomalies. Based on the analysis results, a generated AI model creates the optimal repair procedure. This repair procedure is fed back to the user's visual device via the terminal and displayed in an easy-to-understand format. For example, the anomaly and the tools to be used are visually highlighted.

[0681] Users can perform maintenance and repairs on their devices by following the instructions displayed on the smart glasses' screen. If additional data is sent from the device during the repair process, the server will update the instructions accordingly in real time.

[0682] A concrete example is when a malfunction occurs in the joint of a factory robot. The smart glasses scan the joint, the server identifies the faulty part, and generates appropriate repair instructions. The user can then follow the visual instructions through the glasses to perform tasks such as replacing lubricant or adjusting bolts.

[0683] An example of a prompt message is: "There is a problem with the factory robot. Scan the robot with your smart glasses and highlight the problem area. Identify the cause and provide the optimal repair procedure in real time."

[0684] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0685] Step 1:

[0686] The terminal uses a camera and sensors to collect images and data from industrial machinery. The input is real-time visual data, and the output is high-resolution image data and sensor data. This data is sent directly to the server.

[0687] Step 2:

[0688] The server analyzes the visual data received from the terminal. In this step, computer vision technologies such as OpenCV and TensorFlow are used to identify anomalies from the input image data. The output is information indicating the location and type of the anomaly.

[0689] Step 3:

[0690] The server generates repair procedures using an AI model based on the analysis results. The output from step 2 is used as input, and data processing is performed to generate the optimal repair procedure. The output includes information such as repair procedure manuals and specific repair steps.

[0691] Step 4:

[0692] The server generates repair instructions and sends them to the terminal, which then visually presents them to the user. The input is the generated repair instructions, and the output is a repair guide presented through a visual device. Specifically, it highlights the faulty area and displays each step of the procedure sequentially on the screen.

[0693] Step 5:

[0694] The user performs the actual repair work following the repair procedures presented through the device. The input is the repair procedure presented in step 4, and the output is the result of the repair work performed. Specifically, the user replaces parts or makes adjustments using tools.

[0695] Step 6:

[0696] The device collects additional data during repair and sends it to the server. The input is real-time additional data, and the output is updated visual data. The server uses this data to re-evaluate the procedure and update the repair procedure as needed.

[0697] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0698] As an embodiment of the present invention, the behavior of a system equipped with an emotion engine that recognizes and responds to the user's emotional state, in addition to acquiring and analyzing visual and audio data, will be specifically described below.

[0699] First, the device uses an AR device to collect visual and audio data from the site. By capturing detailed images of the environment and equipment with a camera and collecting surrounding sounds with a microphone, it identifies various data, including any anomalies in the equipment. At the same time, an emotion engine analyzes the user's facial expressions and tone of voice, monitoring the user's emotional state in real time.

[0700] The server analyzes the received visual and audio data. This uses AI-powered computer vision and audio analysis algorithms to detect specific anomalies and diagnose their causes. Based on the analysis results, the AI ​​agent generates appropriate repair procedures. Furthermore, if the emotion engine's analysis indicates that the user is experiencing stress, the server simplifies the repair procedures or adjusts them to provide supplementary information to help the user understand.

[0701] Next, the server sends the generated repair instructions to the device in a format that is appropriate to the user's emotional state. The device then displays a visual repair guide through an AR device. This includes highlighting the location of parts and animations that show the sequence of steps. It can also provide voice assistance in a more user-friendly tone depending on the user's emotional state.

[0702] The user proceeds with the repair by following the displayed guidelines. Based on feedback from the emotion engine, the instructions are further adjusted based on the user's emotional changes, reducing stress and improving work efficiency.

[0703] For example, if the equipment malfunction is complex and the user shows signs of confusion, the emotion engine will detect this. The terminal will then break down each repair step by step, along with easy-to-understand instructions resent from the server. This allows the user to easily grasp the task and perform the work calmly and confidently.

[0704] In this way, by integrating an emotion engine, a system can be realized that efficiently supports equipment repair while taking into account the user's emotional state.

[0705] The following describes the processing flow.

[0706] Step 1:

[0707] The device uses an AR device to acquire visual and audio data from the site. It captures video of the equipment in real time with a camera and records ambient sounds and abnormal sounds with a microphone. Furthermore, an emotion engine analyzes the user's facial expressions and voice tone to recognize their emotional state in real time.

[0708] Step 2:

[0709] The device compresses the acquired visual, audio, and emotional state data and sends it to the server. Data transmission is performed using a secure protocol, allowing for rapid progression to the analysis process.

[0710] Step 3:

[0711] The server analyzes the transmitted data. Image recognition technology identifies abnormal areas from the video data, and audio analysis technology determines the cause of abnormal sounds. Based on these analysis results, an AI agent creates a repair procedure.

[0712] Step 4:

[0713] The server uses an emotion engine to assess the user's emotional state. If the user is feeling confused or stressed, the server makes adjustments, such as simplifying the repair procedure or adding supplementary information.

[0714] Step 5:

[0715] The server sends optimized repair instructions to the device. These instructions are designed to be visually easy to understand and tailored to the user's emotional state.

[0716] Step 6:

[0717] The device visually presents repair procedures to the user through an augmented reality (AR) device. It highlights the location of parts, displays work instructions with animations, and provides clear guidance to the user.

[0718] Step 7:

[0719] Users perform repair work by following instructions received through an AR device. They can proceed with the work without anxiety while receiving feedback based on analysis by an emotion engine.

[0720] Step 8:

[0721] The device continuously records additional visual and emotional data obtained during the work and sends it to the server, updating the repair procedure as needed. Optimal information is provided in real time to support the work process.

[0722] (Example 2)

[0723] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0724] In on-site equipment repair, workers need to quickly and accurately identify equipment malfunctions and understand and execute appropriate repair procedures. However, current systems do not fully utilize visual and auditory information, and lack mechanisms to optimize repair procedures by considering the worker's emotional state. As a result, workers may experience stress and confusion, which can hinder the efficiency of repair work.

[0725] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0726] In this invention, the server includes means for analyzing digital video to identify equipment abnormalities, means for analyzing acoustic data to identify abnormal sounds, and means for analyzing the emotional state of the worker. This makes it possible to identify equipment abnormalities early and to present appropriate repair procedures that reflect the emotional state of the worker.

[0727] "Digital images" are visual information acquired by cameras and other imaging devices that represent the device and its surrounding environment.

[0728] "Analysis" is a process performed to identify anomalies or abnormal sounds based on acquired digital video and audio data, and to pinpoint the cause of a problem.

[0729] A "repair procedure" is a set of instructions outlining the specific steps necessary to correct equipment malfunctions, and by following it, workers can perform effective repairs.

[0730] A "visual display device" is a display device used by workers to confirm repair procedures, and includes AR devices and monitors.

[0731] "Acoustic data" refers to sound information collected through audio input devices such as microphones, and is used to detect abnormal sounds.

[0732] "Emotional state" refers to the internal psychological state of a worker, as judged from their facial expressions, tone of voice, etc., and includes stress and confusion.

[0733] "Adjustment" refers to appropriately modifying repair procedures and supplementary information according to the emotional state of the worker.

[0734] In this embodiment, the system uses digital video and audio data to identify equipment malfunctions and implements a process to support workers. Specifically, the terminal collects digital video and audio data in combination with an AR device. This allows for detailed acquisition of visual information of the site using a camera and recording of audio information, including background sounds, using a microphone.

[0735] The server uses advanced AI algorithms to analyze received digital video and detect anomalies. During this process, it also analyzes acoustic data, utilizing voice analysis technology to identify abnormal sounds. In addition, an emotion engine evaluates the worker's facial expressions and voice tone, analyzing their emotional state in real time.

[0736] Based on the analysis, the server uses a generative AI model to generate appropriate repair procedures. These procedures include the repair process for the faulty area and are customized to be easily understood by the worker. Furthermore, the procedures are adjusted and supplementary information is provided, taking into account the worker's emotional state. The emotion engine optimizes the guidance to reduce stress and confusion.

[0737] For example, if a complex equipment malfunction is detected, the terminal receives easy-to-follow instructions from the server and displays them visually on an AR device. This may include highlighting part locations and animations showing the steps. In addition, voice guidance may be provided, with instructions delivered in a gentle tone that adapts to the user's emotional state.

[0738] An example of input to the generating AI model is a prompt such as, "Use data acquired from an AR device to create a repair guide that takes into account the user's emotional state." Through this prompt, the model can generate detailed repair procedures and provide quick and appropriate assistance to the worker.

[0739] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0740] Step 1:

[0741] The terminal uses an AR device to collect digital video and audio data from the site. Inputs include video captured by a camera and audio recorded by a microphone. This data is used to understand the specific situation and equipment status at the site. Specifically, the terminal continuously acquires video and records audio in real time. The collected digital video and audio data are sent to a server as output.

[0742] Step 2:

[0743] The server analyzes the received digital video to identify equipment malfunctions. The input is digital video transmitted from a terminal. It uses computer vision technology to perform data processing to detect abnormal areas in the video. Specifically, the server uses an image recognition algorithm to identify the abnormal points and outputs the results as vector data.

[0744] Step 3:

[0745] In parallel, the server analyzes the acoustic data. The input is acoustic data transmitted from the terminal. Using an audio analysis algorithm, it removes background noise and then performs data calculations to identify abnormal sounds. Specifically, the server performs frequency analysis of the sound and identifies abnormal patterns. The output is the result of detecting abnormal sounds.

[0746] Step 4:

[0747] The server analyzes the worker's emotional state. Input includes facial expression data and voice tone information received from the terminal. The emotion analysis engine uses this data to perform calculations that evaluate the worker's stress level and level of confusion. Specifically, the server uses a machine learning model to output an index that quantifies the emotional state.

[0748] Step 5:

[0749] The server generates repair procedures based on anomaly detection results and emotion evaluations. Inputs include data on the location of the anomaly, information on abnormal sounds, and evaluation results of the emotion state. A generation AI model is used to perform data processing to adaptively create the repair procedures. Specifically, the server inputs prompt statements into the model and outputs customized repair procedures in text format.

[0750] Step 6:

[0751] The terminal presents repair procedures to the worker. The input is the repair procedure transmitted from the server. Using an AR device, it performs specific actions such as highlighting parts and displaying animations illustrating the procedure. It also provides user-friendly voice guidance that adapts to the worker's emotional state. The output is the repair procedure communicated to the worker both visually and audibly.

[0752] (Application Example 2)

[0753] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0754] Conventional equipment maintenance systems often provide uniform maintenance procedures without considering the emotional state of users, which can lead to decreased user understanding and work efficiency. Furthermore, information used to identify anomalies is often reliant on either visual or auditory cues, resulting in insufficient accuracy in anomaly detection. This can lead to stressful situations for workers and hinder the efficiency of equipment maintenance.

[0755] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0756] In this invention, the server includes a medium for acquiring visual information, means for analyzing the acquired visual information to identify abnormalities in the device, means for analyzing audio information to recognize the user's emotions, and means for generating maintenance procedures based on the analysis results. This enables optimal guidance tailored to the user's emotional state.

[0757] "Visual information" refers to image and video data acquired through cameras and sensors.

[0758] A "medium" refers to a device or apparatus used to acquire or transmit information.

[0759] "Analysis" is a method for processing data and understanding its content and meaning.

[0760] The term "device" is a general term for machines or systems designed to perform a specific function.

[0761] "Abnormal" refers to a state that deviates from the normal condition or function.

[0762] "Audio information" refers to sound data collected through audio input devices such as microphones.

[0763] "User" refers to an individual or group that uses the system.

[0764] "Recognizing emotions" means judging a user's emotional state from their facial expressions and tone of voice.

[0765] "Maintenance procedures" are specific work procedures necessary to keep equipment and devices in a normal state.

[0766] "Guidance" refers to instructions or information provided to encourage users to take action or perform tasks.

[0767] In the system implementing this invention, a robot equipped with an AR device is used to efficiently maintain and manage equipment and machinery within a factory. The AR device is equipped with a camera and a microphone, and acquires visual and audio information in real time. This allows for the collection of detailed data to detect abnormalities in the equipment.

[0768] The server receives this data and uses computer vision technology to analyze the visual information and recognize specific anomalies. Computer vision technologies used include OpenCV. Audio information is analyzed using audio analysis software such as Affectiva. Through this audio analysis, the system grasps the user's emotional state in real time.

[0769] Based on the emotion recognition results, the server generates maintenance procedures, adjusts and optimizes them, and sends them to the robot's guidance system. This uses a generative AI model, which leverages prompts to design optimal guidance. Specific guidance is presented to the user as 3D models and visually clear instructions. A user-friendly voice interface also provides intuitive support.

[0770] For example, if an abnormal operation is detected in a machine on the production line, the server analyzes the abnormality with high accuracy and provides step-by-step guidance so that the operator can easily understand it. If the user's emotions are unstable, the AI ​​will provide further supplementary explanations using prompt messages.

[0771] As an example, here is an example of a prompt statement:

[0772] "Design an AI plan to create machine maintenance procedures within a factory that are adjusted based on the emotional state of the workers."

[0773] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0774] Step 1:

[0775] The terminal uses cameras and microphones mounted on robots patrolling the factory to acquire visual and auditory information in real time. The input data consists of surrounding video and audio, and the output is this digital data. The information acquired by the camera and microphone is detailed and wide-ranging.

[0776] Step 2:

[0777] The server receives visual information transmitted from the terminal and begins analysis using computer vision technology (such as OpenCV). The input is video data received from the terminal, and the output is information about the recognized anomalies. Specifically, it detects specific patterns or changes within the image and identifies equipment malfunctions.

[0778] Step 3:

[0779] The server uses speech analysis software (such as Affectiva) to analyze speech information. The input is speech data received from the terminal, and the output is the result of identifying the user's emotional state. It analyzes emotions from speech tone and manner of speaking and performs specific actions to determine whether the user is feeling tense or stressed.

[0780] Step 4:

[0781] The server generates maintenance procedures based on the analysis results and adjusts the procedures according to the user's emotional state. The input is information about the abnormal location and the emotional state, and the output is an optimized maintenance procedure. A generation AI model is used to design supplementary guidance with prompt messages, specifically to make the step order easier to understand and to add additional explanations.

[0782] Step 5:

[0783] The terminal receives optimized maintenance procedures generated by the server and presents them visually to the worker via an AR display. The input is the optimized maintenance procedure, and the output is a guide that the user can understand visually and audibly. Specifically, it uses 3D models and visual highlighting to show equipment repair procedures in animation.

[0784] Step 6:

[0785] The user performs the actual maintenance work by referring to the provided guide. The input is the guide from the terminal, and the final output is the equipment maintained in a normal state. If the user requires further assistance during the work, the terminal will request additional advice or further instructions as needed.

[0786] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0787] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0788] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0789] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0790] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0791] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0792] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0793] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0794] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0795] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0796] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0797] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0798] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0799] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0800] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0801] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0802] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0803] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0804] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0805] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0806] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0807] The following is further disclosed regarding the embodiments described above.

[0808] (Claim 1)

[0809] A device for acquiring visual data,

[0810] A means of identifying equipment abnormalities by analyzing acquired visual data,

[0811] A means for generating repair procedures based on analysis results,

[0812] A means for presenting the generated repair procedure to the worker via a display device,

[0813] A system that includes this.

[0814] (Claim 2)

[0815] The system according to claim 1, which updates the generated repair procedure based on additional data obtained in real time.

[0816] (Claim 3)

[0817] The system according to claim 1, further comprising means for analyzing audio data and identifying abnormal sounds.

[0818] "Example 1"

[0819] (Claim 1)

[0820] An information acquisition device that acquires visual data and additional information,

[0821] An analysis means for analyzing acquired information to identify equipment abnormalities,

[0822] A means for deriving repair procedures based on analysis results,

[0823] A means of providing the derived repair procedure to the operator through a display device,

[0824] A system that includes this.

[0825] (Claim 2)

[0826] The system according to claim 1, which adjusts the derived repair procedure based on additional information acquired in real time.

[0827] (Claim 3)

[0828] The system according to claim 1, further comprising means for analyzing sound wave data and identifying abnormal sounds.

[0829] "Application Example 1"

[0830] (Claim 1)

[0831] Means for collecting visual information,

[0832] A means for analyzing acquired visual information to identify abnormalities in equipment,

[0833] A means for generating repair procedures based on analysis results,

[0834] A means for presenting the generated procedure to the operator through a visual device,

[0835] A means to monitor the maintenance status of industrial machinery in real time and visually display repair guidelines in case of abnormalities,

[0836] A system that includes this.

[0837] (Claim 2)

[0838] The system according to claim 1, which updates the generated repair procedure based on additional information received in real time.

[0839] (Claim 3)

[0840] The system according to claim 1, further comprising means for analyzing audio information and identifying sounds indicating equipment malfunction.

[0841] "Example 2 of combining an emotion engine"

[0842] (Claim 1)

[0843] Equipment for acquiring digital images,

[0844] A means of identifying equipment abnormalities by analyzing acquired digital images,

[0845] A means for generating repair work procedures based on analysis results,

[0846] A means of showing the generated repair procedure to the worker through a visual presentation device,

[0847] A means of analyzing acoustic data and identifying abnormal sounds,

[0848] A means of analyzing the emotional state of workers,

[0849] A means of adjusting repair procedures based on emotional state,

[0850] A system that includes this.

[0851] (Claim 2)

[0852] The system according to claim 1, which updates the generated repair procedure based on additional information obtained in real time.

[0853] (Claim 3)

[0854] The system according to claim 1, which provides an audio guide that corresponds to the emotional state of the worker when presenting the generated repair procedure.

[0855] "Application example 2 when combining with an emotional engine"

[0856] (Claim 1)

[0857] A medium for acquiring visual information,

[0858] A means of analyzing acquired visual information to identify abnormalities in the device,

[0859] A means of analyzing voice information and recognizing the user's emotions,

[0860] A means for generating maintenance procedures based on analysis results,

[0861] A means for guiding the operator through a display device to generate maintenance procedures,

[0862] A means for optimizing guidance by adjusting maintenance procedures based on emotion recognition results,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, which updates the generated maintenance procedure based on additional information obtained in real time.

[0866] (Claim 3)

[0867] The system according to claim 1, further comprising means for analyzing audio information and identifying abnormal sounds. [Explanation of symbols]

[0868] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A device for acquiring visual data, A means of identifying equipment abnormalities by analyzing acquired visual data, A means for generating repair procedures based on analysis results, A means for presenting the generated repair procedure to the worker via a display device, A system that includes this.

2. The system according to claim 1, which updates the generated repair procedure based on additional data obtained in real time.

3. The system according to claim 1, further comprising means for analyzing audio data and identifying abnormal sounds.

Images