system

Abstract

Provide a system. 【Solution means】 Means for collecting real-time data from sensors and cameras mounted on construction machinery, Means for transmitting the collected real-time data to a control center via a wireless network, Means for analyzing the received real-time data, Artificial intelligence means for generating work instructions based on the analysis results, Means for notifying the generated work instructions to a remote operator, Means for recording the progress of work and the history of interactions during work, Means for generating a work report based on the recorded progress and history, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The improvement of on-site work efficiency and the integrated management of real-time data collection, analysis, instruction, and troubleshooting are particularly important issues in the operation of construction machinery. In the conventional system, manual data analysis and instruction transmission take time and are factors leading to a decline in work efficiency. In addition, although a prompt response is required when a problem occurs, it is difficult to immediately issue appropriate instructions according to the on-site situation. Therefore, there is a need for a system that can perform real-time work instructions and troubleshooting remotely in an integrated manner and realize efficient and accurate work reports.

Means for Solving the Problems

[0005] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. Furthermore, the control center analyzes the received data, and artificial intelligence (AI) generates work instructions based on the analysis results. These work instructions are notified to remote operators, enabling real-time work instructions and troubleshooting. In addition, the system records the progress of the work and the history of interactions during the work, and ultimately generates a work report based on the records. Through these means, the efficiency and accuracy of on-site work are improved.

[0006] "Construction machinery" is a general term for heavy equipment and machinery used at construction sites, and refers to devices used for tasks such as excavation, leveling, and transportation.

[0007] A "sensor" is a device that detects physical quantities and collects them as data, and is used to detect information such as temperature, pressure, vibration, and position.

[0008] A "camera" is a device used to capture video or images, and is used to monitor the situation at a construction site in real time.

[0009] "Real-time data" refers to information collected from sensors and cameras at the present moment, and is data that can be used immediately without delay.

[0010] A "wireless network" is a network technology that allows data to be transmitted without using cables, and is used for sending and receiving data between construction sites and control centers.

[0011] A "control center" is a facility or system that has the central function of processing and analyzing collected data, and issuing work instructions and troubleshooting.

[0012] "Analysis" is the act of thoroughly examining collected data and extracting meaningful information.

[0013] Artificial intelligence (AI) is a system that mimics human intelligence and automatically performs tasks such as data analysis, generation of work instructions, and troubleshooting.

[0014] "Work instructions" are instructions that show the specific way to proceed with a task and the steps involved, and they serve as guidelines for remote operators to follow when performing their work.

[0015] A "remote operator" is a worker who operates construction machinery based on instructions from a control center, even if they are not directly present at the site.

[0016] "Troubleshooting" is the process of identifying the cause of a problem or malfunction and providing an appropriate solution.

[0017] "Progress" refers to indicators or stages that show whether a task is progressing according to plan.

[0018] "History" refers to information that records past work instructions, troubleshooting details, and the progress of work.

[0019] A "work report" is a document that summarizes the progress, results, and troubleshooting of a series of tasks, and is used for reporting and record-keeping purposes. [Brief explanation of the drawing]

[0020] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5]It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0021] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0024] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0025] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0026] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0028] [First Embodiment]

[0029] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0030] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0033] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0036] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0040] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0041] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence (AI) generates work instructions based on the analysis results. These instructions are then notified to remote operators, enabling real-time work instructions and troubleshooting. Furthermore, the system records the progress of the work and the history of interactions, ultimately allowing for the generation of work reports based on these records.

[0042] Specific Examples of the System

[0043] 1. The terminal (construction machinery) is equipped with sensors and cameras to collect information such as ground conditions, equipment operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the cameras acquire visual information.

[0044] 2. The terminal packages the collected real-time data in a compressed format and transmits it to the control center via a wireless network (5G network) using a secure protocol.

[0045] 3. The server (control center) receives and decompresses data transmitted from terminals in real time and converts it into an analyzable format. The received data is then used for analysis, such as analyzing ground conditions and progress using image processing algorithms, and detecting anomalies from sensor data.

[0046] 4. The analyzed data is passed to artificial intelligence (AI), which compares it with the overall work plan to generate specific work instructions based on the analysis results. For example, it automatically creates instructions such as, "To excavate the next area A, excavate the path from position X to position Y."

[0047] 5. The server transmits the generated work instructions to the remote operator using a communication protocol and displays the instructions on the operator's terminal. The instructions are formatted in a clear, easy-to-understand text format.

[0048] 6. When a problem occurs, artificial intelligence (AI) will immediately analyze the situation and provide troubleshooting instructions, such as "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0049] 7. The server meticulously records each work instruction and its results, as well as the history of troubleshooting and its resolution. This record is stored in a database and includes information such as the progress of the work, the operations performed, and the instructions given by the generative AI and their results.

[0050] 8. Recorded progress and historical data are integrated by artificial intelligence (AI) to ultimately generate a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[0051] 9. The server sends a draft of the generated report to the user to assist with review and final editing. The user makes revisions as needed to complete the final work report.

[0052] In this way, the system of the present invention achieves increased efficiency and accurate recording and reporting of work at construction sites. Furthermore, real-time troubleshooting improves safety and work accuracy.

[0053] The following describes the processing flow.

[0054] Step 1:

[0055] The terminal collects real-time data such as ground conditions, machine operation status, and location information using sensors and cameras mounted on construction machinery.

[0056] Step 2:

[0057] The data collected by the device is packaged into a compressed format and transmitted to the control center via a wireless network (5G network) using a secure protocol.

[0058] Step 3:

[0059] The server receives data sent from the terminal in real time and converts the data into a format that can be decompressed and analyzed.

[0060] Step 4:

[0061] The server applies image processing and data analysis algorithms to analyze the received data, detecting ground conditions, machine operation status, and the presence or absence of abnormalities.

[0062] Step 5:

[0063] The generative AI generates specific work instructions based on the analysis results and the overall work plan. For example, it might create an instruction such as, "To excavate the next area A, excavate the path from position X to position Y."

[0064] Step 6:

[0065] The server notifies the remote operator of the generated work instructions. The notification is converted to the appropriate format and displayed on the operator's terminal.

[0066] Step 7:

[0067] When a problem occurs, the generative AI analyzes the situation and provides appropriate troubleshooting instructions. For example, it might send an instruction such as, "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0068] Step 8:

[0069] The server meticulously records each work instruction and its results, as well as a history of troubleshooting and its resolution, and stores this information in a database.

[0070] Step 9:

[0071] The generative AI generates a draft work report based on recorded progress and historical data. For example, it might write: "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[0072] Step 10:

[0073] The server sends a draft of the generated work report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[0074] (Example 1)

[0075] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0076] Modern construction sites demand increased work efficiency and real-time troubleshooting. However, conventional systems handle sensor data collection, analysis, work instruction generation, troubleshooting, and work report creation separately, making integrated work management difficult. Furthermore, the lack of real-time data transmission and immediate analysis hinders rapid response, raising concerns about impacting work progress and safety.

[0077] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0078] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery, means for packaging the collected real-time data in a compressed format and transmitting it to a control center via a wireless network, and means for decompressing the received real-time data and converting it into an analyzable format. This enables integrated and real-time data analysis and work instructions.

[0079] "Construction machinery" refers to large, specialized machines used in civil engineering and construction work.

[0080] A "sensor" is a device that measures physical environmental conditions (temperature, pressure, vibration, etc.) and acquires those values ​​as data.

[0081] A "camera" is a device that captures visual information from its surroundings and saves it in digital format.

[0082] "Real-time data" refers to data that instantly reflects the current state of affairs and is collected and analyzed without delay.

[0083] "Data acquisition means" refers to methods and devices used to acquire necessary data using sensors and cameras.

[0084] A "compression format" is a format that compresses information in order to reduce the data size.

[0085] A "wireless network" is a communication system that uses radio waves to send and receive data.

[0086] A "control center" is a central management hub that receives collected data, performs analysis, and generates instructions.

[0087] "Decompression means" refers to the process or apparatus for restoring compressed data to its original format.

[0088] "Analyzable format" refers to a format in which data is suitable for analysis and measurement after decompression.

[0089] "Analysis means" refers to methods and devices for evaluating collected data and extracting necessary information.

[0090] "Artificial intelligence means" refers to methods or devices that use AI technology to automatically provide work instructions and troubleshoot problems based on analysis results.

[0091] A "communication protocol" refers to the rules and procedures for sending and receiving data.

[0092] A "remote operator" refers to a technician or worker who operates equipment without being physically present at the site.

[0093] "Means for recording history" refers to methods and devices for saving and managing the progress and interactions of past work.

[0094] A "work report" is a document that summarizes the progress and results of a task.

[0095] "Troubleshooting" is the process of identifying the cause of a problem and providing a solution when one occurs.

[0096] "Means of supporting review" refer to methods or devices that enable users to review and correct generated reports, etc.

[0097] This invention is a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence generates work instructions based on the analysis results. These instructions are then notified to remote operators, enabling real-time work instructions and troubleshooting. Furthermore, the system records the progress of the work and the history of interactions, and ultimately generates a work report based on these records.

[0098] 1. Terminals (construction machinery)

[0099] The terminal is equipped with sensors and cameras to collect information such as ground conditions, equipment operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the camera acquires visual information. Specifically, the temperature sensor measures the ground temperature, and the pressure sensor measures the load of the heavy machinery. The camera captures video of the excavation area in real time.

[0100] 2. Data compression and transmission

[0101] The terminal converts the collected real-time data into a compressed format and transmits it to the control center via a wireless network (e.g., a 5G network) using a secure communication protocol. Specifically, it packages the data obtained from each sensor into a single file using a data compression algorithm and transmits the data via a 5G modem.

[0102] 3. Receiving and decompressing data

[0103] The server (control center) receives compressed data sent from the terminal. The received data is reconstructed using a decompression tool. The server's communication module takes in the received data, decompresses the ZIP file, and converts it into JSON format data.

[0104] 4. Data analysis and work instruction generation

[0105] The server passes the decompressed data to analysis tools and algorithms for analysis. For example, it uses an image processing library (e.g., OpenCV) to analyze the ground conditions and detect abnormal cracks and obstacles. The analyzed data is then passed to artificial intelligence (AI) to generate specific work instructions in conjunction with the overall work plan. For example, it automatically generates instructions such as "Excavate from position X to position Y in area A."

[0106] 5. Sending work instructions

[0107] The server sends the generated work instructions to the remote operator using a communication protocol (e.g., MQTT), and displays the instructions on the operator's terminal. The instructions are formatted in a specific text format. The remote operator checks the instructions sent from the server on their terminal's display and sees an instruction such as "Start drilling towards position X."

[0108] 6. Troubleshooting

[0109] Artificial intelligence (AI) instantly analyzes the situation when a problem occurs and provides real-time troubleshooting instructions. For example, it might instruct the machine to "raise the bucket and move backward if the heavy machinery gets stuck on the ground." When an anomaly is detected from sensor data, the AI ​​performs fault tree analysis and generates appropriate troubleshooting steps.

[0110] 7. Recording of work history

[0111] The server meticulously records each work order and its results, as well as the troubleshooting and resolution history, in a database. Work order and result data are saved to the SQL database using INSERT commands. Troubleshooting results are recorded similarly.

[0112] 8. Generation and review of work reports

[0113] Artificial intelligence (AI) integrates recorded progress and historical data to generate a draft work report. For example, it might create a report stating, "Today's excavation work was 80% complete, with two anomalies. The solutions are as follows..." The server sends the generated draft report to the user, who then reviews and makes final edits to complete the final work report. The draft report is generated in PDF format and sent to the user as an email attachment. The user then makes revisions using the editing tool.

[0114] Example of a prompt

[0115] "The construction machine recorded a ground temperature of 35 degrees Celsius from its ground temperature sensor, and the vibration sensor detected strong vibrations. Please use this data to generate the next work instruction."

[0116] In this way, the system of the present invention improves work efficiency and enables accurate recording and reporting at construction sites. Furthermore, real-time troubleshooting improves safety and work accuracy.

[0117] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0118] Specific processing steps of the system program

[0119] Step 1:

[0120] The terminal uses sensors and cameras to collect information such as ground conditions, equipment operating status, and location. Inputs are physical data measured by sensors (temperature, pressure, vibration, etc.) and visual data captured by the camera. Outputs are the collected sensor data and image data. Specifically, the temperature sensor measures the ground temperature, the pressure sensor measures the load of heavy machinery, and the camera acquires visual information.

[0121] Step 2:

[0122] The terminal packages the collected real-time data in a compressed format. The inputs are the sensor data and image data collected in step 1. A compression algorithm is used to efficiently transmit this data, and a compressed data package is obtained as the output. Specifically, the data compression algorithm is used to compress the data obtained from each sensor and package it into a single file.

[0123] Step 3:

[0124] The terminal transmits compressed data to the control center via a wireless network (e.g., a 5G network) using a secure communication protocol. The input is the compressed data generated in step 2. The output is the data transmitted to the control center. Specifically, the data is transmitted via a 5G modem.

[0125] Step 4:

[0126] The server (control center) receives compressed data sent from the terminal. The input is the compressed data sent from the terminal. The output is the received compressed data. Specifically, the server's communication module takes in the received data.

[0127] Step 5:

[0128] The server reconstructs the received compressed data using a decompression tool and converts it into an analyzable format. The input is the compressed data received in step 4. The output is the decompressed and analyzable data. Specifically, it decompresses a ZIP file and converts the data into JSON format.

[0129] Step 6:

[0130] The server passes the decompressed data to analysis tools and algorithms for analysis. The input is the analyzable data decompressed in step 5. The output is the analyzed data and results. Specifically, it uses an image processing library (e.g., OpenCV) to analyze the ground conditions and detect abnormal cracks and obstacles.

[0131] Step 7:

[0132] Artificial intelligence (AI) generates specific work instructions by referring to the analysis results and the overall work plan. The input is the analysis results and data obtained in step 6. The output is the generated work instructions. Specifically, it reads the current progress from the work plan database and generates the optimal work instructions by comparing them with the analysis results.

[0133] Step 8:

[0134] The server sends the generated work instructions to the remote operator using a communication protocol and displays the instructions on the operator's terminal. The input is the work instructions generated in step 7. The output is the work instructions sent to the remote operator. Specifically, the work instructions are converted to JSON format and sent to the operator's tablet terminal via a communication protocol (e.g., MQTT).

[0135] Step 9:

[0136] Artificial intelligence (AI) immediately analyzes the situation when a problem occurs and provides troubleshooting instructions. The input is sensor data and image data collected in real time. The output is specific troubleshooting instructions. Specifically, when an anomaly is detected from the sensor data, the AI ​​performs fault tree analysis and generates appropriate troubleshooting steps.

[0137] Step 10:

[0138] The server meticulously records each work order and its results, as well as troubleshooting and its resolution history, in a database. Inputs include work orders and their results, and troubleshooting steps and results. Outputs are the historical data stored in the database. Specifically, the work order and result data are saved to the SQL database using INSERT commands.

[0139] Step 11:

[0140] Artificial intelligence (AI) integrates recorded progress and historical data to generate a draft work report. The input is progress and historical data recorded in a database. The output is a draft work report. Specifically, it uses natural language generation (NLG) technology to create the report.

[0141] Step 12:

[0142] The server sends a draft of the generated report to the user for review and final editing. The input is the draft of the work report generated in step 11. The output is the draft report sent to the user. Specifically, the server generates the draft report in PDF format and sends it to the user as an email attachment.

[0143] (Application Example 1)

[0144] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0145] There were challenges with conventional technology in improving the operational efficiency of industrial machinery and detecting and responding quickly to anomalies in real time. Furthermore, the generation of response instructions and work reports in the event of an anomaly was not automated, resulting in time losses and human errors.

[0146] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0147] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on industrial machinery, means for transmitting the collected real-time data to a control center via a wireless network, means for analyzing the received real-time data, artificial intelligence means for generating work instructions based on the analysis results, means for notifying a remote operator of the generated work instructions, means for recording the progress of work and the history of interactions during work, means for generating a work report based on the recorded progress and history, means for detecting abnormalities during work and immediately generating response instructions, and means for processing image data acquired by cameras mounted on industrial machinery in real time and detecting abnormal locations. This improves the work efficiency of industrial machinery and enables real-time abnormality detection and rapid response.

[0148] "Industrial machinery" is a general term for automated machines and equipment used in manufacturing and production lines.

[0149] A "sensor" is a device that measures physical quantities such as temperature, pressure, and vibration, and converts them into electrical signals.

[0150] A "camera" is a device that can capture visual information and record and transmit it as digital data.

[0151] "Real-time data" refers to data that indicates the current state of a machine while it is in operation and can be processed or transmitted immediately.

[0152] A "wireless network" is a communication method that uses radio waves to transmit data, and specifically includes Wi-Fi and 5G.

[0153] A "control center" is a facility or system for centrally managing, analyzing, and controlling data from industrial machinery.

[0154] "Means of analysis" refer to processes and devices for breaking down and analyzing received data and extracting useful information.

[0155] "Artificial intelligence means" refers to algorithms and systems that automatically generate judgments and instructions based on collected data.

[0156] A "remote operator" is a person or device that operates a machine or system from a physically distant location.

[0157] "Work progress" refers to information indicating the extent to which planned work has been completed.

[0158] "History" refers to the record of all past work and communications.

[0159] A "work report" is a document that includes details such as the progress of the work and how any problems were handled.

[0160] "Means for detecting abnormalities" refers to a device or method for detecting a state that deviates from normal operating conditions.

[0161] "Means for generating response instructions" refers to a device and algorithm that automatically determines an appropriate response method for a detected anomaly and outputs it as an instruction.

[0162] "Means for processing image data" refers to software and hardware used to analyze images captured by a camera and extract useful information.

[0163] This invention relates to a system that collects real-time data from sensors and cameras mounted on industrial machinery and transmits that data to a control center via a wireless network. Specifically, it is implemented as follows.

[0164] First, industrial machinery is equipped with various sensors such as temperature sensors, pressure sensors, and vibration sensors, as well as cameras. These sensors measure the machine's operating status and environmental conditions in real time, while the cameras capture visual information of the work area. The data collected from the sensors and cameras undergoes basic processing and compression at the terminal.

[0165] Next, the terminal transmits the collected data to the control center via a wireless network (e.g., Wi-Fi or 5G). A server located at the control center decompresses the received data and converts it into an analyzable format. The software used includes libraries for data decompression (e.g., zlib) and libraries for image analysis (e.g., OpenCV, TENSORFLOW®).

[0166] Next, the data, converted into an analyzable format, is analyzed by artificial intelligence (AI). The AI, for example, uses image processing algorithms to detect anomalies in the work area or applies anomaly detection algorithms to sensor data. The analysis results are compared with the overall work plan, and specific work instructions are generated by the AI ​​model. These AI models function as generative AI models, employing, for example, deep learning algorithms.

[0167] The generated work instructions are sent from the server to the remote operator via a wireless network. The instructions are displayed in a specific text format on the terminal used by the operator. For example, an instruction such as "Excavate the path from position X to position Y in order to excavate the next area B" might be issued.

[0168] Furthermore, if an anomaly occurs during operation, the system immediately detects the anomaly and generates a response instruction. This instruction is also notified to the remote operator. For example, a specific response instruction such as "The machine has detected an abnormal temperature, so stop it immediately and begin cooling" is provided.

[0169] All work instructions, their results, and the history of anomaly responses are recorded in a database. This recorded data is later integrated by AI and generated as a work report. The report includes work progress, any anomalies that occurred, and how they were addressed. Users can review the generated report and make revisions as needed.

[0170] Specific example

[0171] Examples of prompts to input into a generative AI model:

[0172] This is real-time monitoring data from automated robots operating within the factory.

[0173] Sensor data: Temperature 30°C, Pressure 1.5, Vibration 0.3

[0174] Camera image: base64_encoded_image_data

[0175] Analysis results: An anomaly was detected in a part of the work line, so the following instructions were generated.

[0176] Instructions: Robot 1 should continue its work, avoiding sensor anomaly area B, and send an alert to the person in charge.

[0177] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0178] Step 1:

[0179] The terminal collects real-time data from sensors and cameras mounted on industrial machinery. Sensors measure physical quantities such as temperature, pressure, and vibration, while cameras capture images of the work area. The collected data is stored as sensor data (temperature, pressure, vibration, etc.) and image data (JPEG format).

[0180] Step 2:

[0181] The terminal performs basic processing on the collected real-time data and then compresses it. For example, it might use the zlib library to compress the data. The compressed data is then prepared to be transmitted over the wireless network as data packets.

[0182] Step 3:

[0183] The device transmits data packets to the control center via a wireless network (e.g., Wi-Fi or 5G). Wireless communication protocols are used, and data security is protected by encryption technology.

[0184] Step 4:

[0185] The server receives the data packets sent from the terminal and decompresses them. This restores the sensor data and image data to their original formats. The zlib library is used here as well.

[0186] Step 5:

[0187] The server analyzes the decompressed data. An anomaly detection algorithm is applied to the sensor data, and image processing algorithms such as OpenCV or TensorFlow are used to detect anomalies in the image data. The analysis results are saved for the next processing step.

[0188] Step 6:

[0189] The server uses a generated AI model to create specific work instructions based on the analysis results. For example, using a deep learning algorithm, it might generate an instruction such as, "To excavate the next area B, excavate a path from position X to position Y." This instruction is formatted in text format.

[0190] Step 7:

[0191] The server notifies the remote operator of the generated work instructions via the wireless network. The instructions are displayed in specific text format on the terminal used by the operator. For example, the instructions may include "Stop the pump and start cooling."

[0192] Step 8:

[0193] The server records the progress of the work and the history of interactions during the work in a database. The recorded data includes each work instruction, its execution result, and a history of how errors were handled.

[0194] Step 9:

[0195] The server generates work reports based on recorded progress and historical data. AI integrates this data and creates a draft of the work report. For example, the report might be in the format of, "Today's work was 90% complete, and 3 anomalies occurred. The specific actions taken are as follows..."

[0196] Specific example

[0197] Examples of prompts to input into a generative AI model:

[0198] This is real-time monitoring data from automated robots operating within the factory.

[0199] Sensor data: Temperature 30°C, Pressure 1.5, Vibration 0.3

[0200] Camera image: base64_encoded_image_data

[0201] Analysis results: An anomaly was detected in a part of the work line, so the following instructions were generated.

[0202] Instructions: Robot 1 should continue its work, avoiding sensor anomaly area B, and send an alert to the person in charge.

[0203] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0204] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence (AI) generates work instructions based on the analysis results. The generated work instructions are notified to the remote operator, enabling real-time work instructions and troubleshooting. The system also records the progress of the work and the history of interactions, and ultimately generates a work report based on the records. Furthermore, this invention incorporates an emotion engine to recognize the user's emotions and adjust the content of work instructions and troubleshooting accordingly.

[0205] Specific Examples of the System

[0206] 1. The terminal (construction machinery) is equipped with sensors and cameras to collect information such as ground conditions, machine operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the cameras acquire visual information.

[0207] 2. The terminal packages the collected real-time data into a compressed format and transmits it to the control center via a wireless network (5G network) using a secure protocol.

[0208] 3. The server (control center) receives data transmitted from terminals in real time and converts the data into a format that can be decompressed and analyzed. The received data is then used for analysis, such as analyzing ground conditions and progress using image processing algorithms, and detecting anomalies from sensor data.

[0209] 4. The analyzed data is passed to artificial intelligence (AI), which compares it with the overall work plan to generate specific work instructions based on the analysis results. For example, it automatically creates instructions such as, "To excavate the next area A, excavate the path from position X to position Y."

[0210] 5. The server notifies the remote operator of the generated work instructions using a communication protocol and displays the instructions on the operator's terminal. The instructions are formatted in a clear, easy-to-understand text format.

[0211] 6. When a problem occurs, the generative AI immediately analyzes the situation and provides troubleshooting instructions, such as "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0212] 7. The server meticulously records and stores in a database the history of each work instruction and its results, as well as troubleshooting and its resolution.

[0213] 8. Recorded progress and historical data are integrated by artificial intelligence (AI) to ultimately generate a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[0214] 9. The server sends a draft of the generated report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[0215] Specific examples of the emotion engine

[0216] 10. The device incorporates an emotion engine that recognizes the user's emotions, analyzing the user's tone of voice, facial expressions, and operation speed during operation to monitor their emotional state in real time.

[0217] 11. The emotion engine collects and analyzes the user's emotional data. For example, if the user's stress level is high or they are confused, the emotion engine recognizes that state.

[0218] 12. The analysis results are sent to the server and used to adjust real-time work instructions and troubleshooting content. For example, if the user is under high stress, the instructions may be made simpler and support messages may be inserted.

[0219] 13. The results of the emotion engine analysis are also reflected in the work report. For example, an assessment of the user's emotional state, such as "The user experienced a high level of stress during today's work and therefore requires support," is added to the report.

[0220] In this way, the system of the present invention not only improves work efficiency, accuracy, and safety at construction sites, but also enables adjustments and support that take into account the user's emotional state. A key feature is that the emotion engine can grasp the user's stress and satisfaction levels in real time, thereby reducing their burden.

[0221] The following describes the processing flow.

[0222] Step 1:

[0223] The terminal (construction machinery) collects real-time data, including ground conditions at the construction site, equipment operating status, and location information, using sensors and cameras. Sensors capture information such as temperature, pressure, and vibration, while cameras acquire visual information.

[0224] Step 2:

[0225] The real-time data collected by the terminal is packaged into a compressed format and transmitted to the control center via a wireless network (5G network) using a secure protocol.

[0226] Step 3:

[0227] The server (control center) receives data sent from the terminal in real time, decompresses the data, and converts it into an analyzable format.

[0228] Step 4:

[0229] The server analyzes the received data, uses image processing algorithms to check the ground conditions and progress, and detects any abnormalities from the sensor data.

[0230] Step 5:

[0231] The generative AI generates specific work instructions based on the analysis results and the overall work plan. For example, it might create an instruction such as, "To excavate the next area A, excavate the path from position X to position Y."

[0232] Step 6:

[0233] The server generates work instructions and notifies the remote operator using a communication protocol, displaying the instructions on the operator's terminal. This ensures that instructions are provided in a specific and easy-to-understand format.

[0234] Step 7:

[0235] When a problem occurs, the generative AI immediately analyzes the situation and provides appropriate troubleshooting instructions. For example, it might issue an instruction such as, "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0236] Step 8:

[0237] The emotion engine collects user emotion data. It monitors the user's emotional state in real time based on factors such as tone of voice, facial expressions, and operation speed, and generates emotion data.

[0238] Step 9:

[0239] The emotion engine analyzes collected emotional data to assess whether the user is experiencing high stress or satisfaction. The analysis results are sent to the server and used to adjust work instructions and troubleshooting content.

[0240] Step 10:

[0241] The server adjusts work instructions and troubleshooting based on emotional data. For example, if a user is in a high-stress state, it will simplify the instructions and add supportive messages.

[0242] Step 11:

[0243] The server meticulously records and stores in a database each work order and its results, troubleshooting history and resolution, and emotional state.

[0244] Step 12:

[0245] The generative AI generates a draft work report based on recorded progress and history data, as well as sentiment data. For example, it might write, "Today's excavation work is 80% complete, and two anomalies occurred. The solutions are as follows..." and include an assessment of the user's sentiment state.

[0246] Step 13:

[0247] The server sends a draft of the generated report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[0248] (Example 2)

[0249] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0250] Real-time data collection and analysis are essential for highly efficient and precise work on construction sites. However, conventional systems often suffer from delays in data collection, analysis, and the generation and notification of work instructions. Furthermore, instructions are not adjusted to take into account the user's emotional state, leading to challenges in operational efficiency and user stress management. These challenges need to be addressed.

[0251] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0252] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery; means for compressing the collected real-time data and transmitting it to a control center via a wireless network; means for converting the received real-time data into a format that can be decompressed and analyzed; artificial intelligence means for generating specific work instructions based on the analyzed data; means for notifying a remote operator of the generated work instructions; means for recording the progress of the work and the history of interactions during the work; means for generating a work report based on the recorded progress and history; means for collecting and analyzing user emotion data; and means for adjusting work instructions based on emotion data. This enables the provision of efficient and highly accurate work instructions in real time, as well as the adjustment of work instructions that take into account the user's emotional state, thereby improving work efficiency and safety.

[0253] "Construction machinery" refers to mechanical devices used to perform civil engineering work, excavation, and other tasks at construction sites.

[0254] A "sensor" is a device that detects the surrounding environment or the state of a machine and outputs that information as an electrical signal.

[0255] A "camera" is a device that captures visual information and records it as digital data.

[0256] "Real-time data" refers to data that is instantly acquired and processed with virtually no delay, reflecting the current state and situation.

[0257] A "wireless network" is a communication network that uses radio waves to send and receive data.

[0258] A "control center" is a base of operations for monitoring and controlling equipment and systems from a remote location.

[0259] "Data compression" is the process of converting large amounts of data so that they can be stored in less memory.

[0260] "Decompression" is the process of restoring compressed data to its original state.

[0261] An "analyzable format" is a format that converts data into a form that can be processed by analytical tools and algorithms.

[0262] "Artificial intelligence means" refers to technologies that analyze large amounts of data, learn from it, and generate appropriate results or instructions.

[0263] "Work instructions" are detailed procedures or commands for performing a specific task.

[0264] A "remote operator" refers to a person or system that operates remotely, without being physically present at the site.

[0265] "Troubleshooting" is the process of diagnosing machine malfunctions or abnormalities and proposing solutions.

[0266] "History" refers to a record of past work and interactions.

[0267] A "work report" is a document that summarizes the work performed, its progress, results, and any problems encountered.

[0268] "Emotional data" refers to information that indicates a user's emotional state, and is data obtained from sources such as voice, facial expressions, and behavior.

[0269] An "emotion engine" is a technology that analyzes a user's emotional data and evaluates their emotional state.

[0270] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery, transmits that data to a control center via a wireless network, and generates work instructions using artificial intelligence (AI) based on the analysis results. This improves the work efficiency and safety at construction sites.

[0271] Terminal hardware and software

[0272] The terminal (construction machinery) is equipped with multiple sensors (e.g., temperature sensors, pressure sensors, vibration sensors) and cameras. These sensors collect information such as ground conditions, machine operation status, and location in real time. The cameras acquire visual data using color and depth cameras. The collected data is compressed using dedicated data compression software (e.g., ZIP).

[0273] Sending data

[0274] The device transmits compressed data to the control center via the 5G network using a secure communication protocol (e.g., HTTPS). This communication ensures both data transmission speed and security.

[0275] Server hardware and software

[0276] The control center is equipped with high-performance servers that receive data transmitted via communication in real time. The received data is decompressed using data decompression software (e.g., Unzip) and converted into an analyzable format. Subsequently, various analysis algorithms (e.g., image processing algorithms, anomaly detection algorithms) are applied to analyze the data.

[0277] AI-generated work instructions

[0278] Based on the analyzed data, artificial intelligence (AI) generates appropriate work instructions. This AI takes into account the progress and conditions of a specific task and gives specific instructions for the next step. For example, it might generate an instruction such as, "To excavate the next area A, excavate from position X to position Y."

[0279] Work instruction notification

[0280] The server notifies the remote operator of the generated work instructions and displays the instructions on the operator's terminal. These instructions are displayed in text format and are easy to understand.

[0281] troubleshooting

[0282] If a problem occurs during operation, the terminal notifies the server of the situation. The server immediately analyzes the data, and the AI ​​generates appropriate troubleshooting steps and notifies the operator. For example, it provides specific instructions such as, "If the heavy machinery gets stuck, raise the bucket and move backward."

[0283] Data recording and work report generation

[0284] The server records in detail each work instruction and its result, as well as the history of troubleshooting and its resolution, and stores them in a database. Based on these records, the AI generates a draft of the work report. For example, it generates a report in the form of "Today's excavation work is 80% complete as planned, and 2 anomalies have occurred. The solutions are as follows...". The server sends this draft report to the user, who reviews it and makes final edits to complete the work report.

[0285] Collection and Analysis of Emotional Data

[0286] The terminal is equipped with an emotion engine that recognizes the user's emotions. It analyzes the tone of the user's voice, facial expressions, operation speed, etc. to monitor the emotional state in real time. The emotion engine collects these data and sends the analysis results to the server.

[0287] Adjustment of Work Instructions Considering Emotional Data

[0288] Based on the analysis results sent from the emotion engine, the server adjusts the content of real-time work instructions and troubleshooting. For example, when the user is in a high-stress state, the instruction content is made concise and support messages are inserted.

[0289] By these technical means, this system can significantly improve work efficiency and safety at the construction site. Furthermore, by considering the user's emotional state, it is possible to reduce stress and provide more appropriate support.

[0290] Examples of Prompt Sentences

[0291] For example, it includes instructions such as "Please excavate the path from position X to position Y to excavate the next area A", and troubleshooting instructions such as "When the heavy equipment is stacked on the ground, raise the bucket and reverse".

[0292] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0293] Step 1:

[0294] Data collection from sensors and cameras using the device

[0295] The device collects real-time data using various built-in sensors (e.g., temperature sensor, pressure sensor, vibration sensor) and a camera. Inputs include the surrounding environment and the operating status of the machine. Sensor data is output in the form of temperature, pressure, vibration, and location information, while the camera captures visual data. The device collects this data and temporarily stores it in its internal storage.

[0296] Specific example: "The device measures the ambient temperature with a temperature sensor and takes pictures of the terrain with a camera."

[0297] Step 2:

[0298] Data compression and transmission by the terminal

[0299] To compress the collected data, the terminal applies a compression algorithm (e.g., ZIP). The inputs are the sensor data and camera data obtained in step 1. The compressed data is sent to the server via the 5G network using a secure communication protocol (e.g., HTTPS). This results in the compressed data being the output of the transmission.

[0300] Specific example: "The device compresses the collected data in ZIP format and sends it to the server using the HTTPS protocol."

[0301] Step 3:

[0302] Receiving and decompressing data by the server.

[0303] The server receives the compressed data transmitted from the terminal. The input is data in a compressed state. After receiving, the server uses data decompression software (e.g., Unzip) to decompress the data and convert it into an analyzable format. As a result, the decompressed data becomes the output.

[0304] Specific example: "The server receives the ZIP file sent from the terminal and decompresses it with Unzip software."

[0305] Step 4:

[0306] Analysis of data by the server

[0307] Based on the decompressed data, the server performs analysis by applying image processing algorithms or anomaly detection algorithms. The input is the decompressed sensor data and camera data. Through the analysis process, analysis results such as the unevenness of the terrain, abnormal temperature points, and pressure fluctuations are output.

[0308] Specific example: "The server uses an image processing algorithm to detect the unevenness of the terrain and an anomaly detection algorithm to identify abnormal points in the temperature data."

[0309] Step 5:

[0310] Generation of work instructions by AI

[0311] Based on the analysis results, the AI generates specific work instructions. The input is the analysis results obtained in Step 4. The AI outputs work instructions such as "To excavate the next area A, please excavate from position X to position Y" in light of the overall work plan.

[0312] Specific example: "The AI indicates a specific route for excavating the next area A based on the analysis results."

[0313] Step 6:

[0314] Notification of work instructions by the server

[0315] The server notifies the remote operator of the generated work instructions. The input is the work instructions obtained in step 5. The work instructions are displayed in text format and sent to the remote control terminal. This allows the operator to check the instructions in real time. The output is the notified work instructions.

[0316] Specific example: "The server notifies the operator of the generated work instructions via SMS and displays the instructions on the terminal's screen."

[0317] Step 7:

[0318] Terminal and server troubleshooting

[0319] If a problem occurs, the terminal notifies the server of the situation. The input is data about the problem. The server immediately analyzes this data, and the AI ​​generates appropriate troubleshooting steps. For example, it might output specific instructions such as, "If the heavy machinery gets stuck, raise the bucket and move backward."

[0320] Specific example: "After receiving abnormal data from a terminal, the server uses AI to generate an instruction to 'raise the bucket and move backward,' and notifies the operator."

[0321] Step 8:

[0322] Data recording by the server

[0323] The server meticulously records each work order and its results, as well as troubleshooting and its resolution history, and stores it in a database. Inputs are work orders, their execution results, and troubleshooting history. The recorded data is then output.

[0324] Specific example: "The server records all work orders and their execution results in the SQL database."

[0325] Step 9:

[0326] AI-powered work report generation

[0327] Based on the recorded data, the AI ​​generates a draft of the work report. The input is the progress data and historical data recorded in step 8. For example, a draft report in the format of "Today's excavation work was 80% complete and two anomalies occurred" will be output.

[0328] Specific example: "The AI ​​analyzes all historical data and generates a draft of the work report titled 'Today's Work Progress and Anomaly Response List'."

[0329] Step 10:

[0330] Server-based report submission and user review.

[0331] The server sends a draft of the generated report to the user. The input is the draft work report generated in step 9. The user reviews and edits it to complete the final work report. The output is the completed work report.

[0332] Specific example: "The server emails a draft of the report to the user, and saves the final version after the user has made edits."

[0333] Step 11:

[0334] Collection of emotional data using devices

[0335] The device uses an emotion engine to analyze the user's voice tone, facial expressions, and operation speed, and collects emotional data. The input is the user's voice and facial expressions during operation. This data is analyzed by the emotion engine, and emotional data is output.

[0336] Specific example: "The device captures the user's voice tone with a microphone and analyzes their facial expressions with a camera."

[0337] Step 12:

[0338] Analysis of emotional data using an emotion engine

[0339] The emotion engine analyzes the collected emotion data to evaluate the user's stress level and emotional state. The input is the emotion data collected in step 11. For example, if the tone of voice is high or the facial expression is strained, it will be evaluated as a "high-stress state," and the result will be output.

[0340] Specific example: "The emotion engine detects that the user's voice tone is high and determines that they are in a high-stress state."

[0341] Step 13:

[0342] Adjusting work instructions based on emotional data from the server.

[0343] The server adjusts real-time work instructions and troubleshooting content based on the analysis results of the emotion engine. The input is the analysis results of the emotion data obtained in step 12. For example, if the user is in a high-stress state, adjusted instructions will be output, with the instructions made simpler and support messages added.

[0344] Specific example: "The server will take into account the high-stress state detected by the emotion engine and add a support message such as 'Please remain calm while operating,' along with concise instructions."

[0345] Through the steps described above, this system provides real-time, efficient, and highly accurate work instructions, and supports operations while taking the user's emotional state into consideration, thereby improving work efficiency and safety at construction sites.

[0346] (Application Example 2)

[0347] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0348] Improving work efficiency and speeding up troubleshooting are crucial challenges in existing factory and construction sites. However, conventional systems lack sufficient real-time data collection and analysis, and in particular, they fail to adjust instructions to take into account the emotions and stress levels of operators. This makes it difficult to monitor work progress and respond immediately to problems, potentially increasing operator stress and confusion.

[0349] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery, means for transmitting the collected real-time data to a control center via a wireless network, means for analyzing the received real-time data, artificial intelligence means for generating work instructions based on the analysis results, means for notifying a remote operator of the generated work instructions, means for recording the progress of the work and the history of interactions during the work, means for generating a work report based on the recorded progress and history, emotion engine means for monitoring the emotional state of the operator, and means for adjusting work instructions based on the emotion data generated by the emotion engine means. This enables not only efficient real-time data collection and analysis and troubleshooting, but also flexible instructions that take into account the operator's emotions.

[0350] A "sensor" is a device that measures physical conditions and outputs them as electrical signals.

[0351] A "camera" is a device that receives light, forms an image, and stores or transmits it as digital data.

[0352] "Real-time data" refers to data that is acquired and transmitted instantaneously without delay.

[0353] A "wireless network" is a means of communication that uses radio waves to send and receive data.

[0354] A "control center" is a centralized control system or location for analyzing and managing collected data.

[0355] "Analysis" is the process of analyzing data and extracting meaning as information.

[0356] "Artificial intelligence" is a technology that uses computer programs to mimic human intelligence, learn from data, and generate work instructions.

[0357] A "remote operator" refers to a person or device that operates or manages a machine or system from a remote location.

[0358] "History" refers to data that records the progress and results of a task or event in chronological order.

[0359] A "work report" is a document that records and summarizes the progress, results, and details of any problems that occurred during a series of tasks.

[0360] An "emotion engine" is software that analyzes the user's voice tone, facial expressions, and operation speed to monitor the user's emotional state in real time.

[0361] This invention relates to a system that improves the work efficiency of robots in a factory and provides instructions that take into account the operator's emotions. Specific embodiments for carrying out this invention are described below.

[0362] Hardware and software

[0363] Hardware:

[0364] Smartphone (iOS / ANDROID®): Used by operators to receive instructions.

[0365] Factory robots: Devices for operating machinery, equipped with numerous sensors and cameras.

[0366] software:

[0367] Languages: Python, Java (registered trademark) (Android), Swift (iOS)

[0368] Frameworks: TensorFlow (AI analysis), OpenCV (image processing), Flask (backend API)

[0369] Network: 5G, Wi-Fi

[0370] Data collection

[0371] There are methods for collecting real-time data from factory robots' sensors and cameras. Sensors measure physical quantities such as temperature, pressure, and vibration, while cameras acquire visual data. This real-time data is transmitted to a server via a wireless network (e.g., 5G).

[0372] Data Analysis

[0373] The server has means to analyze the received real-time data. For example, sensor data is analyzed using an anomaly detection algorithm, and visual data acquired from cameras is analyzed using an image processing algorithm. Software tools such as TensorFlow and OpenCV are used in this analysis process.

[0374] Work order generation

[0375] The server has an artificial intelligence system that generates work instructions based on the analysis results. The AI ​​model compares the analysis results with the overall work plan and generates specific work instructions. For example, it might create instructions such as, "Investigate the following area."

[0376] Instructions, notifications, and coordination

[0377] The generated work instructions are notified to the remote operator using a communication protocol. Furthermore, the server has an emotion engine that monitors the operator's emotional state and adjusts the work instructions based on the emotional data generated by the emotion engine. If the operator is under stress, adjustments are made, such as simplifying the instructions and inserting support messages.

[0378] History recording and report generation

[0379] The server has means to record the progress of work and the history of interactions during the work in detail. It also has means to generate work reports based on the recorded progress and history data. For example, it can automatically generate a report such as, "Today's work was 80% completed as planned, and two anomalies occurred."

[0380] As a concrete example, the following prompt statements can be used:

[0381] "Develop an AI system for monitoring factory work progress and troubleshooting. Collect real-time data from sensors and cameras, analyze it on a remote server to generate work instructions, and adjust those instructions based on user sentiment."

[0382] This enables real-time data collection and analysis, more efficient troubleshooting, and flexible instructions that take operator emotions into consideration.

[0383] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0384] Step 1:

[0385] The terminal (factory robot) collects real-time data such as ground conditions, machine operating status, and location information from its mounted sensors and cameras. The sensors measure physical quantities such as temperature, pressure, and vibration, while the cameras capture visual data. The collected data is packaged in a compressed format. The input consists of machine physical quantities and image data, and the output is real-time data in a compressed format.

[0386] Step 2:

[0387] The terminal transmits collected real-time data to a server via a wireless network (e.g., 5G). A secure protocol is used for transmission, ensuring data confidentiality and integrity. The input is real-time data in a compressed format, and the output is a notification that the data transmission to the server is complete.

[0388] Step 3:

[0389] The server converts received real-time data into a format that can be decompressed and analyzed. To convert it into the format required for analysis, it separates sensor data and image data from a specific data stream and uses them for their respective processing. The input is compressed data, and the output is decompressed sensor data and image data.

[0390] Step 4:

[0391] The server analyzes the decompressed sensor data and image data. First, it applies an anomaly detection algorithm to the sensor data to detect patterns that are different from the norm. Next, it performs image processing on the image data using OpenCV or similar tools to extract specific visual information. The input is the decompressed sensor data and image data, and the output is the anomaly detection results and the visual information extraction results.

[0392] Step 5:

[0393] The server uses an artificial intelligence model (generative AI model) to generate work instructions based on the analysis results. Based on the analysis results and the overall work plan, the AI ​​model generates specific work instructions. For example, it might generate an instruction such as "Investigate the next area A." The input is the anomaly detection results and the visual information extraction results, and the output is the generated work instructions.

[0394] Step 6:

[0395] The server notifies the remote operator's smartphone of the generated work instructions. It uses a communication protocol to notify the instructions in an easily understandable format (e.g., text message). The input is the generated work instructions, and the output is the instruction notification to the remote operator's terminal.

[0396] Step 7:

[0397] The server uses an emotion engine to monitor the operator's emotional state. It analyzes the operator's voice tone, facial expressions, and operating speed, collecting emotional data in real time. The input is the operator's actions and voice data, and the output is the analyzed emotional data.

[0398] Step 8:

[0399] The server adjusts work instructions based on emotional data generated by the emotion engine. For example, if an operator is under high stress, the instructions are simplified and supportive messages are inserted. The input is the analyzed emotional data and existing work instructions, and the output is the adjusted work instructions.

[0400] Step 9:

[0401] The server meticulously records the progress of work and the history of interactions. This history includes data analysis results, work instructions, and sentiment data for each step. Inputs are various analysis results and instructions, while output is a detailed history record.

[0402] Step 10:

[0403] The server generates work reports based on recorded progress and historical data. These reports include progress status, occurrences and responses to anomalies, and the operator's emotional state. For example, it might automatically generate a report stating, "Today's work was 80% complete, and two anomalies occurred." The input is detailed historical records, and the output is the work report.

[0404] As described above, by realizing this invention, it is possible to improve work efficiency and reduce the mental burden on operators.

[0405] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0406] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0407] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0408] [Second Embodiment]

[0409] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0410] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0411] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0412] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0413] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0414] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0415] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0416] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0417] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0418] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0419] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0420] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0421] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence (AI) generates work instructions based on the analysis results. These instructions are then notified to remote operators, enabling real-time work instructions and troubleshooting. Furthermore, the system records the progress of the work and the history of interactions, ultimately allowing for the generation of work reports based on these records.

[0422] Specific Examples of the System

[0423] 1. The terminal (construction machinery) is equipped with sensors and cameras to collect information such as ground conditions, equipment operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the cameras acquire visual information.

[0424] 2. The terminal packages the collected real-time data in a compressed format and transmits it to the control center via a wireless network (5G network) using a secure protocol.

[0425] 3. The server (control center) receives and decompresses data transmitted from terminals in real time and converts it into an analyzable format. The received data is then used for analysis, such as analyzing ground conditions and progress using image processing algorithms, and detecting anomalies from sensor data.

[0426] 4. The analyzed data is passed to artificial intelligence (AI), which compares it with the overall work plan to generate specific work instructions based on the analysis results. For example, it automatically creates instructions such as, "To excavate the next area A, excavate the path from position X to position Y."

[0427] 5. The server transmits the generated work instructions to the remote operator using a communication protocol and displays the instructions on the operator's terminal. The instructions are formatted in a clear, easy-to-understand text format.

[0428] 6. When a problem occurs, artificial intelligence (AI) will immediately analyze the situation and provide troubleshooting instructions, such as "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0429] 7. The server meticulously records each work instruction and its results, as well as the history of troubleshooting and its resolution. This record is stored in a database and includes information such as the progress of the work, the operations performed, and the instructions given by the generative AI and their results.

[0430] 8. Recorded progress and historical data are integrated by artificial intelligence (AI) to ultimately generate a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[0431] 9. The server sends a draft of the generated report to the user to assist with review and final editing. The user makes revisions as needed to complete the final work report.

[0432] In this way, the system of the present invention achieves increased efficiency and accurate recording and reporting of work at construction sites. Furthermore, real-time troubleshooting improves safety and work accuracy.

[0433] The following describes the processing flow.

[0434] Step 1:

[0435] The terminal collects real-time data such as ground conditions, machine operation status, and location information using sensors and cameras mounted on construction machinery.

[0436] Step 2:

[0437] The data collected by the device is packaged into a compressed format and transmitted to the control center via a wireless network (5G network) using a secure protocol.

[0438] Step 3:

[0439] The server receives data sent from the terminal in real time and converts the data into a format that can be decompressed and analyzed.

[0440] Step 4:

[0441] The server applies image processing and data analysis algorithms to analyze the received data, detecting ground conditions, machine operation status, and the presence or absence of abnormalities.

[0442] Step 5:

[0443] The generative AI generates specific work instructions based on the analysis results and the overall work plan. For example, it might create an instruction such as, "To excavate the next area A, excavate the path from position X to position Y."

[0444] Step 6:

[0445] The server notifies the remote operator of the generated work instructions. The notification is converted to the appropriate format and displayed on the operator's terminal.

[0446] Step 7:

[0447] When a problem occurs, the generative AI analyzes the situation and provides appropriate troubleshooting instructions. For example, it might send an instruction such as, "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0448] Step 8:

[0449] The server meticulously records each work instruction and its results, as well as a history of troubleshooting and its resolution, and stores this information in a database.

[0450] Step 9:

[0451] The generative AI generates a draft work report based on recorded progress and historical data. For example, it might write: "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[0452] Step 10:

[0453] The server sends a draft of the generated work report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[0454] (Example 1)

[0455] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0456] Modern construction sites demand increased work efficiency and real-time troubleshooting. However, conventional systems handle sensor data collection, analysis, work instruction generation, troubleshooting, and work report creation separately, making integrated work management difficult. Furthermore, the lack of real-time data transmission and immediate analysis hinders rapid response, raising concerns about impacting work progress and safety.

[0457] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0458] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery, means for packaging the collected real-time data in a compressed format and transmitting it to a control center via a wireless network, and means for decompressing the received real-time data and converting it into an analyzable format. This enables integrated and real-time data analysis and work instructions.

[0459] "Construction machinery" refers to large, specialized machines used in civil engineering and construction work.

[0460] A "sensor" is a device that measures physical environmental conditions (temperature, pressure, vibration, etc.) and acquires those values ​​as data.

[0461] A "camera" is a device that captures visual information from its surroundings and saves it in digital format.

[0462] "Real-time data" refers to data that instantly reflects the current state of affairs and is collected and analyzed without delay.

[0463] "Data acquisition means" refers to methods and devices used to acquire necessary data using sensors and cameras.

[0464] A "compression format" is a format that compresses information in order to reduce the data size.

[0465] A "wireless network" is a communication system that uses radio waves to send and receive data.

[0466] A "control center" is a central management hub that receives collected data, performs analysis, and generates instructions.

[0467] "Decompression means" refers to the process or apparatus for restoring compressed data to its original format.

[0468] "Analyzable format" refers to a format in which data is suitable for analysis and measurement after decompression.

[0469] "Analysis means" refers to methods and devices for evaluating collected data and extracting necessary information.

[0470] "Artificial intelligence means" refers to methods or devices that use AI technology to automatically provide work instructions and troubleshoot problems based on analysis results.

[0471] A "communication protocol" refers to the rules and procedures for sending and receiving data.

[0472] A "remote operator" refers to a technician or worker who operates equipment without being physically present at the site.

[0473] "Means for recording history" refers to methods and devices for saving and managing the progress and interactions of past work.

[0474] A "work report" is a document that summarizes the progress and results of a task.

[0475] "Troubleshooting" is the process of identifying the cause of a problem and providing a solution when one occurs.

[0476] "Means of supporting review" refer to methods or devices that enable users to review and correct generated reports, etc.

[0477] This invention is a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence generates work instructions based on the analysis results. These instructions are then notified to remote operators, enabling real-time work instructions and troubleshooting. Furthermore, the system records the progress of the work and the history of interactions, and ultimately generates a work report based on these records.

[0478] 1. Terminals (construction machinery)

[0479] The terminal is equipped with sensors and cameras to collect information such as ground conditions, equipment operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the camera acquires visual information. Specifically, the temperature sensor measures the ground temperature, and the pressure sensor measures the load of the heavy machinery. The camera captures video of the excavation area in real time.

[0480] 2. Data compression and transmission

[0481] The terminal converts the collected real-time data into a compressed format and transmits it to the control center via a wireless network (e.g., a 5G network) using a secure communication protocol. Specifically, it packages the data obtained from each sensor into a single file using a data compression algorithm and transmits the data via a 5G modem.

[0482] 3. Receiving and decompressing data

[0483] The server (control center) receives compressed data sent from the terminal. The received data is reconstructed using a decompression tool. The server's communication module takes in the received data, decompresses the ZIP file, and converts it into JSON format data.

[0484] 4. Data analysis and work instruction generation

[0485] The server passes the decompressed data to analysis tools and algorithms for analysis. For example, it uses an image processing library (e.g., OpenCV) to analyze the ground conditions and detect abnormal cracks and obstacles. The analyzed data is then passed to artificial intelligence (AI) to generate specific work instructions in conjunction with the overall work plan. For example, it automatically generates instructions such as "Excavate from position X to position Y in area A."

[0486] 5. Sending work instructions

[0487] The server sends the generated work instructions to the remote operator using a communication protocol (e.g., MQTT), and displays the instructions on the operator's terminal. The instructions are formatted in a specific text format. The remote operator checks the instructions sent from the server on their terminal's display and sees an instruction such as "Start drilling towards position X."

[0488] 6. Troubleshooting

[0489] Artificial intelligence (AI) instantly analyzes the situation when a problem occurs and provides real-time troubleshooting instructions. For example, it might instruct the machine to "raise the bucket and move backward if the heavy machinery gets stuck on the ground." When an anomaly is detected from sensor data, the AI ​​performs fault tree analysis and generates appropriate troubleshooting steps.

[0490] 7. Recording of work history

[0491] The server meticulously records each work order and its results, as well as the troubleshooting and resolution history, in a database. Work order and result data are saved to the SQL database using INSERT commands. Troubleshooting results are recorded similarly.

[0492] 8. Generation and review of work reports

[0493] Artificial intelligence (AI) integrates recorded progress and historical data to generate a draft work report. For example, it might create a report stating, "Today's excavation work was 80% complete, with two anomalies. The solutions are as follows..." The server sends the generated draft report to the user, who then reviews and makes final edits to complete the final work report. The draft report is generated in PDF format and sent to the user as an email attachment. The user then makes revisions using the editing tool.

[0494] Example of a prompt

[0495] "The construction machine recorded a ground temperature of 35 degrees Celsius from its ground temperature sensor, and the vibration sensor detected strong vibrations. Please use this data to generate the next work instruction."

[0496] In this way, the system of the present invention improves work efficiency and enables accurate recording and reporting at construction sites. Furthermore, real-time troubleshooting improves safety and work accuracy.

[0497] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0498] Specific processing steps of the system program

[0499] Step 1:

[0500] The terminal uses sensors and cameras to collect information such as ground conditions, equipment operating status, and location. Inputs are physical data measured by sensors (temperature, pressure, vibration, etc.) and visual data captured by the camera. Outputs are the collected sensor data and image data. Specifically, the temperature sensor measures the ground temperature, the pressure sensor measures the load of heavy machinery, and the camera acquires visual information.

[0501] Step 2:

[0502] The terminal packages the collected real-time data in a compressed format. The inputs are the sensor data and image data collected in step 1. A compression algorithm is used to efficiently transmit this data, and a compressed data package is obtained as the output. Specifically, the data compression algorithm is used to compress the data obtained from each sensor and package it into a single file.

[0503] Step 3:

[0504] The terminal transmits compressed data to the control center via a wireless network (e.g., a 5G network) using a secure communication protocol. The input is the compressed data generated in step 2. The output is the data transmitted to the control center. Specifically, the data is transmitted via a 5G modem.

[0505] Step 4:

[0506] The server (control center) receives compressed data sent from the terminal. The input is the compressed data sent from the terminal. The output is the received compressed data. Specifically, the server's communication module takes in the received data.

[0507] Step 5:

[0508] The server reconstructs the received compressed data using a decompression tool and converts it into an analyzable format. The input is the compressed data received in step 4. The output is the decompressed and analyzable data. Specifically, it decompresses a ZIP file and converts the data into JSON format.

[0509] Step 6:

[0510] The server passes the decompressed data to analysis tools and algorithms for analysis. The input is the analyzable data decompressed in step 5. The output is the analyzed data and results. Specifically, it uses an image processing library (e.g., OpenCV) to analyze the ground conditions and detect abnormal cracks and obstacles.

[0511] Step 7:

[0512] Artificial intelligence (AI) generates specific work instructions by referring to the analysis results and the overall work plan. The input is the analysis results and data obtained in step 6. The output is the generated work instructions. Specifically, it reads the current progress from the work plan database and generates the optimal work instructions by comparing them with the analysis results.

[0513] Step 8:

[0514] The server sends the generated work instructions to the remote operator using a communication protocol and displays the instructions on the operator's terminal. The input is the work instructions generated in step 7. The output is the work instructions sent to the remote operator. Specifically, the work instructions are converted to JSON format and sent to the operator's tablet terminal via a communication protocol (e.g., MQTT).

[0515] Step 9:

[0516] Artificial intelligence (AI) immediately analyzes the situation when a problem occurs and provides troubleshooting instructions. The input is sensor data and image data collected in real time. The output is specific troubleshooting instructions. Specifically, when an anomaly is detected from the sensor data, the AI ​​performs fault tree analysis and generates appropriate troubleshooting steps.

[0517] Step 10:

[0518] The server meticulously records each work order and its results, as well as troubleshooting and its resolution history, in a database. Inputs include work orders and their results, and troubleshooting steps and results. Outputs are the historical data stored in the database. Specifically, the work order and result data are saved to the SQL database using INSERT commands.

[0519] Step 11:

[0520] Artificial intelligence (AI) integrates recorded progress and historical data to generate a draft work report. The input is progress and historical data recorded in a database. The output is a draft work report. Specifically, it uses natural language generation (NLG) technology to create the report.

[0521] Step 12:

[0522] The server sends a draft of the generated report to the user for review and final editing. The input is the draft of the work report generated in step 11. The output is the draft report sent to the user. Specifically, the server generates the draft report in PDF format and sends it to the user as an email attachment.

[0523] (Application Example 1)

[0524] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0525] There were challenges with conventional technology in improving the operational efficiency of industrial machinery and detecting and responding quickly to anomalies in real time. Furthermore, the generation of response instructions and work reports in the event of an anomaly was not automated, resulting in time losses and human errors.

[0526] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0527] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on industrial machinery, means for transmitting the collected real-time data to a control center via a wireless network, means for analyzing the received real-time data, artificial intelligence means for generating work instructions based on the analysis results, means for notifying a remote operator of the generated work instructions, means for recording the progress of work and the history of interactions during work, means for generating a work report based on the recorded progress and history, means for detecting abnormalities during work and immediately generating response instructions, and means for processing image data acquired by cameras mounted on industrial machinery in real time and detecting abnormal locations. This improves the work efficiency of industrial machinery and enables real-time abnormality detection and rapid response.

[0528] "Industrial machinery" is a general term for automated machines and equipment used in manufacturing and production lines.

[0529] A "sensor" is a device that measures physical quantities such as temperature, pressure, and vibration, and converts them into electrical signals.

[0530] A "camera" is a device that can capture visual information and record and transmit it as digital data.

[0531] "Real-time data" refers to data that indicates the current state of a machine while it is in operation and can be processed or transmitted immediately.

[0532] A "wireless network" is a communication method that uses radio waves to transmit data, and specifically includes Wi-Fi and 5G.

[0533] A "control center" is a facility or system for centrally managing, analyzing, and controlling data from industrial machinery.

[0534] "Means of analysis" refer to processes and devices for breaking down and analyzing received data and extracting useful information.

[0535] "Artificial intelligence means" refers to algorithms and systems that automatically generate judgments and instructions based on collected data.

[0536] A "remote operator" is a person or device that operates a machine or system from a physically distant location.

[0537] "Work progress" refers to information indicating the extent to which planned work has been completed.

[0538] "History" refers to the record of all past work and communications.

[0539] A "work report" is a document that includes details such as the progress of the work and how any problems were handled.

[0540] "Means for detecting abnormalities" refers to a device or method for detecting a state that deviates from normal operating conditions.

[0541] "Means for generating response instructions" refers to a device and algorithm that automatically determines an appropriate response method for a detected anomaly and outputs it as an instruction.

[0542] "Means for processing image data" refers to software and hardware used to analyze images captured by a camera and extract useful information.

[0543] This invention relates to a system that collects real-time data from sensors and cameras mounted on industrial machinery and transmits that data to a control center via a wireless network. Specifically, it is implemented as follows.

[0544] First, industrial machinery is equipped with various sensors such as temperature sensors, pressure sensors, and vibration sensors, as well as cameras. These sensors measure the machine's operating status and environmental conditions in real time, while the cameras capture visual information of the work area. The data collected from the sensors and cameras undergoes basic processing and compression at the terminal.

[0545] Next, the terminal transmits the collected data to the control center via a wireless network (e.g., Wi-Fi or 5G). A server located at the control center decompresses the received data and converts it into an analyzable format. The software used includes libraries for data decompression (e.g., zlib) and libraries for image analysis (e.g., OpenCV, TensorFlow).

[0546] Next, the data, converted into an analyzable format, is analyzed by artificial intelligence (AI). The AI, for example, uses image processing algorithms to detect anomalies in the work area or applies anomaly detection algorithms to sensor data. The analysis results are compared with the overall work plan, and specific work instructions are generated by the AI ​​model. These AI models function as generative AI models, employing, for example, deep learning algorithms.

[0547] The generated work instructions are sent from the server to the remote operator via a wireless network. The instructions are displayed in a specific text format on the terminal used by the operator. For example, an instruction such as "Excavate the path from position X to position Y in order to excavate the next area B" might be issued.

[0548] Furthermore, if an anomaly occurs during operation, the system immediately detects the anomaly and generates a response instruction. This instruction is also notified to the remote operator. For example, a specific response instruction such as "The machine has detected an abnormal temperature, so stop it immediately and begin cooling" is provided.

[0549] All work instructions, their results, and the history of anomaly responses are recorded in a database. This recorded data is later integrated by AI and generated as a work report. The report includes work progress, any anomalies that occurred, and how they were addressed. Users can review the generated report and make revisions as needed.

[0550] Specific example

[0551] Examples of prompts to input into a generative AI model:

[0552] This is real-time monitoring data from automated robots operating within the factory.

[0553] Sensor data: Temperature 30°C, Pressure 1.5, Vibration 0.3

[0554] Camera image: base64_encoded_image_data

[0555] Analysis results: An anomaly was detected in a part of the work line, so the following instructions were generated.

[0556] Instructions: Robot 1 should continue its work, avoiding sensor anomaly area B, and send an alert to the person in charge.

[0557] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0558] Step 1:

[0559] The terminal collects real-time data from sensors and cameras mounted on industrial machinery. Sensors measure physical quantities such as temperature, pressure, and vibration, while cameras capture images of the work area. The collected data is stored as sensor data (temperature, pressure, vibration, etc.) and image data (JPEG format).

[0560] Step 2:

[0561] The terminal performs basic processing on the collected real-time data and then compresses it. For example, it might use the zlib library to compress the data. The compressed data is then prepared to be transmitted over the wireless network as data packets.

[0562] Step 3:

[0563] The device transmits data packets to the control center via a wireless network (e.g., Wi-Fi or 5G). Wireless communication protocols are used, and data security is protected by encryption technology.

[0564] Step 4:

[0565] The server receives the data packets sent from the terminal and decompresses them. This restores the sensor data and image data to their original formats. The zlib library is used here as well.

[0566] Step 5:

[0567] The server analyzes the decompressed data. An anomaly detection algorithm is applied to the sensor data, and image processing algorithms such as OpenCV or TensorFlow are used to detect anomalies in the image data. The analysis results are saved for the next processing step.

[0568] Step 6:

[0569] The server uses a generated AI model to create specific work instructions based on the analysis results. For example, using a deep learning algorithm, it might generate an instruction such as, "To excavate the next area B, excavate a path from position X to position Y." This instruction is formatted in text format.

[0570] Step 7:

[0571] The server notifies the remote operator of the generated work instructions via the wireless network. The instructions are displayed in specific text format on the terminal used by the operator. For example, the instructions may include "Stop the pump and start cooling."

[0572] Step 8:

[0573] The server records the progress of the work and the history of interactions during the work in a database. The recorded data includes each work instruction, its execution result, and a history of how errors were handled.

[0574] Step 9:

[0575] The server generates work reports based on recorded progress and historical data. AI integrates this data and creates a draft of the work report. For example, the report might be in the format of, "Today's work was 90% complete, and 3 anomalies occurred. The specific actions taken are as follows..."

[0576] Specific example

[0577] Examples of prompts to input into a generative AI model:

[0578] This is real-time monitoring data from automated robots operating within the factory.

[0579] Sensor data: Temperature 30°C, Pressure 1.5, Vibration 0.3

[0580] Camera image: base64_encoded_image_data

[0581] Analysis results: An anomaly was detected in a part of the work line, so the following instructions were generated.

[0582] Instructions: Robot 1 should continue its work, avoiding sensor anomaly area B, and send an alert to the person in charge.

[0583] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0584] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence (AI) generates work instructions based on the analysis results. The generated work instructions are notified to the remote operator, enabling real-time work instructions and troubleshooting. The system also records the progress of the work and the history of interactions, and ultimately generates a work report based on the records. Furthermore, this invention incorporates an emotion engine to recognize the user's emotions and adjust the content of work instructions and troubleshooting accordingly.

[0585] Specific Examples of the System

[0586] 1. The terminal (construction machinery) is equipped with sensors and cameras to collect information such as ground conditions, machine operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the cameras acquire visual information.

[0587] 2. The terminal packages the collected real-time data into a compressed format and transmits it to the control center via a wireless network (5G network) using a secure protocol.

[0588] 3. The server (control center) receives data transmitted from terminals in real time and converts the data into a format that can be decompressed and analyzed. The received data is then used for analysis, such as analyzing ground conditions and progress using image processing algorithms, and detecting anomalies from sensor data.

[0589] 4. The analyzed data is passed to artificial intelligence (AI), which compares it with the overall work plan to generate specific work instructions based on the analysis results. For example, it automatically creates instructions such as, "To excavate the next area A, excavate the path from position X to position Y."

[0590] 5. The server notifies the remote operator of the generated work instructions using a communication protocol and displays the instructions on the operator's terminal. The instructions are formatted in a clear, easy-to-understand text format.

[0591] 6. When a problem occurs, the generative AI immediately analyzes the situation and provides troubleshooting instructions, such as "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0592] 7. The server meticulously records and stores in a database the history of each work instruction and its results, as well as troubleshooting and its resolution.

[0593] 8. Recorded progress and historical data are integrated by artificial intelligence (AI) to ultimately generate a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[0594] 9. The server sends a draft of the generated report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[0595] Specific examples of the emotion engine

[0596] 10. The device incorporates an emotion engine that recognizes the user's emotions, analyzing the user's tone of voice, facial expressions, and operation speed during operation to monitor their emotional state in real time.

[0597] 11. The emotion engine collects and analyzes the user's emotional data. For example, if the user's stress level is high or they are confused, the emotion engine recognizes that state.

[0598] 12. The analysis results are sent to the server and used to adjust real-time work instructions and troubleshooting content. For example, if the user is under high stress, the instructions may be made simpler and support messages may be inserted.

[0599] 13. The results of the emotion engine analysis are also reflected in the work report. For example, an assessment of the user's emotional state, such as "The user experienced a high level of stress during today's work and therefore requires support," is added to the report.

[0600] In this way, the system of the present invention not only improves work efficiency, accuracy, and safety at construction sites, but also enables adjustments and support that take into account the user's emotional state. A key feature is that the emotion engine can grasp the user's stress and satisfaction levels in real time, thereby reducing their burden.

[0601] The following describes the processing flow.

[0602] Step 1:

[0603] The terminal (construction machinery) collects real-time data, including ground conditions at the construction site, equipment operating status, and location information, using sensors and cameras. Sensors capture information such as temperature, pressure, and vibration, while cameras acquire visual information.

[0604] Step 2:

[0605] The real-time data collected by the terminal is packaged into a compressed format and transmitted to the control center via a wireless network (5G network) using a secure protocol.

[0606] Step 3:

[0607] The server (control center) receives data sent from the terminal in real time, decompresses the data, and converts it into an analyzable format.

[0608] Step 4:

[0609] The server analyzes the received data, uses image processing algorithms to check the ground conditions and progress, and detects any abnormalities from the sensor data.

[0610] Step 5:

[0611] The generative AI generates specific work instructions based on the analysis results and the overall work plan. For example, it might create an instruction such as, "To excavate the next area A, excavate the path from position X to position Y."

[0612] Step 6:

[0613] The server generates work instructions and notifies the remote operator using a communication protocol, displaying the instructions on the operator's terminal. This ensures that instructions are provided in a specific and easy-to-understand format.

[0614] Step 7:

[0615] When a problem occurs, the generative AI immediately analyzes the situation and provides appropriate troubleshooting instructions. For example, it might issue an instruction such as, "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0616] Step 8:

[0617] The emotion engine collects user emotion data. It monitors the user's emotional state in real time based on factors such as tone of voice, facial expressions, and operation speed, and generates emotion data.

[0618] Step 9:

[0619] The emotion engine analyzes collected emotional data to assess whether the user is experiencing high stress or satisfaction. The analysis results are sent to the server and used to adjust work instructions and troubleshooting content.

[0620] Step 10:

[0621] The server adjusts work instructions and troubleshooting based on emotional data. For example, if a user is in a high-stress state, it will simplify the instructions and add supportive messages.

[0622] Step 11:

[0623] The server meticulously records and stores in a database each work order and its results, troubleshooting history and resolution, and emotional state.

[0624] Step 12:

[0625] The generative AI generates a draft work report based on recorded progress and history data, as well as sentiment data. For example, it might write, "Today's excavation work is 80% complete, and two anomalies occurred. The solutions are as follows..." and include an assessment of the user's sentiment state.

[0626] Step 13:

[0627] The server sends a draft of the generated report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[0628] (Example 2)

[0629] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0630] Real-time data collection and analysis are essential for highly efficient and precise work on construction sites. However, conventional systems often suffer from delays in data collection, analysis, and the generation and notification of work instructions. Furthermore, instructions are not adjusted to take into account the user's emotional state, leading to challenges in operational efficiency and user stress management. These challenges need to be addressed.

[0631] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0632] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery; means for compressing the collected real-time data and transmitting it to a control center via a wireless network; means for converting the received real-time data into a format that can be decompressed and analyzed; artificial intelligence means for generating specific work instructions based on the analyzed data; means for notifying a remote operator of the generated work instructions; means for recording the progress of the work and the history of interactions during the work; means for generating a work report based on the recorded progress and history; means for collecting and analyzing user emotion data; and means for adjusting work instructions based on emotion data. This enables the provision of efficient and highly accurate work instructions in real time, as well as the adjustment of work instructions that take into account the user's emotional state, thereby improving work efficiency and safety.

[0633] "Construction machinery" refers to mechanical devices used to perform civil engineering work, excavation, and other tasks at construction sites.

[0634] A "sensor" is a device that detects the surrounding environment or the state of a machine and outputs that information as an electrical signal.

[0635] A "camera" is a device that captures visual information and records it as digital data.

[0636] "Real-time data" refers to data that is instantly acquired and processed with virtually no delay, reflecting the current state and situation.

[0637] A "wireless network" is a communication network that uses radio waves to send and receive data.

[0638] A "control center" is a base of operations for monitoring and controlling equipment and systems from a remote location.

[0639] "Data compression" is the process of converting large amounts of data so that they can be stored in less memory.

[0640] "Decompression" is the process of restoring compressed data to its original state.

[0641] An "analyzable format" is a format that converts data into a form that can be processed by analytical tools and algorithms.

[0642] "Artificial intelligence means" refers to technologies that analyze large amounts of data, learn from it, and generate appropriate results or instructions.

[0643] "Work instructions" are detailed procedures or commands for performing a specific task.

[0644] A "remote operator" refers to a person or system that operates remotely, without being physically present at the site.

[0645] "Troubleshooting" is the process of diagnosing machine malfunctions or abnormalities and proposing solutions.

[0646] "History" refers to a record of past work and interactions.

[0647] A "work report" is a document that summarizes the work performed, its progress, results, and any problems encountered.

[0648] "Emotional data" refers to information that indicates a user's emotional state, and is data obtained from sources such as voice, facial expressions, and behavior.

[0649] An "emotion engine" is a technology that analyzes a user's emotional data and evaluates their emotional state.

[0650] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery, transmits that data to a control center via a wireless network, and generates work instructions using artificial intelligence (AI) based on the analysis results. This improves the work efficiency and safety at construction sites.

[0651] Terminal hardware and software

[0652] The terminal (construction machinery) is equipped with multiple sensors (e.g., temperature sensors, pressure sensors, vibration sensors) and cameras. These sensors collect information such as ground conditions, machine operation status, and location in real time. The cameras acquire visual data using color and depth cameras. The collected data is compressed using dedicated data compression software (e.g., ZIP).

[0653] Sending data

[0654] The device transmits compressed data to the control center via the 5G network using a secure communication protocol (e.g., HTTPS). This communication ensures both data transmission speed and security.

[0655] Server hardware and software

[0656] The control center is equipped with high-performance servers that receive data transmitted via communication in real time. The received data is decompressed using data decompression software (e.g., Unzip) and converted into an analyzable format. Subsequently, various analysis algorithms (e.g., image processing algorithms, anomaly detection algorithms) are applied to analyze the data.

[0657] AI-generated work instructions

[0658] Based on the analyzed data, artificial intelligence (AI) generates appropriate work instructions. This AI takes into account the progress and conditions of a specific task and gives specific instructions for the next step. For example, it might generate an instruction such as, "To excavate the next area A, excavate from position X to position Y."

[0659] Work instruction notification

[0660] The server notifies the remote operator of the generated work instructions and displays the instructions on the operator's terminal. These instructions are displayed in text format and are easy to understand.

[0661] troubleshooting

[0662] If a problem occurs during operation, the terminal notifies the server of the situation. The server immediately analyzes the data, and the AI ​​generates appropriate troubleshooting steps and notifies the operator. For example, it provides specific instructions such as, "If the heavy machinery gets stuck, raise the bucket and move backward."

[0663] Data recording and work report generation

[0664] The server meticulously records each work instruction and its results, as well as troubleshooting and its resolution history, and stores it in a database. Based on these records, AI generates a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..." The server sends this draft report to the user, who reviews it and makes final edits to complete the work report.

[0665] Collection and analysis of emotional data

[0666] The device incorporates an emotion engine that recognizes the user's emotions, analyzing the user's tone of voice, facial expressions, and operation speed to monitor their emotional state in real time. The emotion engine collects this data and sends the analysis results to a server.

[0667] Adjusting work instructions based on emotional data

[0668] The server adjusts real-time work instructions and troubleshooting content based on the analysis results sent from the emotion engine. For example, if the user is in a high-stress state, the instructions will be simplified and supportive messages will be inserted.

[0669] These technological measures enable this system to significantly improve work efficiency and safety at construction sites. Furthermore, by considering the user's emotional state, it can reduce stress and provide more appropriate support.

[0670] Example of a prompt

[0671] For example, instructions might include, "Excavate a path from position X to position Y to excavate the next area A," or troubleshooting instructions such as, "If the heavy equipment gets stuck on the ground, lift the bucket and move backward."

[0672] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0673] Step 1:

[0674] Data collection from sensors and cameras using the device

[0675] The device collects real-time data using various built-in sensors (e.g., temperature sensor, pressure sensor, vibration sensor) and a camera. Inputs include the surrounding environment and the operating status of the machine. Sensor data is output in the form of temperature, pressure, vibration, and location information, while the camera captures visual data. The device collects this data and temporarily stores it in its internal storage.

[0676] Specific example: "The device measures the ambient temperature with a temperature sensor and takes pictures of the terrain with a camera."

[0677] Step 2:

[0678] Data compression and transmission by the terminal

[0679] To compress the collected data, the terminal applies a compression algorithm (e.g., ZIP). The inputs are the sensor data and camera data obtained in step 1. The compressed data is sent to the server via the 5G network using a secure communication protocol (e.g., HTTPS). This results in the compressed data being the output of the transmission.

[0680] Specific example: "The device compresses the collected data in ZIP format and sends it to the server using the HTTPS protocol."

[0681] Step 3:

[0682] Receiving and decompressing data by the server.

[0683] The server receives compressed data sent from the terminal. The input is compressed data. After receiving the data, the server uses data decompression software (e.g., Unzip) to decompress it and convert it into an analyzable format. The decompressed data then becomes the output.

[0684] Specific example: "The server receives a ZIP file from the terminal and unzips it using Unzip software."

[0685] Step 4:

[0686] Server-based data analysis

[0687] Based on the decompressed data, the server performs analysis by applying image processing algorithms and anomaly detection algorithms. The input consists of decompressed sensor data and camera data. After the analysis process, the server outputs analysis results such as terrain irregularities, temperature anomalies, and pressure fluctuations.

[0688] Specific example: "The server uses an image processing algorithm to detect topographic irregularities and an anomaly detection algorithm to identify anomalies in the temperature data."

[0689] Step 5:

[0690] AI-generated work instructions

[0691] Based on the analysis results, the AI ​​generates specific work instructions. The input is the analysis results obtained in step 4. The AI ​​compares these with the overall work plan and outputs work instructions such as, "To excavate the next area A, excavate from position X to position Y."

[0692] Specific example: "Based on the analysis results, the AI ​​instructs on a specific route for excavating the next area A."

[0693] Step 6:

[0694] Server-based work instruction notification

[0695] The server notifies the remote operator of the generated work instructions. The input is the work instructions obtained in step 5. The work instructions are displayed in text format and sent to the remote control terminal. This allows the operator to check the instructions in real time. The output is the notified work instructions.

[0696] Specific example: "The server notifies the operator of the generated work instructions via SMS and displays the instructions on the terminal's screen."

[0697] Step 7:

[0698] Terminal and server troubleshooting

[0699] If a problem occurs, the terminal notifies the server of the situation. The input is data about the problem. The server immediately analyzes this data, and the AI ​​generates appropriate troubleshooting steps. For example, it might output specific instructions such as, "If the heavy machinery gets stuck, raise the bucket and move backward."

[0700] Specific example: "After receiving abnormal data from a terminal, the server uses AI to generate an instruction to 'raise the bucket and move backward,' and notifies the operator."

[0701] Step 8:

[0702] Data recording by the server

[0703] The server meticulously records each work order and its results, as well as troubleshooting and its resolution history, and stores it in a database. Inputs are work orders, their execution results, and troubleshooting history. The recorded data is then output.

[0704] Specific example: "The server records all work orders and their execution results in the SQL database."

[0705] Step 9:

[0706] AI-powered work report generation

[0707] Based on the recorded data, the AI ​​generates a draft of the work report. The input is the progress data and historical data recorded in step 8. For example, a draft report in the format of "Today's excavation work was 80% complete and two anomalies occurred" will be output.

[0708] Specific example: "The AI ​​analyzes all historical data and generates a draft of the work report titled 'Today's Work Progress and Anomaly Response List'."

[0709] Step 10:

[0710] Server-based report submission and user review.

[0711] The server sends a draft of the generated report to the user. The input is the draft work report generated in step 9. The user reviews and edits it to complete the final work report. The output is the completed work report.

[0712] Specific example: "The server emails a draft of the report to the user, and saves the final version after the user has made edits."

[0713] Step 11:

[0714] Collection of emotional data using devices

[0715] The device uses an emotion engine to analyze the user's voice tone, facial expressions, and operation speed, and collects emotional data. The input is the user's voice and facial expressions during operation. This data is analyzed by the emotion engine, and emotional data is output.

[0716] Specific example: "The device captures the user's voice tone with a microphone and analyzes their facial expressions with a camera."

[0717] Step 12:

[0718] Analysis of emotional data using an emotion engine

[0719] The emotion engine analyzes the collected emotion data to evaluate the user's stress level and emotional state. The input is the emotion data collected in step 11. For example, if the tone of voice is high or the facial expression is strained, it will be evaluated as a "high-stress state," and the result will be output.

[0720] Specific example: "The emotion engine detects that the user's voice tone is high and determines that they are in a high-stress state."

[0721] Step 13:

[0722] Adjusting work instructions based on emotional data from the server.

[0723] The server adjusts real-time work instructions and troubleshooting content based on the analysis results of the emotion engine. The input is the analysis results of the emotion data obtained in step 12. For example, if the user is in a high-stress state, adjusted instructions will be output, with the instructions made simpler and support messages added.

[0724] Specific example: "The server will take into account the high-stress state detected by the emotion engine and add a support message such as 'Please remain calm while operating,' along with concise instructions."

[0725] Through the steps described above, this system provides real-time, efficient, and highly accurate work instructions, and supports operations while taking the user's emotional state into consideration, thereby improving work efficiency and safety at construction sites.

[0726] (Application Example 2)

[0727] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0728] Improving work efficiency and speeding up troubleshooting are crucial challenges in existing factory and construction sites. However, conventional systems lack sufficient real-time data collection and analysis, and in particular, they fail to adjust instructions to take into account the emotions and stress levels of operators. This makes it difficult to monitor work progress and respond immediately to problems, potentially increasing operator stress and confusion.

[0729] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery, means for transmitting the collected real-time data to a control center via a wireless network, means for analyzing the received real-time data, artificial intelligence means for generating work instructions based on the analysis results, means for notifying a remote operator of the generated work instructions, means for recording the progress of the work and the history of interactions during the work, means for generating a work report based on the recorded progress and history, emotion engine means for monitoring the emotional state of the operator, and means for adjusting work instructions based on the emotion data generated by the emotion engine means. This enables not only efficient real-time data collection and analysis and troubleshooting, but also flexible instructions that take into account the operator's emotions.

[0730] A "sensor" is a device that measures physical conditions and outputs them as electrical signals.

[0731] A "camera" is a device that receives light, forms an image, and stores or transmits it as digital data.

[0732] "Real-time data" refers to data that is acquired and transmitted instantaneously without delay.

[0733] A "wireless network" is a means of communication that uses radio waves to send and receive data.

[0734] A "control center" is a centralized control system or location for analyzing and managing collected data.

[0735] "Analysis" is the process of analyzing data and extracting meaning as information.

[0736] "Artificial intelligence" is a technology that uses computer programs to mimic human intelligence, learn from data, and generate work instructions.

[0737] A "remote operator" refers to a person or device that operates or manages a machine or system from a remote location.

[0738] "History" refers to data that records the progress and results of a task or event in chronological order.

[0739] A "work report" is a document that records and summarizes the progress, results, and details of any problems that occurred during a series of tasks.

[0740] An "emotion engine" is software that analyzes the user's voice tone, facial expressions, and operation speed to monitor the user's emotional state in real time.

[0741] This invention relates to a system that improves the work efficiency of robots in a factory and provides instructions that take into account the operator's emotions. Specific embodiments for carrying out this invention are described below.

[0742] Hardware and software

[0743] Hardware:

[0744] Smartphone (iOS / Android): Used by the operator to receive instructions.

[0745] Factory robots: Devices for operating machinery, equipped with numerous sensors and cameras.

[0746] software:

[0747] Languages: Python, Java (Android), Swift (iOS)

[0748] Frameworks: TensorFlow (AI analysis), OpenCV (image processing), Flask (backend API)

[0749] Network: 5G, Wi-Fi

[0750] Data collection

[0751] There are methods for collecting real-time data from factory robots' sensors and cameras. Sensors measure physical quantities such as temperature, pressure, and vibration, while cameras acquire visual data. This real-time data is transmitted to a server via a wireless network (e.g., 5G).

[0752] Data Analysis

[0753] The server has means to analyze the received real-time data. For example, sensor data is analyzed using an anomaly detection algorithm, and visual data acquired from cameras is analyzed using an image processing algorithm. Software tools such as TensorFlow and OpenCV are used in this analysis process.

[0754] Work order generation

[0755] The server has an artificial intelligence system that generates work instructions based on the analysis results. The AI ​​model compares the analysis results with the overall work plan and generates specific work instructions. For example, it might create instructions such as, "Investigate the following area."

[0756] Instructions, notifications, and coordination

[0757] The generated work instructions are notified to the remote operator using a communication protocol. Furthermore, the server has an emotion engine that monitors the operator's emotional state and adjusts the work instructions based on the emotional data generated by the emotion engine. If the operator is under stress, adjustments are made, such as simplifying the instructions and inserting support messages.

[0758] History recording and report generation

[0759] The server has means to record the progress of work and the history of interactions during the work in detail. It also has means to generate work reports based on the recorded progress and history data. For example, it can automatically generate a report such as, "Today's work was 80% completed as planned, and two anomalies occurred."

[0760] As a concrete example, the following prompt statements can be used:

[0761] "Develop an AI system for monitoring factory work progress and troubleshooting. Collect real-time data from sensors and cameras, analyze it on a remote server to generate work instructions, and adjust those instructions based on user sentiment."

[0762] This enables real-time data collection and analysis, more efficient troubleshooting, and flexible instructions that take operator emotions into consideration.

[0763] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0764] Step 1:

[0765] The terminal (factory robot) collects real-time data such as ground conditions, machine operating status, and location information from its mounted sensors and cameras. The sensors measure physical quantities such as temperature, pressure, and vibration, while the cameras capture visual data. The collected data is packaged in a compressed format. The input consists of machine physical quantities and image data, and the output is real-time data in a compressed format.

[0766] Step 2:

[0767] The terminal transmits collected real-time data to a server via a wireless network (e.g., 5G). A secure protocol is used for transmission, ensuring data confidentiality and integrity. The input is real-time data in a compressed format, and the output is a notification that the data transmission to the server is complete.

[0768] Step 3:

[0769] The server converts received real-time data into a format that can be decompressed and analyzed. To convert it into the format required for analysis, it separates sensor data and image data from a specific data stream and uses them for their respective processing. The input is compressed data, and the output is decompressed sensor data and image data.

[0770] Step 4:

[0771] The server analyzes the decompressed sensor data and image data. First, it applies an anomaly detection algorithm to the sensor data to detect patterns that are different from the norm. Next, it performs image processing on the image data using OpenCV or similar tools to extract specific visual information. The input is the decompressed sensor data and image data, and the output is the anomaly detection results and the visual information extraction results.

[0772] Step 5:

[0773] The server uses an artificial intelligence model (generative AI model) to generate work instructions based on the analysis results. Based on the analysis results and the overall work plan, the AI ​​model generates specific work instructions. For example, it might generate an instruction such as "Investigate the next area A." The input is the anomaly detection results and the visual information extraction results, and the output is the generated work instructions.

[0774] Step 6:

[0775] The server notifies the remote operator's smartphone of the generated work instructions. It uses a communication protocol to notify the instructions in an easily understandable format (e.g., text message). The input is the generated work instructions, and the output is the instruction notification to the remote operator's terminal.

[0776] Step 7:

[0777] The server uses an emotion engine to monitor the operator's emotional state. It analyzes the operator's voice tone, facial expressions, and operating speed, collecting emotional data in real time. The input is the operator's actions and voice data, and the output is the analyzed emotional data.

[0778] Step 8:

[0779] The server adjusts work instructions based on emotional data generated by the emotion engine. For example, if an operator is under high stress, the instructions are simplified and supportive messages are inserted. The input is the analyzed emotional data and existing work instructions, and the output is the adjusted work instructions.

[0780] Step 9:

[0781] The server meticulously records the progress of work and the history of interactions. This history includes data analysis results, work instructions, and sentiment data for each step. Inputs are various analysis results and instructions, while output is a detailed history record.

[0782] Step 10:

[0783] The server generates work reports based on recorded progress and historical data. These reports include progress status, occurrences and responses to anomalies, and the operator's emotional state. For example, it might automatically generate a report stating, "Today's work was 80% complete, and two anomalies occurred." The input is detailed historical records, and the output is the work report.

[0784] As described above, by realizing this invention, it is possible to improve work efficiency and reduce the mental burden on operators.

[0785] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0786] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0787] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0788] [Third Embodiment]

[0789] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0790] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0791] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0792] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0793] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0794] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0795] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0796] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0797] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0798] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0799] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0800] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0801] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence (AI) generates work instructions based on the analysis results. These instructions are then notified to remote operators, enabling real-time work instructions and troubleshooting. Furthermore, the system records the progress of the work and the history of interactions, ultimately allowing for the generation of work reports based on these records.

[0802] Specific Examples of the System

[0803] 1. The terminal (construction machinery) is equipped with sensors and cameras to collect information such as ground conditions, equipment operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the cameras acquire visual information.

[0804] 2. The terminal packages the collected real-time data in a compressed format and transmits it to the control center via a wireless network (5G network) using a secure protocol.

[0805] 3. The server (control center) receives and decompresses data transmitted from terminals in real time and converts it into an analyzable format. The received data is then used for analysis, such as analyzing ground conditions and progress using image processing algorithms, and detecting anomalies from sensor data.

[0806] 4. The analyzed data is passed to artificial intelligence (AI), which compares it with the overall work plan to generate specific work instructions based on the analysis results. For example, it automatically creates instructions such as, "To excavate the next area A, excavate the path from position X to position Y."

[0807] 5. The server transmits the generated work instructions to the remote operator using a communication protocol and displays the instructions on the operator's terminal. The instructions are formatted in a clear, easy-to-understand text format.

[0808] 6. When a problem occurs, artificial intelligence (AI) will immediately analyze the situation and provide troubleshooting instructions, such as "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0809] 7. The server meticulously records each work instruction and its results, as well as the history of troubleshooting and its resolution. This record is stored in a database and includes information such as the progress of the work, the operations performed, and the instructions given by the generative AI and their results.

[0810] 8. Recorded progress and historical data are integrated by artificial intelligence (AI) to ultimately generate a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[0811] 9. The server sends a draft of the generated report to the user to assist with review and final editing. The user makes revisions as needed to complete the final work report.

[0812] In this way, the system of the present invention achieves increased efficiency and accurate recording and reporting of work at construction sites. Furthermore, real-time troubleshooting improves safety and work accuracy.

[0813] The following describes the processing flow.

[0814] Step 1:

[0815] The terminal collects real-time data such as ground conditions, machine operation status, and location information using sensors and cameras mounted on construction machinery.

[0816] Step 2:

[0817] The data collected by the device is packaged into a compressed format and transmitted to the control center via a wireless network (5G network) using a secure protocol.

[0818] Step 3:

[0819] The server receives data sent from the terminal in real time and converts the data into a format that can be decompressed and analyzed.

[0820] Step 4:

[0821] The server applies image processing and data analysis algorithms to analyze the received data, detecting ground conditions, machine operation status, and the presence or absence of abnormalities.

[0822] Step 5:

[0823] The generative AI generates specific work instructions based on the analysis results and the overall work plan. For example, it might create an instruction such as, "To excavate the next area A, excavate the path from position X to position Y."

[0824] Step 6:

[0825] The server notifies the remote operator of the generated work instructions. The notification is converted to the appropriate format and displayed on the operator's terminal.

[0826] Step 7:

[0827] When a problem occurs, the generative AI analyzes the situation and provides appropriate troubleshooting instructions. For example, it might send an instruction such as, "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0828] Step 8:

[0829] The server meticulously records each work instruction and its results, as well as a history of troubleshooting and its resolution, and stores this information in a database.

[0830] Step 9:

[0831] The generative AI generates a draft work report based on recorded progress and historical data. For example, it might write: "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[0832] Step 10:

[0833] The server sends a draft of the generated work report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[0834] (Example 1)

[0835] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0836] Modern construction sites demand increased work efficiency and real-time troubleshooting. However, conventional systems handle sensor data collection, analysis, work instruction generation, troubleshooting, and work report creation separately, making integrated work management difficult. Furthermore, the lack of real-time data transmission and immediate analysis hinders rapid response, raising concerns about impacting work progress and safety.

[0837] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0838] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery, means for packaging the collected real-time data in a compressed format and transmitting it to a control center via a wireless network, and means for decompressing the received real-time data and converting it into an analyzable format. This enables integrated and real-time data analysis and work instructions.

[0839] "Construction machinery" refers to large, specialized machines used in civil engineering and construction work.

[0840] A "sensor" is a device that measures physical environmental conditions (temperature, pressure, vibration, etc.) and acquires those values ​​as data.

[0841] A "camera" is a device that captures visual information from its surroundings and saves it in digital format.

[0842] "Real-time data" refers to data that instantly reflects the current state of affairs and is collected and analyzed without delay.

[0843] "Data acquisition means" refers to methods and devices used to acquire necessary data using sensors and cameras.

[0844] A "compression format" is a format that compresses information in order to reduce the data size.

[0845] A "wireless network" is a communication system that uses radio waves to send and receive data.

[0846] A "control center" is a central management hub that receives collected data, performs analysis, and generates instructions.

[0847] "Decompression means" refers to the process or apparatus for restoring compressed data to its original format.

[0848] "Analyzable format" refers to a format in which data is suitable for analysis and measurement after decompression.

[0849] "Analysis means" refers to methods and devices for evaluating collected data and extracting necessary information.

[0850] "Artificial intelligence means" refers to methods or devices that use AI technology to automatically provide work instructions and troubleshoot problems based on analysis results.

[0851] A "communication protocol" refers to the rules and procedures for sending and receiving data.

[0852] A "remote operator" refers to a technician or worker who operates equipment without being physically present at the site.

[0853] "Means for recording history" refers to methods and devices for saving and managing the progress and interactions of past work.

[0854] A "work report" is a document that summarizes the progress and results of a task.

[0855] "Troubleshooting" is the process of identifying the cause of a problem and providing a solution when one occurs.

[0856] "Means of supporting review" refer to methods or devices that enable users to review and correct generated reports, etc.

[0857] This invention is a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence generates work instructions based on the analysis results. These instructions are then notified to remote operators, enabling real-time work instructions and troubleshooting. Furthermore, the system records the progress of the work and the history of interactions, and ultimately generates a work report based on these records.

[0858] 1. Terminals (construction machinery)

[0859] The terminal is equipped with sensors and cameras to collect information such as ground conditions, equipment operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the camera acquires visual information. Specifically, the temperature sensor measures the ground temperature, and the pressure sensor measures the load of the heavy machinery. The camera captures video of the excavation area in real time.

[0860] 2. Data compression and transmission

[0861] The terminal converts the collected real-time data into a compressed format and transmits it to the control center via a wireless network (e.g., a 5G network) using a secure communication protocol. Specifically, it packages the data obtained from each sensor into a single file using a data compression algorithm and transmits the data via a 5G modem.

[0862] 3. Receiving and decompressing data

[0863] The server (control center) receives compressed data sent from the terminal. The received data is reconstructed using a decompression tool. The server's communication module takes in the received data, decompresses the ZIP file, and converts it into JSON format data.

[0864] 4. Data analysis and work instruction generation

[0865] The server passes the decompressed data to analysis tools and algorithms for analysis. For example, it uses an image processing library (e.g., OpenCV) to analyze the ground conditions and detect abnormal cracks and obstacles. The analyzed data is then passed to artificial intelligence (AI) to generate specific work instructions in conjunction with the overall work plan. For example, it automatically generates instructions such as "Excavate from position X to position Y in area A."

[0866] 5. Sending work instructions

[0867] The server sends the generated work instructions to the remote operator using a communication protocol (e.g., MQTT), and displays the instructions on the operator's terminal. The instructions are formatted in a specific text format. The remote operator checks the instructions sent from the server on their terminal's display and sees an instruction such as "Start drilling towards position X."

[0868] 6. Troubleshooting

[0869] Artificial intelligence (AI) instantly analyzes the situation when a problem occurs and provides real-time troubleshooting instructions. For example, it might instruct the machine to "raise the bucket and move backward if the heavy machinery gets stuck on the ground." When an anomaly is detected from sensor data, the AI ​​performs fault tree analysis and generates appropriate troubleshooting steps.

[0870] 7. Recording of work history

[0871] The server meticulously records each work order and its results, as well as the troubleshooting and resolution history, in a database. Work order and result data are saved to the SQL database using INSERT commands. Troubleshooting results are recorded similarly.

[0872] 8. Generation and review of work reports

[0873] Artificial intelligence (AI) integrates recorded progress and historical data to generate a draft work report. For example, it might create a report stating, "Today's excavation work was 80% complete, with two anomalies. The solutions are as follows..." The server sends the generated draft report to the user, who then reviews and makes final edits to complete the final work report. The draft report is generated in PDF format and sent to the user as an email attachment. The user then makes revisions using the editing tool.

[0874] Example of a prompt

[0875] "The construction machine recorded a ground temperature of 35 degrees Celsius from its ground temperature sensor, and the vibration sensor detected strong vibrations. Please use this data to generate the next work instruction."

[0876] In this way, the system of the present invention improves work efficiency and enables accurate recording and reporting at construction sites. Furthermore, real-time troubleshooting improves safety and work accuracy.

[0877] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0878] Specific processing steps of the system program

[0879] Step 1:

[0880] The terminal uses sensors and cameras to collect information such as ground conditions, equipment operating status, and location. Inputs are physical data measured by sensors (temperature, pressure, vibration, etc.) and visual data captured by the camera. Outputs are the collected sensor data and image data. Specifically, the temperature sensor measures the ground temperature, the pressure sensor measures the load of heavy machinery, and the camera acquires visual information.

[0881] Step 2:

[0882] The terminal packages the collected real-time data in a compressed format. The inputs are the sensor data and image data collected in step 1. A compression algorithm is used to efficiently transmit this data, and a compressed data package is obtained as the output. Specifically, the data compression algorithm is used to compress the data obtained from each sensor and package it into a single file.

[0883] Step 3:

[0884] The terminal transmits compressed data to the control center via a wireless network (e.g., a 5G network) using a secure communication protocol. The input is the compressed data generated in step 2. The output is the data transmitted to the control center. Specifically, the data is transmitted via a 5G modem.

[0885] Step 4:

[0886] The server (control center) receives compressed data sent from the terminal. The input is the compressed data sent from the terminal. The output is the received compressed data. Specifically, the server's communication module takes in the received data.

[0887] Step 5:

[0888] The server reconstructs the received compressed data using a decompression tool and converts it into an analyzable format. The input is the compressed data received in step 4. The output is the decompressed and analyzable data. Specifically, it decompresses a ZIP file and converts the data into JSON format.

[0889] Step 6:

[0890] The server passes the decompressed data to analysis tools and algorithms for analysis. The input is the analyzable data decompressed in step 5. The output is the analyzed data and results. Specifically, it uses an image processing library (e.g., OpenCV) to analyze the ground conditions and detect abnormal cracks and obstacles.

[0891] Step 7:

[0892] Artificial intelligence (AI) generates specific work instructions by referring to the analysis results and the overall work plan. The input is the analysis results and data obtained in step 6. The output is the generated work instructions. Specifically, it reads the current progress from the work plan database and generates the optimal work instructions by comparing them with the analysis results.

[0893] Step 8:

[0894] The server sends the generated work instructions to the remote operator using a communication protocol and displays the instructions on the operator's terminal. The input is the work instructions generated in step 7. The output is the work instructions sent to the remote operator. Specifically, the work instructions are converted to JSON format and sent to the operator's tablet terminal via a communication protocol (e.g., MQTT).

[0895] Step 9:

[0896] Artificial intelligence (AI) immediately analyzes the situation when a problem occurs and provides troubleshooting instructions. The input is sensor data and image data collected in real time. The output is specific troubleshooting instructions. Specifically, when an anomaly is detected from the sensor data, the AI ​​performs fault tree analysis and generates appropriate troubleshooting steps.

[0897] Step 10:

[0898] The server meticulously records each work order and its results, as well as troubleshooting and its resolution history, in a database. Inputs include work orders and their results, and troubleshooting steps and results. Outputs are the historical data stored in the database. Specifically, the work order and result data are saved to the SQL database using INSERT commands.

[0899] Step 11:

[0900] Artificial intelligence (AI) integrates recorded progress and historical data to generate a draft work report. The input is progress and historical data recorded in a database. The output is a draft work report. Specifically, it uses natural language generation (NLG) technology to create the report.

[0901] Step 12:

[0902] The server sends a draft of the generated report to the user for review and final editing. The input is the draft of the work report generated in step 11. The output is the draft report sent to the user. Specifically, the server generates the draft report in PDF format and sends it to the user as an email attachment.

[0903] (Application Example 1)

[0904] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0905] There were challenges with conventional technology in improving the operational efficiency of industrial machinery and detecting and responding quickly to anomalies in real time. Furthermore, the generation of response instructions and work reports in the event of an anomaly was not automated, resulting in time losses and human errors.

[0906] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0907] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on industrial machinery, means for transmitting the collected real-time data to a control center via a wireless network, means for analyzing the received real-time data, artificial intelligence means for generating work instructions based on the analysis results, means for notifying a remote operator of the generated work instructions, means for recording the progress of work and the history of interactions during work, means for generating a work report based on the recorded progress and history, means for detecting abnormalities during work and immediately generating response instructions, and means for processing image data acquired by cameras mounted on industrial machinery in real time and detecting abnormal locations. This improves the work efficiency of industrial machinery and enables real-time abnormality detection and rapid response.

[0908] "Industrial machinery" is a general term for automated machines and equipment used in manufacturing and production lines.

[0909] A "sensor" is a device that measures physical quantities such as temperature, pressure, and vibration, and converts them into electrical signals.

[0910] A "camera" is a device that can capture visual information and record and transmit it as digital data.

[0911] "Real-time data" refers to data that indicates the current state of a machine while it is in operation and can be processed or transmitted immediately.

[0912] A "wireless network" is a communication method that uses radio waves to transmit data, and specifically includes Wi-Fi and 5G.

[0913] A "control center" is a facility or system for centrally managing, analyzing, and controlling data from industrial machinery.

[0914] "Means of analysis" refer to processes and devices for breaking down and analyzing received data and extracting useful information.

[0915] "Artificial intelligence means" refers to algorithms and systems that automatically generate judgments and instructions based on collected data.

[0916] A "remote operator" is a person or device that operates a machine or system from a physically distant location.

[0917] "Work progress" refers to information indicating the extent to which planned work has been completed.

[0918] "History" refers to the record of all past work and communications.

[0919] A "work report" is a document that includes details such as the progress of the work and how any problems were handled.

[0920] "Means for detecting abnormalities" refers to a device or method for detecting a state that deviates from normal operating conditions.

[0921] "Means for generating response instructions" refers to a device and algorithm that automatically determines an appropriate response method for a detected anomaly and outputs it as an instruction.

[0922] "Means for processing image data" refers to software and hardware used to analyze images captured by a camera and extract useful information.

[0923] This invention relates to a system that collects real-time data from sensors and cameras mounted on industrial machinery and transmits that data to a control center via a wireless network. Specifically, it is implemented as follows.

[0924] First, industrial machinery is equipped with various sensors such as temperature sensors, pressure sensors, and vibration sensors, as well as cameras. These sensors measure the machine's operating status and environmental conditions in real time, while the cameras capture visual information of the work area. The data collected from the sensors and cameras undergoes basic processing and compression at the terminal.

[0925] Next, the terminal transmits the collected data to the control center via a wireless network (e.g., Wi-Fi or 5G). A server located at the control center decompresses the received data and converts it into an analyzable format. The software used includes libraries for data decompression (e.g., zlib) and libraries for image analysis (e.g., OpenCV, TensorFlow).

[0926] Next, the data, converted into an analyzable format, is analyzed by artificial intelligence (AI). The AI, for example, uses image processing algorithms to detect anomalies in the work area or applies anomaly detection algorithms to sensor data. The analysis results are compared with the overall work plan, and specific work instructions are generated by the AI ​​model. These AI models function as generative AI models, employing, for example, deep learning algorithms.

[0927] The generated work instructions are sent from the server to the remote operator via a wireless network. The instructions are displayed in a specific text format on the terminal used by the operator. For example, an instruction such as "Excavate the path from position X to position Y in order to excavate the next area B" might be issued.

[0928] Furthermore, if an anomaly occurs during operation, the system immediately detects the anomaly and generates a response instruction. This instruction is also notified to the remote operator. For example, a specific response instruction such as "The machine has detected an abnormal temperature, so stop it immediately and begin cooling" is provided.

[0929] All work instructions, their results, and the history of anomaly responses are recorded in a database. This recorded data is later integrated by AI and generated as a work report. The report includes work progress, any anomalies that occurred, and how they were addressed. Users can review the generated report and make revisions as needed.

[0930] Specific example

[0931] Examples of prompts to input into a generative AI model:

[0932] This is real-time monitoring data from automated robots operating within the factory.

[0933] Sensor data: Temperature 30°C, Pressure 1.5, Vibration 0.3

[0934] Camera image: base64_encoded_image_data

[0935] Analysis results: An anomaly was detected in a part of the work line, so the following instructions were generated.

[0936] Instructions: Robot 1 should continue its work, avoiding sensor anomaly area B, and send an alert to the person in charge.

[0937] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0938] Step 1:

[0939] The terminal collects real-time data from sensors and cameras mounted on industrial machinery. Sensors measure physical quantities such as temperature, pressure, and vibration, while cameras capture images of the work area. The collected data is stored as sensor data (temperature, pressure, vibration, etc.) and image data (JPEG format).

[0940] Step 2:

[0941] The terminal performs basic processing on the collected real-time data and then compresses it. For example, it might use the zlib library to compress the data. The compressed data is then prepared to be transmitted over the wireless network as data packets.

[0942] Step 3:

[0943] The device transmits data packets to the control center via a wireless network (e.g., Wi-Fi or 5G). Wireless communication protocols are used, and data security is protected by encryption technology.

[0944] Step 4:

[0945] The server receives the data packets sent from the terminal and decompresses them. This restores the sensor data and image data to their original formats. The zlib library is used here as well.

[0946] Step 5:

[0947] The server analyzes the decompressed data. An anomaly detection algorithm is applied to the sensor data, and image processing algorithms such as OpenCV or TensorFlow are used to detect anomalies in the image data. The analysis results are saved for the next processing step.

[0948] Step 6:

[0949] The server uses a generated AI model to create specific work instructions based on the analysis results. For example, using a deep learning algorithm, it might generate an instruction such as, "To excavate the next area B, excavate a path from position X to position Y." This instruction is formatted in text format.

[0950] Step 7:

[0951] The server notifies the remote operator of the generated work instructions via the wireless network. The instructions are displayed in specific text format on the terminal used by the operator. For example, the instructions may include "Stop the pump and start cooling."

[0952] Step 8:

[0953] The server records the progress of the work and the history of interactions during the work in a database. The recorded data includes each work instruction, its execution result, and a history of how errors were handled.

[0954] Step 9:

[0955] The server generates work reports based on recorded progress and historical data. AI integrates this data and creates a draft of the work report. For example, the report might be in the format of, "Today's work was 90% complete, and 3 anomalies occurred. The specific actions taken are as follows..."

[0956] Specific example

[0957] Examples of prompts to input into a generative AI model:

[0958] This is real-time monitoring data from automated robots operating within the factory.

[0959] Sensor data: Temperature 30°C, Pressure 1.5, Vibration 0.3

[0960] Camera image: base64_encoded_image_data

[0961] Analysis results: An anomaly was detected in a part of the work line, so the following instructions were generated.

[0962] Instructions: Robot 1 should continue its work, avoiding sensor anomaly area B, and send an alert to the person in charge.

[0963] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0964] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence (AI) generates work instructions based on the analysis results. The generated work instructions are notified to the remote operator, enabling real-time work instructions and troubleshooting. The system also records the progress of the work and the history of interactions, and ultimately generates a work report based on the records. Furthermore, this invention incorporates an emotion engine to recognize the user's emotions and adjust the content of work instructions and troubleshooting accordingly.

[0965] Specific Examples of the System

[0966] 1. The terminal (construction machinery) is equipped with sensors and cameras to collect information such as ground conditions, machine operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the cameras acquire visual information.

[0967] 2. The terminal packages the collected real-time data into a compressed format and transmits it to the control center via a wireless network (5G network) using a secure protocol.

[0968] 3. The server (control center) receives data transmitted from terminals in real time and converts the data into a format that can be decompressed and analyzed. The received data is then used for analysis, such as analyzing ground conditions and progress using image processing algorithms, and detecting anomalies from sensor data.

[0969] 4. The analyzed data is passed to artificial intelligence (AI), which compares it with the overall work plan to generate specific work instructions based on the analysis results. For example, it automatically creates instructions such as, "To excavate the next area A, excavate the path from position X to position Y."

[0970] 5. The server notifies the remote operator of the generated work instructions using a communication protocol and displays the instructions on the operator's terminal. The instructions are formatted in a clear, easy-to-understand text format.

[0971] 6. When a problem occurs, the generative AI immediately analyzes the situation and provides troubleshooting instructions, such as "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0972] 7. The server meticulously records and stores in a database the history of each work instruction and its results, as well as troubleshooting and its resolution.

[0973] 8. Recorded progress and historical data are integrated by artificial intelligence (AI) to ultimately generate a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[0974] 9. The server sends a draft of the generated report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[0975] Specific examples of the emotion engine

[0976] 10. The device incorporates an emotion engine that recognizes the user's emotions, analyzing the user's tone of voice, facial expressions, and operation speed during operation to monitor their emotional state in real time.

[0977] 11. The emotion engine collects and analyzes the user's emotional data. For example, if the user's stress level is high or they are confused, the emotion engine recognizes that state.

[0978] 12. The analysis results are sent to the server and used to adjust real-time work instructions and troubleshooting content. For example, if the user is under high stress, the instructions may be made simpler and support messages may be inserted.

[0979] 13. The results of the emotion engine analysis are also reflected in the work report. For example, an assessment of the user's emotional state, such as "The user experienced a high level of stress during today's work and therefore requires support," is added to the report.

[0980] In this way, the system of the present invention not only improves work efficiency, accuracy, and safety at construction sites, but also enables adjustments and support that take into account the user's emotional state. A key feature is that the emotion engine can grasp the user's stress and satisfaction levels in real time, thereby reducing their burden.

[0981] The following describes the processing flow.

[0982] Step 1:

[0983] The terminal (construction machinery) collects real-time data, including ground conditions at the construction site, equipment operating status, and location information, using sensors and cameras. Sensors capture information such as temperature, pressure, and vibration, while cameras acquire visual information.

[0984] Step 2:

[0985] The real-time data collected by the terminal is packaged into a compressed format and transmitted to the control center via a wireless network (5G network) using a secure protocol.

[0986] Step 3:

[0987] The server (control center) receives data sent from the terminal in real time, decompresses the data, and converts it into an analyzable format.

[0988] Step 4:

[0989] The server analyzes the received data, uses image processing algorithms to check the ground conditions and progress, and detects any abnormalities from the sensor data.

[0990] Step 5:

[0991] The generative AI generates specific work instructions based on the analysis results and the overall work plan. For example, it might create an instruction such as, "To excavate the next area A, excavate the path from position X to position Y."

[0992] Step 6:

[0993] The server generates work instructions and notifies the remote operator using a communication protocol, displaying the instructions on the operator's terminal. This ensures that instructions are provided in a specific and easy-to-understand format.

[0994] Step 7:

[0995] When a problem occurs, the generative AI immediately analyzes the situation and provides appropriate troubleshooting instructions. For example, it might issue an instruction such as, "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[0996] Step 8:

[0997] The emotion engine collects user emotion data. It monitors the user's emotional state in real time based on factors such as tone of voice, facial expressions, and operation speed, and generates emotion data.

[0998] Step 9:

[0999] The emotion engine analyzes collected emotional data to assess whether the user is experiencing high stress or satisfaction. The analysis results are sent to the server and used to adjust work instructions and troubleshooting content.

[1000] Step 10:

[1001] The server adjusts work instructions and troubleshooting based on emotional data. For example, if a user is in a high-stress state, it will simplify the instructions and add supportive messages.

[1002] Step 11:

[1003] The server meticulously records and stores in a database each work order and its results, troubleshooting history and resolution, and emotional state.

[1004] Step 12:

[1005] The generative AI generates a draft work report based on recorded progress and history data, as well as sentiment data. For example, it might write, "Today's excavation work is 80% complete, and two anomalies occurred. The solutions are as follows..." and include an assessment of the user's sentiment state.

[1006] Step 13:

[1007] The server sends a draft of the generated report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[1008] (Example 2)

[1009] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1010] Real-time data collection and analysis are essential for highly efficient and precise work on construction sites. However, conventional systems often suffer from delays in data collection, analysis, and the generation and notification of work instructions. Furthermore, instructions are not adjusted to take into account the user's emotional state, leading to challenges in operational efficiency and user stress management. These challenges need to be addressed.

[1011] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1012] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery; means for compressing the collected real-time data and transmitting it to a control center via a wireless network; means for converting the received real-time data into a format that can be decompressed and analyzed; artificial intelligence means for generating specific work instructions based on the analyzed data; means for notifying a remote operator of the generated work instructions; means for recording the progress of the work and the history of interactions during the work; means for generating a work report based on the recorded progress and history; means for collecting and analyzing user emotion data; and means for adjusting work instructions based on emotion data. This enables the provision of efficient and highly accurate work instructions in real time, as well as the adjustment of work instructions that take into account the user's emotional state, thereby improving work efficiency and safety.

[1013] "Construction machinery" refers to mechanical devices used to perform civil engineering work, excavation, and other tasks at construction sites.

[1014] A "sensor" is a device that detects the surrounding environment or the state of a machine and outputs that information as an electrical signal.

[1015] A "camera" is a device that captures visual information and records it as digital data.

[1016] "Real-time data" refers to data that is instantly acquired and processed with virtually no delay, reflecting the current state and situation.

[1017] A "wireless network" is a communication network that uses radio waves to send and receive data.

[1018] A "control center" is a base of operations for monitoring and controlling equipment and systems from a remote location.

[1019] "Data compression" is the process of converting large amounts of data so that they can be stored in less memory.

[1020] "Decompression" is the process of restoring compressed data to its original state.

[1021] An "analyzable format" is a format that converts data into a form that can be processed by analytical tools and algorithms.

[1022] "Artificial intelligence means" refers to technologies that analyze large amounts of data, learn from it, and generate appropriate results or instructions.

[1023] "Work instructions" are detailed procedures or commands for performing a specific task.

[1024] A "remote operator" refers to a person or system that operates remotely, without being physically present at the site.

[1025] "Troubleshooting" is the process of diagnosing machine malfunctions or abnormalities and proposing solutions.

[1026] "History" refers to a record of past work and interactions.

[1027] A "work report" is a document that summarizes the work performed, its progress, results, and any problems encountered.

[1028] "Emotional data" refers to information that indicates a user's emotional state, and is data obtained from sources such as voice, facial expressions, and behavior.

[1029] An "emotion engine" is a technology that analyzes a user's emotional data and evaluates their emotional state.

[1030] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery, transmits that data to a control center via a wireless network, and generates work instructions using artificial intelligence (AI) based on the analysis results. This improves the work efficiency and safety at construction sites.

[1031] Terminal hardware and software

[1032] The terminal (construction machinery) is equipped with multiple sensors (e.g., temperature sensors, pressure sensors, vibration sensors) and cameras. These sensors collect information such as ground conditions, machine operation status, and location in real time. The cameras acquire visual data using color and depth cameras. The collected data is compressed using dedicated data compression software (e.g., ZIP).

[1033] Sending data

[1034] The device transmits compressed data to the control center via the 5G network using a secure communication protocol (e.g., HTTPS). This communication ensures both data transmission speed and security.

[1035] Server hardware and software

[1036] The control center is equipped with high-performance servers that receive data transmitted via communication in real time. The received data is decompressed using data decompression software (e.g., Unzip) and converted into an analyzable format. Subsequently, various analysis algorithms (e.g., image processing algorithms, anomaly detection algorithms) are applied to analyze the data.

[1037] AI-generated work instructions

[1038] Based on the analyzed data, artificial intelligence (AI) generates appropriate work instructions. This AI takes into account the progress and conditions of a specific task and gives specific instructions for the next step. For example, it might generate an instruction such as, "To excavate the next area A, excavate from position X to position Y."

[1039] Work instruction notification

[1040] The server notifies the remote operator of the generated work instructions and displays the instructions on the operator's terminal. These instructions are displayed in text format and are easy to understand.

[1041] troubleshooting

[1042] If a problem occurs during operation, the terminal notifies the server of the situation. The server immediately analyzes the data, and the AI ​​generates appropriate troubleshooting steps and notifies the operator. For example, it provides specific instructions such as, "If the heavy machinery gets stuck, raise the bucket and move backward."

[1043] Data recording and work report generation

[1044] The server meticulously records each work instruction and its results, as well as troubleshooting and its resolution history, and stores it in a database. Based on these records, AI generates a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..." The server sends this draft report to the user, who reviews it and makes final edits to complete the work report.

[1045] Collection and analysis of emotional data

[1046] The device incorporates an emotion engine that recognizes the user's emotions, analyzing the user's tone of voice, facial expressions, and operation speed to monitor their emotional state in real time. The emotion engine collects this data and sends the analysis results to a server.

[1047] Adjusting work instructions based on emotional data

[1048] The server adjusts real-time work instructions and troubleshooting content based on the analysis results sent from the emotion engine. For example, if the user is in a high-stress state, the instructions will be simplified and supportive messages will be inserted.

[1049] These technological measures enable this system to significantly improve work efficiency and safety at construction sites. Furthermore, by considering the user's emotional state, it can reduce stress and provide more appropriate support.

[1050] Example of a prompt

[1051] For example, instructions might include, "Excavate a path from position X to position Y to excavate the next area A," or troubleshooting instructions such as, "If the heavy equipment gets stuck on the ground, lift the bucket and move backward."

[1052] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1053] Step 1:

[1054] Data collection from sensors and cameras using the device

[1055] The device collects real-time data using various built-in sensors (e.g., temperature sensor, pressure sensor, vibration sensor) and a camera. Inputs include the surrounding environment and the operating status of the machine. Sensor data is output in the form of temperature, pressure, vibration, and location information, while the camera captures visual data. The device collects this data and temporarily stores it in its internal storage.

[1056] Specific example: "The device measures the ambient temperature with a temperature sensor and takes pictures of the terrain with a camera."

[1057] Step 2:

[1058] Data compression and transmission by the terminal

[1059] To compress the collected data, the terminal applies a compression algorithm (e.g., ZIP). The inputs are the sensor data and camera data obtained in step 1. The compressed data is sent to the server via the 5G network using a secure communication protocol (e.g., HTTPS). This results in the compressed data being the output of the transmission.

[1060] Specific example: "The device compresses the collected data in ZIP format and sends it to the server using the HTTPS protocol."

[1061] Step 3:

[1062] Receiving and decompressing data by the server.

[1063] The server receives compressed data sent from the terminal. The input is compressed data. After receiving the data, the server uses data decompression software (e.g., Unzip) to decompress it and convert it into an analyzable format. The decompressed data then becomes the output.

[1064] Specific example: "The server receives a ZIP file from the terminal and unzips it using Unzip software."

[1065] Step 4:

[1066] Server-based data analysis

[1067] Based on the decompressed data, the server performs analysis by applying image processing algorithms and anomaly detection algorithms. The input consists of decompressed sensor data and camera data. After the analysis process, the server outputs analysis results such as terrain irregularities, temperature anomalies, and pressure fluctuations.

[1068] Specific example: "The server uses an image processing algorithm to detect topographic irregularities and an anomaly detection algorithm to identify anomalies in the temperature data."

[1069] Step 5:

[1070] AI-generated work instructions

[1071] Based on the analysis results, the AI ​​generates specific work instructions. The input is the analysis results obtained in step 4. The AI ​​compares these with the overall work plan and outputs work instructions such as, "To excavate the next area A, excavate from position X to position Y."

[1072] Specific example: "Based on the analysis results, the AI ​​instructs on a specific route for excavating the next area A."

[1073] Step 6:

[1074] Server-based work instruction notification

[1075] The server notifies the remote operator of the generated work instructions. The input is the work instructions obtained in step 5. The work instructions are displayed in text format and sent to the remote control terminal. This allows the operator to check the instructions in real time. The output is the notified work instructions.

[1076] Specific example: "The server notifies the operator of the generated work instructions via SMS and displays the instructions on the terminal's screen."

[1077] Step 7:

[1078] Terminal and server troubleshooting

[1079] If a problem occurs, the terminal notifies the server of the situation. The input is data about the problem. The server immediately analyzes this data, and the AI ​​generates appropriate troubleshooting steps. For example, it might output specific instructions such as, "If the heavy machinery gets stuck, raise the bucket and move backward."

[1080] Specific example: "After receiving abnormal data from a terminal, the server uses AI to generate an instruction to 'raise the bucket and move backward,' and notifies the operator."

[1081] Step 8:

[1082] Data recording by the server

[1083] The server meticulously records each work order and its results, as well as troubleshooting and its resolution history, and stores it in a database. Inputs are work orders, their execution results, and troubleshooting history. The recorded data is then output.

[1084] Specific example: "The server records all work orders and their execution results in the SQL database."

[1085] Step 9:

[1086] AI-powered work report generation

[1087] Based on the recorded data, the AI ​​generates a draft of the work report. The input is the progress data and historical data recorded in step 8. For example, a draft report in the format of "Today's excavation work was 80% complete and two anomalies occurred" will be output.

[1088] Specific example: "The AI ​​analyzes all historical data and generates a draft of the work report titled 'Today's Work Progress and Anomaly Response List'."

[1089] Step 10:

[1090] Server-based report submission and user review.

[1091] The server sends a draft of the generated report to the user. The input is the draft work report generated in step 9. The user reviews and edits it to complete the final work report. The output is the completed work report.

[1092] Specific example: "The server emails a draft of the report to the user, and saves the final version after the user has made edits."

[1093] Step 11:

[1094] Collection of emotional data using devices

[1095] The device uses an emotion engine to analyze the user's voice tone, facial expressions, and operation speed, and collects emotional data. The input is the user's voice and facial expressions during operation. This data is analyzed by the emotion engine, and emotional data is output.

[1096] Specific example: "The device captures the user's voice tone with a microphone and analyzes their facial expressions with a camera."

[1097] Step 12:

[1098] Analysis of emotional data using an emotion engine

[1099] The emotion engine analyzes the collected emotion data to evaluate the user's stress level and emotional state. The input is the emotion data collected in step 11. For example, if the tone of voice is high or the facial expression is strained, it will be evaluated as a "high-stress state," and the result will be output.

[1100] Specific example: "The emotion engine detects that the user's voice tone is high and determines that they are in a high-stress state."

[1101] Step 13:

[1102] Adjusting work instructions based on emotional data from the server.

[1103] The server adjusts real-time work instructions and troubleshooting content based on the analysis results of the emotion engine. The input is the analysis results of the emotion data obtained in step 12. For example, if the user is in a high-stress state, adjusted instructions will be output, with the instructions made simpler and support messages added.

[1104] Specific example: "The server will take into account the high-stress state detected by the emotion engine and add a support message such as 'Please remain calm while operating,' along with concise instructions."

[1105] Through the steps described above, this system provides real-time, efficient, and highly accurate work instructions, and supports operations while taking the user's emotional state into consideration, thereby improving work efficiency and safety at construction sites.

[1106] (Application Example 2)

[1107] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1108] Improving work efficiency and speeding up troubleshooting are crucial challenges in existing factory and construction sites. However, conventional systems lack sufficient real-time data collection and analysis, and in particular, they fail to adjust instructions to take into account the emotions and stress levels of operators. This makes it difficult to monitor work progress and respond immediately to problems, potentially increasing operator stress and confusion.

[1109] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery, means for transmitting the collected real-time data to a control center via a wireless network, means for analyzing the received real-time data, artificial intelligence means for generating work instructions based on the analysis results, means for notifying a remote operator of the generated work instructions, means for recording the progress of the work and the history of interactions during the work, means for generating a work report based on the recorded progress and history, emotion engine means for monitoring the emotional state of the operator, and means for adjusting work instructions based on the emotion data generated by the emotion engine means. This enables not only efficient real-time data collection and analysis and troubleshooting, but also flexible instructions that take into account the operator's emotions.

[1110] A "sensor" is a device that measures physical conditions and outputs them as electrical signals.

[1111] A "camera" is a device that receives light, forms an image, and stores or transmits it as digital data.

[1112] "Real-time data" refers to data that is acquired and transmitted instantaneously without delay.

[1113] A "wireless network" is a means of communication that uses radio waves to send and receive data.

[1114] A "control center" is a centralized control system or location for analyzing and managing collected data.

[1115] "Analysis" is the process of analyzing data and extracting meaning as information.

[1116] "Artificial intelligence" is a technology that uses computer programs to mimic human intelligence, learn from data, and generate work instructions.

[1117] A "remote operator" refers to a person or device that operates or manages a machine or system from a remote location.

[1118] "History" refers to data that records the progress and results of a task or event in chronological order.

[1119] A "work report" is a document that records and summarizes the progress, results, and details of any problems that occurred during a series of tasks.

[1120] An "emotion engine" is software that analyzes the user's voice tone, facial expressions, and operation speed to monitor the user's emotional state in real time.

[1121] This invention relates to a system that improves the work efficiency of robots in a factory and provides instructions that take into account the operator's emotions. Specific embodiments for carrying out this invention are described below.

[1122] Hardware and software

[1123] Hardware:

[1124] Smartphone (iOS / Android): Used by the operator to receive instructions.

[1125] Factory robots: Devices for operating machinery, equipped with numerous sensors and cameras.

[1126] software:

[1127] Languages: Python, Java (Android), Swift (iOS)

[1128] Frameworks: TensorFlow (AI analysis), OpenCV (image processing), Flask (backend API)

[1129] Network: 5G, Wi-Fi

[1130] Data collection

[1131] There are methods for collecting real-time data from factory robots' sensors and cameras. Sensors measure physical quantities such as temperature, pressure, and vibration, while cameras acquire visual data. This real-time data is transmitted to a server via a wireless network (e.g., 5G).

[1132] Data Analysis

[1133] The server has means to analyze the received real-time data. For example, sensor data is analyzed using an anomaly detection algorithm, and visual data acquired from cameras is analyzed using an image processing algorithm. Software tools such as TensorFlow and OpenCV are used in this analysis process.

[1134] Work order generation

[1135] The server has an artificial intelligence system that generates work instructions based on the analysis results. The AI ​​model compares the analysis results with the overall work plan and generates specific work instructions. For example, it might create instructions such as, "Investigate the following area."

[1136] Instructions, notifications, and coordination

[1137] The generated work instructions are notified to the remote operator using a communication protocol. Furthermore, the server has an emotion engine that monitors the operator's emotional state and adjusts the work instructions based on the emotional data generated by the emotion engine. If the operator is under stress, adjustments are made, such as simplifying the instructions and inserting support messages.

[1138] History recording and report generation

[1139] The server has means to record the progress of work and the history of interactions during the work in detail. It also has means to generate work reports based on the recorded progress and history data. For example, it can automatically generate a report such as, "Today's work was 80% completed as planned, and two anomalies occurred."

[1140] As a concrete example, the following prompt statements can be used:

[1141] "Develop an AI system for monitoring factory work progress and troubleshooting. Collect real-time data from sensors and cameras, analyze it on a remote server to generate work instructions, and adjust those instructions based on user sentiment."

[1142] This enables real-time data collection and analysis, more efficient troubleshooting, and flexible instructions that take operator emotions into consideration.

[1143] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1144] Step 1:

[1145] The terminal (factory robot) collects real-time data such as ground conditions, machine operating status, and location information from its mounted sensors and cameras. The sensors measure physical quantities such as temperature, pressure, and vibration, while the cameras capture visual data. The collected data is packaged in a compressed format. The input consists of machine physical quantities and image data, and the output is real-time data in a compressed format.

[1146] Step 2:

[1147] The terminal transmits collected real-time data to a server via a wireless network (e.g., 5G). A secure protocol is used for transmission, ensuring data confidentiality and integrity. The input is real-time data in a compressed format, and the output is a notification that the data transmission to the server is complete.

[1148] Step 3:

[1149] The server converts received real-time data into a format that can be decompressed and analyzed. To convert it into the format required for analysis, it separates sensor data and image data from a specific data stream and uses them for their respective processing. The input is compressed data, and the output is decompressed sensor data and image data.

[1150] Step 4:

[1151] The server analyzes the decompressed sensor data and image data. First, it applies an anomaly detection algorithm to the sensor data to detect patterns that are different from the norm. Next, it performs image processing on the image data using OpenCV or similar tools to extract specific visual information. The input is the decompressed sensor data and image data, and the output is the anomaly detection results and the visual information extraction results.

[1152] Step 5:

[1153] The server uses an artificial intelligence model (generative AI model) to generate work instructions based on the analysis results. Based on the analysis results and the overall work plan, the AI ​​model generates specific work instructions. For example, it might generate an instruction such as "Investigate the next area A." The input is the anomaly detection results and the visual information extraction results, and the output is the generated work instructions.

[1154] Step 6:

[1155] The server notifies the remote operator's smartphone of the generated work instructions. It uses a communication protocol to notify the instructions in an easily understandable format (e.g., text message). The input is the generated work instructions, and the output is the instruction notification to the remote operator's terminal.

[1156] Step 7:

[1157] The server uses an emotion engine to monitor the operator's emotional state. It analyzes the operator's voice tone, facial expressions, and operating speed, collecting emotional data in real time. The input is the operator's actions and voice data, and the output is the analyzed emotional data.

[1158] Step 8:

[1159] The server adjusts work instructions based on emotional data generated by the emotion engine. For example, if an operator is under high stress, the instructions are simplified and supportive messages are inserted. The input is the analyzed emotional data and existing work instructions, and the output is the adjusted work instructions.

[1160] Step 9:

[1161] The server meticulously records the progress of work and the history of interactions. This history includes data analysis results, work instructions, and sentiment data for each step. Inputs are various analysis results and instructions, while output is a detailed history record.

[1162] Step 10:

[1163] The server generates work reports based on recorded progress and historical data. These reports include progress status, occurrences and responses to anomalies, and the operator's emotional state. For example, it might automatically generate a report stating, "Today's work was 80% complete, and two anomalies occurred." The input is detailed historical records, and the output is the work report.

[1164] As described above, by realizing this invention, it is possible to improve work efficiency and reduce the mental burden on operators.

[1165] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1166] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1167] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1168] [Fourth Embodiment]

[1169] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1170] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1171] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1172] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1173] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1174] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1175] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1176] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1177] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1178] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1179] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1180] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1181] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1182] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence (AI) generates work instructions based on the analysis results. These instructions are then notified to remote operators, enabling real-time work instructions and troubleshooting. Furthermore, the system records the progress of the work and the history of interactions, ultimately allowing for the generation of work reports based on these records.

[1183] Specific Examples of the System

[1184] 1. The terminal (construction machinery) is equipped with sensors and cameras to collect information such as ground conditions, equipment operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the cameras acquire visual information.

[1185] 2. The terminal packages the collected real-time data in a compressed format and transmits it to the control center via a wireless network (5G network) using a secure protocol.

[1186] 3. The server (control center) receives and decompresses data transmitted from terminals in real time and converts it into an analyzable format. The received data is then used for analysis, such as analyzing ground conditions and progress using image processing algorithms, and detecting anomalies from sensor data.

[1187] 4. The analyzed data is passed to artificial intelligence (AI), which compares it with the overall work plan to generate specific work instructions based on the analysis results. For example, it automatically creates instructions such as, "To excavate the next area A, excavate the path from position X to position Y."

[1188] 5. The server transmits the generated work instructions to the remote operator using a communication protocol and displays the instructions on the operator's terminal. The instructions are formatted in a clear, easy-to-understand text format.

[1189] 6. When a problem occurs, artificial intelligence (AI) will immediately analyze the situation and provide troubleshooting instructions, such as "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[1190] 7. The server meticulously records each work instruction and its results, as well as the history of troubleshooting and its resolution. This record is stored in a database and includes information such as the progress of the work, the operations performed, and the instructions given by the generative AI and their results.

[1191] 8. Recorded progress and historical data are integrated by artificial intelligence (AI) to ultimately generate a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[1192] 9. The server sends a draft of the generated report to the user to assist with review and final editing. The user makes revisions as needed to complete the final work report.

[1193] In this way, the system of the present invention achieves increased efficiency and accurate recording and reporting of work at construction sites. Furthermore, real-time troubleshooting improves safety and work accuracy.

[1194] The following describes the processing flow.

[1195] Step 1:

[1196] The terminal collects real-time data such as ground conditions, machine operation status, and location information using sensors and cameras mounted on construction machinery.

[1197] Step 2:

[1198] The data collected by the device is packaged into a compressed format and transmitted to the control center via a wireless network (5G network) using a secure protocol.

[1199] Step 3:

[1200] The server receives data sent from the terminal in real time and converts the data into a format that can be decompressed and analyzed.

[1201] Step 4:

[1202] The server applies image processing and data analysis algorithms to analyze the received data, detecting ground conditions, machine operation status, and the presence or absence of abnormalities.

[1203] Step 5:

[1204] The generative AI generates specific work instructions based on the analysis results and the overall work plan. For example, it might create an instruction such as, "To excavate the next area A, excavate the path from position X to position Y."

[1205] Step 6:

[1206] The server notifies the remote operator of the generated work instructions. The notification is converted to the appropriate format and displayed on the operator's terminal.

[1207] Step 7:

[1208] When a problem occurs, the generative AI analyzes the situation and provides appropriate troubleshooting instructions. For example, it might send an instruction such as, "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[1209] Step 8:

[1210] The server meticulously records each work instruction and its results, as well as a history of troubleshooting and its resolution, and stores this information in a database.

[1211] Step 9:

[1212] The generative AI generates a draft work report based on recorded progress and historical data. For example, it might write: "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[1213] Step 10:

[1214] The server sends a draft of the generated work report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[1215] (Example 1)

[1216] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1217] Modern construction sites demand increased work efficiency and real-time troubleshooting. However, conventional systems handle sensor data collection, analysis, work instruction generation, troubleshooting, and work report creation separately, making integrated work management difficult. Furthermore, the lack of real-time data transmission and immediate analysis hinders rapid response, raising concerns about impacting work progress and safety.

[1218] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1219] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery, means for packaging the collected real-time data in a compressed format and transmitting it to a control center via a wireless network, and means for decompressing the received real-time data and converting it into an analyzable format. This enables integrated and real-time data analysis and work instructions.

[1220] "Construction machinery" refers to large, specialized machines used in civil engineering and construction work.

[1221] A "sensor" is a device that measures physical environmental conditions (temperature, pressure, vibration, etc.) and acquires those values ​​as data.

[1222] A "camera" is a device that captures visual information from its surroundings and saves it in digital format.

[1223] "Real-time data" refers to data that instantly reflects the current state of affairs and is collected and analyzed without delay.

[1224] "Data acquisition means" refers to methods and devices used to acquire necessary data using sensors and cameras.

[1225] A "compression format" is a format that compresses information in order to reduce the data size.

[1226] A "wireless network" is a communication system that uses radio waves to send and receive data.

[1227] A "control center" is a central management hub that receives collected data, performs analysis, and generates instructions.

[1228] "Decompression means" refers to the process or apparatus for restoring compressed data to its original format.

[1229] "Analyzable format" refers to a format in which data is suitable for analysis and measurement after decompression.

[1230] "Analysis means" refers to methods and devices for evaluating collected data and extracting necessary information.

[1231] "Artificial intelligence means" refers to methods or devices that use AI technology to automatically provide work instructions and troubleshoot problems based on analysis results.

[1232] A "communication protocol" refers to the rules and procedures for sending and receiving data.

[1233] A "remote operator" refers to a technician or worker who operates equipment without being physically present at the site.

[1234] "Means for recording history" refers to methods and devices for saving and managing the progress and interactions of past work.

[1235] A "work report" is a document that summarizes the progress and results of a task.

[1236] "Troubleshooting" is the process of identifying the cause of a problem and providing a solution when one occurs.

[1237] "Means of supporting review" refer to methods or devices that enable users to review and correct generated reports, etc.

[1238] This invention is a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence generates work instructions based on the analysis results. These instructions are then notified to remote operators, enabling real-time work instructions and troubleshooting. Furthermore, the system records the progress of the work and the history of interactions, and ultimately generates a work report based on these records.

[1239] 1. Terminals (construction machinery)

[1240] The terminal is equipped with sensors and cameras to collect information such as ground conditions, equipment operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the camera acquires visual information. Specifically, the temperature sensor measures the ground temperature, and the pressure sensor measures the load of the heavy machinery. The camera captures video of the excavation area in real time.

[1241] 2. Data compression and transmission

[1242] The terminal converts the collected real-time data into a compressed format and transmits it to the control center via a wireless network (e.g., a 5G network) using a secure communication protocol. Specifically, it packages the data obtained from each sensor into a single file using a data compression algorithm and transmits the data via a 5G modem.

[1243] 3. Receiving and decompressing data

[1244] The server (control center) receives compressed data sent from the terminal. The received data is reconstructed using a decompression tool. The server's communication module takes in the received data, decompresses the ZIP file, and converts it into JSON format data.

[1245] 4. Data analysis and work instruction generation

[1246] The server passes the decompressed data to analysis tools and algorithms for analysis. For example, it uses an image processing library (e.g., OpenCV) to analyze the ground conditions and detect abnormal cracks and obstacles. The analyzed data is then passed to artificial intelligence (AI) to generate specific work instructions in conjunction with the overall work plan. For example, it automatically generates instructions such as "Excavate from position X to position Y in area A."

[1247] 5. Sending work instructions

[1248] The server sends the generated work instructions to the remote operator using a communication protocol (e.g., MQTT), and displays the instructions on the operator's terminal. The instructions are formatted in a specific text format. The remote operator checks the instructions sent from the server on their terminal's display and sees an instruction such as "Start drilling towards position X."

[1249] 6. Troubleshooting

[1250] Artificial intelligence (AI) instantly analyzes the situation when a problem occurs and provides real-time troubleshooting instructions. For example, it might instruct the machine to "raise the bucket and move backward if the heavy machinery gets stuck on the ground." When an anomaly is detected from sensor data, the AI ​​performs fault tree analysis and generates appropriate troubleshooting steps.

[1251] 7. Recording of work history

[1252] The server meticulously records each work order and its results, as well as the troubleshooting and resolution history, in a database. Work order and result data are saved to the SQL database using INSERT commands. Troubleshooting results are recorded similarly.

[1253] 8. Generation and review of work reports

[1254] Artificial intelligence (AI) integrates recorded progress and historical data to generate a draft work report. For example, it might create a report stating, "Today's excavation work was 80% complete, with two anomalies. The solutions are as follows..." The server sends the generated draft report to the user, who then reviews and makes final edits to complete the final work report. The draft report is generated in PDF format and sent to the user as an email attachment. The user then makes revisions using the editing tool.

[1255] Example of a prompt

[1256] "The construction machine recorded a ground temperature of 35 degrees Celsius from its ground temperature sensor, and the vibration sensor detected strong vibrations. Please use this data to generate the next work instruction."

[1257] In this way, the system of the present invention improves work efficiency and enables accurate recording and reporting at construction sites. Furthermore, real-time troubleshooting improves safety and work accuracy.

[1258] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1259] Specific processing steps of the system program

[1260] Step 1:

[1261] The terminal uses sensors and cameras to collect information such as ground conditions, equipment operating status, and location. Inputs are physical data measured by sensors (temperature, pressure, vibration, etc.) and visual data captured by the camera. Outputs are the collected sensor data and image data. Specifically, the temperature sensor measures the ground temperature, the pressure sensor measures the load of heavy machinery, and the camera acquires visual information.

[1262] Step 2:

[1263] The terminal packages the collected real-time data in a compressed format. The inputs are the sensor data and image data collected in step 1. A compression algorithm is used to efficiently transmit this data, and a compressed data package is obtained as the output. Specifically, the data compression algorithm is used to compress the data obtained from each sensor and package it into a single file.

[1264] Step 3:

[1265] The terminal transmits compressed data to the control center via a wireless network (e.g., a 5G network) using a secure communication protocol. The input is the compressed data generated in step 2. The output is the data transmitted to the control center. Specifically, the data is transmitted via a 5G modem.

[1266] Step 4:

[1267] The server (control center) receives compressed data sent from the terminal. The input is the compressed data sent from the terminal. The output is the received compressed data. Specifically, the server's communication module takes in the received data.

[1268] Step 5:

[1269] The server reconstructs the received compressed data using a decompression tool and converts it into an analyzable format. The input is the compressed data received in step 4. The output is the decompressed and analyzable data. Specifically, it decompresses a ZIP file and converts the data into JSON format.

[1270] Step 6:

[1271] The server passes the decompressed data to analysis tools and algorithms for analysis. The input is the analyzable data decompressed in step 5. The output is the analyzed data and results. Specifically, it uses an image processing library (e.g., OpenCV) to analyze the ground conditions and detect abnormal cracks and obstacles.

[1272] Step 7:

[1273] Artificial intelligence (AI) generates specific work instructions by referring to the analysis results and the overall work plan. The input is the analysis results and data obtained in step 6. The output is the generated work instructions. Specifically, it reads the current progress from the work plan database and generates the optimal work instructions by comparing them with the analysis results.

[1274] Step 8:

[1275] The server sends the generated work instructions to the remote operator using a communication protocol and displays the instructions on the operator's terminal. The input is the work instructions generated in step 7. The output is the work instructions sent to the remote operator. Specifically, the work instructions are converted to JSON format and sent to the operator's tablet terminal via a communication protocol (e.g., MQTT).

[1276] Step 9:

[1277] Artificial intelligence (AI) immediately analyzes the situation when a problem occurs and provides troubleshooting instructions. The input is sensor data and image data collected in real time. The output is specific troubleshooting instructions. Specifically, when an anomaly is detected from the sensor data, the AI ​​performs fault tree analysis and generates appropriate troubleshooting steps.

[1278] Step 10:

[1279] The server meticulously records each work order and its results, as well as troubleshooting and its resolution history, in a database. Inputs include work orders and their results, and troubleshooting steps and results. Outputs are the historical data stored in the database. Specifically, the work order and result data are saved to the SQL database using INSERT commands.

[1280] Step 11:

[1281] Artificial intelligence (AI) integrates recorded progress and historical data to generate a draft work report. The input is progress and historical data recorded in a database. The output is a draft work report. Specifically, it uses natural language generation (NLG) technology to create the report.

[1282] Step 12:

[1283] The server sends a draft of the generated report to the user for review and final editing. The input is the draft of the work report generated in step 11. The output is the draft report sent to the user. Specifically, the server generates the draft report in PDF format and sends it to the user as an email attachment.

[1284] (Application Example 1)

[1285] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1286] There were challenges with conventional technology in improving the operational efficiency of industrial machinery and detecting and responding quickly to anomalies in real time. Furthermore, the generation of response instructions and work reports in the event of an anomaly was not automated, resulting in time losses and human errors.

[1287] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1288] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on industrial machinery, means for transmitting the collected real-time data to a control center via a wireless network, means for analyzing the received real-time data, artificial intelligence means for generating work instructions based on the analysis results, means for notifying a remote operator of the generated work instructions, means for recording the progress of work and the history of interactions during work, means for generating a work report based on the recorded progress and history, means for detecting abnormalities during work and immediately generating response instructions, and means for processing image data acquired by cameras mounted on industrial machinery in real time and detecting abnormal locations. This improves the work efficiency of industrial machinery and enables real-time abnormality detection and rapid response.

[1289] "Industrial machinery" is a general term for automated machines and equipment used in manufacturing and production lines.

[1290] A "sensor" is a device that measures physical quantities such as temperature, pressure, and vibration, and converts them into electrical signals.

[1291] A "camera" is a device that can capture visual information and record and transmit it as digital data.

[1292] "Real-time data" refers to data that indicates the current state of a machine while it is in operation and can be processed or transmitted immediately.

[1293] A "wireless network" is a communication method that uses radio waves to transmit data, and specifically includes Wi-Fi and 5G.

[1294] A "control center" is a facility or system for centrally managing, analyzing, and controlling data from industrial machinery.

[1295] "Means of analysis" refer to processes and devices for breaking down and analyzing received data and extracting useful information.

[1296] "Artificial intelligence means" refers to algorithms and systems that automatically generate judgments and instructions based on collected data.

[1297] A "remote operator" is a person or device that operates a machine or system from a physically distant location.

[1298] "Work progress" refers to information indicating the extent to which planned work has been completed.

[1299] "History" refers to the record of all past work and communications.

[1300] A "work report" is a document that includes details such as the progress of the work and how any problems were handled.

[1301] "Means for detecting abnormalities" refers to a device or method for detecting a state that deviates from normal operating conditions.

[1302] "Means for generating response instructions" refers to a device and algorithm that automatically determines an appropriate response method for a detected anomaly and outputs it as an instruction.

[1303] "Means for processing image data" refers to software and hardware used to analyze images captured by a camera and extract useful information.

[1304] This invention relates to a system that collects real-time data from sensors and cameras mounted on industrial machinery and transmits that data to a control center via a wireless network. Specifically, it is implemented as follows.

[1305] First, industrial machinery is equipped with various sensors such as temperature sensors, pressure sensors, and vibration sensors, as well as cameras. These sensors measure the machine's operating status and environmental conditions in real time, while the cameras capture visual information of the work area. The data collected from the sensors and cameras undergoes basic processing and compression at the terminal.

[1306] Next, the terminal transmits the collected data to the control center via a wireless network (e.g., Wi-Fi or 5G). A server located at the control center decompresses the received data and converts it into an analyzable format. The software used includes libraries for data decompression (e.g., zlib) and libraries for image analysis (e.g., OpenCV, TensorFlow).

[1307] Next, the data, converted into an analyzable format, is analyzed by artificial intelligence (AI). The AI, for example, uses image processing algorithms to detect anomalies in the work area or applies anomaly detection algorithms to sensor data. The analysis results are compared with the overall work plan, and specific work instructions are generated by the AI ​​model. These AI models function as generative AI models, employing, for example, deep learning algorithms.

[1308] The generated work instructions are sent from the server to the remote operator via a wireless network. The instructions are displayed in a specific text format on the terminal used by the operator. For example, an instruction such as "Excavate the path from position X to position Y in order to excavate the next area B" might be issued.

[1309] Furthermore, if an anomaly occurs during operation, the system immediately detects the anomaly and generates a response instruction. This instruction is also notified to the remote operator. For example, a specific response instruction such as "The machine has detected an abnormal temperature, so stop it immediately and begin cooling" is provided.

[1310] All work instructions, their results, and the history of anomaly responses are recorded in a database. This recorded data is later integrated by AI and generated as a work report. The report includes work progress, any anomalies that occurred, and how they were addressed. Users can review the generated report and make revisions as needed.

[1311] Specific example

[1312] Examples of prompts to input into a generative AI model:

[1313] This is real-time monitoring data from automated robots operating within the factory.

[1314] Sensor data: Temperature 30°C, Pressure 1.5, Vibration 0.3

[1315] Camera image: base64_encoded_image_data

[1316] Analysis results: An anomaly was detected in a part of the work line, so the following instructions were generated.

[1317] Instructions: Robot 1 should continue its work, avoiding sensor anomaly area B, and send an alert to the person in charge.

[1318] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1319] Step 1:

[1320] The terminal collects real-time data from sensors and cameras mounted on industrial machinery. Sensors measure physical quantities such as temperature, pressure, and vibration, while cameras capture images of the work area. The collected data is stored as sensor data (temperature, pressure, vibration, etc.) and image data (JPEG format).

[1321] Step 2:

[1322] The terminal performs basic processing on the collected real-time data and then compresses it. For example, it might use the zlib library to compress the data. The compressed data is then prepared to be transmitted over the wireless network as data packets.

[1323] Step 3:

[1324] The device transmits data packets to the control center via a wireless network (e.g., Wi-Fi or 5G). Wireless communication protocols are used, and data security is protected by encryption technology.

[1325] Step 4:

[1326] The server receives the data packets sent from the terminal and decompresses them. This restores the sensor data and image data to their original formats. The zlib library is used here as well.

[1327] Step 5:

[1328] The server analyzes the decompressed data. An anomaly detection algorithm is applied to the sensor data, and image processing algorithms such as OpenCV or TensorFlow are used to detect anomalies in the image data. The analysis results are saved for the next processing step.

[1329] Step 6:

[1330] The server uses a generated AI model to create specific work instructions based on the analysis results. For example, using a deep learning algorithm, it might generate an instruction such as, "To excavate the next area B, excavate a path from position X to position Y." This instruction is formatted in text format.

[1331] Step 7:

[1332] The server notifies the remote operator of the generated work instructions via the wireless network. The instructions are displayed in specific text format on the terminal used by the operator. For example, the instructions may include "Stop the pump and start cooling."

[1333] Step 8:

[1334] The server records the progress of the work and the history of interactions during the work in a database. The recorded data includes each work instruction, its execution result, and a history of how errors were handled.

[1335] Step 9:

[1336] The server generates work reports based on recorded progress and historical data. AI integrates this data and creates a draft of the work report. For example, the report might be in the format of, "Today's work was 90% complete, and 3 anomalies occurred. The specific actions taken are as follows..."

[1337] Specific example

[1338] Examples of prompts to input into a generative AI model:

[1339] This is real-time monitoring data from automated robots operating within the factory.

[1340] Sensor data: Temperature 30°C, Pressure 1.5, Vibration 0.3

[1341] Camera image: base64_encoded_image_data

[1342] Analysis results: An anomaly was detected in a part of the work line, so the following instructions were generated.

[1343] Instructions: Robot 1 should continue its work, avoiding sensor anomaly area B, and send an alert to the person in charge.

[1344] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1345] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery and transmits that data to a control center via a wireless network. The control center analyzes the received data, and artificial intelligence (AI) generates work instructions based on the analysis results. The generated work instructions are notified to the remote operator, enabling real-time work instructions and troubleshooting. The system also records the progress of the work and the history of interactions, and ultimately generates a work report based on the records. Furthermore, this invention incorporates an emotion engine to recognize the user's emotions and adjust the content of work instructions and troubleshooting accordingly.

[1346] Specific Examples of the System

[1347] 1. The terminal (construction machinery) is equipped with sensors and cameras to collect information such as ground conditions, machine operating status, and location. These sensors capture information such as temperature, pressure, and vibration, while the cameras acquire visual information.

[1348] 2. The terminal packages the collected real-time data into a compressed format and transmits it to the control center via a wireless network (5G network) using a secure protocol.

[1349] 3. The server (control center) receives data transmitted from terminals in real time and converts the data into a format that can be decompressed and analyzed. The received data is then used for analysis, such as analyzing ground conditions and progress using image processing algorithms, and detecting anomalies from sensor data.

[1350] 4. The analyzed data is passed to artificial intelligence (AI), which compares it with the overall work plan to generate specific work instructions based on the analysis results. For example, it automatically creates instructions such as, "To excavate the next area A, excavate the path from position X to position Y."

[1351] 5. The server notifies the remote operator of the generated work instructions using a communication protocol and displays the instructions on the operator's terminal. The instructions are formatted in a clear, easy-to-understand text format.

[1352] 6. When a problem occurs, the generative AI immediately analyzes the situation and provides troubleshooting instructions, such as "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[1353] 7. The server meticulously records and stores in a database the history of each work instruction and its results, as well as troubleshooting and its resolution.

[1354] 8. Recorded progress and historical data are integrated by artificial intelligence (AI) to ultimately generate a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..."

[1355] 9. The server sends a draft of the generated report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[1356] Specific examples of the emotion engine

[1357] 10. The device incorporates an emotion engine that recognizes the user's emotions, analyzing the user's tone of voice, facial expressions, and operation speed during operation to monitor their emotional state in real time.

[1358] 11. The emotion engine collects and analyzes the user's emotional data. For example, if the user's stress level is high or they are confused, the emotion engine recognizes that state.

[1359] 12. The analysis results are sent to the server and used to adjust real-time work instructions and troubleshooting content. For example, if the user is under high stress, the instructions may be made simpler and support messages may be inserted.

[1360] 13. The results of the emotion engine analysis are also reflected in the work report. For example, an assessment of the user's emotional state, such as "The user experienced a high level of stress during today's work and therefore requires support," is added to the report.

[1361] In this way, the system of the present invention not only improves work efficiency, accuracy, and safety at construction sites, but also enables adjustments and support that take into account the user's emotional state. A key feature is that the emotion engine can grasp the user's stress and satisfaction levels in real time, thereby reducing their burden.

[1362] The following describes the processing flow.

[1363] Step 1:

[1364] The terminal (construction machinery) collects real-time data, including ground conditions at the construction site, equipment operating status, and location information, using sensors and cameras. Sensors capture information such as temperature, pressure, and vibration, while cameras acquire visual information.

[1365] Step 2:

[1366] The real-time data collected by the terminal is packaged into a compressed format and transmitted to the control center via a wireless network (5G network) using a secure protocol.

[1367] Step 3:

[1368] The server (control center) receives data sent from the terminal in real time, decompresses the data, and converts it into an analyzable format.

[1369] Step 4:

[1370] The server analyzes the received data, uses image processing algorithms to check the ground conditions and progress, and detects any abnormalities from the sensor data.

[1371] Step 5:

[1372] The generative AI generates specific work instructions based on the analysis results and the overall work plan. For example, it might create an instruction such as, "To excavate the next area A, excavate the path from position X to position Y."

[1373] Step 6:

[1374] The server generates work instructions and notifies the remote operator using a communication protocol, displaying the instructions on the operator's terminal. This ensures that instructions are provided in a specific and easy-to-understand format.

[1375] Step 7:

[1376] When a problem occurs, the generative AI immediately analyzes the situation and provides appropriate troubleshooting instructions. For example, it might issue an instruction such as, "If the heavy machinery gets stuck on the ground, raise the bucket and move backward."

[1377] Step 8:

[1378] The emotion engine collects user emotion data. It monitors the user's emotional state in real time based on factors such as tone of voice, facial expressions, and operation speed, and generates emotion data.

[1379] Step 9:

[1380] The emotion engine analyzes collected emotional data to assess whether the user is experiencing high stress or satisfaction. The analysis results are sent to the server and used to adjust work instructions and troubleshooting content.

[1381] Step 10:

[1382] The server adjusts work instructions and troubleshooting based on emotional data. For example, if a user is in a high-stress state, it will simplify the instructions and add supportive messages.

[1383] Step 11:

[1384] The server meticulously records and stores in a database each work order and its results, troubleshooting history and resolution, and emotional state.

[1385] Step 12:

[1386] The generative AI generates a draft work report based on recorded progress and history data, as well as sentiment data. For example, it might write, "Today's excavation work is 80% complete, and two anomalies occurred. The solutions are as follows..." and include an assessment of the user's sentiment state.

[1387] Step 13:

[1388] The server sends a draft of the generated report to the user, who then reviews and makes final edits. They revise as needed to complete the final work report.

[1389] (Example 2)

[1390] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1391] Real-time data collection and analysis are essential for highly efficient and precise work on construction sites. However, conventional systems often suffer from delays in data collection, analysis, and the generation and notification of work instructions. Furthermore, instructions are not adjusted to take into account the user's emotional state, leading to challenges in operational efficiency and user stress management. These challenges need to be addressed.

[1392] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1393] In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery; means for compressing the collected real-time data and transmitting it to a control center via a wireless network; means for converting the received real-time data into a format that can be decompressed and analyzed; artificial intelligence means for generating specific work instructions based on the analyzed data; means for notifying a remote operator of the generated work instructions; means for recording the progress of the work and the history of interactions during the work; means for generating a work report based on the recorded progress and history; means for collecting and analyzing user emotion data; and means for adjusting work instructions based on emotion data. This enables the provision of efficient and highly accurate work instructions in real time, as well as the adjustment of work instructions that take into account the user's emotional state, thereby improving work efficiency and safety.

[1394] "Construction machinery" refers to mechanical devices used to perform civil engineering work, excavation, and other tasks at construction sites.

[1395] A "sensor" is a device that detects the surrounding environment or the state of a machine and outputs that information as an electrical signal.

[1396] A "camera" is a device that captures visual information and records it as digital data.

[1397] "Real-time data" refers to data that is instantly acquired and processed with virtually no delay, reflecting the current state and situation.

[1398] A "wireless network" is a communication network that uses radio waves to send and receive data.

[1399] A "control center" is a base of operations for monitoring and controlling equipment and systems from a remote location.

[1400] "Data compression" is the process of converting large amounts of data so that they can be stored in less memory.

[1401] "Decompression" is the process of restoring compressed data to its original state.

[1402] An "analyzable format" is a format that converts data into a form that can be processed by analytical tools and algorithms.

[1403] "Artificial intelligence means" refers to technologies that analyze large amounts of data, learn from it, and generate appropriate results or instructions.

[1404] "Work instructions" are detailed procedures or commands for performing a specific task.

[1405] A "remote operator" refers to a person or system that operates remotely, without being physically present at the site.

[1406] "Troubleshooting" is the process of diagnosing machine malfunctions or abnormalities and proposing solutions.

[1407] "History" refers to a record of past work and interactions.

[1408] A "work report" is a document that summarizes the work performed, its progress, results, and any problems encountered.

[1409] "Emotional data" refers to information that indicates a user's emotional state, and is data obtained from sources such as voice, facial expressions, and behavior.

[1410] An "emotion engine" is a technology that analyzes a user's emotional data and evaluates their emotional state.

[1411] This invention relates to a system that collects real-time data from sensors and cameras mounted on construction machinery, transmits that data to a control center via a wireless network, and generates work instructions using artificial intelligence (AI) based on the analysis results. This improves the work efficiency and safety at construction sites.

[1412] Terminal hardware and software

[1413] The terminal (construction machinery) is equipped with multiple sensors (e.g., temperature sensors, pressure sensors, vibration sensors) and cameras. These sensors collect information such as ground conditions, machine operation status, and location in real time. The cameras acquire visual data using color and depth cameras. The collected data is compressed using dedicated data compression software (e.g., ZIP).

[1414] Sending data

[1415] The device transmits compressed data to the control center via the 5G network using a secure communication protocol (e.g., HTTPS). This communication ensures both data transmission speed and security.

[1416] Server hardware and software

[1417] The control center is equipped with high-performance servers that receive data transmitted via communication in real time. The received data is decompressed using data decompression software (e.g., Unzip) and converted into an analyzable format. Subsequently, various analysis algorithms (e.g., image processing algorithms, anomaly detection algorithms) are applied to analyze the data.

[1418] AI-generated work instructions

[1419] Based on the analyzed data, artificial intelligence (AI) generates appropriate work instructions. This AI takes into account the progress and conditions of a specific task and gives specific instructions for the next step. For example, it might generate an instruction such as, "To excavate the next area A, excavate from position X to position Y."

[1420] Work instruction notification

[1421] The server notifies the remote operator of the generated work instructions and displays the instructions on the operator's terminal. These instructions are displayed in text format and are easy to understand.

[1422] troubleshooting

[1423] If a problem occurs during operation, the terminal notifies the server of the situation. The server immediately analyzes the data, and the AI ​​generates appropriate troubleshooting steps and notifies the operator. For example, it provides specific instructions such as, "If the heavy machinery gets stuck, raise the bucket and move backward."

[1424] Data recording and work report generation

[1425] The server meticulously records each work instruction and its results, as well as troubleshooting and its resolution history, and stores it in a database. Based on these records, AI generates a draft of the work report. For example, it might generate a report in the format of, "Today's excavation work was 80% complete as planned, and two anomalies occurred. The solutions are as follows..." The server sends this draft report to the user, who reviews it and makes final edits to complete the work report.

[1426] Collection and analysis of emotional data

[1427] The device incorporates an emotion engine that recognizes the user's emotions, analyzing the user's tone of voice, facial expressions, and operation speed to monitor their emotional state in real time. The emotion engine collects this data and sends the analysis results to a server.

[1428] Adjusting work instructions based on emotional data

[1429] The server adjusts real-time work instructions and troubleshooting content based on the analysis results sent from the emotion engine. For example, if the user is in a high-stress state, the instructions will be simplified and supportive messages will be inserted.

[1430] These technological measures enable this system to significantly improve work efficiency and safety at construction sites. Furthermore, by considering the user's emotional state, it can reduce stress and provide more appropriate support.

[1431] Example of a prompt

[1432] For example, instructions might include, "Excavate a path from position X to position Y to excavate the next area A," or troubleshooting instructions such as, "If the heavy equipment gets stuck on the ground, lift the bucket and move backward."

[1433] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1434] Step 1:

[1435] Data collection from sensors and cameras using the device

[1436] The device collects real-time data using various built-in sensors (e.g., temperature sensor, pressure sensor, vibration sensor) and a camera. Inputs include the surrounding environment and the operating status of the machine. Sensor data is output in the form of temperature, pressure, vibration, and location information, while the camera captures visual data. The device collects this data and temporarily stores it in its internal storage.

[1437] Specific example: "The device measures the ambient temperature with a temperature sensor and takes pictures of the terrain with a camera."

[1438] Step 2:

[1439] Data compression and transmission by the terminal

[1440] To compress the collected data, the terminal applies a compression algorithm (e.g., ZIP). The inputs are the sensor data and camera data obtained in step 1. The compressed data is sent to the server via the 5G network using a secure communication protocol (e.g., HTTPS). This results in the compressed data being the output of the transmission.

[1441] Specific example: "The device compresses the collected data in ZIP format and sends it to the server using the HTTPS protocol."

[1442] Step 3:

[1443] Receiving and decompressing data by the server.

[1444] The server receives compressed data sent from the terminal. The input is compressed data. After receiving the data, the server uses data decompression software (e.g., Unzip) to decompress it and convert it into an analyzable format. The decompressed data then becomes the output.

[1445] Specific example: "The server receives a ZIP file from the terminal and unzips it using Unzip software."

[1446] Step 4:

[1447] Server-based data analysis

[1448] Based on the decompressed data, the server performs analysis by applying image processing algorithms and anomaly detection algorithms. The input consists of decompressed sensor data and camera data. After the analysis process, the server outputs analysis results such as terrain irregularities, temperature anomalies, and pressure fluctuations.

[1449] Specific example: "The server uses an image processing algorithm to detect topographic irregularities and an anomaly detection algorithm to identify anomalies in the temperature data."

[1450] Step 5:

[1451] AI-generated work instructions

[1452] Based on the analysis results, the AI ​​generates specific work instructions. The input is the analysis results obtained in step 4. The AI ​​compares these with the overall work plan and outputs work instructions such as, "To excavate the next area A, excavate from position X to position Y."

[1453] Specific example: "Based on the analysis results, the AI ​​instructs on a specific route for excavating the next area A."

[1454] Step 6:

[1455] Server-based work instruction notification

[1456] The server notifies the remote operator of the generated work instructions. The input is the work instructions obtained in step 5. The work instructions are displayed in text format and sent to the remote control terminal. This allows the operator to check the instructions in real time. The output is the notified work instructions.

[1457] Specific example: "The server notifies the operator of the generated work instructions via SMS and displays the instructions on the terminal's screen."

[1458] Step 7:

[1459] Terminal and server troubleshooting

[1460] If a problem occurs, the terminal notifies the server of the situation. The input is data about the problem. The server immediately analyzes this data, and the AI ​​generates appropriate troubleshooting steps. For example, it might output specific instructions such as, "If the heavy machinery gets stuck, raise the bucket and move backward."

[1461] Specific example: "After receiving abnormal data from a terminal, the server uses AI to generate an instruction to 'raise the bucket and move backward,' and notifies the operator."

[1462] Step 8:

[1463] Data recording by the server

[1464] The server meticulously records each work order and its results, as well as troubleshooting and its resolution history, and stores it in a database. Inputs are work orders, their execution results, and troubleshooting history. The recorded data is then output.

[1465] Specific example: "The server records all work orders and their execution results in the SQL database."

[1466] Step 9:

[1467] AI-powered work report generation

[1468] Based on the recorded data, the AI ​​generates a draft of the work report. The input is the progress data and historical data recorded in step 8. For example, a draft report in the format of "Today's excavation work was 80% complete and two anomalies occurred" will be output.

[1469] Specific example: "The AI ​​analyzes all historical data and generates a draft of the work report titled 'Today's Work Progress and Anomaly Response List'."

[1470] Step 10:

[1471] Server-based report submission and user review.

[1472] The server sends a draft of the generated report to the user. The input is the draft work report generated in step 9. The user reviews and edits it to complete the final work report. The output is the completed work report.

[1473] Specific example: "The server emails a draft of the report to the user, and saves the final version after the user has made edits."

[1474] Step 11:

[1475] Collection of emotional data using devices

[1476] The device uses an emotion engine to analyze the user's voice tone, facial expressions, and operation speed, and collects emotional data. The input is the user's voice and facial expressions during operation. This data is analyzed by the emotion engine, and emotional data is output.

[1477] Specific example: "The device captures the user's voice tone with a microphone and analyzes their facial expressions with a camera."

[1478] Step 12:

[1479] Analysis of emotional data using an emotion engine

[1480] The emotion engine analyzes the collected emotion data to evaluate the user's stress level and emotional state. The input is the emotion data collected in step 11. For example, if the tone of voice is high or the facial expression is strained, it will be evaluated as a "high-stress state," and the result will be output.

[1481] Specific example: "The emotion engine detects that the user's voice tone is high and determines that they are in a high-stress state."

[1482] Step 13:

[1483] Adjusting work instructions based on emotional data from the server.

[1484] The server adjusts real-time work instructions and troubleshooting content based on the analysis results of the emotion engine. The input is the analysis results of the emotion data obtained in step 12. For example, if the user is in a high-stress state, adjusted instructions will be output, with the instructions made simpler and support messages added.

[1485] Specific example: "The server will take into account the high-stress state detected by the emotion engine and add a support message such as 'Please remain calm while operating,' along with concise instructions."

[1486] Through the steps described above, this system provides real-time, efficient, and highly accurate work instructions, and supports operations while taking the user's emotional state into consideration, thereby improving work efficiency and safety at construction sites.

[1487] (Application Example 2)

[1488] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1489] Improving work efficiency and speeding up troubleshooting are crucial challenges in existing factory and construction sites. However, conventional systems lack sufficient real-time data collection and analysis, and in particular, they fail to adjust instructions to take into account the emotions and stress levels of operators. This makes it difficult to monitor work progress and respond immediately to problems, potentially increasing operator stress and confusion.

[1490] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting real-time data from sensors and cameras mounted on construction machinery, means for transmitting the collected real-time data to a control center via a wireless network, means for analyzing the received real-time data, artificial intelligence means for generating work instructions based on the analysis results, means for notifying a remote operator of the generated work instructions, means for recording the progress of the work and the history of interactions during the work, means for generating a work report based on the recorded progress and history, emotion engine means for monitoring the emotional state of the operator, and means for adjusting work instructions based on the emotion data generated by the emotion engine means. This enables not only efficient real-time data collection and analysis and troubleshooting, but also flexible instructions that take into account the operator's emotions.

[1491] A "sensor" is a device that measures physical conditions and outputs them as electrical signals.

[1492] A "camera" is a device that receives light, forms an image, and stores or transmits it as digital data.

[1493] "Real-time data" refers to data that is acquired and transmitted instantaneously without delay.

[1494] A "wireless network" is a means of communication that uses radio waves to send and receive data.

[1495] A "control center" is a centralized control system or location for analyzing and managing collected data.

[1496] "Analysis" is the process of analyzing data and extracting meaning as information.

[1497] "Artificial intelligence" is a technology that uses computer programs to mimic human intelligence, learn from data, and generate work instructions.

[1498] A "remote operator" refers to a person or device that operates or manages a machine or system from a remote location.

[1499] "History" refers to data that records the progress and results of a task or event in chronological order.

[1500] A "work report" is a document that records and summarizes the progress, results, and details of any problems that occurred during a series of tasks.

[1501] An "emotion engine" is software that analyzes the user's voice tone, facial expressions, and operation speed to monitor the user's emotional state in real time.

[1502] This invention relates to a system that improves the work efficiency of robots in a factory and provides instructions that take into account the operator's emotions. Specific embodiments for carrying out this invention are described below.

[1503] Hardware and software

[1504] Hardware:

[1505] Smartphone (iOS / Android): Used by the operator to receive instructions.

[1506] Factory robots: Devices for operating machinery, equipped with numerous sensors and cameras.

[1507] software:

[1508] Languages: Python, Java (Android), Swift (iOS)

[1509] Frameworks: TensorFlow (AI analysis), OpenCV (image processing), Flask (backend API)

[1510] Network: 5G, Wi-Fi

[1511] Data collection

[1512] There are methods for collecting real-time data from factory robots' sensors and cameras. Sensors measure physical quantities such as temperature, pressure, and vibration, while cameras acquire visual data. This real-time data is transmitted to a server via a wireless network (e.g., 5G).

[1513] Data Analysis

[1514] The server has means to analyze the received real-time data. For example, sensor data is analyzed using an anomaly detection algorithm, and visual data acquired from cameras is analyzed using an image processing algorithm. Software tools such as TensorFlow and OpenCV are used in this analysis process.

[1515] Work order generation

[1516] The server has an artificial intelligence system that generates work instructions based on the analysis results. The AI ​​model compares the analysis results with the overall work plan and generates specific work instructions. For example, it might create instructions such as, "Investigate the following area."

[1517] Instructions, notifications, and coordination

[1518] The generated work instructions are notified to the remote operator using a communication protocol. Furthermore, the server has an emotion engine that monitors the operator's emotional state and adjusts the work instructions based on the emotional data generated by the emotion engine. If the operator is under stress, adjustments are made, such as simplifying the instructions and inserting support messages.

[1519] History recording and report generation

[1520] The server has means to record the progress of work and the history of interactions during the work in detail. It also has means to generate work reports based on the recorded progress and history data. For example, it can automatically generate a report such as, "Today's work was 80% completed as planned, and two anomalies occurred."

[1521] As a concrete example, the following prompt statements can be used:

[1522] "Develop an AI system for monitoring factory work progress and troubleshooting. Collect real-time data from sensors and cameras, analyze it on a remote server to generate work instructions, and adjust those instructions based on user sentiment."

[1523] This enables real-time data collection and analysis, more efficient troubleshooting, and flexible instructions that take operator emotions into consideration.

[1524] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1525] Step 1:

[1526] The terminal (factory robot) collects real-time data such as ground conditions, machine operating status, and location information from its mounted sensors and cameras. The sensors measure physical quantities such as temperature, pressure, and vibration, while the cameras capture visual data. The collected data is packaged in a compressed format. The input consists of machine physical quantities and image data, and the output is real-time data in a compressed format.

[1527] Step 2:

[1528] The terminal transmits collected real-time data to a server via a wireless network (e.g., 5G). A secure protocol is used for transmission, ensuring data confidentiality and integrity. The input is real-time data in a compressed format, and the output is a notification that the data transmission to the server is complete.

[1529] Step 3:

[1530] The server converts received real-time data into a format that can be decompressed and analyzed. To convert it into the format required for analysis, it separates sensor data and image data from a specific data stream and uses them for their respective processing. The input is compressed data, and the output is decompressed sensor data and image data.

[1531] Step 4:

[1532] The server analyzes the decompressed sensor data and image data. First, it applies an anomaly detection algorithm to the sensor data to detect patterns that are different from the norm. Next, it performs image processing on the image data using OpenCV or similar tools to extract specific visual information. The input is the decompressed sensor data and image data, and the output is the anomaly detection results and the visual information extraction results.

[1533] Step 5:

[1534] The server uses an artificial intelligence model (generative AI model) to generate work instructions based on the analysis results. Based on the analysis results and the overall work plan, the AI ​​model generates specific work instructions. For example, it might generate an instruction such as "Investigate the next area A." The input is the anomaly detection results and the visual information extraction results, and the output is the generated work instructions.

[1535] Step 6:

[1536] The server notifies the remote operator's smartphone of the generated work instructions. It uses a communication protocol to notify the instructions in an easily understandable format (e.g., text message). The input is the generated work instructions, and the output is the instruction notification to the remote operator's terminal.

[1537] Step 7:

[1538] The server uses an emotion engine to monitor the operator's emotional state. It analyzes the operator's voice tone, facial expressions, and operating speed, collecting emotional data in real time. The input is the operator's actions and voice data, and the output is the analyzed emotional data.

[1539] Step 8:

[1540] The server adjusts work instructions based on emotional data generated by the emotion engine. For example, if an operator is under high stress, the instructions are simplified and supportive messages are inserted. The input is the analyzed emotional data and existing work instructions, and the output is the adjusted work instructions.

[1541] Step 9:

[1542] The server meticulously records the progress of work and the history of interactions. This history includes data analysis results, work instructions, and sentiment data for each step. Inputs are various analysis results and instructions, while output is a detailed history record.

[1543] Step 10:

[1544] The server generates work reports based on recorded progress and historical data. These reports include progress status, occurrences and responses to anomalies, and the operator's emotional state. For example, it might automatically generate a report stating, "Today's work was 80% complete, and two anomalies occurred." The input is detailed historical records, and the output is the work report.

[1545] As described above, by realizing this invention, it is possible to improve work efficiency and reduce the mental burden on operators.

[1546] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1547] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1548] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1549] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1550] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1551] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1552] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1553] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1554] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1555] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1556] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1557] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1558] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The proce...

Claims

1. A means for collecting real-time data from sensors and cameras mounted on construction machinery, A means for transmitting the collected real-time data to a control center via a wireless network, A means of analyzing the received real-time data, An artificial intelligence means for generating work instructions based on analysis results, A means of notifying the remote operator of the generated work instructions, A means for recording the progress of the work and the history of communication during the work, Means for generating work reports based on recorded progress and history, A system that includes this.

2. The system according to claim 1, further comprising means for troubleshooting based on analysis results and proposing solutions.

3. The system according to claim 1, further comprising means for generating a draft of a work report based on recorded data and sending it to a user to assist in review.

Images