System

Abstract

A system is provided.SOLUTION: A system including means for capturing an image or a moving image of on-site work, means for transmitting the captured image or moving image to a server, means for formatting data into a state that can be analyzed by the server, means for analyzing the image or moving image using generative artificial intelligence in the server to identify a hot spot, means for generating information on the identified hot spot in a chart / text / voice format, means for transmitting the generated feedback data to a terminal, and means for presenting feedback to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Safety patrols for on-site work involve humans visiting the site and pointing out dangerous areas, which poses challenges in terms of consistent evaluation and the time and effort required. Patrolling is also difficult, particularly at regional bases, making it difficult to respond quickly. This has led to a need to reduce evaluation errors and improve the efficiency of safety patrols. Furthermore, detailed feedback on work and skill improvement are necessary, but achieving these is difficult. The purpose of this invention is to solve these challenges and realize more efficient and consistent safety patrols. [Means for solving the problem]

[0005] This invention provides a system for streamlining safety patrols for on-site work. Specifically, the system includes a means for capturing images and videos of on-site work, a means for transmitting the captured images and videos to a server, a means for formatting the data so that it can be analyzed by the server, a means for analyzing the images and videos on the server using generative artificial intelligence to identify hazardous areas, a means for generating information about the identified hazardous areas in diagram, text, and audio formats, a means for transmitting the generated feedback data to a terminal, and a means for presenting the feedback to the user. This eliminates the problem of evaluation errors, provides consistent feedback, reduces travel time and labor pressure, and enables efficient safety patrols for all work. Furthermore, the generative artificial intelligence model learns procedures and drawings in advance and uses them to analyze on-site work, providing more specific and intuitive feedback. Furthermore, the feedback data can be saved in log format and used for later analysis and improvement, contributing to safety measures and skill improvement.

[0006] "On-site work" refers to work that is physically performed at a construction or manufacturing site.

[0007] "Means for taking images and videos" refers to camera equipment or a terminal equipped with such functionality for recording the status of on-site work.

[0008] "Server" refers to a computer system that receives, stores, and processes information sent from a terminal via a network.

[0009] "Means for formatting data" refers to a processing method for converting received image or video data into a format suitable for analysis.

[0010] "Generative artificial intelligence (AI)" refers to algorithms and models that can analyze images and videos, assess situations, and identify dangerous areas.

[0011] "Means of analysis" refers to the processing methods and technologies used to analyze the contents of captured images and videos and identify dangerous areas.

[0012] "Dangerous areas" refer to places or situations where there are problems with work safety and the possibility of causing accidents or health damage.

[0013] "Means of generating information in the form of diagrams, text, and audio" refers to a method of generating information that can clearly communicate the details of identified dangerous areas visually and audibly.

[0014] "Terminals" refers to digital devices such as smartphones, tablets, and wearable devices carried by workers and supervisors.

[0015] "Feedback data" refers to information and instructions about dangerous areas generated based on the analysis results.

[0016] "Log format" refers to a data format that stores records of a series of operations and events in chronological order. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a system for improving the efficiency of safety patrols for on-site work. This system is realized by taking images and videos of on-site work, sending them to a server, analyzing them, and providing the results to the site as feedback.

[0039] composition

[0040] 1. Taking pictures and videos

[0041] Users take pictures of the work situation on-site using devices such as smartphones or tablets. It is recommended to use a high-resolution camera that can record even complex work situations.

[0042] 2. Data transmission

[0043] The device sends the captured images and videos to the server via the network. Before sending, an appropriate compression method is used to maintain the quality of the image and video data.

[0044] 3. Data Formatting

[0045] The server converts the received data into a format suitable for analysis, specifically by extracting the necessary frames from images and videos and performing preprocessing.

[0046] 4. Analysis using generative artificial intelligence (AI)

[0047] The server uses pre-trained generative AI models to analyze the content of images and videos, for example, assessing the safety of scaffolding or the operation status of machinery based on work area procedures and safety standards.

[0048] 5. Identifying dangerous areas

[0049] Based on the analysis results performed by the server, dangerous areas are identified in real time, such as areas where handrails are not installed during work at height or where protective covers are not attached to equipment.

[0050] 6. Generate feedback

[0051] The server generates feedback data on identified hazards in the form of diagrams, text, and audio, including image highlights of the hazards and specific instructions for countermeasures.

[0052] 7. Sending and Providing Feedback

[0053] The server sends the generated feedback data to the terminal, which receives it and presents it to the user in the form of diagrams, text, and audio, allowing on-site workers to immediately recognize dangerous areas and take appropriate action.

[0054] 8. Feedback logging

[0055] The server stores each feedback in a detailed log format, which can be used for future analysis and improvement planning.

[0056] Specific examples

[0057] For example, when performing construction work at height, workers use their smartphones to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative artificial intelligence model. The model analyzes scaffolding defects and missing safety equipment and generates specific feedback, such as "There is a risk of falling because handrails are not installed." This feedback is sent to the worker's device in real time and displayed as visual alerts and audio guidance. Workers can immediately understand the situation and take necessary measures.

[0058] In this way, the system of the present invention allows patrols to be carried out efficiently and consistently to improve the safety of on-site operations.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] Before starting work on-site, users prepare their smartphone, tablet, or other device, launch the camera app, and, if necessary, organize the surrounding environment to make it easier to take photos.

[0062] Step 2:

[0063] Users can take video and still images of the progress and environment of on-site work, recording areas that require particular attention from multiple angles, allowing them to grasp the overall situation.

[0064] Step 3:

[0065] The device temporarily stores captured images and videos in local storage, compressing and converting the data as needed to maintain quality.

[0066] Step 4:

[0067] The device sends the captured images and videos to the server via the network. During the transmission, the progress is displayed for the user to check.

[0068] Step 5:

[0069] The server stores the received image and video data and formats it into an analyzable format, for example by extracting the necessary frames and performing preprocessing.

[0070] Step 6:

[0071] The server inputs image and video data into the generative AI model and begins analysis. The model has been trained in advance and can determine the work situation based on procedure manuals and drawings.

[0072] Step 7:

[0073] The generative AI model analyzes the content of images and videos to identify hazards, such as missing handrails or missing safety equipment, and returns the results.

[0074] Step 8:

[0075] Based on the analysis results, the server generates information on dangerous areas in the form of diagrams, text, and audio, highlighting specific areas and providing detailed explanations of appropriate countermeasures.

[0076] Step 9:

[0077] The server transmits the generated feedback data to the device in real time, which includes image, text, and audio information.

[0078] Step 10:

[0079] The device then presents the received feedback data to the user, for example by highlighting dangerous areas on the screen and providing specific precautions using text and audio guidance.

[0080] Step 11:

[0081] The server stores each feedback in a detailed log, providing data for later analysis and development of improvements.

[0082] Step 12:

[0083] Based on the feedback provided, users can take appropriate measures to address dangerous areas at the site, such as installing handrails and ensuring that safety equipment is worn.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] Safety patrols at on-site work sites consume a lot of human resources, are inefficient, and are difficult to implement consistently. Traditional methods are unable to respond quickly to changes in the work environment, which delays the identification of dangerous areas and immediate response. Furthermore, recording feedback and analyzing it for future use are also manual processes, which means it takes a long time to develop improvement measures.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes a means for capturing images and videos of on-site work, a means for transmitting the captured images and videos via a network, a means for converting the received image and video data into an analyzable format, a means for analyzing the content of the images and videos using a generative AI model and identifying hazardous areas, a means for generating information on the identified hazardous areas in diagram, text, and audio formats, a means for transmitting the generated feedback data back to the terminal, and a means for presenting the feedback to the user. This enables the efficiency and consistency of on-site work safety patrols. Furthermore, real-time identification of hazardous areas and instructions for response can be achieved, and the automated recording of feedback content can be used for future analysis and the development of improvement measures.

[0089] "Field work" refers to work such as construction, maintenance, and safety inspections carried out at a specific location.

[0090] "Images and videos" are visual data recorded using a photographic device such as a camera, and are used to visually evaluate the work situation.

[0091] "Network" means a communications infrastructure for transmitting data, including the Internet and an internal company network.

[0092] "Server" refers to a computing system that receives, processes, or transmits data.

[0093] A "terminal" is a device that a user directly uses, such as a smartphone or tablet.

[0094] A "generative AI model" is an artificial intelligence model that has been trained to perform specific tasks in advance and is used to analyze the content of images and videos.

[0095] "Analysis" is the process of dissecting data to find meaning and patterns.

[0096] "Hazardous point" refers to an area or condition in the work environment that does not meet safety standards and that presents a risk of ingestion.

[0097] "Feedback" is information generated based on the analysis results, and is used to alert the user or give instructions for improvement.

[0098] A "log" is a collection of data that records system operations and analysis results, and is used for later analysis and verification.

[0099] MODE FOR CARRYING OUT THE INVENTION

[0100] This invention is a system for improving the efficiency of safety patrols at on-site work, and automates the process of taking, transmitting, analyzing, and providing feedback on images and videos of on-site work. Below, we will explain each component of this system and its specific operation.

[0101] Hardware and software used

[0102] 1. Hardware

[0103] The devices that users use are smartphones and tablets, which are equipped with high-resolution cameras.

[0104] A server is a high performance computer system.

[0105] 2. Software

[0106] OpenCV is used as the image processing library.

[0107] The generative AI model is built using TENSORFLOW® or PyTorch.

[0108] Use standard communication protocols (e.g. HTTP / HTTPS) for network communication.

[0109] System Overview

[0110] 1. Taking pictures and videos

[0111] Users use their devices to take photos of the work being done on-site, and the captured data is recorded in detail using a high-resolution camera.

[0112] 2. Data transmission

[0113] The device sends the captured images and videos to the server over the network. Before transmission, the data is compressed using a compression algorithm (e.g., JPEG or H.264).

[0114] 3. Data format conversion

[0115] The server converts the received data into a format suitable for analysis. Specifically, it extracts the necessary frames from images or videos and performs preprocessing using an image processing library such as OpenCV.

[0116] 4. Analysis using generative AI models

[0117] The server uses a pre-trained generative AI model to analyze the content of images and videos, and the analysis is based on work area procedures and safety standards to assess the safety of the work.

[0118] 5. Identifying dangerous areas

[0119] Based on the analysis results, the server identifies dangerous areas, such as areas where handrails are not installed when working at height, or areas where protective covers are not attached to equipment.

[0120] 6. Generate feedback

[0121] The server generates graphical, textual, and audio feedback data about identified hot spots, including image highlights of hot spots and specific instructions for countermeasures.

[0122] 7. Sending and Providing Feedback

[0123] The server then sends the generated feedback data to the terminal, which receives it and presents it to the user in the form of diagrams, text, and audio, allowing on-site workers to immediately recognize dangerous areas and take appropriate action.

[0124] 8. Feedback logging

[0125] The server stores each feedback in a detailed log format, which can be used for later analysis and development of improvement measures.

[0126] Specific examples

[0127] For example, when performing construction work at height, workers use their smartphones to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative AI model. The model analyzes scaffolding defects and missing safety equipment, generating specific feedback such as "There is a risk of falling because handrails are not installed." This feedback is sent to the worker's device in real time and displayed as visual alerts and audio guidance. Workers can immediately understand the situation and take necessary measures.

[0128] Prompt Sentence Examples

[0129] "This is an image and video analysis system for safety confirmation at construction sites. It checks the safety of working at heights, identifies dangerous areas such as missing handrails or missing safety equipment, and provides feedback in real time. This can improve safety at work sites."

[0130] This is a specific mode for carrying out the invention.

[0131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0132] Step 1:

[0133] Taking pictures and videos

[0134] Users take photos of the work situation on-site using a smartphone or tablet.

[0135] Input: On-site working environment.

[0136] Specific actions: High-resolution cameras are used to capture detailed images of the work area and potential hazards.

[0137] Output: High resolution image and video data.

[0138] Step 2:

[0139] Sending data

[0140] The device sends the captured images and videos to a server via the network.

[0141] Input: High-resolution image and video data captured.

[0142] Specific operation: Before transmitting data, the amount of data is reduced using a compression algorithm (e.g., JPEG or H.264). The compressed data is sent to the server using a network communication protocol (e.g., HTTP / HTTPS).

[0143] Output: Compressed image and video data is sent to the server.

[0144] Step 3:

[0145] Data format conversion

[0146] The server converts the received data into a format suitable for analysis.

[0147] Input: Compressed image and video data.

[0148] Specific operation: Using an image processing library such as OpenCV, extract important frames from the video data and save them as image data. Also, perform preprocessing such as resizing and filtering of the image data.

[0149] Output: Image data in a format suitable for analysis.

[0150] Step 4:

[0151] Analysis using generative AI models

[0152] The server uses a pre-trained generative AI model to analyze the content of images and videos.

[0153] Input: Image data in a format suitable for analysis.

[0154] How it works: Apply generative AI models built with TensorFlow or PyTorch to analyze image data, specifically assessing it against work area procedures and safety standards.

[0155] Output: Analysis results (e.g., identification of dangerous areas and non-fulfillment of safety standards).

[0156] Step 5:

[0157] Identifying dangerous areas

[0158] The server identifies dangerous areas based on the analysis results.

[0159] Input: Analysis results from the generative AI model.

[0160] Specific actions: Review the analysis results and identify and list areas where handrails have not been installed when working at height, areas where protective covers have not been installed on equipment, etc.

[0161] Output: A list of identified hazards.

[0162] Step 6:

[0163] Generate feedback

[0164] The server generates feedback data in the form of diagrams, text, and audio about identified dangerous areas.

[0165] Input: A list of identified hazards.

[0166] Specific actions: Highlight images of dangerous areas and create diagrams, text, and audio data containing specific instructions for countermeasures.

[0167] Output: Feedback data (e.g., in diagram, text, or audio format).

[0168] Step 7:

[0169] Sending and Providing Feedback

[0170] The server transmits the generated feedback data to the terminal.

[0171] Input: Feedback data.

[0172] Specific operation: The generated feedback data is sent to the terminal using a network communication protocol. The terminal receives the data and presents it to the user visually and audibly.

[0173] Output: Feedback information presented to the user (e.g., image highlighting of dangerous areas, specific instructions for countermeasures).

[0174] Step 8:

[0175] Feedback logging

[0176] The server stores each feedback in a detailed log format.

[0177] Input: Feedback data.

[0178] Specific operation: Feedback content is saved in log format using SQLite or a NoSQL database (e.g. MongoDB).

[0179] Output: A saved feedback log that can be used for later analysis and development of improvement measures.

[0180] (Application example 1)

[0181] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0182] Ensuring safety during on-site work is extremely important for protecting the lives of workers, but conventional methods lack consistency and efficiency. In particular, it is difficult to identify dangerous areas and take immediate action, and feedback management is insufficient. Therefore, a system is needed to ensure the safety of on-site work in real time and efficiently identify dangerous areas.

[0183] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0184] In this invention, the server includes means for formatting images and videos so that they can be analyzed, means for analyzing images and videos using generative artificial intelligence to identify dangerous areas, means for generating information about the identified dangerous areas in the form of diagrams, text, and audio, and means for presenting feedback data by audio and notifying the user of the dangerous areas and countermeasures. This makes it possible to ensure the safety of on-site work in real time, efficiently identify dangerous areas, and respond immediately.

[0185] "On-site work" refers to all work performed at a specific physical location, primarily including indoor and outdoor work activities such as construction sites and factories.

[0186] "Means for taking images or videos" means a device that has the function of capturing still images or videos using a photographing device such as a camera or video camera.

[0187] "Server" refers to the computer system utilized to receive, process, analyze data, and generate feedback.

[0188] "Generative artificial intelligence" refers to algorithms and models that learn, analyze, and make decisions based on large amounts of data.

[0189] "Dangerous areas" refer to locations or situations at work sites where there is a high possibility of accidents or disasters occurring.

[0190] "Feedback data" is a general term for information generated based on analysis results and conveyed to workers. This can be in the form of charts, text, audio, etc.

[0191] A "terminal" is a device used by a user, such as a smartphone, tablet, or PC.

[0192] "Means for presenting feedback" is a general term for devices and programs that have the function of conveying generated feedback data to the user visually or audibly.

[0193] A "log" is a full record of a system's operation history and analysis results, and refers to data that can be used for later analysis and improvement.

[0194] The present invention is a system for improving the efficiency of safety patrols within a factory. This system uses a robot that automatically patrols the factory to identify dangerous areas and provide feedback to the user in real time. The system is implemented using the following hardware and software.

[0195] Hardware

[0196] 1. Camera: A high-resolution camera (e.g., Logitech C920) is used to capture the environment and work conditions within the factory.

[0197] 2. Robots: Robots capable of autonomous patrol (e.g., Boston Dynamics Spot) patrol the factory and transport cameras.

[0198] 3. Server: A server (e.g., AWS (registered trademark) EC2) is used for data analysis and feedback generation.

[0199] software

[0200] 1. OpenCV: Used for handling images and videos and preprocessing for analysis.

[0201] 2. Flask: Used to receive data on the server and build APIs.

[0202] 3. pyttsx3: A speech synthesis library for presenting feedback data audibly.

[0203] 4. Generative AI model: A machine learning model that analyzes images and videos and identifies dangerous areas (example: a custom model using TensorFlow or PyTorch).

[0204] Data processing and calculation

[0205] 1. Shooting and data transmission:

[0206] The robot patrols the factory, capturing images of the equipment and work environment with a camera. The captured images and video data are sent to a server via a network. At this stage, an appropriate compression method is used to maintain data quality.

[0207] 2. Data formatting and analysis:

[0208] The server formats the received image and video data so that it can be analyzed. Specifically, it uses OpenCV to preprocess the images and extract the necessary frames. It then analyzes the data using a pre-trained generative AI model to identify dangerous areas.

[0209] 3. Generating and Presenting Feedback:

[0210] Based on the analysis results, the server generates feedback data for identified dangerous areas. The feedback is generated in the form of diagrams, text, and audio and sent to the device. On the device side, the feedback is presented to the user via audio using pyttsx3, instantly notifying the user of dangerous areas and countermeasures.

[0211] Examples of concrete examples and prompts

[0212] For example, if a machine cover is loose in a factory, a robot will take a photo of this area and send the image to a server. The server will then use a generative AI model to analyze the image and generate feedback such as "The cover on machine X is loose. Please repair it immediately," and notify the robot and workers in real time.

[0213] Example prompt sentence:

[0214] "Analyze the provided images and assess the safety of the facility. Identify any dangerous areas or areas that need improvement and provide specific countermeasures."

[0215] In this way, the system of the present invention allows for efficient and consistent safety patrols within the factory, improving worker safety.

[0216] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0217] Step 1:

[0218] Input: Images and videos of on-site work taken by cameras

[0219] Processing: The robot autonomously patrols the factory, taking photos and videos of the work environment and equipment using a high-resolution camera.

[0220] Output: Captured image and video data

[0221] Specific operation: The robot moves along a pre-set patrol route, stopping periodically to take photos using its camera. The acquired data is temporarily stored in the terminal.

[0222] Step 2:

[0223] Input: Captured image and video data

[0224] Processing: Image and video data is sent to the server via the network. The data is compressed before being sent.

[0225] Output: Compressed image and video data sent to the server

[0226] What it does: It uses a compression algorithm to reduce the size of image and video data stored on the device and then uploads it to a server via Wi-Fi or a wired network.

[0227] Step 3:

[0228] Input: Compressed image and video data

[0229] Processing: The server converts the received data into an analyzable format, specifically by using OpenCV to preprocess the image and extract the necessary frames.

[0230] Output: Image and video data in a parsable format

[0231] Specific operation: The server performs preprocessing such as image resizing and noise reduction, and extracts the information necessary for analysis.

[0232] Step 4:

[0233] Input: Image and video data in a parsable format

[0234] Processing: Generative AI models are used to analyze images and videos to identify hazards.

[0235] Output: Identification of dangerous areas

[0236] What it does: The server runs a generative AI model (e.g., a custom model using TensorFlow or PyTorch) to analyze image and video data. Specifically, the model evaluates preprocessed images and videos to detect hazards and anomalies.

[0237] Step 5:

[0238] Input: Hazardous area identification results

[0239] Processing: Generate feedback in the form of diagrams, text, or audio based on the identified hazards. Send the generated feedback data to the device.

[0240] Output: Generated feedback data

[0241] Specific operation: The server generates feedback in the form of diagrams, text, and audio guides. For example, it generates audio files using pyttsx3 and uses them to notify users of dangerous areas. The generated feedback data is sent to the device via API.

[0242] Step 6:

[0243] Input: Feedback data sent to the device

[0244] Processing: The terminal presents the received feedback data to the user.

[0245] Output: Feedback information presented to the user

[0246] What it does: The device analyzes the feedback data and displays it in a user-facing interface. Specifically, audio guides are played using pyttsx3, and visual hazard warnings are also displayed on the screen.

[0247] Step 7:

[0248] Input: Feedback information

[0249] Processing: Feedback data is stored in a detailed log format for later analysis and improvement.

[0250] Output: Feedback data saved as a log

[0251] Specific operation: The server classifies all feedback data by date, time, location, content, etc. and stores it in a database. This log data is organized for later reference.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] This invention combines a system for improving the efficiency of on-site safety patrols with an emotion engine that recognizes the user's emotions. This system recognizes the user's emotional state during the process of photographing, analyzing, and providing feedback, thereby improving the safety and efficiency of work by providing corresponding feedback.

[0254] composition

[0255] 1. Taking pictures and videos

[0256] Users take pictures of the work situation on-site using devices such as smartphones or tablets. High-resolution cameras are used to take pictures from multiple angles, allowing for a detailed understanding of the situation.

[0257] 2. Data transmission

[0258] The device sends the captured images and videos to the server via the network, where any necessary compression and format conversion is performed to maintain data quality during transmission.

[0259] 3. Data Formatting

[0260] The server converts the received data into a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[0261] 4. Analysis using generative artificial intelligence (AI)

[0262] The server uses a pre-trained generative AI model to analyze the content of images and videos, which evaluates the safety of work based on procedures and drawings and identifies dangerous areas.

[0263] 5. Identifying dangerous areas

[0264] Based on the analysis results, the server identifies dangerous areas, such as missing handrails when working at height or defects in the protective covers of machinery, in real time.

[0265] 6. User Emotion Recognition

[0266] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize their emotional state. For example, it can detect tension or anxiety.

[0267] 7. Generate feedback

[0268] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state, including flexible responses according to the user's emotional state.

[0269] 8. Sending and Providing Feedback

[0270] The server sends the generated feedback data to the terminal, which then presents the received data to the user and provides specific safety instructions using diagrams, text, and audio guidance.

[0271] 9. Feedback logging

[0272] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[0273] Specific examples

[0274] For example, when working at height, a worker (user) uses a smartphone to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative artificial intelligence model. As a result of the analysis, it is identified that a handrail is not installed, posing a risk. At the same time, the device's emotion engine analyzes the worker's facial expressions to detect tension or anxiety.

[0275] Based on the information about dangerous locations and the user's emotional state, the server generates specific, emotionally sensitive feedback such as, "There is a risk of falling because there are no handrails installed. Please stay calm and take appropriate measures." This feedback is sent to the worker's device in real time and displayed through visual highlights and audio guidance, allowing the worker to immediately recognize the situation and take appropriate measures.

[0276] In this way, the system of the present invention improves the safety of on-site work and enables flexible responses that take into account the emotional state of the user, thereby improving both work efficiency and safety.

[0277] The processing flow will be explained below.

[0278] Step 1:

[0279] Before starting work on-site, users prepare a device such as a smartphone or tablet and launch a camera app. They organize the surrounding environment of the subject to be photographed so that they can take appropriate photos.

[0280] Step 2:

[0281] Users should take video and still images of the work environment and the work in progress, taking care to record in detail any areas that may be particularly dangerous or important points during the work.

[0282] Step 3:

[0283] The device stores captured images and videos in local storage, compressing and converting formats as needed to maintain data quality.

[0284] Step 4:

[0285] The device transmits the saved data to the server via the network, and the progress of the transmission is displayed to the user in real time.

[0286] Step 5:

[0287] The server stores the received image and video data and formats it in a format suitable for analysis. Specifically, it extracts the necessary frames from the images and performs preprocessing to remove noise.

[0288] Step 6:

[0289] The server inputs the data into a generative artificial intelligence (AI) model and begins analysis. The model analyzes the work content based on pre-trained procedures and drawings.

[0290] Step 7:

[0291] Generative AI models analyze the content of images and videos to identify hazards, such as missing handrails or defective protective covers on equipment, in real time.

[0292] Step 8:

[0293] The device uses a built-in emotion engine to analyze the user's facial expressions and voice to recognize their emotional state, automatically detecting emotions such as tension, anxiety, and surprise.

[0294] Step 9:

[0295] Based on the analysis results, the server generates feedback in the form of diagrams, text, and audio, taking into account the user's emotional state and the information on dangerous areas. The feedback is specific and easy to understand, taking into consideration the user's emotions.

[0296] Step 10:

[0297] The server sends the generated feedback data to the terminal in real time, so that the feedback content is reflected immediately to the user.

[0298] Step 11:

[0299] The device then presents the received feedback data to the user, using visual highlights and audio guidance to provide detailed explanations of dangerous areas and how to respond.

[0300] Step 12:

[0301] The server stores each feedback and the user's emotional state in a detailed log format, which makes it easier to analyze and develop improvement measures later.

[0302] Step 13:

[0303] Based on the feedback provided, users can take appropriate measures to address dangerous areas on-site, such as installing handrails or checking safety equipment.

[0304] In this way, by providing feedback that takes into account the user's emotional state, the system significantly improves the safety and efficiency of on-site work.

[0305] Example 2

[0306] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0307] Ensuring the safety of on-site work requires rapid and accurate identification of dangerous areas and appropriate feedback. However, current systems face the problem that these processes take time and make it difficult to respond flexibly while taking into account the user's emotional state. In particular, when a worker is feeling nervous or anxious, feedback that ignores that psychological state not only fails to improve safety but may actually reduce work efficiency. The present invention aims to solve these problems.

[0308] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for formatting captured video data into an analyzable state, a means for analyzing the video using the generated artificial intelligence model and identifying dangerous areas, and a means for generating feedback based on the user's emotional state. This increases the safety of on-site work and enables flexible response according to the user's emotional state.

[0309] "Field work" refers to all work activities performed by workers on-site.

[0310] "Footage" refers to images or videos that record the status of on-site work.

[0311] "Server" refers to a computer system that receives, processes, and stores data over a network.

[0312] A "terminal" is a device operated by a user, and refers to a mobile information terminal such as a smartphone or tablet.

[0313] "Means for formatting into an analyzable state" refers to the process of converting received video data into a format suitable for analysis.

[0314] "Generative artificial intelligence" refers to an AI model that has learned large amounts of data in advance and has the ability to perform analysis on specific tasks.

[0315] "Generating" refers to the process of creating feedback such as text, diagrams, or audio based on specific information.

[0316] "Means for recognizing the user's emotions" refers to a mechanism for determining the user's psychological state based on data such as facial expressions and voice.

[0317] "Feedback" refers to advice and warning messages provided to users based on analysis results and emotion recognition.

[0318] "Means for storing in log format" refers to the process of recording the generated feedback data and the user's emotional state and storing it in a format that can be analyzed at a later date.

[0319] MODE FOR CARRYING OUT THE INVENTION

[0320] This system automates a series of processes that acquires on-site video, analyzes it, and provides feedback to improve the safety of on-site work. Specific hardware and software combinations include the following:

[0321] Hardware

[0322] Users use devices such as smartphones and tablets.

[0323] The server uses a high-performance computer system, enabling it to process large amounts of data quickly.

[0324] software

[0325] The device is equipped with a high-resolution camera, a microphone for recording audio, and emotion recognition software.

[0326] The server is equipped with a generative artificial intelligence model, enabling complex image and video analysis.

[0327] Data processing and calculation

[0328] 1. The user uses the device to capture high-resolution images of the worksite, including images and videos from multiple angles.

[0329] 2. The device compresses and converts the acquired video data into an appropriate format (JPEG or MP4).

[0330] 3. The terminal sends the compressed and converted data to the server via the Internet.

[0331] 4. The server preprocesses the received data to make it analyzable, including extracting and formatting images and video frames.

[0332] 5. The server analyzes the preprocessed data using the generative AI model and identifies dangerous areas on-site based on procedures and drawings.

[0333] 6. The device uses emotion recognition software to analyze the user's facial expressions and voice and evaluate the user's emotional state.

[0334] 7. The server generates feedback by taking into consideration the information about dangerous areas and the user's emotional state. This feedback includes text, diagrams, and audio guidance.

[0335] 8. The server sends the generated feedback data to the terminal.

[0336] 9. The device presents the received feedback to the user visually and audibly.

[0337] 10. The server stores the sent feedback data and the user's emotional state in log format and uses it for later analysis and improvement.

[0338] Specific examples

[0339] For example, when working at height, a worker (user) uses a smartphone to film the scaffolding and work environment. The device converts this video data into an appropriate format and sends it to a server via the Internet. The server preprocesses the received data and analyzes it using a generative AI model. This identifies dangerous areas, such as areas without handrails. At the same time, the device analyzes the worker's facial expressions and voice using emotion recognition software to detect tension or anxiety. Based on the dangerous areas and emotion data, the server generates feedback such as, "There is a risk of falling because there are no handrails. Please remain calm and take appropriate measures," and sends it to the device. The device then presents this feedback to the worker visually and audibly, allowing the worker to take immediate action.

[0340] This not only improves the safety of on-site work, but also enables flexible responses according to the user's emotional state, which is expected to improve both work efficiency and safety.

[0341] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0342] Step 1:

[0343] Users use devices such as smartphones and tablets to capture images of the work site. The input is image and video data of the site. Specifically, a high-resolution camera is used to capture images of the site from multiple angles, capturing multiple detailed videos. This video data is used for subsequent analysis.

[0344] Step 2:

[0345] The device compresses the captured image and video data and converts it into a format suitable for analysis (e.g., JPEG or MP4). The input is the video data acquired in step 1, and the output is compressed image or video data. Specifically, a data compression algorithm is applied to reduce the file size.

[0346] Step 3:

[0347] The device sends compressed image and video data to a server over the Internet. The input is the compressed data, and the output is the state the data will be in when it arrives at the server. Error checking is performed to ensure reliability during data transfer.

[0348] Step 4:

[0349] The server checks the integrity of the received data and converts it into a format that can be analyzed. The input is the data received in step 3, and the output is the preprocessed data. Specifically, it performs an error check, then extracts images or video frames and performs any necessary preprocessing.

[0350] Step 5:

[0351] The server loads a pre-trained generative AI model and begins analyzing the pre-processed data as input. The output is the analysis results, which are information used to evaluate the safety of the work. Specifically, the work content is analyzed based on procedures and drawings, and dangerous areas are identified.

[0352] Step 6:

[0353] The server identifies dangerous areas based on the analysis results. The input is the analysis results of the generative AI model, and the output is a list of identified dangerous areas. Specifically, it detects missing handrails when working at height and defects in the protective covers of machinery.

[0354] Step 7:

[0355] The device collects the user's facial expressions and voice using a built-in camera and microphone. The input is the user's facial and voice data, and the output is their emotional state analyzed by emotion recognition software. Specifically, the device measures tension and anxiety by recognizing the user's face and analyzing their voice tone.

[0356] Step 8:

[0357] The server generates feedback based on the identified dangerous locations and the user's emotional state. The input is a list of dangerous locations and the user's emotional state, and the output is the generated feedback data. Specifically, safety instructions and advice are created as text, diagrams, and audio guides.

[0358] Step 9:

[0359] The server sends the generated feedback data to the terminal. The input is the feedback data, and the output is the data transmission to the terminal.

[0360] Step 10:

[0361] The device presents the received feedback to the user. The input is the feedback data, and the output is the visual and audio feedback provided to the user. Specifically, it displays diagrams and text on the screen and plays audio guidance if necessary.

[0362] Step 11:

[0363] The server stores the sent feedback and the user's emotional state in log format. The input is feedback and emotional data, and the output is log data. This allows data to be accumulated for later analysis and improvement.

[0364] (Application example 2)

[0365] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0366] In on-site work, it is necessary to ensure work safety while also taking into consideration the emotional state of workers. However, current systems only evaluate the safety of work and lack the functionality to recognize the emotional state of workers and provide appropriate feedback based on that. As a result, there is an issue in that they are unable to respond appropriately to workers who are feeling tense or anxious.

[0367] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0368] In this invention, the server includes means for taking images and videos of on-site work, means for transmitting the captured images and videos to the server, means for formatting the data so that it can be analyzed by the server, means for analyzing the images and videos in the server using artificial intelligence to identify hazardous areas, means for generating information about the identified hazardous areas in the form of diagrams, text, and audio, means for transmitting the generated feedback data to the terminal, means for presenting feedback to the user, means for analyzing the user's facial expressions and voice to recognize the user's emotional state, and means for generating feedback based on the user's emotional state. This makes it possible to provide flexible and appropriate feedback that takes into consideration not only the safety of the work but also the emotional state of the worker.

[0369] "On-site work" refers to work activities that take place at a designated location.

[0370] "Means for taking images or videos" refers to equipment or systems for collecting visual information using devices, including cameras.

[0371] "Means for sending to the server" refers to the communication protocols and devices used to transfer collected data to the server via a network.

[0372] "Means for formatting data in an analyzable form" refers to the process or software used to convert collected data into a form that can be efficiently analyzed by the server.

[0373] "Means for analyzing images and videos using generative artificial intelligence" refers to a system that uses a pre-trained AI model to evaluate collected visual information and extract the necessary information.

[0374] "Means for identifying dangerous areas" refers to algorithms or programs for recognizing risk areas in the workplace based on the analysis results.

[0375] "Means for generating feedback in the form of diagrams, text, or audio" refers to technologies that allow users to visualize and / or voice information.

[0376] The "means for transmitting generated feedback data to the terminal" refers to a communication means for quickly transferring the generated feedback information to the user's device.

[0377] The term "means for presenting feedback to the user" refers to a function for displaying feedback information so that the user can check it via the device.

[0378] "Means of analyzing the user's facial expressions and voice to recognize their emotional state" refers to an AI engine that determines the user's emotions based on data acquired by a camera and microphone.

[0379] The "means for generating feedback based on the user's emotional state" refers to a system for proposing countermeasures according to the recognized emotions and generating them as feedback information.

[0380] This invention is a system that streamlines safety patrols for on-site work and takes into account the user's emotional state. The system uses high-resolution cameras, microphones, servers, and terminals (smartphones, tablets, factory robots, etc.) as hardware. The software includes an emotion recognition engine, generative AI model, data formatter, communication protocol, and feedback generation algorithm.

[0381] System Operation Overview

[0382] Data collection

[0383] Users use the device's built-in high-resolution camera and microphone to collect images, video, and audio data of on-site work, providing a detailed picture of the work environment and the worker's situation.

[0384] Data transmission

[0385] The devices transmit the collected data to a server in real time using a communication protocol that compresses the data and converts it into a suitable format.

[0386] Data Format

[0387] The server formats the transmitted data in a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[0388] Analysis by generative artificial intelligence

[0389] The server uses a pre-trained generative AI model to analyze the collected images and videos. This allows it to compare the images and videos with work procedures and drawings, evaluate the safety of the work, and identify dangerous areas. An emotion recognition engine is also used to identify the user's emotional state from their facial expressions and voice.

[0390] Generate feedback

[0391] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state. For example, it combines information such as "It is dangerous to work at height because there are no handrails installed" with instructions such as "Please stay calm and check" if the worker is nervous.

[0392] Sending and Providing Feedback

[0393] The generated feedback data is sent from the server to the terminal. The terminal then presents the received feedback to the user visually (on screen) and audibly, allowing workers to immediately understand dangerous areas and take appropriate measures.

[0394] Feedback logging

[0395] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[0396] Specific examples

[0397] For example, when working at height, a worker (user) uses a smartphone to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative AI model. The analysis identifies a hazard due to the absence of handrails. At the same time, the device's emotion engine analyzes the worker's facial expressions to detect tension or anxiety. Based on the hazard information and the user's emotional state, the server generates specific, emotion-sensitive feedback, such as, "There is a risk of falling because handrails are not installed. Please remain calm and take appropriate measures." This feedback is sent to the worker's device in real time and displayed through visual highlights and audio guidance. The worker can immediately recognize the situation and take appropriate measures.

[0398] Specific examples of prompts for the generative AI model to use

[0399] "Analyze image data of work at height to identify workers' emotional states (tension, anxiety, etc.) and dangerous areas."

[0400] "Analyze the voice data, determine the worker's emotional state from the tone and content of their voice, and generate appropriate feedback."

[0401] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0402] Step 1:

[0403] Users use the device's built-in high-resolution camera and microphone to collect images, video, and audio data of on-site work.

[0404] Input: User-activated camera and microphone, local video and audio

[0405] Output: High-resolution image data, video data, audio data

[0406] Step 2:

[0407] The devices transmit the collected data to a server in real time using a communication protocol that compresses the data and converts it into a suitable format.

[0408] Input: High-resolution image data, video data, audio data

[0409] Output: Compressed image data, video data, and audio data are sent to the server.

[0410] Step 3:

[0411] The server formats the transmitted data in a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[0412] Input: Compressed image data, video data, audio data

[0413] Output: Parsable formatted image frames and audio data

[0414] Step 4:

[0415] The server uses a pre-trained generative AI model to analyze the collected images and videos, compare them with work procedures and drawings, evaluate the safety of the work, and identify any hazards.

[0416] Input: Image frames and audio data formatted in a parseable format

[0417] Output: Information on identified dangerous areas (e.g., areas where handrails are not installed)

[0418] Step 5:

[0419] The server simultaneously operates an emotion recognition engine to identify the user's emotional state from their facial expressions and voice.

[0420] Input: Image frames and audio data formatted in a parseable format

[0421] Output: Information about the user's emotional state (e.g., tension, anxiety)

[0422] Step 6:

[0423] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state.

[0424] Input: Information on dangerous areas, information on the user's emotional state

[0425] Output: Feedback data (in diagram, text, audio format)

[0426] Step 7:

[0427] The generated feedback data is sent from the server to the terminal, which then presents the received feedback to the user visually (on screen) or audibly.

[0428] Input: Feedback data (in diagram, text, audio format)

[0429] Output: Feedback presented through the device's screen and speakers

[0430] Step 8:

[0431] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[0432] Input: Feedback data, information on the user's emotional state

[0433] Output: Saved log data

[0434] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0435] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0436] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0437] [Second embodiment]

[0438] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0439] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0440] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0441] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0442] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0443] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0444] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0445] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0446] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0447] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0448] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0449] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0450] This invention is a system for improving the efficiency of safety patrols for on-site work. This system is realized by taking images and videos of on-site work, sending them to a server, analyzing them, and providing the results to the site as feedback.

[0451] composition

[0452] 1. Taking pictures and videos

[0453] Users take pictures of the work situation on-site using devices such as smartphones or tablets. It is recommended to use a high-resolution camera that can record even complex work situations.

[0454] 2. Data transmission

[0455] The device sends the captured images and videos to the server via the network. Before sending, an appropriate compression method is used to maintain the quality of the image and video data.

[0456] 3. Data Formatting

[0457] The server converts the received data into a format suitable for analysis, specifically by extracting the necessary frames from images and videos and performing preprocessing.

[0458] 4. Analysis using generative artificial intelligence (AI)

[0459] The server uses pre-trained generative AI models to analyze the content of images and videos, for example, assessing the safety of scaffolding or the operation status of machinery based on work area procedures and safety standards.

[0460] 5. Identifying dangerous areas

[0461] Based on the analysis results performed by the server, dangerous areas are identified in real time, such as areas where handrails are not installed during work at height or where protective covers are not attached to equipment.

[0462] 6. Generate feedback

[0463] The server generates feedback data on identified hazards in the form of diagrams, text, and audio, including image highlights of the hazards and specific instructions for countermeasures.

[0464] 7. Sending and Providing Feedback

[0465] The server sends the generated feedback data to the terminal, which receives it and presents it to the user in the form of diagrams, text, and audio, allowing on-site workers to immediately recognize dangerous areas and take appropriate action.

[0466] 8. Feedback logging

[0467] The server stores each feedback in a detailed log format, which can be used for future analysis and improvement planning.

[0468] Specific examples

[0469] For example, when performing construction work at height, workers use their smartphones to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative artificial intelligence model. The model analyzes scaffolding defects and missing safety equipment and generates specific feedback, such as "There is a risk of falling because handrails are not installed." This feedback is sent to the worker's device in real time and displayed as visual alerts and audio guidance. Workers can immediately understand the situation and take necessary measures.

[0470] In this way, the system of the present invention allows patrols to be carried out efficiently and consistently to improve the safety of on-site operations.

[0471] The processing flow will be explained below.

[0472] Step 1:

[0473] Before starting work on-site, users prepare their smartphone, tablet, or other device, launch the camera app, and, if necessary, organize the surrounding environment to make it easier to take photos.

[0474] Step 2:

[0475] Users can take video and still images of the progress and environment of on-site work, recording areas that require particular attention from multiple angles, allowing them to grasp the overall situation.

[0476] Step 3:

[0477] The device temporarily stores captured images and videos in local storage, compressing and converting the data as needed to maintain quality.

[0478] Step 4:

[0479] The device sends the captured images and videos to the server via the network. During the transmission, the progress is displayed for the user to check.

[0480] Step 5:

[0481] The server stores the received image and video data and formats it into an analyzable format, for example by extracting the necessary frames and performing preprocessing.

[0482] Step 6:

[0483] The server inputs image and video data into the generative AI model and begins analysis. The model has been trained in advance and can determine the work situation based on instructions and drawings.

[0484] Step 7:

[0485] The generative AI model analyzes the content of images and videos to identify hazards, such as missing handrails or missing safety equipment, and returns the results.

[0486] Step 8:

[0487] Based on the analysis results, the server generates information on dangerous areas in the form of diagrams, text, and audio, highlighting specific areas and providing detailed explanations of appropriate countermeasures.

[0488] Step 9:

[0489] The server transmits the generated feedback data to the device in real time, which includes image, text, and audio information.

[0490] Step 10:

[0491] The device then presents the received feedback data to the user, for example by highlighting dangerous areas on the screen and providing specific precautions using text and audio guidance.

[0492] Step 11:

[0493] The server stores each feedback in a detailed log, providing data for later analysis and development of improvements.

[0494] Step 12:

[0495] Based on the feedback provided, users can take appropriate measures to address dangerous areas at the site, such as installing handrails and ensuring that safety equipment is worn.

[0496] Example 1

[0497] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0498] Safety patrols at on-site work sites consume a lot of human resources, are inefficient, and are difficult to implement consistently. Traditional methods are unable to respond quickly to changes in the work environment, which delays the identification of dangerous areas and immediate response. Furthermore, recording feedback and analyzing it for future use are also manual processes, which means it takes a long time to develop improvement measures.

[0499] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0500] In this invention, the server includes a means for capturing images and videos of on-site work, a means for transmitting the captured images and videos via a network, a means for converting the received image and video data into an analyzable format, a means for analyzing the content of the images and videos using a generative AI model and identifying hazardous areas, a means for generating information on the identified hazardous areas in diagram, text, and audio formats, a means for transmitting the generated feedback data back to the terminal, and a means for presenting the feedback to the user. This enables the efficiency and consistency of on-site work safety patrols. Furthermore, real-time identification of hazardous areas and instructions for response can be achieved, and the automated recording of feedback content can be used for future analysis and the development of improvement measures.

[0501] "Field work" refers to work such as construction, maintenance, and safety inspections carried out at a specific location.

[0502] "Images and videos" are visual data recorded using a photographic device such as a camera, and are used to visually evaluate the work situation.

[0503] "Network" means a communications infrastructure for transmitting data, including the Internet and an internal company network.

[0504] "Server" refers to a computing system that receives, processes, or transmits data.

[0505] A "terminal" is a device that a user directly uses, such as a smartphone or tablet.

[0506] A "generative AI model" is an artificial intelligence model that has been trained to perform specific tasks in advance and is used to analyze the content of images and videos.

[0507] "Analysis" is the process of dissecting data to find meaning and patterns.

[0508] "Hazardous point" refers to an area or condition in the work environment that does not meet safety standards and that presents a risk of ingestion.

[0509] "Feedback" is information generated based on the analysis results, and is used to alert the user or give instructions for improvement.

[0510] A "log" is a collection of data that records system operations and analysis results, and is used for later analysis and verification.

[0511] MODE FOR CARRYING OUT THE INVENTION

[0512] This invention is a system for improving the efficiency of safety patrols at on-site work, and automates the process of taking, transmitting, analyzing, and providing feedback on images and videos of on-site work. Below, we will explain each component of this system and its specific operation.

[0513] Hardware and software used

[0514] 1. Hardware

[0515] The devices that users use are smartphones and tablets, which are equipped with high-resolution cameras.

[0516] A server is a high performance computer system.

[0517] 2. Software

[0518] OpenCV is used as the image processing library.

[0519] Generative AI models are built using TensorFlow or PyTorch.

[0520] Use standard communication protocols (e.g. HTTP / HTTPS) for network communication.

[0521] System Overview

[0522] 1. Taking pictures and videos

[0523] Users use their devices to take photos of the work being done on-site, and the captured data is recorded in detail using a high-resolution camera.

[0524] 2. Data transmission

[0525] The device sends the captured images and videos to the server over the network. Before transmission, the data is compressed using a compression algorithm (e.g., JPEG or H.264).

[0526] 3. Data format conversion

[0527] The server converts the received data into a format suitable for analysis. Specifically, it extracts the necessary frames from images or videos and performs preprocessing using an image processing library such as OpenCV.

[0528] 4. Analysis using generative AI models

[0529] The server uses a pre-trained generative AI model to analyze the content of images and videos, and the analysis is based on work area procedures and safety standards to assess the safety of the work.

[0530] 5. Identifying dangerous areas

[0531] Based on the analysis results, the server identifies dangerous areas, such as areas where handrails are not installed when working at height, or areas where protective covers are not attached to equipment.

[0532] 6. Generate feedback

[0533] The server generates graphical, textual, and audio feedback data about identified hot spots, including image highlights of hot spots and specific instructions for countermeasures.

[0534] 7. Sending and Providing Feedback

[0535] The server then sends the generated feedback data to the terminal, which receives it and presents it to the user in the form of diagrams, text, and audio, allowing on-site workers to immediately recognize dangerous areas and take appropriate action.

[0536] 8. Feedback logging

[0537] The server stores each feedback in a detailed log format, which can be used for later analysis and development of improvement measures.

[0538] Specific examples

[0539] For example, when performing construction work at height, workers use their smartphones to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative AI model. The model analyzes scaffolding defects and missing safety equipment, generating specific feedback such as "There is a risk of falling because handrails are not installed." This feedback is sent to the worker's device in real time and displayed as visual alerts and audio guidance. Workers can immediately understand the situation and take necessary measures.

[0540] Prompt Sentence Examples

[0541] "This is an image and video analysis system for safety confirmation at construction sites. It checks the safety of working at heights, identifies dangerous areas such as missing handrails or missing safety equipment, and provides feedback in real time. This can improve safety at work sites."

[0542] This is a specific mode for carrying out the invention.

[0543] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0544] Step 1:

[0545] Taking pictures and videos

[0546] Users take photos of the work situation on-site using a smartphone or tablet.

[0547] Input: On-site working environment.

[0548] Specific actions: High-resolution cameras are used to capture detailed images of work areas and potential hazards.

[0549] Output: High resolution image and video data.

[0550] Step 2:

[0551] Sending data

[0552] The device sends the captured images and videos to a server via the network.

[0553] Input: High-resolution image and video data captured.

[0554] Specific operation: Before transmitting data, the amount of data is reduced using a compression algorithm (e.g., JPEG or H.264). The compressed data is sent to the server using a network communication protocol (e.g., HTTP / HTTPS).

[0555] Output: Compressed image and video data is sent to the server.

[0556] Step 3:

[0557] Data format conversion

[0558] The server converts the received data into a format suitable for analysis.

[0559] Input: Compressed image and video data.

[0560] Specific operation: Using an image processing library such as OpenCV, extract important frames from the video data and save them as image data. Also, perform preprocessing such as resizing and filtering of the image data.

[0561] Output: Image data in a format suitable for analysis.

[0562] Step 4:

[0563] Analysis using generative AI models

[0564] The server uses a pre-trained generative AI model to analyze the content of images and videos.

[0565] Input: Image data in a format suitable for analysis.

[0566] How it works: Apply generative AI models built with TensorFlow or PyTorch to analyze image data, specifically assessing it against work area procedures and safety standards.

[0567] Output: Analysis results (e.g., identification of dangerous areas and non-fulfillment of safety standards).

[0568] Step 5:

[0569] Identifying dangerous areas

[0570] The server identifies dangerous areas based on the analysis results.

[0571] Input: Analysis results from the generative AI model.

[0572] Specific actions: Review the analysis results and identify and list areas where handrails have not been installed when working at height, areas where protective covers have not been installed on equipment, etc.

[0573] Output: A list of identified hazards.

[0574] Step 6:

[0575] Generate feedback

[0576] The server generates feedback data in the form of diagrams, text, and audio about identified dangerous areas.

[0577] Input: A list of identified hazards.

[0578] Specific actions: Highlight images of dangerous areas and create diagrams, text, and audio data containing specific instructions for countermeasures.

[0579] Output: Feedback data (e.g., in diagram, text, or audio format).

[0580] Step 7:

[0581] Sending and Providing Feedback

[0582] The server transmits the generated feedback data to the terminal.

[0583] Input: Feedback data.

[0584] Specific operation: The generated feedback data is sent to the terminal using a network communication protocol. The terminal receives the feedback data and presents it to the user visually and audibly.

[0585] Output: Feedback information presented to the user (e.g., image highlighting of dangerous areas, specific instructions for countermeasures).

[0586] Step 8:

[0587] Feedback logging

[0588] The server stores each feedback in a detailed log format.

[0589] Input: Feedback data.

[0590] Specific operation: Feedback content is saved in log format using SQLite or a NoSQL database (e.g. MongoDB).

[0591] Output: A saved feedback log that can be used for later analysis and development of improvement measures.

[0592] (Application example 1)

[0593] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0594] Ensuring safety during on-site work is extremely important for protecting the lives of workers, but conventional methods lack consistency and efficiency. In particular, it is difficult to identify dangerous areas and take immediate action, and feedback management is insufficient. Therefore, a system is needed to ensure the safety of on-site work in real time and efficiently identify dangerous areas.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0596] In this invention, the server includes means for formatting images and videos so that they can be analyzed, means for analyzing images and videos using generative artificial intelligence to identify dangerous areas, means for generating information about the identified dangerous areas in the form of diagrams, text, and audio, and means for presenting feedback data by audio and notifying the user of the dangerous areas and countermeasures. This makes it possible to ensure the safety of on-site work in real time, efficiently identify dangerous areas, and respond immediately.

[0597] "On-site work" refers to all work performed at a specific physical location, primarily including indoor and outdoor work activities such as construction sites and factories.

[0598] "Means for taking images or videos" means a device that has the function of capturing still images or videos using a photographing device such as a camera or video camera.

[0599] "Server" refers to the computer system utilized to receive, process, analyze data, and generate feedback.

[0600] "Generative artificial intelligence" refers to algorithms and models that learn, analyze, and make decisions based on large amounts of data.

[0601] "Dangerous areas" refer to locations or situations at work sites where there is a high possibility of accidents or disasters occurring.

[0602] "Feedback data" is a general term for information generated based on analysis results and conveyed to workers. This can be in the form of charts, text, audio, etc.

[0603] A "terminal" is a device used by a user, such as a smartphone, tablet, or PC.

[0604] "Means for presenting feedback" is a general term for devices and programs that have the function of conveying generated feedback data to the user visually or audibly.

[0605] A "log" is a full record of a system's operation history and analysis results, and refers to data that can be used for later analysis and improvement.

[0606] The present invention is a system for improving the efficiency of safety patrols within a factory. This system uses a robot that automatically patrols the factory to identify dangerous areas and provide feedback to the user in real time. The system is implemented using the following hardware and software.

[0607] Hardware

[0608] 1. Camera: A high-resolution camera (e.g., Logitech C920) is used to capture the environment and work conditions within the factory.

[0609] 2. Robots: Robots capable of autonomous patrol (e.g., Boston Dynamics Spot) patrol the factory and transport cameras.

[0610] 3. Server: A server (e.g. AWS EC2) is used for data analysis and feedback generation.

[0611] software

[0612] 1. OpenCV: Used for handling images and videos and preprocessing for analysis.

[0613] 2. Flask: Used to receive data on the server and build APIs.

[0614] 3. pyttsx3: A speech synthesis library for presenting feedback data audibly.

[0615] 4. Generative AI model: A machine learning model that analyzes images and videos and identifies dangerous areas (example: a custom model using TensorFlow or PyTorch).

[0616] Data processing and calculation

[0617] 1. Shooting and data transmission:

[0618] The robot patrols the factory, capturing images of the equipment and work environment with a camera. The captured images and video data are sent to a server via a network. At this stage, an appropriate compression method is used to maintain data quality.

[0619] 2. Data formatting and analysis:

[0620] The server formats the received image and video data so that it can be analyzed. Specifically, it uses OpenCV to preprocess the images and extract the necessary frames. It then analyzes the data using a pre-trained generative AI model to identify dangerous areas.

[0621] 3. Generating and Presenting Feedback:

[0622] Based on the analysis results, the server generates feedback data for identified dangerous areas. The feedback is generated in the form of diagrams, text, and audio and sent to the device. On the device side, the feedback is presented to the user via audio using pyttsx3, instantly notifying the user of dangerous areas and countermeasures.

[0623] Examples of concrete examples and prompts

[0624] For example, if a machine cover is loose in a factory, a robot will take a photo of this area and send the image to a server. The server will then use a generative AI model to analyze the image and generate feedback such as "The cover on machine X is loose. Please repair it immediately," and notify the robot and workers in real time.

[0625] Example prompt sentence:

[0626] "Analyze the provided images and assess the safety of the facility. Identify any dangerous areas or areas that need improvement and provide specific countermeasures."

[0627] In this way, the system of the present invention allows for efficient and consistent safety patrols within the factory, improving worker safety.

[0628] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0629] Step 1:

[0630] Input: Images and videos of on-site work taken by cameras

[0631] Processing: The robot autonomously patrols the factory, taking photos and videos of the work environment and equipment using a high-resolution camera.

[0632] Output: Captured image and video data

[0633] Specific operation: The robot moves along a pre-set patrol route, stopping periodically to take photos using its camera. The acquired data is temporarily stored in the terminal.

[0634] Step 2:

[0635] Input: Captured image and video data

[0636] Processing: Image and video data is sent to the server via the network. The data is compressed before being sent.

[0637] Output: Compressed image and video data sent to the server

[0638] What it does: It uses a compression algorithm to reduce the size of image and video data stored on the device and then uploads it to a server via Wi-Fi or a wired network.

[0639] Step 3:

[0640] Input: Compressed image and video data

[0641] Processing: The server converts the received data into an analyzable format, specifically by using OpenCV to preprocess the image and extract the necessary frames.

[0642] Output: Image and video data in a parsable format

[0643] Specific operation: The server performs preprocessing such as image resizing and noise reduction, and extracts the information necessary for analysis.

[0644] Step 4:

[0645] Input: Image and video data in a parsable format

[0646] Processing: Generative AI models are used to analyze images and videos to identify hazards.

[0647] Output: Identification of dangerous areas

[0648] What it does: The server runs a generative AI model (e.g., a custom model using TensorFlow or PyTorch) to analyze image and video data. Specifically, the model evaluates preprocessed images and videos to detect hazards and anomalies.

[0649] Step 5:

[0650] Input: Hazardous area identification results

[0651] Processing: Generate feedback in the form of diagrams, text, or audio based on the identified hazards. Send the generated feedback data to the device.

[0652] Output: Generated feedback data

[0653] Specific operation: The server generates feedback in the form of diagrams, text, and audio guides. For example, it generates audio files using pyttsx3 and uses them to notify users of dangerous areas. The generated feedback data is sent to the device via API.

[0654] Step 6:

[0655] Input: Feedback data sent to the device

[0656] Processing: The terminal presents the received feedback data to the user.

[0657] Output: Feedback information presented to the user

[0658] What it does: The device analyzes the feedback data and displays it in a user-facing interface. Specifically, audio guides are played using pyttsx3, and visual hazard warnings are also displayed on the screen.

[0659] Step 7:

[0660] Input: Feedback information

[0661] Processing: Feedback data is stored in a detailed log format for later analysis and improvement.

[0662] Output: Feedback data saved as a log

[0663] Specific operation: The server classifies all feedback data by date, time, location, content, etc. and stores it in a database. This log data is organized for later reference.

[0664] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0665] This invention combines a system for improving the efficiency of on-site safety patrols with an emotion engine that recognizes the user's emotions. This system recognizes the user's emotional state during the process of photographing, analyzing, and providing feedback, thereby improving the safety and efficiency of work by providing corresponding feedback.

[0666] composition

[0667] 1. Taking pictures and videos

[0668] Users take pictures of the work situation on-site using devices such as smartphones or tablets. High-resolution cameras are used to take pictures from multiple angles, allowing for a detailed understanding of the situation.

[0669] 2. Data transmission

[0670] The device sends the captured images and videos to the server via the network, where any necessary compression and format conversion is performed to maintain data quality during transmission.

[0671] 3. Data Formatting

[0672] The server converts the received data into a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[0673] 4. Analysis using generative artificial intelligence (AI)

[0674] The server uses a pre-trained generative AI model to analyze the content of images and videos, which evaluates the safety of work based on procedures and drawings and identifies dangerous areas.

[0675] 5. Identifying dangerous areas

[0676] Based on the analysis results, the server identifies dangerous areas, such as missing handrails when working at height or defects in the protective covers of machinery, in real time.

[0677] 6. User Emotion Recognition

[0678] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize their emotional state. For example, it can detect tension or anxiety.

[0679] 7. Generate feedback

[0680] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state, including flexible responses according to the user's emotional state.

[0681] 8. Sending and Providing Feedback

[0682] The server sends the generated feedback data to the terminal, which then presents the received data to the user and provides specific safety instructions using diagrams, text, and audio guidance.

[0683] 9. Feedback logging

[0684] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[0685] Specific examples

[0686] For example, when working at height, a worker (user) uses a smartphone to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative artificial intelligence model. As a result of the analysis, it is identified that a handrail is not installed, posing a risk. At the same time, the device's emotion engine analyzes the worker's facial expressions to detect tension or anxiety.

[0687] Based on the information about dangerous locations and the user's emotional state, the server generates specific, emotionally sensitive feedback such as, "There is a risk of falling because there are no handrails installed. Please stay calm and take appropriate measures." This feedback is sent to the worker's device in real time and displayed through visual highlights and audio guidance, allowing the worker to immediately recognize the situation and take appropriate measures.

[0688] In this way, the system of the present invention improves the safety of on-site work and enables flexible responses that take into account the emotional state of the user, thereby improving both work efficiency and safety.

[0689] The processing flow will be explained below.

[0690] Step 1:

[0691] Before starting work on-site, users prepare a device such as a smartphone or tablet and launch a camera app. They organize the surrounding environment of the subject to be photographed so that they can take appropriate photos.

[0692] Step 2:

[0693] Users should take video and still images of the work environment and the work in progress, taking care to record in detail any areas that may be particularly dangerous or important points during the work.

[0694] Step 3:

[0695] The device stores captured images and videos in local storage, compressing and converting formats as needed to maintain data quality.

[0696] Step 4:

[0697] The device transmits the saved data to the server via the network, and the progress of the transmission is displayed to the user in real time.

[0698] Step 5:

[0699] The server stores the received image and video data and formats it in a format suitable for analysis. Specifically, it extracts the necessary frames from the images and performs preprocessing to remove noise.

[0700] Step 6:

[0701] The server inputs the data into a generative artificial intelligence (AI) model and begins analysis. The model analyzes the work content based on pre-trained procedures and drawings.

[0702] Step 7:

[0703] Generative AI models analyze the content of images and videos to identify hazards, such as missing handrails or defective protective covers on equipment, in real time.

[0704] Step 8:

[0705] The device uses a built-in emotion engine to analyze the user's facial expressions and voice to recognize their emotional state, automatically detecting emotions such as tension, anxiety, and surprise.

[0706] Step 9:

[0707] Based on the analysis results, the server generates feedback in the form of diagrams, text, and audio, taking into account the user's emotional state and the information on dangerous areas. The feedback is specific and easy to understand, taking into consideration the user's emotions.

[0708] Step 10:

[0709] The server sends the generated feedback data to the terminal in real time, so that the feedback content is reflected immediately to the user.

[0710] Step 11:

[0711] The device then presents the received feedback data to the user, using visual highlights and audio guidance to provide detailed explanations of dangerous areas and how to respond.

[0712] Step 12:

[0713] The server stores each feedback and the user's emotional state in a detailed log format, which makes it easier to analyze and develop improvement measures later.

[0714] Step 13:

[0715] Based on the feedback provided, users can take appropriate measures to address dangerous areas on-site, such as installing handrails or checking safety equipment.

[0716] In this way, by providing feedback that takes into account the user's emotional state, the system significantly improves the safety and efficiency of on-site work.

[0717] Example 2

[0718] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0719] Ensuring the safety of on-site work requires rapid and accurate identification of dangerous areas and appropriate feedback. However, current systems face the problem that these processes take time and make it difficult to respond flexibly while taking into account the user's emotional state. In particular, when a worker is feeling nervous or anxious, feedback that ignores that psychological state not only fails to improve safety but may actually reduce work efficiency. The present invention aims to solve these problems.

[0720] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for formatting captured video data into an analyzable state, a means for analyzing the video using the generated artificial intelligence model and identifying dangerous areas, and a means for generating feedback based on the user's emotional state. This increases the safety of on-site work and enables flexible response according to the user's emotional state.

[0721] "Field work" refers to all work activities performed by workers on-site.

[0722] "Footage" refers to images or videos that record the status of on-site work.

[0723] "Server" refers to a computer system that receives, processes, and stores data over a network.

[0724] A "terminal" is a device operated by a user, and refers to a mobile information terminal such as a smartphone or tablet.

[0725] "Means for formatting into an analyzable state" refers to the process of converting received video data into a format suitable for analysis.

[0726] "Generative artificial intelligence" refers to an AI model that has learned large amounts of data in advance and has the ability to perform analysis on specific tasks.

[0727] "Generating" refers to the process of creating feedback such as text, diagrams, or audio based on specific information.

[0728] "Means for recognizing the user's emotions" refers to a mechanism for determining the user's psychological state based on data such as facial expressions and voice.

[0729] "Feedback" refers to advice and warning messages provided to users based on analysis results and emotion recognition.

[0730] "Means for storing in log format" refers to the process of recording the generated feedback data and the user's emotional state and storing it in a format that can be analyzed at a later date.

[0731] MODE FOR CARRYING OUT THE INVENTION

[0732] This system automates a series of processes that acquires on-site video, analyzes it, and provides feedback to improve the safety of on-site work. Specific hardware and software combinations include the following:

[0733] Hardware

[0734] Users use devices such as smartphones and tablets.

[0735] The server uses a high-performance computer system, enabling it to process large amounts of data quickly.

[0736] software

[0737] The device is equipped with a high-resolution camera, a microphone for recording audio, and emotion recognition software.

[0738] The server is equipped with a generative artificial intelligence model, enabling complex image and video analysis.

[0739] Data processing and calculation

[0740] 1. The user uses the device to capture high-resolution images of the worksite, including images and videos from multiple angles.

[0741] 2. The device compresses and converts the acquired video data into an appropriate format (JPEG or MP4).

[0742] 3. The terminal sends the compressed and converted data to the server via the Internet.

[0743] 4. The server preprocesses the received data to make it analyzable, including extracting and formatting images and video frames.

[0744] 5. The server analyzes the preprocessed data using the generative AI model and identifies dangerous areas on-site based on procedures and drawings.

[0745] 6. The device uses emotion recognition software to analyze the user's facial expressions and voice and evaluate the user's emotional state.

[0746] 7. The server generates feedback by taking into consideration the information about dangerous areas and the user's emotional state. This feedback includes text, diagrams, and audio guidance.

[0747] 8. The server sends the generated feedback data to the terminal.

[0748] 9. The device presents the received feedback to the user visually and audibly.

[0749] 10. The server stores the sent feedback data and the user's emotional state in log format and uses it for later analysis and improvement.

[0750] Specific examples

[0751] For example, when working at height, a worker (user) uses a smartphone to film the scaffolding and work environment. The device converts this video data into an appropriate format and sends it to a server via the Internet. The server preprocesses the received data and analyzes it using a generative AI model. This identifies dangerous areas, such as areas without handrails. At the same time, the device analyzes the worker's facial expressions and voice using emotion recognition software to detect tension or anxiety. Based on the dangerous areas and emotion data, the server generates feedback such as, "There is a risk of falling because there are no handrails. Please remain calm and take appropriate measures," and sends it to the device. The device then presents this feedback to the worker visually and audibly, allowing the worker to take immediate action.

[0752] This not only improves the safety of on-site work, but also enables flexible responses according to the user's emotional state, which is expected to improve both work efficiency and safety.

[0753] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0754] Step 1:

[0755] Users use devices such as smartphones and tablets to capture images of the work site. The input is image and video data of the site. Specifically, a high-resolution camera is used to capture images of the site from multiple angles, capturing multiple detailed videos. This video data is used for subsequent analysis.

[0756] Step 2:

[0757] The device compresses the captured image and video data and converts it into a format suitable for analysis (e.g., JPEG or MP4). The input is the video data acquired in step 1, and the output is compressed image or video data. Specifically, a data compression algorithm is applied to reduce the file size.

[0758] Step 3:

[0759] The device sends compressed image and video data to a server over the Internet. The input is the compressed data, and the output is the state the data will be in when it arrives at the server. Error checking is performed to ensure reliability during data transfer.

[0760] Step 4:

[0761] The server checks the integrity of the received data and converts it into a format that can be analyzed. The input is the data received in step 3, and the output is the preprocessed data. Specifically, it performs an error check, then extracts images or video frames and performs any necessary preprocessing.

[0762] Step 5:

[0763] The server loads a pre-trained generative AI model and begins analyzing the pre-processed data as input. The output is the analysis results, which are information used to evaluate the safety of the work. Specifically, the work content is analyzed based on procedures and drawings, and dangerous areas are identified.

[0764] Step 6:

[0765] The server identifies dangerous areas based on the analysis results. The input is the analysis results of the generative AI model, and the output is a list of identified dangerous areas. Specifically, it detects missing handrails when working at height and defects in the protective covers of machinery.

[0766] Step 7:

[0767] The device collects the user's facial expressions and voice using a built-in camera and microphone. The input is the user's facial and voice data, and the output is their emotional state analyzed by emotion recognition software. Specifically, the device measures tension and anxiety by recognizing the user's face and analyzing their voice tone.

[0768] Step 8:

[0769] The server generates feedback based on the identified dangerous locations and the user's emotional state. The input is a list of dangerous locations and the user's emotional state, and the output is the generated feedback data. Specifically, safety instructions and advice are created as text, diagrams, and audio guides.

[0770] Step 9:

[0771] The server sends the generated feedback data to the terminal. The input is the feedback data, and the output is the data transmission to the terminal.

[0772] Step 10:

[0773] The device presents the received feedback to the user. The input is the feedback data, and the output is the visual and audio feedback provided to the user. Specifically, it displays diagrams and text on the screen and plays audio guidance if necessary.

[0774] Step 11:

[0775] The server stores the sent feedback and the user's emotional state in log format. The input is feedback and emotional data, and the output is log data. This allows data to be accumulated for later analysis and improvement.

[0776] (Application example 2)

[0777] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0778] In on-site work, it is necessary to ensure work safety while also taking into consideration the emotional state of workers. However, current systems only evaluate the safety of work and lack the functionality to recognize the emotional state of workers and provide appropriate feedback based on that. As a result, there is an issue in that they are unable to respond appropriately to workers who are feeling tense or anxious.

[0779] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0780] In this invention, the server includes means for taking images and videos of on-site work, means for transmitting the captured images and videos to the server, means for formatting the data so that it can be analyzed by the server, means for analyzing the images and videos in the server using artificial intelligence to identify hazardous areas, means for generating information about the identified hazardous areas in the form of diagrams, text, and audio, means for transmitting the generated feedback data to the terminal, means for presenting feedback to the user, means for analyzing the user's facial expressions and voice to recognize the user's emotional state, and means for generating feedback based on the user's emotional state. This makes it possible to provide flexible and appropriate feedback that takes into consideration not only the safety of the work but also the emotional state of the worker.

[0781] "On-site work" refers to work activities that take place at a designated location.

[0782] "Means for taking images or videos" refers to equipment or systems for collecting visual information using devices, including cameras.

[0783] "Means for sending to the server" refers to the communication protocols and devices used to transfer collected data to the server via a network.

[0784] "Means for formatting data in an analyzable form" refers to the process or software used to convert collected data into a form that can be efficiently analyzed by the server.

[0785] "Means for analyzing images and videos using generative artificial intelligence" refers to a system that uses a pre-trained AI model to evaluate collected visual information and extract the necessary information.

[0786] "Means for identifying dangerous areas" refers to algorithms or programs for recognizing risk areas in the workplace based on the analysis results.

[0787] "Means for generating feedback in the form of diagrams, text, or audio" refers to technologies that allow users to visualize and / or voice information.

[0788] The "means for transmitting generated feedback data to the terminal" refers to a communication means for quickly transferring the generated feedback information to the user's device.

[0789] The term "means for presenting feedback to the user" refers to a function for displaying feedback information so that the user can check it via the device.

[0790] "Means of analyzing the user's facial expressions and voice to recognize their emotional state" refers to an AI engine that determines the user's emotions based on data acquired by a camera and microphone.

[0791] The "means for generating feedback based on the user's emotional state" refers to a system for proposing countermeasures according to the recognized emotions and generating them as feedback information.

[0792] This invention is a system that streamlines safety patrols for on-site work and takes into account the user's emotional state. The system uses high-resolution cameras, microphones, servers, and terminals (smartphones, tablets, factory robots, etc.) as hardware. The software includes an emotion recognition engine, generative AI model, data formatter, communication protocol, and feedback generation algorithm.

[0793] System Operation Overview

[0794] Data collection

[0795] Users use the device's built-in high-resolution camera and microphone to collect images, video, and audio data of on-site work, providing a detailed picture of the work environment and the worker's situation.

[0796] Data transmission

[0797] The devices transmit the collected data to a server in real time using a communication protocol that compresses the data and converts it into a suitable format.

[0798] Data Format

[0799] The server formats the transmitted data in a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[0800] Analysis by generative artificial intelligence

[0801] The server uses a pre-trained generative AI model to analyze the collected images and videos. This allows it to compare the images and videos with work procedures and drawings, evaluate the safety of the work, and identify dangerous areas. An emotion recognition engine is also used to identify the user's emotional state from their facial expressions and voice.

[0802] Generate feedback

[0803] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state. For example, it combines information such as "It is dangerous to work at height because there are no handrails installed" with instructions such as "Please stay calm and check" if the worker is nervous.

[0804] Sending and Providing Feedback

[0805] The generated feedback data is sent from the server to the terminal. The terminal then presents the received feedback to the user visually (on screen) and audibly, allowing workers to immediately understand dangerous areas and take appropriate measures.

[0806] Feedback logging

[0807] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[0808] Specific examples

[0809] For example, when working at height, a worker (user) uses a smartphone to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative AI model. The analysis identifies a hazard due to the absence of handrails. At the same time, the device's emotion engine analyzes the worker's facial expressions to detect tension or anxiety. Based on the hazard information and the user's emotional state, the server generates specific, emotion-sensitive feedback, such as, "There is a risk of falling because handrails are not installed. Please remain calm and take appropriate measures." This feedback is sent to the worker's device in real time and displayed through visual highlights and audio guidance. The worker can immediately recognize the situation and take appropriate measures.

[0810] Specific examples of prompts for the generative AI model to use

[0811] "Analyze image data of work at height to identify workers' emotional states (tension, anxiety, etc.) and dangerous areas."

[0812] "Analyze the voice data, determine the worker's emotional state from the tone and content of their voice, and generate appropriate feedback."

[0813] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0814] Step 1:

[0815] Users use the device's built-in high-resolution camera and microphone to collect images, video, and audio data of on-site work.

[0816] Input: User-activated camera and microphone, local video and audio

[0817] Output: High-resolution image data, video data, audio data

[0818] Step 2:

[0819] The devices transmit the collected data to a server in real time using a communication protocol that compresses the data and converts it into a suitable format.

[0820] Input: High-resolution image data, video data, audio data

[0821] Output: Compressed image data, video data, and audio data are sent to the server.

[0822] Step 3:

[0823] The server formats the transmitted data in a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[0824] Input: Compressed image data, video data, audio data

[0825] Output: Parsable formatted image frames and audio data

[0826] Step 4:

[0827] The server uses a pre-trained generative AI model to analyze the collected images and videos, compare them with work procedures and drawings, evaluate the safety of the work, and identify any hazards.

[0828] Input: Image frames and audio data formatted in a parseable format

[0829] Output: Information on identified dangerous areas (e.g., areas where handrails are not installed)

[0830] Step 5:

[0831] The server simultaneously operates an emotion recognition engine to identify the user's emotional state from their facial expressions and voice.

[0832] Input: Image frames and audio data formatted in a parseable format

[0833] Output: Information about the user's emotional state (e.g., tension, anxiety)

[0834] Step 6:

[0835] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state.

[0836] Input: Information on dangerous areas, information on the user's emotional state

[0837] Output: Feedback data (in diagram, text, audio format)

[0838] Step 7:

[0839] The generated feedback data is sent from the server to the terminal, which then presents the received feedback to the user visually (on screen) or audibly.

[0840] Input: Feedback data (in diagram, text, audio format)

[0841] Output: Feedback presented through the device's screen and speakers

[0842] Step 8:

[0843] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[0844] Input: Feedback data, information on the user's emotional state

[0845] Output: Saved log data

[0846] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0847] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0848] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0849] [Third embodiment]

[0850] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0851] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0852] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0853] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0854] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0855] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0856] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0857] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0858] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0859] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0860] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0861] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0862] This invention is a system for improving the efficiency of safety patrols for on-site work. This system is realized by taking images and videos of on-site work, sending them to a server, analyzing them, and providing the results to the site as feedback.

[0863] composition

[0864] 1. Taking pictures and videos

[0865] Users take pictures of the work situation on-site using devices such as smartphones or tablets. It is recommended to use a high-resolution camera that can record even complex work situations.

[0866] 2. Data transmission

[0867] The device sends the captured images and videos to the server via the network. Before sending, an appropriate compression method is used to maintain the quality of the image and video data.

[0868] 3. Data Formatting

[0869] The server converts the received data into a format suitable for analysis, specifically by extracting the necessary frames from images and videos and performing preprocessing.

[0870] 4. Analysis using generative artificial intelligence (AI)

[0871] The server uses pre-trained generative AI models to analyze the content of images and videos, for example, assessing the safety of scaffolding or the operation status of machinery based on work area procedures and safety standards.

[0872] 5. Identifying dangerous areas

[0873] Based on the analysis results performed by the server, dangerous areas are identified in real time, such as areas where handrails are not installed during work at height or where protective covers are not attached to equipment.

[0874] 6. Generate feedback

[0875] The server generates feedback data on identified hazards in the form of diagrams, text, and audio, including image highlights of the hazards and specific instructions for countermeasures.

[0876] 7. Sending and Providing Feedback

[0877] The server sends the generated feedback data to the terminal, which receives it and presents it to the user in the form of diagrams, text, and audio, allowing on-site workers to immediately recognize dangerous areas and take appropriate action.

[0878] 8. Feedback logging

[0879] The server stores each feedback in a detailed log format, which can be used for future analysis and improvement planning.

[0880] Specific examples

[0881] For example, when performing construction work at height, workers use their smartphones to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative artificial intelligence model. The model analyzes scaffolding defects and missing safety equipment and generates specific feedback, such as "There is a risk of falling because handrails are not installed." This feedback is sent to the worker's device in real time and displayed as visual alerts and audio guidance. Workers can immediately understand the situation and take necessary measures.

[0882] In this way, the system of the present invention allows patrols to be carried out efficiently and consistently to improve the safety of on-site operations.

[0883] The processing flow will be explained below.

[0884] Step 1:

[0885] Before starting work on-site, users prepare their smartphone, tablet, or other device, launch the camera app, and, if necessary, organize the surrounding environment to make it easier to take photos.

[0886] Step 2:

[0887] Users can take video and still images of the progress and environment of on-site work, recording areas that require particular attention from multiple angles, allowing them to grasp the overall situation.

[0888] Step 3:

[0889] The device temporarily stores captured images and videos in local storage, compressing and converting the data as needed to maintain quality.

[0890] Step 4:

[0891] The device sends the captured images and videos to the server via the network. During the transmission, the progress is displayed for the user to check.

[0892] Step 5:

[0893] The server stores the received image and video data and formats it into an analyzable format, for example by extracting the necessary frames and performing preprocessing.

[0894] Step 6:

[0895] The server inputs image and video data into the generative AI model and begins analysis. The model has been trained in advance and can determine the work situation based on instructions and drawings.

[0896] Step 7:

[0897] The generative AI model analyzes the content of images and videos to identify hazards, such as missing handrails or missing safety equipment, and returns the results.

[0898] Step 8:

[0899] Based on the analysis results, the server generates information on dangerous areas in the form of diagrams, text, and audio, highlighting specific areas and providing detailed explanations of appropriate countermeasures.

[0900] Step 9:

[0901] The server transmits the generated feedback data to the device in real time, which includes image, text, and audio information.

[0902] Step 10:

[0903] The device then presents the received feedback data to the user, for example by highlighting dangerous areas on the screen and providing specific precautions using text and audio guidance.

[0904] Step 11:

[0905] The server stores each feedback in a detailed log, providing data for later analysis and development of improvements.

[0906] Step 12:

[0907] Based on the feedback provided, users can take appropriate measures to address dangerous areas at the site, such as installing handrails and ensuring that safety equipment is worn.

[0908] Example 1

[0909] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0910] Safety patrols at on-site work sites consume a lot of human resources, are inefficient, and are difficult to implement consistently. Traditional methods are unable to respond quickly to changes in the work environment, which delays the identification of dangerous areas and immediate response. Furthermore, recording feedback and analyzing it for future use are also manual processes, which means it takes a long time to develop improvement measures.

[0911] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0912] In this invention, the server includes a means for capturing images and videos of on-site work, a means for transmitting the captured images and videos via a network, a means for converting the received image and video data into an analyzable format, a means for analyzing the content of the images and videos using a generative AI model and identifying hazardous areas, a means for generating information on the identified hazardous areas in diagram, text, and audio formats, a means for transmitting the generated feedback data back to the terminal, and a means for presenting the feedback to the user. This enables the efficiency and consistency of on-site work safety patrols. Furthermore, real-time identification of hazardous areas and instructions for response can be achieved, and the automated recording of feedback content can be used for future analysis and the development of improvement measures.

[0913] "Field work" refers to work such as construction, maintenance, and safety inspections carried out at a specific location.

[0914] "Images and videos" are visual data recorded using a photographic device such as a camera, and are used to visually evaluate the work situation.

[0915] "Network" means a communications infrastructure for transmitting data, including the Internet and an internal company network.

[0916] "Server" refers to a computing system that receives, processes, or transmits data.

[0917] A "terminal" is a device that a user directly uses, such as a smartphone or tablet.

[0918] A "generative AI model" is an artificial intelligence model that has been trained to perform specific tasks in advance and is used to analyze the content of images and videos.

[0919] "Analysis" is the process of dissecting data to find meaning and patterns.

[0920] "Hazardous point" refers to an area or condition in the work environment that does not meet safety standards and that presents a risk of ingestion.

[0921] "Feedback" is information generated based on the analysis results, and is used to alert the user or give instructions for improvement.

[0922] A "log" is a collection of data that records system operations and analysis results, and is used for later analysis and verification.

[0923] MODE FOR CARRYING OUT THE INVENTION

[0924] This invention is a system for improving the efficiency of safety patrols at on-site work, and automates the process of taking, transmitting, analyzing, and providing feedback on images and videos of on-site work. Below, we will explain each component of this system and its specific operation.

[0925] Hardware and software used

[0926] 1. Hardware

[0927] The devices that users use are smartphones and tablets, which are equipped with high-resolution cameras.

[0928] A server is a high performance computer system.

[0929] 2. Software

[0930] OpenCV is used as the image processing library.

[0931] Generative AI models are built using TensorFlow or PyTorch.

[0932] Use standard communication protocols (e.g. HTTP / HTTPS) for network communication.

[0933] System Overview

[0934] 1. Taking pictures and videos

[0935] Users use their devices to take photos of the work being done on-site, and the captured data is recorded in detail using a high-resolution camera.

[0936] 2. Data transmission

[0937] The device sends the captured images and videos to the server over the network. Before transmission, the data is compressed using a compression algorithm (e.g., JPEG or H.264).

[0938] 3. Data format conversion

[0939] The server converts the received data into a format suitable for analysis. Specifically, it extracts the necessary frames from images or videos and performs preprocessing using an image processing library such as OpenCV.

[0940] 4. Analysis using generative AI models

[0941] The server uses a pre-trained generative AI model to analyze the content of images and videos, and the analysis is based on work area procedures and safety standards to assess the safety of the work.

[0942] 5. Identifying dangerous areas

[0943] Based on the analysis results, the server identifies dangerous areas, such as areas where handrails are not installed when working at height, or areas where protective covers are not attached to equipment.

[0944] 6. Generate feedback

[0945] The server generates graphical, textual, and audio feedback data about identified hot spots, including image highlights of hot spots and specific instructions for countermeasures.

[0946] 7. Sending and Providing Feedback

[0947] The server then sends the generated feedback data to the terminal, which receives it and presents it to the user in the form of diagrams, text, and audio, allowing on-site workers to immediately recognize dangerous areas and take appropriate action.

[0948] 8. Feedback logging

[0949] The server stores each feedback in a detailed log format, which can be used for later analysis and development of improvement measures.

[0950] Specific examples

[0951] For example, when performing construction work at height, workers use their smartphones to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative AI model. The model analyzes scaffolding defects and missing safety equipment, generating specific feedback such as "There is a risk of falling because handrails are not installed." This feedback is sent to the worker's device in real time and displayed as visual alerts and audio guidance. Workers can immediately understand the situation and take necessary measures.

[0952] Prompt Sentence Examples

[0953] "This is an image and video analysis system for safety confirmation at construction sites. It checks the safety of working at heights, identifies dangerous areas such as missing handrails or missing safety equipment, and provides feedback in real time. This can improve safety at work sites."

[0954] This is a specific mode for carrying out the invention.

[0955] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0956] Step 1:

[0957] Taking pictures and videos

[0958] Users take photos of the work situation on-site using a smartphone or tablet.

[0959] Input: On-site working environment.

[0960] Specific actions: High-resolution cameras are used to capture detailed images of work areas and potential hazards.

[0961] Output: High resolution image and video data.

[0962] Step 2:

[0963] Sending data

[0964] The device sends the captured images and videos to a server via the network.

[0965] Input: High-resolution image and video data captured.

[0966] Specific operation: Before transmitting data, the amount of data is reduced using a compression algorithm (e.g., JPEG or H.264). The compressed data is sent to the server using a network communication protocol (e.g., HTTP / HTTPS).

[0967] Output: Compressed image and video data is sent to the server.

[0968] Step 3:

[0969] Data format conversion

[0970] The server converts the received data into a format suitable for analysis.

[0971] Input: Compressed image and video data.

[0972] Specific operation: Using an image processing library such as OpenCV, extract important frames from the video data and save them as image data. Also, perform preprocessing such as resizing and filtering of the image data.

[0973] Output: Image data in a format suitable for analysis.

[0974] Step 4:

[0975] Analysis using generative AI models

[0976] The server uses a pre-trained generative AI model to analyze the content of images and videos.

[0977] Input: Image data in a format suitable for analysis.

[0978] How it works: Apply generative AI models built with TensorFlow or PyTorch to analyze image data, specifically assessing it against work area procedures and safety standards.

[0979] Output: Analysis results (e.g., identification of dangerous areas and non-fulfillment of safety standards).

[0980] Step 5:

[0981] Identifying dangerous areas

[0982] The server identifies dangerous areas based on the analysis results.

[0983] Input: Analysis results from the generative AI model.

[0984] Specific actions: Review the analysis results and identify and list areas where handrails have not been installed when working at height, areas where protective covers have not been installed on equipment, etc.

[0985] Output: A list of identified hazards.

[0986] Step 6:

[0987] Generate feedback

[0988] The server generates feedback data in the form of diagrams, text, and audio about identified dangerous areas.

[0989] Input: A list of identified hazards.

[0990] Specific actions: Highlight images of dangerous areas and create diagrams, text, and audio data containing specific instructions for countermeasures.

[0991] Output: Feedback data (e.g., in diagram, text, or audio format).

[0992] Step 7:

[0993] Sending and Providing Feedback

[0994] The server transmits the generated feedback data to the terminal.

[0995] Input: Feedback data.

[0996] Specific operation: The generated feedback data is sent to the terminal using a network communication protocol. The terminal receives the feedback data and presents it to the user visually and audibly.

[0997] Output: Feedback information presented to the user (e.g., image highlighting of dangerous areas, specific instructions for countermeasures).

[0998] Step 8:

[0999] Feedback logging

[1000] The server stores each feedback in a detailed log format.

[1001] Input: Feedback data.

[1002] Specific operation: Feedback content is saved in log format using SQLite or a NoSQL database (e.g. MongoDB).

[1003] Output: A saved feedback log that can be used for later analysis and development of improvement measures.

[1004] (Application example 1)

[1005] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1006] Ensuring safety during on-site work is extremely important for protecting the lives of workers, but conventional methods lack consistency and efficiency. In particular, it is difficult to identify dangerous areas and take immediate action, and feedback management is insufficient. Therefore, a system is needed to ensure the safety of on-site work in real time and efficiently identify dangerous areas.

[1007] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1008] In this invention, the server includes means for formatting images and videos so that they can be analyzed, means for analyzing images and videos using generative artificial intelligence to identify dangerous areas, means for generating information about the identified dangerous areas in the form of diagrams, text, and audio, and means for presenting feedback data by audio and notifying the user of the dangerous areas and countermeasures. This makes it possible to ensure the safety of on-site work in real time, efficiently identify dangerous areas, and respond immediately.

[1009] "On-site work" refers to all work performed at a specific physical location, primarily including indoor and outdoor work activities such as construction sites and factories.

[1010] "Means for taking images or videos" means a device that has the function of capturing still images or videos using a photographing device such as a camera or video camera.

[1011] "Server" refers to the computer system utilized to receive, process, analyze data, and generate feedback.

[1012] "Generative artificial intelligence" refers to algorithms and models that learn, analyze, and make decisions based on large amounts of data.

[1013] "Dangerous areas" refer to locations or situations at work sites where there is a high possibility of accidents or disasters occurring.

[1014] "Feedback data" is a general term for information generated based on analysis results and conveyed to workers. This can be in the form of charts, text, audio, etc.

[1015] A "terminal" is a device used by a user, such as a smartphone, tablet, or PC.

[1016] "Means for presenting feedback" is a general term for devices and programs that have the function of conveying generated feedback data to the user visually or audibly.

[1017] A "log" is a full record of a system's operation history and analysis results, and refers to data that can be used for later analysis and improvement.

[1018] The present invention is a system for improving the efficiency of safety patrols within a factory. This system uses a robot that automatically patrols the factory to identify dangerous areas and provide feedback to the user in real time. The system is implemented using the following hardware and software.

[1019] Hardware

[1020] 1. Camera: A high-resolution camera (e.g., Logitech C920) is used to capture the environment and work conditions within the factory.

[1021] 2. Robots: Robots capable of autonomous patrol (e.g., Boston Dynamics Spot) patrol the factory and transport cameras.

[1022] 3. Server: A server (e.g. AWS EC2) is used for data analysis and feedback generation.

[1023] software

[1024] 1. OpenCV: Used for handling images and videos and preprocessing for analysis.

[1025] 2. Flask: Used to receive data on the server and build APIs.

[1026] 3. pyttsx3: A speech synthesis library for presenting feedback data audibly.

[1027] 4. Generative AI model: A machine learning model that analyzes images and videos and identifies dangerous areas (example: a custom model using TensorFlow or PyTorch).

[1028] Data processing and calculation

[1029] 1. Shooting and data transmission:

[1030] The robot patrols the factory, capturing images of the equipment and work environment with its camera. The captured images and video data are sent to a server via a network. At this stage, an appropriate compression method is used to maintain data quality.

[1031] 2. Data formatting and analysis:

[1032] The server formats the received image and video data so that it can be analyzed. Specifically, it uses OpenCV to preprocess the images and extract the necessary frames. It then analyzes the data using a pre-trained generative AI model to identify dangerous areas.

[1033] 3. Generating and Presenting Feedback:

[1034] Based on the analysis results, the server generates feedback data for identified dangerous areas. The feedback is generated in the form of diagrams, text, and audio and sent to the device. On the device side, the feedback is presented to the user via audio using pyttsx3, instantly notifying the user of dangerous areas and countermeasures.

[1035] Examples of concrete examples and prompts

[1036] For example, if a machine cover is loose in a factory, a robot will take a photo of this area and send the image to a server. The server will then use a generative AI model to analyze the image and generate feedback such as "The cover on machine X is loose. Please repair it immediately," and notify the robot and workers in real time.

[1037] Example prompt sentence:

[1038] "Analyze the provided images and assess the safety of the facility. Identify any dangerous areas or areas that need improvement and provide specific countermeasures."

[1039] In this way, the system of the present invention allows for efficient and consistent safety patrols within the factory, improving worker safety.

[1040] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1041] Step 1:

[1042] Input: Images and videos of on-site work taken by cameras

[1043] Processing: The robot autonomously patrols the factory, taking photos and videos of the work environment and equipment using a high-resolution camera.

[1044] Output: Captured image and video data

[1045] Specific operation: The robot moves along a pre-set patrol route, stopping periodically to take photos using its camera. The acquired data is temporarily stored in the terminal.

[1046] Step 2:

[1047] Input: Captured image and video data

[1048] Processing: Image and video data is sent to the server via the network. The data is compressed before being sent.

[1049] Output: Compressed image and video data sent to the server

[1050] What it does: It uses a compression algorithm to reduce the size of image and video data stored on the device and then uploads it to a server via Wi-Fi or a wired network.

[1051] Step 3:

[1052] Input: Compressed image and video data

[1053] Processing: The server converts the received data into an analyzable format, specifically by using OpenCV to preprocess the image and extract the necessary frames.

[1054] Output: Image and video data in a parsable format

[1055] Specific operation: The server performs preprocessing such as image resizing and noise reduction, and extracts the information necessary for analysis.

[1056] Step 4:

[1057] Input: Image and video data in a parsable format

[1058] Processing: Generative AI models are used to analyze images and videos to identify hazards.

[1059] Output: Identification of dangerous areas

[1060] What it does: The server runs a generative AI model (e.g., a custom model using TensorFlow or PyTorch) to analyze image and video data. Specifically, the model evaluates preprocessed images and videos to detect hazards and anomalies.

[1061] Step 5:

[1062] Input: Hazardous area identification results

[1063] Processing: Generate feedback in the form of diagrams, text, or audio based on the identified hazards. Send the generated feedback data to the device.

[1064] Output: Generated feedback data

[1065] Specific operation: The server generates feedback in the form of diagrams, text, and audio guides. For example, it generates audio files using pyttsx3 and uses them to notify users of dangerous areas. The generated feedback data is sent to the device via API.

[1066] Step 6:

[1067] Input: Feedback data sent to the device

[1068] Processing: The terminal presents the received feedback data to the user.

[1069] Output: Feedback information presented to the user

[1070] What it does: The device analyzes the feedback data and displays it in a user-facing interface. Specifically, audio guides are played using pyttsx3, and visual hazard warnings are also displayed on the screen.

[1071] Step 7:

[1072] Input: Feedback information

[1073] Processing: Feedback data is stored in a detailed log format for later analysis and improvement.

[1074] Output: Feedback data saved as a log

[1075] Specific operation: The server classifies all feedback data by date, time, location, content, etc. and stores it in a database. This log data is organized for later reference.

[1076] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1077] This invention combines a system for improving the efficiency of on-site safety patrols with an emotion engine that recognizes the user's emotions. This system recognizes the user's emotional state during the process of photographing, analyzing, and providing feedback, thereby improving the safety and efficiency of work by providing corresponding feedback.

[1078] composition

[1079] 1. Taking pictures and videos

[1080] Users take pictures of the work situation on-site using devices such as smartphones or tablets. High-resolution cameras are used to take pictures from multiple angles, allowing for a detailed understanding of the situation.

[1081] 2. Data transmission

[1082] The device sends the captured images and videos to the server via the network, where any necessary compression and format conversion is performed to maintain data quality during transmission.

[1083] 3. Data Formatting

[1084] The server converts the received data into a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[1085] 4. Analysis using generative artificial intelligence (AI)

[1086] The server uses a pre-trained generative AI model to analyze the content of images and videos, which evaluates the safety of work based on procedures and drawings and identifies dangerous areas.

[1087] 5. Identifying dangerous areas

[1088] Based on the analysis results, the server identifies dangerous areas, such as missing handrails when working at height or defects in the protective covers of machinery, in real time.

[1089] 6. User Emotion Recognition

[1090] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize their emotional state. For example, it can detect tension or anxiety.

[1091] 7. Generate feedback

[1092] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state, including flexible responses according to the user's emotional state.

[1093] 8. Sending and Providing Feedback

[1094] The server sends the generated feedback data to the terminal, which then presents the received data to the user and provides specific safety instructions using diagrams, text, and audio guidance.

[1095] 9. Feedback logging

[1096] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[1097] Specific examples

[1098] For example, when working at height, a worker (user) uses a smartphone to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative artificial intelligence model. As a result of the analysis, it is identified that a handrail is not installed, posing a risk. At the same time, the device's emotion engine analyzes the worker's facial expressions to detect tension or anxiety.

[1099] Based on the information about dangerous locations and the user's emotional state, the server generates specific, emotionally sensitive feedback such as, "There is a risk of falling because there are no handrails installed. Please stay calm and take appropriate measures." This feedback is sent to the worker's device in real time and displayed through visual highlights and audio guidance, allowing the worker to immediately recognize the situation and take appropriate measures.

[1100] In this way, the system of the present invention improves the safety of on-site work and enables flexible responses that take into account the emotional state of the user, thereby improving both work efficiency and safety.

[1101] The processing flow will be explained below.

[1102] Step 1:

[1103] Before starting work on-site, users prepare a device such as a smartphone or tablet and launch a camera app. They organize the surrounding environment of the subject to be photographed so that they can take appropriate photos.

[1104] Step 2:

[1105] Users should take video and still images of the work environment and the work in progress, taking care to record in detail any areas that may be particularly dangerous or important points during the work.

[1106] Step 3:

[1107] The device stores captured images and videos in local storage, compressing and converting formats as needed to maintain data quality.

[1108] Step 4:

[1109] The device transmits the saved data to the server via the network, and the progress of the transmission is displayed to the user in real time.

[1110] Step 5:

[1111] The server stores the received image and video data and formats it in a format suitable for analysis. Specifically, it extracts the necessary frames from the images and performs preprocessing to remove noise.

[1112] Step 6:

[1113] The server inputs the data into a generative artificial intelligence (AI) model and begins analysis. The model analyzes the work content based on pre-trained procedures and drawings.

[1114] Step 7:

[1115] Generative AI models analyze the content of images and videos to identify hazards, such as missing handrails or defective protective covers on equipment, in real time.

[1116] Step 8:

[1117] The device uses a built-in emotion engine to analyze the user's facial expressions and voice to recognize their emotional state, automatically detecting emotions such as tension, anxiety, and surprise.

[1118] Step 9:

[1119] Based on the analysis results, the server generates feedback in the form of diagrams, text, and audio, taking into account the user's emotional state and the information on dangerous areas. The feedback is specific and easy to understand, taking into consideration the user's emotions.

[1120] Step 10:

[1121] The server sends the generated feedback data to the terminal in real time, so that the feedback content is reflected immediately to the user.

[1122] Step 11:

[1123] The device then presents the received feedback data to the user, using visual highlights and audio guidance to provide detailed explanations of dangerous areas and how to respond.

[1124] Step 12:

[1125] The server stores each feedback and the user's emotional state in a detailed log format, which makes it easier to analyze and develop improvement measures later.

[1126] Step 13:

[1127] Based on the feedback provided, users can take appropriate measures to address dangerous areas on-site, such as installing handrails or checking safety equipment.

[1128] In this way, by providing feedback that takes into account the user's emotional state, the system significantly improves the safety and efficiency of on-site work.

[1129] Example 2

[1130] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1131] Ensuring the safety of on-site work requires rapid and accurate identification of dangerous areas and appropriate feedback. However, current systems face the problem that these processes take time and make it difficult to respond flexibly while taking into account the user's emotional state. In particular, when a worker is feeling nervous or anxious, feedback that ignores that psychological state not only fails to improve safety but may actually reduce work efficiency. The present invention aims to solve these problems.

[1132] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for formatting captured video data into an analyzable state, a means for analyzing the video using the generated artificial intelligence model and identifying dangerous areas, and a means for generating feedback based on the user's emotional state. This increases the safety of on-site work and enables flexible response according to the user's emotional state.

[1133] "Field work" refers to all work activities performed by workers on-site.

[1134] "Footage" refers to images or videos that record the status of on-site work.

[1135] "Server" refers to a computer system that receives, processes, and stores data over a network.

[1136] A "terminal" is a device operated by a user, and refers to a mobile information terminal such as a smartphone or tablet.

[1137] "Means for formatting into an analyzable state" refers to the process of converting received video data into a format suitable for analysis.

[1138] "Generative artificial intelligence" refers to an AI model that has learned large amounts of data in advance and has the ability to perform analysis on specific tasks.

[1139] "Generating" refers to the process of creating feedback such as text, diagrams, or audio based on specific information.

[1140] "Means for recognizing the user's emotions" refers to a mechanism for determining the user's psychological state based on data such as facial expressions and voice.

[1141] "Feedback" refers to advice and warning messages provided to users based on analysis results and emotion recognition.

[1142] "Means for storing in log format" refers to the process of recording the generated feedback data and the user's emotional state and storing it in a format that can be analyzed at a later date.

[1143] MODE FOR CARRYING OUT THE INVENTION

[1144] This system automates a series of processes that acquires on-site video, analyzes it, and provides feedback to improve the safety of on-site work. The specific hardware and software combinations used are as follows:

[1145] Hardware

[1146] Users use devices such as smartphones and tablets.

[1147] The server uses a high-performance computer system, enabling it to process large amounts of data quickly.

[1148] software

[1149] The device is equipped with a high-resolution camera, a microphone for recording audio, and emotion recognition software.

[1150] The server is equipped with a generative artificial intelligence model, enabling complex image and video analysis.

[1151] Data processing and calculation

[1152] 1. The user uses the device to capture high-resolution images of the worksite, including images and videos from multiple angles.

[1153] 2. The device compresses and converts the acquired video data into an appropriate format (JPEG or MP4).

[1154] 3. The terminal sends the compressed and converted data to the server via the Internet.

[1155] 4. The server preprocesses the received data to make it analyzable, including extracting and formatting images and video frames.

[1156] 5. The server analyzes the preprocessed data using the generative AI model and identifies dangerous areas on-site based on procedures and drawings.

[1157] 6. The device uses emotion recognition software to analyze the user's facial expressions and voice and evaluate the user's emotional state.

[1158] 7. The server generates feedback by taking into consideration the information about dangerous areas and the user's emotional state. This feedback includes text, diagrams, and audio guidance.

[1159] 8. The server sends the generated feedback data to the terminal.

[1160] 9. The device presents the received feedback to the user visually and audibly.

[1161] 10. The server stores the sent feedback data and the user's emotional state in log format and uses it for later analysis and improvement.

[1162] Specific examples

[1163] For example, when working at height, a worker (user) uses a smartphone to film the scaffolding and work environment. The device converts this video data into an appropriate format and sends it to a server via the Internet. The server preprocesses the received data and analyzes it using a generative AI model. This identifies dangerous areas, such as areas without handrails. At the same time, the device analyzes the worker's facial expressions and voice using emotion recognition software to detect tension or anxiety. Based on the dangerous areas and emotion data, the server generates feedback such as, "There is a risk of falling because there are no handrails. Please remain calm and take appropriate measures," and sends it to the device. The device then presents this feedback to the worker visually and audibly, allowing the worker to take immediate action.

[1164] This not only improves the safety of on-site work, but also enables flexible responses according to the user's emotional state, which is expected to improve both work efficiency and safety.

[1165] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1166] Step 1:

[1167] Users use devices such as smartphones and tablets to capture images of the work situation at the site. The input is image and video data of the site. Specifically, a high-resolution camera is used to capture images of the site from multiple angles, capturing multiple detailed videos. This video data is used for subsequent analysis.

[1168] Step 2:

[1169] The device compresses the captured image and video data and converts it into a format suitable for analysis (e.g., JPEG or MP4). The input is the video data acquired in step 1, and the output is compressed image or video data. Specifically, a data compression algorithm is applied to reduce the file size.

[1170] Step 3:

[1171] The device sends compressed image and video data to a server over the Internet. The input is the compressed data, and the output is the state the data will be in when it arrives at the server. Error checking is performed to ensure reliability during data transfer.

[1172] Step 4:

[1173] The server checks the integrity of the received data and converts it into a format that can be analyzed. The input is the data received in step 3, and the output is the preprocessed data. Specifically, it performs an error check, then extracts images or video frames and performs any necessary preprocessing.

[1174] Step 5:

[1175] The server loads a pre-trained generative AI model and begins analyzing the pre-processed data as input. The output is the analysis results, which are information used to evaluate the safety of the work. Specifically, the work content is analyzed based on procedures and drawings, and dangerous areas are identified.

[1176] Step 6:

[1177] The server identifies dangerous areas based on the analysis results. The input is the analysis results of the generative AI model, and the output is a list of identified dangerous areas. Specifically, it detects missing handrails when working at height and defects in the protective covers of machinery.

[1178] Step 7:

[1179] The device collects the user's facial expressions and voice using a built-in camera and microphone. The input is the user's facial and voice data, and the output is their emotional state analyzed by emotion recognition software. Specifically, the device measures tension and anxiety by recognizing the user's face and analyzing their voice tone.

[1180] Step 8:

[1181] The server generates feedback based on the identified dangerous locations and the user's emotional state. The input is a list of dangerous locations and the user's emotional state, and the output is the generated feedback data. Specifically, safety instructions and advice are created as text, diagrams, and audio guides.

[1182] Step 9:

[1183] The server sends the generated feedback data to the terminal. The input is the feedback data, and the output is the data transmission to the terminal.

[1184] Step 10:

[1185] The device presents the received feedback to the user. The input is the feedback data, and the output is the visual and audio feedback provided to the user. Specifically, it displays diagrams and text on the screen and plays audio guidance as needed.

[1186] Step 11:

[1187] The server stores the sent feedback and the user's emotional state in log format. The input is feedback and emotional data, and the output is log data. This allows data to be accumulated for later analysis and improvement.

[1188] (Application example 2)

[1189] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1190] In on-site work, it is necessary to ensure work safety while also taking into consideration the emotional state of workers. However, current systems only evaluate the safety of work and lack the functionality to recognize the emotional state of workers and provide appropriate feedback based on that. As a result, there is an issue in that they are unable to respond appropriately to workers who are feeling tense or anxious.

[1191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1192] In this invention, the server includes means for taking images and videos of on-site work, means for transmitting the captured images and videos to the server, means for formatting the data so that it can be analyzed by the server, means for analyzing the images and videos in the server using artificial intelligence to identify hazardous areas, means for generating information about the identified hazardous areas in the form of diagrams, text, and audio, means for transmitting the generated feedback data to the terminal, means for presenting feedback to the user, means for analyzing the user's facial expressions and voice to recognize the user's emotional state, and means for generating feedback based on the user's emotional state. This makes it possible to provide flexible and appropriate feedback that takes into consideration not only the safety of the work but also the emotional state of the worker.

[1193] "On-site work" refers to work activities that take place at a designated location.

[1194] "Means for taking images or videos" refers to equipment or systems for collecting visual information using devices, including cameras.

[1195] "Means for sending to the server" refers to the communication protocols and devices used to transfer collected data to the server via a network.

[1196] "Means for formatting data in an analyzable form" refers to the process or software used to convert collected data into a form that can be efficiently analyzed by the server.

[1197] "Means for analyzing images and videos using generative artificial intelligence" refers to a system that uses a pre-trained AI model to evaluate collected visual information and extract the necessary information.

[1198] "Means for identifying dangerous areas" refers to algorithms or programs for recognizing risk areas in the workplace based on the analysis results.

[1199] "Means for generating feedback in the form of diagrams, text, or audio" refers to technologies that allow users to visualize and / or voice information.

[1200] The "means for transmitting generated feedback data to the terminal" refers to a communication means for quickly transferring the generated feedback information to the user's device.

[1201] The term "means for presenting feedback to the user" refers to a function for displaying feedback information so that the user can check it via the device.

[1202] "Means of analyzing the user's facial expressions and voice to recognize their emotional state" refers to an AI engine that determines the user's emotions based on data acquired by a camera and microphone.

[1203] The "means for generating feedback based on the user's emotional state" refers to a system for proposing countermeasures according to the recognized emotions and generating them as feedback information.

[1204] This invention is a system that streamlines safety patrols for on-site work and takes into account the user's emotional state. The system uses high-resolution cameras, microphones, servers, and terminals (smartphones, tablets, factory robots, etc.) as hardware. The software includes an emotion recognition engine, generative AI model, data formatter, communication protocol, and feedback generation algorithm.

[1205] System Operation Overview

[1206] Data collection

[1207] Users use the device's built-in high-resolution camera and microphone to collect images, video, and audio data of on-site work, providing detailed information about the work environment and the worker's situation.

[1208] Data transmission

[1209] The devices transmit the collected data to a server in real time using a communication protocol that compresses the data and converts it into a suitable format.

[1210] Data Format

[1211] The server formats the transmitted data in a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[1212] Analysis by generative artificial intelligence

[1213] The server uses a pre-trained generative AI model to analyze the collected images and videos. This allows it to compare the images and videos with work procedures and drawings, evaluate the safety of the work, and identify dangerous areas. An emotion recognition engine is also used to identify the user's emotional state from their facial expressions and voice.

[1214] Generate feedback

[1215] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state. For example, it combines information such as "It is dangerous to work at height because there are no handrails installed" with instructions such as "Please stay calm and check" if the worker is nervous.

[1216] Sending and Providing Feedback

[1217] The generated feedback data is sent from the server to the terminal. The terminal then presents the received feedback to the user visually (on screen) and audibly, allowing workers to immediately understand dangerous areas and take appropriate measures.

[1218] Feedback logging

[1219] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[1220] Specific examples

[1221] For example, when working at height, a worker (user) uses a smartphone to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative AI model. The analysis identifies a hazard due to the absence of handrails. At the same time, the device's emotion engine analyzes the worker's facial expressions to detect tension or anxiety. Based on the hazard information and the user's emotional state, the server generates specific, emotion-sensitive feedback, such as, "There is a risk of falling because handrails are not installed. Please remain calm and take appropriate measures." This feedback is sent to the worker's device in real time and displayed through visual highlights and audio guidance. The worker can immediately recognize the situation and take appropriate measures.

[1222] Specific examples of prompts for the generative AI model to use

[1223] "Analyze image data of work at height to identify workers' emotional states (tension, anxiety, etc.) and dangerous areas."

[1224] "Analyze the voice data, determine the worker's emotional state from the tone and content of their voice, and generate appropriate feedback."

[1225] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1226] Step 1:

[1227] Users use the device's built-in high-resolution camera and microphone to collect images, videos, and audio data of on-site work.

[1228] Input: User-activated camera and microphone, local video and audio

[1229] Output: High-resolution image data, video data, audio data

[1230] Step 2:

[1231] The devices transmit the collected data to a server in real time using a communication protocol that compresses the data and converts it into a suitable format.

[1232] Input: High-resolution image data, video data, audio data

[1233] Output: Compressed image data, video data, and audio data are sent to the server.

[1234] Step 3:

[1235] The server formats the transmitted data in a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[1236] Input: Compressed image data, video data, audio data

[1237] Output: Parsable formatted image frames and audio data

[1238] Step 4:

[1239] The server uses a pre-trained generative AI model to analyze the collected images and videos, compare them with work procedures and drawings, evaluate the safety of the work, and identify any hazards.

[1240] Input: Image frames and audio data formatted in a parseable format

[1241] Output: Information on identified dangerous areas (e.g., areas where handrails are not installed)

[1242] Step 5:

[1243] The server simultaneously operates an emotion recognition engine to identify the user's emotional state from their facial expressions and voice.

[1244] Input: Image frames and audio data formatted in a parseable format

[1245] Output: Information about the user's emotional state (e.g., tension, anxiety)

[1246] Step 6:

[1247] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state.

[1248] Input: Information on dangerous areas, information on the user's emotional state

[1249] Output: Feedback data (in diagram, text, audio format)

[1250] Step 7:

[1251] The generated feedback data is sent from the server to the terminal, which then presents the received feedback to the user visually (on screen) or audibly.

[1252] Input: Feedback data (in diagram, text, audio format)

[1253] Output: Feedback presented through the device's screen and speakers

[1254] Step 8:

[1255] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[1256] Input: Feedback data, information on the user's emotional state

[1257] Output: Saved log data

[1258] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1259] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1260] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1261] [Fourth embodiment]

[1262] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1263] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1264] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1265] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1266] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1267] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1268] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1269] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1270] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1271] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1272] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1273] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1274] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1275] This invention is a system for improving the efficiency of safety patrols for on-site work. This system is realized by taking images and videos of on-site work, sending them to a server, analyzing them, and providing the results to the site as feedback.

[1276] composition

[1277] 1. Taking pictures and videos

[1278] Users take pictures of the work situation on-site using devices such as smartphones or tablets. It is recommended to use a high-resolution camera that can record even complex work situations.

[1279] 2. Data transmission

[1280] The device sends the captured images and videos to the server via the network. Before sending, an appropriate compression method is used to maintain the quality of the image and video data.

[1281] 3. Data Formatting

[1282] The server converts the received data into a format suitable for analysis, specifically by extracting the necessary frames from images and videos and performing preprocessing.

[1283] 4. Analysis using generative artificial intelligence (AI)

[1284] The server uses pre-trained generative AI models to analyze the content of images and videos, for example, assessing the safety of scaffolding or the operation status of machinery based on work area procedures and safety standards.

[1285] 5. Identifying dangerous areas

[1286] Based on the analysis results performed by the server, dangerous areas are identified in real time, such as areas where handrails are not installed during work at height or where protective covers are not attached to equipment.

[1287] 6. Generate feedback

[1288] The server generates feedback data on identified hazards in the form of diagrams, text, and audio, including image highlights of the hazards and specific instructions for countermeasures.

[1289] 7. Sending and Providing Feedback

[1290] The server sends the generated feedback data to the terminal, which receives it and presents it to the user in the form of diagrams, text, and audio, allowing on-site workers to immediately recognize dangerous areas and take appropriate action.

[1291] 8. Feedback logging

[1292] The server stores each feedback in a detailed log format, which can be used for future analysis and improvement planning.

[1293] Specific examples

[1294] For example, when performing construction work at height, workers use their smartphones to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative artificial intelligence model. The model analyzes scaffolding defects and missing safety equipment and generates specific feedback, such as "There is a risk of falling because handrails are not installed." This feedback is sent to the worker's device in real time and displayed as visual alerts and audio guidance. Workers can immediately understand the situation and take necessary measures.

[1295] In this way, the system of the present invention allows patrols to be carried out efficiently and consistently to improve the safety of on-site operations.

[1296] The processing flow will be explained below.

[1297] Step 1:

[1298] Before starting work on-site, users prepare their smartphone, tablet, or other device, launch the camera app, and, if necessary, organize the surrounding environment to make it easier to take photos.

[1299] Step 2:

[1300] Users can take video and still images of the progress and environment of on-site work, recording areas that require particular attention from multiple angles, allowing them to grasp the overall situation.

[1301] Step 3:

[1302] The device temporarily stores captured images and videos in local storage, compressing and converting the data as needed to maintain quality.

[1303] Step 4:

[1304] The device sends the captured images and videos to the server via the network. During the transmission, the progress is displayed for the user to check.

[1305] Step 5:

[1306] The server stores the received image and video data and formats it into an analyzable format, for example by extracting the necessary frames and performing preprocessing.

[1307] Step 6:

[1308] The server inputs image and video data into the generative AI model and begins analysis. The model has been trained in advance and can determine the work situation based on instructions and drawings.

[1309] Step 7:

[1310] The generative AI model analyzes the content of images and videos to identify hazards, such as missing handrails or missing safety equipment, and returns the results.

[1311] Step 8:

[1312] Based on the analysis results, the server generates information on dangerous areas in the form of diagrams, text, and audio, highlighting specific areas and providing detailed explanations of appropriate countermeasures.

[1313] Step 9:

[1314] The server transmits the generated feedback data to the device in real time, which includes image, text, and audio information.

[1315] Step 10:

[1316] The device then presents the received feedback data to the user, for example by highlighting dangerous areas on the screen and providing specific precautions with text and audio guidance.

[1317] Step 11:

[1318] The server stores each feedback in a detailed log, providing data for later analysis and development of improvements.

[1319] Step 12:

[1320] Based on the feedback provided, users can take appropriate measures to address dangerous areas at the site, such as installing handrails and ensuring that safety equipment is worn.

[1321] Example 1

[1322] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1323] Safety patrols at on-site work sites consume a lot of human resources, are inefficient, and are difficult to implement consistently. Traditional methods are unable to respond quickly to changes in the work environment, which delays the identification of dangerous areas and immediate response. Furthermore, recording feedback and analyzing it for future use are also manual processes, which means it takes a long time to develop improvement measures.

[1324] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1325] In this invention, the server includes a means for capturing images and videos of on-site work, a means for transmitting the captured images and videos via a network, a means for converting the received image and video data into an analyzable format, a means for analyzing the content of the images and videos using a generative AI model and identifying hazardous areas, a means for generating information on the identified hazardous areas in diagram, text, and audio formats, a means for transmitting the generated feedback data back to the terminal, and a means for presenting the feedback to the user. This enables the efficiency and consistency of on-site work safety patrols. Furthermore, real-time identification of hazardous areas and instructions for response can be achieved, and the automated recording of feedback content can be used for future analysis and the development of improvement measures.

[1326] "Field work" refers to work such as construction, maintenance, and safety inspections carried out at a specific location.

[1327] "Images and videos" are visual data recorded using a photographic device such as a camera, and are used to visually evaluate the work situation.

[1328] "Network" means a communications infrastructure for transmitting data, including the Internet and an internal company network.

[1329] "Server" refers to a computing system that receives, processes, or transmits data.

[1330] A "terminal" is a device that a user directly uses, such as a smartphone or tablet.

[1331] A "generative AI model" is an artificial intelligence model that has been trained to perform specific tasks in advance and is used to analyze the content of images and videos.

[1332] "Analysis" is the process of dissecting data to find meaning and patterns.

[1333] "Hazardous point" refers to an area or condition in the work environment that does not meet safety standards and that presents a risk of ingestion.

[1334] "Feedback" is information generated based on the analysis results, and is used to alert the user or give instructions for improvement.

[1335] A "log" is a collection of data that records system operations and analysis results, and is used for later analysis and verification.

[1336] MODE FOR CARRYING OUT THE INVENTION

[1337] This invention is a system for improving the efficiency of safety patrols at on-site work, and automates the process of taking, transmitting, analyzing, and providing feedback on images and videos of on-site work. Below, we will explain each component of this system and its specific operation.

[1338] Hardware and software used

[1339] 1. Hardware

[1340] The devices that users use are smartphones and tablets, which are equipped with high-resolution cameras.

[1341] A server is a high performance computer system.

[1342] 2. Software

[1343] OpenCV is used as the image processing library.

[1344] Generative AI models are built using TensorFlow or PyTorch.

[1345] Use standard communication protocols (e.g. HTTP / HTTPS) for network communication.

[1346] System Overview

[1347] 1. Taking pictures and videos

[1348] Users use their devices to take photos of the work being done on-site, and the captured data is recorded in detail using a high-resolution camera.

[1349] 2. Data transmission

[1350] The device sends the captured images and videos to the server over the network. Before transmission, the data is compressed using a compression algorithm (e.g., JPEG or H.264).

[1351] 3. Data format conversion

[1352] The server converts the received data into a format suitable for analysis. Specifically, it extracts the necessary frames from images or videos and performs preprocessing using an image processing library such as OpenCV.

[1353] 4. Analysis using generative AI models

[1354] The server uses a pre-trained generative AI model to analyze the content of images and videos, and the analysis is based on work area procedures and safety standards to assess the safety of the work.

[1355] 5. Identifying dangerous areas

[1356] Based on the analysis results, the server identifies dangerous areas, such as areas where handrails are not installed when working at height, or areas where protective covers are not attached to equipment.

[1357] 6. Generate feedback

[1358] The server generates graphical, textual, and audio feedback data about identified hot spots, including image highlights of hot spots and specific instructions for countermeasures.

[1359] 7. Sending and Providing Feedback

[1360] The server then sends the generated feedback data to the terminal, which receives it and presents it to the user in the form of diagrams, text, and audio, allowing on-site workers to immediately recognize dangerous areas and take appropriate action.

[1361] 8. Feedback logging

[1362] The server stores each feedback in a detailed log format, which can be used for later analysis and development of improvement measures.

[1363] Specific examples

[1364] For example, when performing construction work at height, workers use their smartphones to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative AI model. The model analyzes scaffolding defects and missing safety equipment, generating specific feedback such as "There is a risk of falling because handrails are not installed." This feedback is sent to the worker's device in real time and displayed as visual alerts and audio guidance. Workers can immediately understand the situation and take necessary measures.

[1365] Prompt Sentence Examples

[1366] "This is an image and video analysis system for safety confirmation at construction sites. It checks the safety of working at heights, identifies dangerous areas such as missing handrails or missing safety equipment, and provides feedback in real time. This can improve safety at work sites."

[1367] This is a specific mode for carrying out the invention.

[1368] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1369] Step 1:

[1370] Taking pictures and videos

[1371] Users take photos of the work situation on-site using a smartphone or tablet.

[1372] Input: On-site working environment.

[1373] Specific actions: High-resolution cameras are used to capture detailed images of work areas and potential hazards.

[1374] Output: High resolution image and video data.

[1375] Step 2:

[1376] Sending data

[1377] The device sends the captured images and videos to a server via the network.

[1378] Input: High-resolution image and video data captured.

[1379] Specific operation: Before transmitting data, the amount of data is reduced using a compression algorithm (e.g., JPEG or H.264). The compressed data is sent to the server using a network communication protocol (e.g., HTTP / HTTPS).

[1380] Output: Compressed image and video data is sent to the server.

[1381] Step 3:

[1382] Data format conversion

[1383] The server converts the received data into a format suitable for analysis.

[1384] Input: Compressed image and video data.

[1385] Specific operation: Using an image processing library such as OpenCV, extract important frames from the video data and save them as image data. Also, perform preprocessing such as resizing and filtering of the image data.

[1386] Output: Image data in a format suitable for analysis.

[1387] Step 4:

[1388] Analysis using generative AI models

[1389] The server uses a pre-trained generative AI model to analyze the content of images and videos.

[1390] Input: Image data in a format suitable for analysis.

[1391] How it works: Apply generative AI models built with TensorFlow or PyTorch to analyze image data, specifically assessing it against work area procedures and safety standards.

[1392] Output: Analysis results (e.g., identification of dangerous areas and non-fulfillment of safety standards).

[1393] Step 5:

[1394] Identifying dangerous areas

[1395] The server identifies dangerous areas based on the analysis results.

[1396] Input: Analysis results from the generative AI model.

[1397] Specific actions: Review the analysis results and identify and list areas where handrails have not been installed when working at height, areas where protective covers have not been installed on equipment, etc.

[1398] Output: A list of identified hazards.

[1399] Step 6:

[1400] Generate feedback

[1401] The server generates feedback data in the form of diagrams, text, and audio about identified dangerous areas.

[1402] Input: A list of identified hazards.

[1403] Specific actions: Highlight images of dangerous areas and create diagrams, text, and audio data containing specific instructions for countermeasures.

[1404] Output: Feedback data (e.g., in diagram, text, or audio format).

[1405] Step 7:

[1406] Sending and Providing Feedback

[1407] The server transmits the generated feedback data to the terminal.

[1408] Input: Feedback data.

[1409] Specific operation: The generated feedback data is sent to the terminal using a network communication protocol. The terminal receives the data and presents it to the user visually and audibly.

[1410] Output: Feedback information presented to the user (e.g., image highlighting of dangerous areas, specific instructions for countermeasures).

[1411] Step 8:

[1412] Feedback logging

[1413] The server stores each feedback in a detailed log format.

[1414] Input: Feedback data.

[1415] Specific operation: Feedback content is saved in log format using SQLite or a NoSQL database (e.g. MongoDB).

[1416] Output: A saved feedback log that can be used for later analysis and development of improvement measures.

[1417] (Application example 1)

[1418] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1419] Ensuring safety during on-site work is extremely important for protecting the lives of workers, but conventional methods lack consistency and efficiency. In particular, it is difficult to identify dangerous areas and take immediate action, and feedback management is insufficient. Therefore, a system is needed to ensure the safety of on-site work in real time and efficiently identify dangerous areas.

[1420] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1421] In this invention, the server includes means for formatting images and videos so that they can be analyzed, means for analyzing images and videos using generative artificial intelligence to identify dangerous areas, means for generating information about the identified dangerous areas in the form of diagrams, text, and audio, and means for presenting feedback data by audio and notifying the user of the dangerous areas and countermeasures. This makes it possible to ensure the safety of on-site work in real time, efficiently identify dangerous areas, and respond immediately.

[1422] "On-site work" refers to all work performed at a specific physical location, primarily including indoor and outdoor work activities such as construction sites and factories.

[1423] "Means for taking images or videos" means a device that has the function of capturing still images or videos using a photographing device such as a camera or video camera.

[1424] "Server" refers to the computer system utilized to receive, process, analyze data, and generate feedback.

[1425] "Generative artificial intelligence" refers to algorithms and models that learn, analyze, and make decisions based on large amounts of data.

[1426] "Dangerous areas" refer to locations or situations at work sites where there is a high possibility of accidents or disasters occurring.

[1427] "Feedback data" is a general term for information generated based on analysis results and conveyed to workers. This can be in the form of charts, text, audio, etc.

[1428] A "terminal" is a device used by a user, such as a smartphone, tablet, or PC.

[1429] "Means for presenting feedback" is a general term for devices and programs that have the function of conveying generated feedback data to the user visually or audibly.

[1430] A "log" is a full record of a system's operation history and analysis results, and refers to data that can be used for later analysis and improvement.

[1431] The present invention is a system for improving the efficiency of safety patrols within a factory. This system uses a robot that automatically patrols the factory to identify dangerous areas and provide feedback to the user in real time. The system is implemented using the following hardware and software.

[1432] Hardware

[1433] 1. Camera: A high-resolution camera (e.g., Logitech C920) is used to capture the environment and work conditions within the factory.

[1434] 2. Robots: Robots capable of autonomous patrol (e.g., Boston Dynamics Spot) patrol the factory and transport cameras.

[1435] 3. Server: A server (e.g. AWS EC2) is used for data analysis and feedback generation.

[1436] software

[1437] 1. OpenCV: Used for handling images and videos and preprocessing for analysis.

[1438] 2. Flask: Used to receive data on the server and build APIs.

[1439] 3. pyttsx3: A speech synthesis library for presenting feedback data audibly.

[1440] 4. Generative AI model: A machine learning model that analyzes images and videos and identifies dangerous areas (example: a custom model using TensorFlow or PyTorch).

[1441] Data processing and calculation

[1442] 1. Shooting and data transmission:

[1443] The robot patrols the factory, capturing images of the equipment and work environment with its camera. The captured images and video data are sent to a server via a network. At this stage, an appropriate compression method is used to maintain data quality.

[1444] 2. Data formatting and analysis:

[1445] The server formats the received image and video data so that it can be analyzed. Specifically, it uses OpenCV to preprocess the images and extract the necessary frames. It then analyzes the data using a pre-trained generative AI model to identify dangerous areas.

[1446] 3. Generating and Presenting Feedback:

[1447] Based on the analysis results, the server generates feedback data for identified dangerous areas. The feedback is generated in the form of diagrams, text, and audio and sent to the device. On the device side, the feedback is presented to the user via audio using pyttsx3, instantly notifying the user of dangerous areas and countermeasures.

[1448] Examples of concrete examples and prompts

[1449] For example, if a machine cover is loose in a factory, a robot will take a photo of this area and send the image to a server. The server will then use a generative AI model to analyze the image and generate feedback such as "The cover on machine X is loose. Please repair it immediately," and notify the robot and workers in real time.

[1450] Example prompt sentence:

[1451] "Analyze the provided images and assess the safety of the facility. Identify any dangerous areas or areas that need improvement and provide specific countermeasures."

[1452] In this way, the system of the present invention allows for efficient and consistent safety patrols within the factory, improving worker safety.

[1453] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1454] Step 1:

[1455] Input: Images and videos of on-site work taken by cameras

[1456] Processing: The robot autonomously patrols the factory, taking photos and videos of the work environment and equipment using a high-resolution camera.

[1457] Output: Captured image and video data

[1458] Specific operation: The robot moves along a pre-set patrol route, stopping periodically to take photos using its camera. The acquired data is temporarily stored in the terminal.

[1459] Step 2:

[1460] Input: Captured image and video data

[1461] Processing: Image and video data is sent to the server via the network. The data is compressed before being sent.

[1462] Output: Compressed image and video data sent to the server

[1463] What it does: It uses a compression algorithm to reduce the size of image and video data stored on the device and then uploads it to a server via Wi-Fi or a wired network.

[1464] Step 3:

[1465] Input: Compressed image and video data

[1466] Processing: The server converts the received data into an analyzable format, specifically by using OpenCV to preprocess the image and extract the necessary frames.

[1467] Output: Image and video data in a parsable format

[1468] Specific operation: The server performs preprocessing such as image resizing and noise reduction, and extracts the information necessary for analysis.

[1469] Step 4:

[1470] Input: Image and video data in a parsable format

[1471] Processing: Generative AI models are used to analyze images and videos to identify hazards.

[1472] Output: Identification of dangerous areas

[1473] What it does: The server runs a generative AI model (e.g., a custom model using TensorFlow or PyTorch) to analyze image and video data. Specifically, the model evaluates preprocessed images and videos to detect hazards and anomalies.

[1474] Step 5:

[1475] Input: Hazardous area identification results

[1476] Processing: Generate feedback in the form of diagrams, text, or audio based on the identified hazards. Send the generated feedback data to the device.

[1477] Output: Generated feedback data

[1478] Specific operation: The server generates feedback in the form of diagrams, text, and audio guides. For example, it generates audio files using pyttsx3 and uses them to notify users of dangerous areas. The generated feedback data is sent to the device via API.

[1479] Step 6:

[1480] Input: Feedback data sent to the device

[1481] Processing: The terminal presents the received feedback data to the user.

[1482] Output: Feedback information presented to the user

[1483] What it does: The device analyzes the feedback data and displays it in a user-facing interface. Specifically, audio guides are played using pyttsx3, and visual hazard warnings are also displayed on the screen.

[1484] Step 7:

[1485] Input: Feedback information

[1486] Processing: Feedback data is stored in a detailed log format for later analysis and improvement.

[1487] Output: Feedback data saved as a log

[1488] Specific operation: The server classifies all feedback data by date, time, location, content, etc. and stores it in a database. This log data is organized for later reference.

[1489] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1490] This invention combines a system for improving the efficiency of on-site safety patrols with an emotion engine that recognizes the user's emotions. This system recognizes the user's emotional state during the process of photographing, analyzing, and providing feedback, thereby improving the safety and efficiency of work by providing corresponding feedback.

[1491] composition

[1492] 1. Taking pictures and videos

[1493] Users take pictures of the work situation on-site using devices such as smartphones or tablets. High-resolution cameras are used to take pictures from multiple angles, allowing for a detailed understanding of the situation.

[1494] 2. Data transmission

[1495] The device sends the captured images and videos to the server via the network, where any necessary compression and format conversion is performed to maintain data quality during transmission.

[1496] 3. Data Formatting

[1497] The server converts the received data into a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[1498] 4. Analysis using generative artificial intelligence (AI)

[1499] The server uses a pre-trained generative AI model to analyze the content of images and videos, which evaluates the safety of work based on procedures and drawings and identifies dangerous areas.

[1500] 5. Identifying dangerous areas

[1501] Based on the analysis results, the server identifies dangerous areas, such as missing handrails when working at height or defects in the protective covers of machinery, in real time.

[1502] 6. User Emotion Recognition

[1503] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize their emotional state. For example, it can detect tension or anxiety.

[1504] 7. Generate feedback

[1505] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state, including flexible responses according to the user's emotional state.

[1506] 8. Sending and Providing Feedback

[1507] The server sends the generated feedback data to the terminal, which then presents the received data to the user and provides specific safety instructions using diagrams, text, and audio guidance.

[1508] 9. Feedback logging

[1509] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[1510] Specific examples

[1511] For example, when working at height, a worker (user) uses a smartphone to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative artificial intelligence model. As a result of the analysis, it is identified that a handrail is not installed, posing a risk. At the same time, the device's emotion engine analyzes the worker's facial expressions to detect tension or anxiety.

[1512] Based on the information about dangerous locations and the user's emotional state, the server generates specific, emotionally sensitive feedback such as, "There is a risk of falling because there are no handrails installed. Please stay calm and take appropriate measures." This feedback is sent to the worker's device in real time and displayed through visual highlights and audio guidance, allowing the worker to immediately recognize the situation and take appropriate measures.

[1513] In this way, the system of the present invention improves the safety of on-site work and enables flexible responses that take into account the emotional state of the user, thereby improving both work efficiency and safety.

[1514] The processing flow will be explained below.

[1515] Step 1:

[1516] Before starting work on-site, users prepare a device such as a smartphone or tablet and launch a camera app. They organize the surrounding environment of the subject to be photographed so that they can take appropriate photos.

[1517] Step 2:

[1518] Users should take video and still images of the work environment and the work in progress, taking care to record in detail any areas that may be particularly dangerous or important points during the work.

[1519] Step 3:

[1520] The device stores captured images and videos in local storage, compressing and converting formats as needed to maintain data quality.

[1521] Step 4:

[1522] The device transmits the saved data to the server via the network, and the progress of the transmission is displayed to the user in real time.

[1523] Step 5:

[1524] The server stores the received image and video data and formats it in a format suitable for analysis. Specifically, it extracts the necessary frames from the images and performs preprocessing to remove noise.

[1525] Step 6:

[1526] The server inputs the data into a generative artificial intelligence (AI) model and begins analysis. The model analyzes the work content based on pre-trained procedures and drawings.

[1527] Step 7:

[1528] Generative AI models analyze the content of images and videos to identify hazards, such as missing handrails or defective protective covers on equipment, in real time.

[1529] Step 8:

[1530] The device uses a built-in emotion engine to analyze the user's facial expressions and voice to recognize their emotional state, automatically detecting emotions such as tension, anxiety, and surprise.

[1531] Step 9:

[1532] Based on the analysis results, the server generates feedback in the form of diagrams, text, and audio, taking into account the user's emotional state and the information on dangerous areas. The feedback is specific and easy to understand, taking into consideration the user's emotions.

[1533] Step 10:

[1534] The server sends the generated feedback data to the terminal in real time, so that the feedback content is reflected immediately to the user.

[1535] Step 11:

[1536] The device then presents the received feedback data to the user, using visual highlights and audio guidance to provide detailed explanations of dangerous areas and how to respond.

[1537] Step 12:

[1538] The server stores each feedback and the user's emotional state in a detailed log format, which makes it easier to analyze and develop improvement measures later.

[1539] Step 13:

[1540] Based on the feedback provided, users can take appropriate measures to address dangerous areas on-site, such as installing handrails or checking safety equipment.

[1541] In this way, by providing feedback that takes into account the user's emotional state, the system significantly improves the safety and efficiency of on-site work.

[1542] Example 2

[1543] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1544] Ensuring the safety of on-site work requires rapid and accurate identification of dangerous areas and appropriate feedback. However, current systems face the problem that these processes take time and make it difficult to respond flexibly while taking into account the user's emotional state. In particular, when a worker is feeling nervous or anxious, feedback that ignores that psychological state not only fails to improve safety but may actually reduce work efficiency. The present invention aims to solve these problems.

[1545] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for formatting captured video data into an analyzable state, a means for analyzing the video using the generated artificial intelligence model and identifying dangerous areas, and a means for generating feedback based on the user's emotional state. This increases the safety of on-site work and enables flexible response according to the user's emotional state.

[1546] "Field work" refers to all work activities performed by workers on-site.

[1547] "Footage" refers to images or videos that record the status of on-site work.

[1548] "Server" refers to a computer system that receives, processes, and stores data over a network.

[1549] A "terminal" is a device operated by a user, and refers to a mobile information terminal such as a smartphone or tablet.

[1550] "Means for formatting into an analyzable state" refers to the process of converting received video data into a format suitable for analysis.

[1551] "Generative artificial intelligence" refers to an AI model that has learned large amounts of data in advance and has the ability to perform analysis on specific tasks.

[1552] "Generating" refers to the process of creating feedback such as text, diagrams, or audio based on specific information.

[1553] "Means for recognizing the user's emotions" refers to a mechanism for determining the user's psychological state based on data such as facial expressions and voice.

[1554] "Feedback" refers to advice and warning messages provided to users based on analysis results and emotion recognition.

[1555] "Means for storing in log format" refers to the process of recording the generated feedback data and the user's emotional state and storing it in a format that can be analyzed at a later date.

[1556] MODE FOR CARRYING OUT THE INVENTION

[1557] This system automates a series of processes that acquires on-site video, analyzes it, and provides feedback to improve the safety of on-site work. The specific hardware and software combinations used are as follows:

[1558] Hardware

[1559] Users use devices such as smartphones and tablets.

[1560] The server uses a high-performance computer system, enabling it to process large amounts of data quickly.

[1561] software

[1562] The device is equipped with a high-resolution camera, a microphone for recording audio, and emotion recognition software.

[1563] The server is equipped with a generative artificial intelligence model, enabling complex image and video analysis.

[1564] Data processing and calculation

[1565] 1. The user uses the device to capture high-resolution images of the worksite, including images and videos from multiple angles.

[1566] 2. The device compresses and converts the acquired video data into an appropriate format (JPEG or MP4).

[1567] 3. The terminal sends the compressed and converted data to the server via the Internet.

[1568] 4. The server preprocesses the received data to make it analyzable, including extracting and formatting images and video frames.

[1569] 5. The server analyzes the preprocessed data using the generative AI model and identifies dangerous areas on-site based on procedures and drawings.

[1570] 6. The device uses emotion recognition software to analyze the user's facial expressions and voice and evaluate the user's emotional state.

[1571] 7. The server generates feedback by taking into consideration the information about dangerous areas and the user's emotional state. This feedback includes text, diagrams, and audio guidance.

[1572] 8. The server sends the generated feedback data to the terminal.

[1573] 9. The device presents the received feedback to the user visually and audibly.

[1574] 10. The server stores the sent feedback data and the user's emotional state in log format and uses it for later analysis and improvement.

[1575] Specific examples

[1576] For example, when working at height, a worker (user) uses a smartphone to film the scaffolding and work environment. The device converts this video data into an appropriate format and sends it to a server via the Internet. The server preprocesses the received data and analyzes it using a generative AI model. This identifies dangerous areas, such as areas without handrails. At the same time, the device analyzes the worker's facial expressions and voice using emotion recognition software to detect tension or anxiety. Based on the dangerous areas and emotion data, the server generates feedback such as, "There is a risk of falling because there are no handrails. Please remain calm and take appropriate measures," and sends it to the device. The device then presents this feedback to the worker visually and audibly, allowing the worker to take immediate action.

[1577] This not only improves the safety of on-site work, but also enables flexible responses according to the user's emotional state, which is expected to improve both work efficiency and safety.

[1578] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1579] Step 1:

[1580] Users use devices such as smartphones and tablets to capture images of the work situation at the site. The input is image and video data of the site. Specifically, a high-resolution camera is used to capture images of the site from multiple angles, capturing multiple detailed videos. This video data is used for subsequent analysis.

[1581] Step 2:

[1582] The device compresses the captured image and video data and converts it into a format suitable for analysis (e.g., JPEG or MP4). The input is the video data acquired in step 1, and the output is compressed image or video data. Specifically, a data compression algorithm is applied to reduce the file size.

[1583] Step 3:

[1584] The device sends compressed image and video data to a server over the Internet. The input is the compressed data, and the output is the state the data will be in when it arrives at the server. Error checking is performed to ensure reliability during data transfer.

[1585] Step 4:

[1586] The server checks the integrity of the received data and converts it into a format that can be analyzed. The input is the data received in step 3, and the output is the preprocessed data. Specifically, it performs an error check, then extracts images or video frames and performs any necessary preprocessing.

[1587] Step 5:

[1588] The server loads a pre-trained generative AI model and begins analyzing the pre-processed data as input. The output is the analysis results, which are information used to evaluate the safety of the work. Specifically, the work content is analyzed based on procedures and drawings, and dangerous areas are identified.

[1589] Step 6:

[1590] The server identifies dangerous areas based on the analysis results. The input is the analysis results of the generative AI model, and the output is a list of identified dangerous areas. Specifically, it detects missing handrails when working at height and defects in the protective covers of machinery.

[1591] Step 7:

[1592] The device collects the user's facial expressions and voice using a built-in camera and microphone. The input is the user's facial and voice data, and the output is their emotional state analyzed by emotion recognition software. Specifically, the device measures tension and anxiety by recognizing the user's face and analyzing their voice tone.

[1593] Step 8:

[1594] The server generates feedback based on the identified dangerous locations and the user's emotional state. The input is a list of dangerous locations and the user's emotional state, and the output is the generated feedback data. Specifically, safety instructions and advice are created as text, diagrams, and audio guides.

[1595] Step 9:

[1596] The server sends the generated feedback data to the terminal. The input is the feedback data, and the output is the data transmission to the terminal.

[1597] Step 10:

[1598] The device presents the received feedback to the user. The input is the feedback data, and the output is the visual and audio feedback provided to the user. Specifically, it displays diagrams and text on the screen and plays audio guidance as needed.

[1599] Step 11:

[1600] The server stores the sent feedback and the user's emotional state in log format. The input is feedback and emotional data, and the output is log data. This allows data to be accumulated for later analysis and improvement.

[1601] (Application example 2)

[1602] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1603] In on-site work, it is necessary to ensure work safety while also taking into consideration the emotional state of workers. However, current systems only evaluate the safety of work and lack the functionality to recognize the emotional state of workers and provide appropriate feedback based on that. As a result, there is an issue in that they are unable to respond appropriately to workers who are feeling tense or anxious.

[1604] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1605] In this invention, the server includes means for taking images and videos of on-site work, means for transmitting the captured images and videos to the server, means for formatting the data so that it can be analyzed by the server, means for analyzing the images and videos in the server using artificial intelligence to identify hazardous areas, means for generating information about the identified hazardous areas in the form of diagrams, text, and audio, means for transmitting the generated feedback data to the terminal, means for presenting feedback to the user, means for analyzing the user's facial expressions and voice to recognize the user's emotional state, and means for generating feedback based on the user's emotional state. This makes it possible to provide flexible and appropriate feedback that takes into consideration not only the safety of the work but also the emotional state of the worker.

[1606] "On-site work" refers to work activities that take place at a designated location.

[1607] "Means for taking images or videos" refers to equipment or systems for collecting visual information using devices, including cameras.

[1608] "Means for sending to the server" refers to the communication protocols and devices used to transfer collected data to the server via a network.

[1609] "Means for formatting data in an analyzable form" refers to the process or software used to convert collected data into a form that can be efficiently analyzed by the server.

[1610] "Means for analyzing images and videos using generative artificial intelligence" refers to a system that uses a pre-trained AI model to evaluate collected visual information and extract the necessary information.

[1611] "Means for identifying dangerous areas" refers to algorithms or programs for recognizing risk areas in the workplace based on the analysis results.

[1612] "Means for generating feedback in the form of diagrams, text, or audio" refers to technologies that allow users to visualize and / or voice information.

[1613] The "means for transmitting generated feedback data to the terminal" refers to a communication means for quickly transferring the generated feedback information to the user's device.

[1614] The term "means for presenting feedback to the user" refers to a function for displaying feedback information so that the user can check it via the device.

[1615] "Means of analyzing the user's facial expressions and voice to recognize their emotional state" refers to an AI engine that determines the user's emotions based on data acquired by a camera and microphone.

[1616] The "means for generating feedback based on the user's emotional state" refers to a system for proposing countermeasures according to the recognized emotions and generating them as feedback information.

[1617] This invention is a system that streamlines safety patrols for on-site work and takes into account the user's emotional state. The system uses high-resolution cameras, microphones, servers, and terminals (smartphones, tablets, factory robots, etc.) as hardware. The software includes an emotion recognition engine, generative AI model, data formatter, communication protocol, and feedback generation algorithm.

[1618] System Operation Overview

[1619] Data collection

[1620] Users use the device's built-in high-resolution camera and microphone to collect images, video, and audio data of on-site work, providing detailed information about the work environment and the worker's situation.

[1621] Data transmission

[1622] The devices transmit the collected data to a server in real time using a communication protocol that compresses the data and converts it into a suitable format.

[1623] Data Format

[1624] The server formats the transmitted data in a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[1625] Analysis by generative artificial intelligence

[1626] The server uses a pre-trained generative AI model to analyze the collected images and videos. This allows it to compare the images and videos with work procedures and drawings, evaluate the safety of the work, and identify dangerous areas. An emotion recognition engine is also used to identify the user's emotional state from their facial expressions and voice.

[1627] Generate feedback

[1628] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state. For example, it combines information such as "It is dangerous to work at height because there are no handrails installed" with instructions such as "Please stay calm and check" if the worker is nervous.

[1629] Sending and Providing Feedback

[1630] The generated feedback data is sent from the server to the terminal. The terminal then presents the received feedback to the user visually (on screen) and audibly, allowing workers to immediately understand dangerous areas and take appropriate measures.

[1631] Feedback logging

[1632] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[1633] Specific examples

[1634] For example, when working at height, a worker (user) uses a smartphone to take pictures of the scaffolding and work environment. These images and videos are sent from the device to a server, where they are analyzed using a generative AI model. The analysis identifies a hazard due to the absence of handrails. At the same time, the device's emotion engine analyzes the worker's facial expressions to detect tension or anxiety. Based on the hazard information and the user's emotional state, the server generates specific, emotion-sensitive feedback, such as, "There is a risk of falling because handrails are not installed. Please remain calm and take appropriate measures." This feedback is sent to the worker's device in real time and displayed through visual highlights and audio guidance. The worker can immediately recognize the situation and take appropriate measures.

[1635] Specific examples of prompts for the generative AI model to use

[1636] "Analyze image data of work at height to identify workers' emotional states (tension, anxiety, etc.) and dangerous areas."

[1637] "Analyze the voice data, determine the worker's emotional state from the tone and content of their voice, and generate appropriate feedback."

[1638] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1639] Step 1:

[1640] Users use the device's built-in high-resolution camera and microphone to collect images, videos, and audio data of on-site work.

[1641] Input: User-activated camera and microphone, local video and audio

[1642] Output: High-resolution image data, video data, audio data

[1643] Step 2:

[1644] The devices transmit the collected data to a server in real time using a communication protocol that compresses the data and converts it into a suitable format.

[1645] Input: High-resolution image data, video data, audio data

[1646] Output: Compressed image data, video data, and audio data are sent to the server.

[1647] Step 3:

[1648] The server formats the transmitted data in a format suitable for analysis, specifically by extracting and preprocessing images and video frames.

[1649] Input: Compressed image data, video data, audio data

[1650] Output: Parsable formatted image frames and audio data

[1651] Step 4:

[1652] The server uses a pre-trained generative AI model to analyze the collected images and videos, compare them with work procedures and drawings, evaluate the safety of the work, and identify any hazards.

[1653] Input: Image frames and audio data formatted in a parseable format

[1654] Output: Information on identified dangerous areas (e.g., areas where handrails are not installed)

[1655] Step 5:

[1656] The server simultaneously operates an emotion recognition engine to identify the user's emotional state from their facial expressions and voice.

[1657] Input: Image frames and audio data formatted in a parseable format

[1658] Output: Information about the user's emotional state (e.g., tension, anxiety)

[1659] Step 6:

[1660] The server generates feedback in the form of diagrams, text, and audio based on the identified dangerous areas and the user's emotional state.

[1661] Input: Information on dangerous areas, information on the user's emotional state

[1662] Output: Feedback data (in diagram, text, audio format)

[1663] Step 7:

[1664] The generated feedback data is sent from the server to the terminal, which then presents the received feedback to the user visually (on screen) or audibly.

[1665] Input: Feedback data (in diagram, text, audio format)

[1666] Output: Feedback presented through the device's screen and speakers

[1667] Step 8:

[1668] The server stores each feedback and the user's emotional state in a detailed log format, providing data for future analysis and improvement planning.

[1669] Input: Feedback data, information on the user's emotional state

[1670] Output: Saved log data

[1671] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1672] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1673] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1674] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1675] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1676] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1677] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1678] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1679] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1680] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1681] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1682] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1683] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1684] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1685] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1686] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1687] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1688] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1689] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1690] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1691] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1692] The following is further disclosed regarding the above embodiment.

[1693] (Claim 1)

[1694] A means of taking images and videos of on-site work;

[1695] A means for sending captured images and videos to a server;

[1696] a means of formatting the data so that it can be parsed by the server;

[1697] A method for analyzing images and videos using artificial intelligence generated on the server and identifying dangerous areas;

[1698] A means for generating information on identified hazardous locations in diagram, text, and audio formats;

[1699] means for transmitting the generated feedback data to the terminal;

[1700] means for presenting feedback to the user;

[1701] A system including:

[1702] (Claim 2)

[1703] The system according to claim 1, characterized in that the generative artificial intelligence model learns procedure manuals and drawings in advance and uses them to analyze on-site work.

[1704] (Claim 3)

[1705] The system according to claim 1, further comprising means for saving the feedback data in log format and utilizing it for later analysis and improvement measures.

[1706] "Example 1"

[1707] (Claim 1)

[1708] A means of taking images and videos of on-site work;

[1709] A means for transmitting the captured images and videos via a network;

[1710] A means for converting received image and video data into an analyzable format;

[1711] A means to analyze the content of images and videos using generative AI models and identify dangerous areas,

[1712] A means for generating information on identified hazardous locations in diagram, text, and audio formats;

[1713] means for transmitting the generated feedback data to the terminal again;

[1714] means for presenting feedback to the user;

[1715] A system including:

[1716] (Claim 2)

[1717] The system described in claim 1, characterized in that the generative AI model learns work procedures and safety standards in advance and uses them to analyze on-site work.

[1718] (Claim 3)

[1719] The system according to claim 1, further comprising means for saving the feedback data in log format and utilizing it for later analysis and improvement measures.

[1720] "Application Example 1"

[1721] (Claim 1)

[1722] A means of taking images and videos of on-site work;

[1723] A means for sending captured images and videos to a server;

[1724] a means of formatting the data so that it can be parsed by the server;

[1725] A method for analyzing images and videos using artificial intelligence generated on the server and identifying dangerous areas;

[1726] A means for generating information on identified hazardous locations in diagram, text, and audio formats;

[1727] means for transmitting the generated feedback data to the terminal;

[1728] means for presenting feedback to the user;

[1729] a means for presenting the feedback data by voice and notifying the user of dangerous locations and countermeasures;

[1730] A system including:

[1731] (Claim 2)

[1732] The system according to claim 1, characterized in that the generative artificial intelligence model learns procedure manuals and drawings in advance and uses them to analyze on-site work.

[1733] (Claim 3)

[1734] The system according to claim 1, further comprising means for saving the feedback data in log format and utilizing it for later analysis and improvement measures.

[1735] "Example 2: Combining Emotion Engines"

[1736] (Claim 1)

[1737] a means for obtaining footage of on-site work;

[1738] means for transmitting the acquired video to a server;

[1739] a means of formatting the data so that it can be parsed by the server;

[1740] A means for analyzing the video using artificial intelligence generated on the server and identifying dangerous areas;

[1741] means for recognizing a user's emotion;

[1742] a means for generating feedback based on information about the identified dangerous spot and the user's emotional state;

[1743] means for transmitting the generated feedback data to the terminal;

[1744] means for presenting feedback to the user;

[1745] A system including:

[1746] (Claim 2)

[1747] The system according to claim 1, characterized in that the generative artificial intelligence model learns standard operating procedures and design drawings in advance and uses them to analyze on-site work.

[1748] (Claim 3)

[1749] The system according to claim 1, further comprising means for saving the feedback data in log format and utilizing it for later analysis and improvement measures.

[1750] "Application example 2 when combining emotion engines"

[1751] (Claim 1)

[1752] A means of taking images and videos of on-site work;

[1753] A means for sending captured images and videos to a server;

[1754] a means of formatting the data so that it can be parsed by the server;

[1755] A method for analyzing images and videos using artificial intelligence generated on the server and identifying dangerous areas;

[1756] A means for generating information on identified hazardous locations in diagram, text, and audio formats;

[1757] means for transmitting the generated feedback data to the terminal;

[1758] means for presenting feedback to the user;

[1759] A means for analyzing a user's facial expression and voice to recognize the user's emotional state;

[1760] means for generating feedback based on the user's emotional state;

[1761] A system including:

[1762] (Claim 2)

[1763] The system according to claim 1, characterized in that the generative artificial intelligence model learns procedure manuals and drawings in advance and uses them to analyze on-site work.

[1764] (Claim 3)

[1765] The system according to claim 1, further comprising means for saving the feedback data in log format and utilizing it for later analysis and improvement measures. [Explanation of symbols]

[1766] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of taking images and videos of on-site work; A means for sending captured images and videos to a server; a means of formatting the data so that it can be parsed by the server; A method for analyzing images and videos using artificial intelligence generated on the server and identifying dangerous areas; a means for generating information on the identified dangerous locations in diagram, text, and audio formats; means for transmitting the generated feedback data to the terminal; means for presenting feedback to the user; A system including:

2. 2. The system according to claim 1, wherein the generative artificial intelligence model learns in advance from procedure manuals and drawings and uses them to analyze on-site work.

3. 2. The system according to claim 1, further comprising means for saving the feedback data in a log format and utilizing the saved data for later analysis and improvement measures.

Images