system

Abstract

We provide the system. [Solution] Photography methods targeting infrastructure, An artificial intelligence image analysis method that analyzes captured images to identify defects, A deterioration evaluation means for evaluating the deterioration state of a structure based on defect information identified by the artificial intelligence image analysis means, A priority determination means for calculating the priority of repairs based on the deterioration evaluation means, A report generation and notification system that generates an evaluation report and notifies the administrator, A means for automatically creating a schedule for periodic inspections, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] While the aging of infrastructure progresses, efficient and accurate inspections are required. However, the conventional manual inspection method has problems of requiring time and manpower and having insufficient accuracy. Also, the management of inspection results and the planning of the next inspection rely on manual work, which hinders the efficiency of operations. To solve these problems, an automated high-precision inspection system is required.

Means for Solving the Problems

[0005] This invention provides a technology that precisely identifies defects from captured images by utilizing an automated imaging means for infrastructure and an image analysis means equipped with artificial intelligence. Furthermore, it aims to improve the efficiency of inspection work by providing a priority determination means that evaluates the deterioration state based on the analysis results and automatically determines the priority of repairs. In addition, it aims to solve conventional problems by providing an inspection scheduling means that automatically generates an evaluation report from the inspection results and notifies the administrator, and also automatically creates a schedule for periodic inspections.

[0006] "Infrastructure" refers to facilities and equipment that make up the social infrastructure, and mainly includes roads, tunnels, bridges, etc.

[0007] "Means of capturing images" refers to devices such as cameras and sensors used to acquire images of infrastructure.

[0008] "Artificial intelligence image analysis means" refers to a process or system that has the technology to automatically identify defects from captured images using an image analysis algorithm.

[0009] A "defect" refers to a section of damage or deterioration that could potentially affect the structure or function of the infrastructure.

[0010] "Deterioration assessment means" refers to a method or device for evaluating the overall condition and degree of deterioration of infrastructure based on identified defect information.

[0011] "Priority determination means" refers to the criteria and methods used to determine the urgency and importance of a defective part that requires repair.

[0012] "Report generation and notification means" refers to systems and devices that create visualized reports for administrators based on analysis results and deterioration assessments, and communicate that information.

[0013] "Inspection scheduling means" refers to a system or device that automatically determines the date and time of the next inspection and efficiently plans the inspection. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a processor with a reference number (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0018] In the following embodiments, a RAM (Random Access Memory) with a reference number is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a storage with a reference number is one or more non-volatile storage devices that store various programs and various parameters, etc.. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides a system that integrates several key elements to enable efficient and accurate inspection of infrastructure. This system functions through the collaboration of three parties: a terminal, a server, and a user.

[0036] First, the terminal acquires images of the infrastructure at the site. The terminal is equipped with a camera, and metadata such as location information and time of capture is added to the captured images. This allows for precise identification of where and when the images were taken. These image data are transmitted to the server via secure communication.

[0037] Next, the server processes the received images. First, an image analysis module feeds the image data into an AI model to identify defects on the infrastructure surface. This AI model is trained on a vast amount of training data and can detect cracks and damage quickly and with high accuracy. As a result, the locations where defects exist are marked at the pixel level, and detailed information such as the size and shape of the defects is extracted.

[0038] Subsequently, the server evaluates the state of deterioration based on these analysis results. Past inspection data and environmental conditions (e.g., weather information) are also taken into consideration to assess the overall health of the infrastructure. This evaluation determines the priority of areas that require repair.

[0039] In the report generation step, the server generates a detailed degradation assessment report. The report includes detected defects, their priority, and recommended repair actions, providing administrators with information to gain an overall understanding of the structural health. Users receive this report from the server and use it to consider countermeasures and plan future maintenance.

[0040] Furthermore, the server automatically sets the next inspection schedule. This allows for continuous monitoring of the infrastructure's status and enables safe and efficient management.

[0041] Specific example

[0042] For example, in the case of bridge inspections, the terminal takes pictures of each section of the bridge, and the acquired images are sent to the server. The server analyzes the images and identifies the location and severity of any cracks. This information is compiled into a deterioration assessment report, and if it is determined that repairs are of high priority, prompt action is required. Upon receiving the report, the user works quickly in cooperation with the repair team to carry out the work. In addition, the next inspection schedule set by the server is automatically added to the user's digital calendar, promoting planned inspection activities.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The device captures images of the infrastructure. By adding metadata such as location information and the date and time of capture, the source of the image is clearly identified.

[0046] Step 2:

[0047] The device sends the captured image to the server using a secure communication protocol. Here, the data is compressed during transmission, improving transmission speed and reducing network load.

[0048] Step 3:

[0049] The server passes the received images to the AI ​​image analysis module. First, image preprocessing is performed, such as adjusting the resolution and removing noise. This improves the accuracy of the analysis.

[0050] Step 4:

[0051] The AI ​​image analysis module analyzes pre-processed images to identify defects appearing on the infrastructure surface. Specifically, it detects areas of cracks and damage and records their location, size, and shape in detail.

[0052] Step 5:

[0053] The server evaluates the deterioration status of the structure based on the analysis results. This evaluation also takes into account past inspection data and environmental factors. The degree of deterioration is scored, and an evaluation report is created that lists the scores.

[0054] Step 6:

[0055] The server determines repair priorities based on degradation assessments. Priorities are tiered from those requiring immediate attention to those that only require routine maintenance, which helps administrators make informed decisions.

[0056] Step 7:

[0057] The server generates an evaluation report and notifies the user in PDF or digital dashboard format. The report includes information such as the structural integrity, repair recommendations, and priority lists.

[0058] Step 8:

[0059] The server automatically sets the next maintenance schedule and notifies the user. This schedule is optimized based on the infrastructure status and synchronized with the user's digital calendar.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] Infrastructure inspection work is inefficient due to its reliance on manual processes, leading to high risks of overlooking defects and delayed assessments. Furthermore, the manual management of inspection data makes accurate assessment of deterioration and the development of appropriate repair plans difficult. To address these challenges and improve infrastructure safety, an efficient and accurate automated inspection system is required.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes an artificial intelligence analysis means for analyzing images to identify infrastructure defects at the pixel level, a deterioration evaluation means for comprehensively evaluating the deterioration state of the infrastructure based on the identified defect information, and a report generation and notification means for generating a deterioration evaluation report and providing it to the relevant parties. This automates infrastructure inspection work, enabling efficient and accurate detection and evaluation of defects.

[0065] "Image acquisition means" refers to a device that has the function of taking images of infrastructure and acquiring related location information and time information of the images taken.

[0066] "Data transmission means" refers to a device or system that has the function of transmitting acquired image data to a server via a secure communication protocol.

[0067] An "artificial intelligence analysis means" is a system that has the function of analyzing images using an AI model to identify infrastructure defects contained in the acquired image at the pixel level.

[0068] A "deterioration evaluation method" is a system that has the function of evaluating the deterioration state of the entire infrastructure based on defect information identified by artificial intelligence analysis methods.

[0069] A "priority calculation means" is a system that has the function of determining the priority of repairs based on a deterioration evaluation means.

[0070] A "report generation and notification means" is a device or system that has the function of generating a report based on the results of a deterioration assessment and providing it to the relevant parties.

[0071] A "scheduling method" is a system that has the function of automatically planning and setting the date for the next inspection.

[0072] As an embodiment of the present invention, an information processing system for efficiently inspecting infrastructure is provided. This system consists of three main components: a terminal, a server, and a user.

[0073] The device consists of hardware equipped with a high-resolution camera and takes images in the infrastructure field. Location and time information are automatically added to the captured images. The device transmits this image data to the server via a secure protocol (e.g., HTTPS).

[0074] The server is a high-performance computer system that analyzes received image data. The server uses a generative AI model to perform image analysis to identify defects within the images. This AI model is trained on a large amount of training data and can quickly and accurately identify cracks and damage at the pixel level. Based on the analyzed data, the server further considers past inspection records and external environmental data to assess the deterioration status of the infrastructure. Based on the assessment results, it calculates repair priorities and generates a detailed deterioration assessment report. The server has the function to notify the user of this report and the next inspection schedule.

[0075] Users receive these notifications and plan repairs and maintenance based on the information provided. Users refer to reports and instruct repair work if necessary. Users also conduct planned inspections according to the automatically set next inspection schedule. This process makes infrastructure inspections more efficient and effective than before.

[0076] As a concrete example, in bridge inspections, the terminal acquires images of the bridge from various angles, and these images are sent to the server. The server analyzes the images and identifies defects such as cracks. A report based on this information is promptly notified to the user, allowing for necessary actions to be taken quickly. An example of a prompt message would be, "Input the section images of this bridge into the AI ​​model to identify the location and severity of defects and generate a report."

[0077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0078] Step 1:

[0079] The terminal acquires images of the infrastructure on-site. The input consists of visual data captured by the terminal's camera, as well as GPS location and timestamp information. Based on this information, metadata is added to the images, preparing image data with precisely identified location and timestamps.

[0080] Step 2:

[0081] The device transmits the acquired image data to the server via a secure communication protocol. The input consists of the captured image and its associated metadata. The device compresses the data, optimizes communication, and minimizes latency before sending it to the server. After transmission, a transmission log is stored on the device.

[0082] Step 3:

[0083] The server takes in image data received from the terminal for analysis. The input consists of a set of images and their metadata sent from the terminal. The server uses a generative AI model to detect surface defects in the infrastructure through an image analysis module. The AI ​​model performs pixel-level analysis on the image data, identifies the defective areas, and outputs analysis data with those areas marked.

[0084] Step 4:

[0085] The server evaluates the deterioration status of the infrastructure based on the analysis results. The input data consists of analyzed image data, past inspection records, and environmental data. The server integrates these and performs calculations to evaluate the health of the infrastructure, generating a list of prioritized areas requiring repair as an evaluation result.

[0086] Step 5:

[0087] The server generates a detailed report based on the degradation assessment results. The inputs used are the degradation assessment results and priority information. The server organizes the information, creates a report in a format easily understood by stakeholders, and electronically notifies the user. As a result, the user receives a detailed report as output, which allows them to consider future countermeasures.

[0088] Step 6:

[0089] Users receive reports sent from the server and use them to develop repair plans. They refer to the report information as input and collaborate with the repair team to proceed with specific actions. At the same time, users can check the next inspection schedule set by the server and obtain output to plan ongoing infrastructure inspections.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] Early detection and repair of deterioration and damage to equipment and machinery in manufacturing is crucial for improving quality and maintaining productivity. However, conventional inspection methods rely on human verification, which limits the frequency and accuracy of inspections, making early problem detection difficult. In particular, large factories need to efficiently monitor multiple machines and pieces of equipment, but managing them is extremely labor-intensive and inefficient.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] In this invention, the server includes a means for capturing images, an artificial intelligence image analysis means, a means for evaluating deterioration, a means for determining priority, a means for generating and notifying reports, and a means for scheduling inspections. This makes it possible to quickly and accurately identify wear and damage to multiple pieces of equipment and machinery, determine the priority of necessary repairs, and automatically create a periodic inspection schedule.

[0095] "Equipment and machinery" refers to the collective term for production equipment and manufacturing machinery used in factories and manufacturing industries, and these are the items whose deterioration and damage are monitored.

[0096] "Photography means" refers to devices used to acquire images of equipment or machinery, mainly cameras and sensors, and also has the function of adding metadata such as location information and time of shooting.

[0097] "Artificial intelligence image analysis means" refers to an analysis function that uses AI technology to analyze acquired images and identify wear and damage to equipment and machinery contained therein.

[0098] A "deterioration evaluation method" is a function that evaluates the deterioration state of equipment and machinery based on wear and damage information identified by artificial intelligence image analysis methods.

[0099] The "priority determination means" is a function for quantitatively calculating the priority of areas requiring repair based on the deterioration evaluation means.

[0100] The "report generation and notification means" is a function for creating a report based on the results of the deterioration assessment and notifying administrators and responsible persons of it.

[0101] The "inspection scheduling means" is a function that automatically generates a schedule for periodic inspections based on the condition of the equipment and machinery.

[0102] This system aims to quickly and accurately identify wear and tear and damage to equipment and machinery within a factory, and to support appropriate repair planning. Its embodiments are described in detail below.

[0103] First, the terminal is equipped with a high-resolution camera, which is used to periodically acquire image data of equipment and machinery. The camera has the function of adding location information and time of capture as metadata to the image. This terminal will be implemented in devices suitable for the factory environment, such as robots and smart glasses. Next, this image data is securely transmitted to a server using tamper-evident communication technology.

[0104] The server analyzes the collected image data using a highly efficient data processing system. This analysis utilizes AI image analysis models and libraries such as TENSORFLOW® and PyTorch. These AI models identify defects such as wear and damage within the images. Based on the analysis results, the degree of deterioration is evaluated, and repair priorities are determined. This process is carried out using algorithms that take into account historical data and weather conditions.

[0105] Subsequently, a deterioration assessment report is automatically generated. The report includes identified problem areas, priorities, and specific measures to be taken if repairs are necessary, and is notified to managers and relevant personnel.

[0106] The server also generates an inspection schedule and automatically sets the next inspection date. This schedule is directly synchronized with the administrator's digital calendar.

[0107] As a concrete example, consider a scenario where a conveyor belt in a factory is regularly photographed, and wear is detected through AI analysis. This information is compiled into a report, and a notification is sent to the person in charge promptly urging them to replace the parts.

[0108] An example of a prompt message for the generated AI model is: "Based on the following image, identify wear and damage to the machinery and equipment to be inspected. Output detailed information and priority, and if repairs are needed, indicate the recommended actions."

[0109] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0110] Step 1:

[0111] The terminal captures high-resolution images of equipment and machinery within the factory. During this process, location and timestamp information are added to the image data as metadata. The input is physical factory equipment, and the output is image data with location and timestamp information added. Specifically, the camera captures the deterioration status of conveyor belts and press machines, and then adds GPS information and timestamps.

[0112] Step 2:

[0113] The device transmits captured image data to the server based on a secure communication protocol. This transmission involves encryption to maintain data integrity. The input is the image data collected by the device, and the output is the encrypted image data received by the server. Specifically, the SSL / TLS protocol is used to transmit the image data and prevent unauthorized access.

[0114] Step 3:

[0115] The server analyzes the received image data using an AI model. The AI ​​model includes a generative AI model that identifies defects in the input image. The input is encrypted image data received by the server, which is decrypted and then analyzed. The output is labeled data indicating areas of wear and damage. Specifically, an AI trained with the TensorFlow library identifies worn areas of the belt in the image on a pixel-by-pixel basis.

[0116] Step 4:

[0117] The server evaluates the equipment's deterioration status and determines repair priorities based on defect information identified by AI. The input is labeled analysis data, and the output is repair items categorized by priority. Specifically, certain thresholds are set, and priority ranks are assigned according to the degree of deterioration.

[0118] Step 5:

[0119] The server generates a detailed report based on the evaluation results and notifies administrators and users. This report includes information on areas requiring repair and recommended actions based on priority. The input is repair item data with assigned priorities, and the output is a structured report. Specifically, the report is generated in PDF format and delivered to users via email or a dedicated application.

[0120] Step 6:

[0121] The server calculates the next inspection schedule and automatically registers it in the user's digital calendar. Inputs are equipment degradation assessment data and existing maintenance schedules, and output is an updated calendar event. Specifically, it uses the Google® Calendar API to automatically add appointments and facilitate the next inspection.

[0122] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0123] This invention provides a system that streamlines inspection work while taking into account the user's emotional state by combining an emotion engine with an infrastructure inspection system. The system mainly consists of three aspects: physical infrastructure inspection, AI-based image analysis and deterioration assessment, and recognition of the user's emotional state.

[0124] First, the device captures images of the infrastructure on-site and adds location information and the date and time of capture as metadata. This data is compressed and sent to the server via a secure communication protocol.

[0125] Next, the server processes the received images using an AI image analysis module to identify defects. Based on the analysis data, it evaluates the degree of deterioration and determines the repair priority. The detailed evaluation report generated in this process is notified to the administrator, and the next inspection schedule is automatically set.

[0126] Furthermore, this invention incorporates an emotion engine that evaluates the user's emotional state in real time. Emotional data is acquired through built-in sensors (e.g., camera and microphone) of the terminal used by the user during inspections, and the user's stress and fatigue levels are analyzed. This information is used to adjust the priority of inspection tasks and is linked with report generation and notification means to provide appropriate feedback.

[0127] As a concrete example, when inspecting a bridge, the user takes images using a tablet equipped with emotion recognition capabilities. If the emotion engine detects the user's stress level from their facial expressions and voice, the server automatically adjusts the next task and schedule to prevent the work from becoming excessively burdensome. This allows the user to perform inspection tasks efficiently while reducing stress.

[0128] In this way, the present invention provides an advanced infrastructure inspection system that combines high-precision inspection using AI with flexible work management that takes into account the user's emotional state.

[0129] The following describes the processing flow.

[0130] Step 1:

[0131] The device captures images of the infrastructure on-site. The device has GPS and timestamp capabilities to add location information and the date and time of capture, and this metadata is embedded in the image data.

[0132] Step 2:

[0133] The device compresses the image data it acquires and sends it to the server using a secure communication protocol. Encryption is applied during data transmission to maintain communication security.

[0134] Step 3:

[0135] The server passes the received images to the AI ​​image analysis module for preprocessing. This preprocessing includes noise reduction and image resolution adjustment, which improves the accuracy of the image analysis.

[0136] Step 4:

[0137] An AI image analysis module processes images to identify defects such as cracks and damage on the infrastructure. Information on detected defects, including location, size, and shape, is recorded.

[0138] Step 5:

[0139] The server evaluates the deterioration status of structures based on defect information. The evaluation takes into account past inspection data and local environmental information, and the deterioration status is scored.

[0140] Step 6:

[0141] The server uses the degradation assessment results to determine repair priorities. These priorities are ranked according to the need for repair, and sections requiring immediate attention are identified.

[0142] Step 7:

[0143] The server generates an evaluation report and notifies the administrator. The report includes the deterioration assessment results, repair priorities, and recommended plans for the next inspection. Notifications are sent via email or a dedicated dashboard.

[0144] Step 8:

[0145] The emotion engine evaluates the user's emotional state in real time. It uses the device's camera and microphone to analyze the user's facial expressions and voice tone, and calculates stress and fatigue levels.

[0146] Step 9:

[0147] The server dynamically adjusts the assignment of inspection tasks based on data from the emotion engine. If a user's stress level is high, it automatically reduces the workload or distributes tasks to other workers.

[0148] Step 10:

[0149] The server creates the next inspection schedule and notifies the user. The inspection date, time, and frequency are adjusted considering the user's emotional state, optimizing the user's workload.

[0150] (Example 2)

[0151] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0152] While conventional infrastructure inspection systems could assess the physical deterioration of structures, they did not adequately consider the psychological burden on inspectors or the efficiency of their work. In particular, increased stress and fatigue among workers can negatively impact the quality of work and their health. Therefore, there is a need to conduct highly accurate infrastructure inspections while simultaneously reducing the psychological burden on workers and improving work efficiency.

[0153] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0154] In this invention, the server includes artificial intelligence image analysis means, degradation evaluation means, priority determination means, emotion evaluation means, and feedback provision means. This enables precise defect detection and degradation evaluation of infrastructure, as well as real-time adjustment of work schedules and feedback based on the emotional state of workers. This reduces the psychological burden on workers and improves work efficiency.

[0155] "Infrastructure" refers to social infrastructure facilities such as bridges, roads, and tunnels, and inspections are necessary to maintain the safety and functionality of these structures.

[0156] "Shooting equipment" refers to a device that has the function of acquiring image data at infrastructure sites and adds metadata including location information and the date and time of shooting.

[0157] "Artificial intelligence image analysis means" refers to artificial intelligence technology used to identify and analyze defects in infrastructure using captured image data.

[0158] "Deterioration evaluation means" refers to a technology that determines and evaluates the degree of deterioration of a structure based on defect information identified by artificial intelligence image analysis means.

[0159] A "priority determination method" refers to a technology that calculates the necessity and urgency of infrastructure repairs based on the results of deterioration assessment methods, and then determines their priority.

[0160] "Report generation and notification means" refers to technology that, in conjunction with priority determination means, aggregates deterioration assessment results and priority information and provides them to administrators as reports.

[0161] "Inspection scheduling means" refers to technology that automatically creates schedules to efficiently plan and manage periodic inspections of infrastructure.

[0162] "Emotional assessment tools" refer to technologies used to monitor workers' emotional states in real time and to assess their stress and fatigue levels.

[0163] "Feedback provision means" refers to technology that has the function of adjusting workload and suggesting improvements based on the worker's state obtained through emotional evaluation means.

[0164] This invention is a system for efficiently performing infrastructure inspection work and aims to reduce the psychological burden on workers. The system consists of the following three main components.

[0165] First, the terminal is used in the field and has the function of capturing images of infrastructure. Specifically, it uses a mobile device equipped with a camera and GPS function to add location information and the date and time of capture as metadata to the image. This data is then efficiently transmitted to the server using data compression technology after capture.

[0166] Next, the server plays a central role in processing the received image data. The server includes image analysis modules utilizing AI technologies such as TensorFlow to perform detailed image analysis. This analysis identifies minor defects and uses that information to perform a degradation assessment. The assessment results are generated as a report in JSON format and notified to the administrator. In addition, a schedule for the next inspection is automatically set.

[0167] Furthermore, the system incorporates an emotion assessment mechanism that monitors the user's emotional state in real time. The user's device is equipped with a camera and microphone, and these sensors collect the user's facial expressions and voice information. This information is then used with analysis libraries such as Librosa to assess stress and fatigue levels. The assessment results are passed to a feedback system, which provides appropriate work instructions and break suggestions based on the stress level.

[0168] As a concrete example, if a user takes an image using a tablet device while inspecting a bridge, and the emotion engine detects high stress from the user's facial expression, the server will automatically revise the schedule and suggest ways to reduce the workload. In this case, an example of a prompt message would be, "If the user is in a high-stress state and a high-temperature environment while inspecting the bridge, please indicate how the work schedule should be changed."

[0169] Thus, the system of the present invention combines advanced image analysis technology and emotion analysis technology, making it possible to perform infrastructure inspection work efficiently and safely.

[0170] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0171] Step 1:

[0172] The device captures images of infrastructure on-site and acquires image data. The input is video from the device's camera sensor, and the output is an image file in JPEG format. During this process, location information is obtained using the built-in GPS, and the date and time of capture are recorded using the system clock, and these are added to the image metadata.

[0173] Step 2:

[0174] The device compresses the acquired image data using a compression algorithm to reduce its size and sends it to the server using the HTTPS protocol to ensure security. Input consists of JPEG images and metadata, while output is a compressed data packet. A checksum is generated before transmission to guarantee data integrity.

[0175] Step 3:

[0176] The server passes the received image data to an AI image analysis module. The input is a compressed JPEG image, and the output is analyzed defect information. The AI ​​model, using TensorFlow, identifies problematic defects in the image at the pixel level and highlights areas that require particular attention.

[0177] Step 4:

[0178] The server performs a deterioration assessment using the analysis results from the AI ​​image analysis module. The input is defect information, and the output is a deterioration assessment report. For example, it quantifies the width of cracks and the spread of rust to measure the degree of deterioration. Based on these results, it calculates a repair priority score.

[0179] Step 5:

[0180] The server evaluates the user's emotional state in real time using emotion assessment tools. Input is sensor information from the terminal's camera and microphone, and output is stress and fatigue scores. Voice and facial expression data are analyzed using Librosa and OpenCV to determine the user's psychological state.

[0181] Step 6:

[0182] The server automatically adjusts the schedule for the next task based on the analysis results and emotional assessment. The input is the deterioration assessment score and emotional state score, and the output is the updated task schedule. It suggests reducing the workload or adding breaks as needed.

[0183] Step 7:

[0184] The server ultimately provides all results to the user as feedback. Inputs are updated schedules and sentiment feedback information, while outputs are specific instructions and suggestions displayed on the user's tablet. This ensures that the work is not overly burdensome and allows for efficient work execution.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] Infrastructure inspections require considerable effort and time, and efficiency can decrease depending on the psychological state of the workers. Furthermore, accurately assessing deterioration and defects can be difficult, potentially leading to errors in prioritizing repairs. In this context, there is a need to accurately understand the condition of the infrastructure while reducing the psychological burden on workers and improving work efficiency.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes an imaging device for infrastructure, a machine learning image analysis device that processes the captured images to identify defects, and an adjustment device that uses an emotion recognition engine to analyze the operator's psychological state in real time and adjusts the work based on this information. This enables precise evaluation of the infrastructure's condition, reduces the operator's psychological burden, and improves work efficiency.

[0190] A "photography device" is a device that acquires images of infrastructure and generates visual data necessary for detailed defect identification and deterioration assessment.

[0191] A "machine learning image analysis device" is a device that uses AI technology to automatically identify defects based on captured images.

[0192] A "condition evaluation device" is a device that evaluates the deterioration state of a structure based on defect information obtained by a machine learning image analysis device.

[0193] A "priority determination device" is a device that determines the priority of repairs based on the evaluated deterioration status.

[0194] A "report generation and notification device" is a device that automatically generates reports based on evaluation results and priorities, and notifies relevant parties.

[0195] A "inspection planning device" is a device that automatically creates a schedule for periodic inspections.

[0196] An "emotion recognition engine" is a technology that analyzes the psychological state of workers in real time and adjusts their work content and schedule as needed.

[0197] The system for realizing this application primarily includes hardware such as an imaging device, image analysis device, condition evaluation device, priority determination device, report generation and notification device, inspection planning device, and emotion recognition engine. These devices work together to efficiently perform infrastructure inspections and adjust work according to the psychological state of the workers.

[0198] The server receives image data acquired from the imaging device using edge AI devices such as NVIDIA Jetson and Intel Movidius. Image analysis is performed using machine learning models with TensorFlow or PyTorch to identify defects in the structure. Subsequently, a condition evaluation device analyzes the defect information and evaluates the deterioration state of the structure. Next, a priority determination device calculates the repair priority based on this evaluation result.

[0199] Furthermore, the report generation and notification device generates a report summarizing the results and notifies the administrator. The inspection planning device automatically sets the schedule for the next periodic inspection. In addition, the emotion recognition engine evaluates the worker's psychological state in real time and adjusts the work content and schedule as needed.

[0200] As a concrete example, if a robot in a factory takes images of equipment and detects an anomaly, that information is immediately reported to the manager. At the same time, an emotion recognition engine measures the fatigue level of workers and suggests breaks to alleviate excessive strain. This entire process comprehensively supports improved work efficiency and the maintenance of workers' physical and mental health.

[0201] Examples of prompts to input into a generative AI model:

[0202] "Use AI to analyze images taken by factory equipment inspection robots to detect deterioration and abnormalities. Also, evaluate employee stress levels based on emotional data and provide appropriate feedback."

[0203] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0204] Step 1:

[0205] The terminal acquires images using a camera at the infrastructure site. At this time, location information and time of capture are added to the image as metadata, and the data is compressed. The input consists of infrastructure image data, location information, and time data, while the output is compressed image data.

[0206] Step 2:

[0207] The terminal transmits compressed image data to the server using a secure communication protocol. The input is compressed image data, and the output is the accurate transmission of data to the server.

[0208] Step 3:

[0209] The server processes the received image data using a machine learning image analysis system. It utilizes frameworks such as TensorFlow and PyTorch to perform image analysis to identify defects. The input is the received image data, and the output is the identified defect information.

[0210] Step 4:

[0211] The server evaluates the degradation state using a condition evaluation device based on defect information obtained from image analysis. The input is defect information, and the output is the evaluated degradation state data.

[0212] Step 5:

[0213] The server calculates repair priorities using a priority determination device based on condition evaluation data. The input is deterioration status data, and the output is the repair priority.

[0214] Step 6:

[0215] The server generates a report using a report generation and notification device based on the evaluation results and priorities, and notifies the administrator. The input is the evaluation results and priority data, and the output is a notification of the generated report to the administrator.

[0216] Step 7:

[0217] The server automatically generates the next scheduled inspection schedule using the inspection planning device. The input is the evaluation results and priority data, and the output is the automatically generated schedule.

[0218] Step 8:

[0219] The server uses an emotion recognition engine to monitor the user's psychological state in real time and evaluate stress and fatigue. The input is the user's emotional data, and the output is the evaluated stress level.

[0220] Step 9:

[0221] The server provides feedback that adjusts the work schedule and content based on the user's psychological state. The input is the assessed stress level, and the output is the adjusted work feedback.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] This invention provides a system that integrates several key elements to enable efficient and accurate inspection of infrastructure. This system functions through the collaboration of three parties: a terminal, a server, and a user.

[0239] First, the terminal acquires images of the infrastructure at the site. The terminal is equipped with a camera, and metadata such as location information and time of capture is added to the captured images. This allows for precise identification of where and when the images were taken. These image data are transmitted to the server via secure communication.

[0240] Next, the server processes the received images. First, an image analysis module feeds the image data into an AI model to identify defects on the infrastructure surface. This AI model is trained on a vast amount of training data and can detect cracks and damage quickly and with high accuracy. As a result, the locations where defects exist are marked at the pixel level, and detailed information such as the size and shape of the defects is extracted.

[0241] Subsequently, the server evaluates the state of deterioration based on these analysis results. Past inspection data and environmental conditions (e.g., weather information) are also taken into consideration to assess the overall health of the infrastructure. This evaluation determines the priority of areas that require repair.

[0242] In the report generation step, the server generates a detailed degradation assessment report. The report includes detected defects, their priority, and recommended repair actions, providing administrators with information to gain an overall understanding of the structural health. Users receive this report from the server and use it to consider countermeasures and plan future maintenance.

[0243] Furthermore, the server automatically sets the next inspection schedule. This allows for continuous monitoring of the infrastructure's status and enables safe and efficient management.

[0244] Specific example

[0245] For example, in the case of bridge inspections, the terminal takes pictures of each section of the bridge, and the acquired images are sent to the server. The server analyzes the images and identifies the location and severity of any cracks. This information is compiled into a deterioration assessment report, and if it is determined that repairs are of high priority, prompt action is required. Upon receiving the report, the user works quickly in cooperation with the repair team to carry out the work. In addition, the next inspection schedule set by the server is automatically added to the user's digital calendar, promoting planned inspection activities.

[0246] The following describes the processing flow.

[0247] Step 1:

[0248] The device captures images of the infrastructure. By adding metadata such as location information and the date and time of capture, the source of the image is clearly identified.

[0249] Step 2:

[0250] The device sends the captured image to the server using a secure communication protocol. Here, the data is compressed during transmission, improving transmission speed and reducing network load.

[0251] Step 3:

[0252] The server passes the received images to the AI ​​image analysis module. First, image preprocessing is performed, such as adjusting the resolution and removing noise. This improves the accuracy of the analysis.

[0253] Step 4:

[0254] The AI ​​image analysis module analyzes pre-processed images to identify defects appearing on the infrastructure surface. Specifically, it detects areas of cracks and damage and records their location, size, and shape in detail.

[0255] Step 5:

[0256] The server evaluates the deterioration status of the structure based on the analysis results. This evaluation also takes into account past inspection data and environmental factors. The degree of deterioration is scored, and an evaluation report is created that lists the scores.

[0257] Step 6:

[0258] The server determines repair priorities based on degradation assessments. Priorities are tiered from those requiring immediate attention to those that only require routine maintenance, which helps administrators make informed decisions.

[0259] Step 7:

[0260] The server generates an evaluation report and notifies the user in PDF or digital dashboard format. The report includes information such as the structural integrity, repair recommendations, and priority lists.

[0261] Step 8:

[0262] The server automatically sets the next maintenance schedule and notifies the user. This schedule is optimized based on the infrastructure status and synchronized with the user's digital calendar.

[0263] (Example 1)

[0264] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0265] Infrastructure inspection work is inefficient due to its reliance on manual processes, leading to high risks of overlooking defects and delayed assessments. Furthermore, the manual management of inspection data makes accurate assessment of deterioration and the development of appropriate repair plans difficult. To address these challenges and improve infrastructure safety, an efficient and accurate automated inspection system is required.

[0266] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0267] In this invention, the server includes an artificial intelligence analysis means for analyzing images to identify infrastructure defects at the pixel level, a deterioration evaluation means for comprehensively evaluating the deterioration state of the infrastructure based on the identified defect information, and a report generation and notification means for generating a deterioration evaluation report and providing it to the relevant parties. This automates infrastructure inspection work, enabling efficient and accurate detection and evaluation of defects.

[0268] "Image acquisition means" refers to a device that has the function of taking images of infrastructure and acquiring related location information and time information of the images taken.

[0269] "Data transmission means" refers to a device or system that has the function of transmitting acquired image data to a server via a secure communication protocol.

[0270] An "artificial intelligence analysis means" is a system that has the function of analyzing images using an AI model to identify infrastructure defects contained in the acquired image at the pixel level.

[0271] A "deterioration evaluation method" is a system that has the function of evaluating the deterioration state of the entire infrastructure based on defect information identified by artificial intelligence analysis methods.

[0272] A "priority calculation means" is a system that has the function of determining the priority of repairs based on a deterioration evaluation means.

[0273] A "report generation and notification means" is a device or system that has the function of generating a report based on the results of a deterioration assessment and providing it to the relevant parties.

[0274] A "scheduling method" is a system that has the function of automatically planning and setting the date for the next inspection.

[0275] As an embodiment of the present invention, an information processing system for efficiently inspecting infrastructure is provided. This system consists of three main components: a terminal, a server, and a user.

[0276] The device consists of hardware equipped with a high-resolution camera and takes images in the infrastructure field. Location and time information are automatically added to the captured images. The device transmits this image data to the server via a secure protocol (e.g., HTTPS).

[0277] The server is a high-performance computer system that analyzes received image data. The server uses a generative AI model to perform image analysis to identify defects within the images. This AI model is trained on a large amount of training data and can quickly and accurately identify cracks and damage at the pixel level. Based on the analyzed data, the server further considers past inspection records and external environmental data to assess the deterioration status of the infrastructure. Based on the assessment results, it calculates repair priorities and generates a detailed deterioration assessment report. The server has the function to notify the user of this report and the next inspection schedule.

[0278] Users receive these notifications and plan repairs and maintenance based on the information provided. Users refer to reports and instruct repair work if necessary. Users also conduct planned inspections according to the automatically set next inspection schedule. This process makes infrastructure inspections more efficient and effective than before.

[0279] As a specific example, in the inspection of a bridge, the terminal acquires images of the bridge from various angles, and the images are transmitted to the server. The server analyzes them to identify defects such as cracks. The report created based on that information is promptly notified to the user, and the necessary countermeasures are taken promptly. An example of a prompt sentence is, "Please input the image of this section of the bridge into the AI model, identify the location and severity of the defect, and generate a report."

[0280] The flow of the specific process in Example 1 will be described using FIG. 11.

[0281] Step 1:

[0282] The terminal acquires an image of the infrastructure at the site. As input, it uses the visual data captured by the camera of the terminal, the position information by GPS, and the shooting time information. Based on this information, metadata is added to the image to prepare image data with the shooting location and time accurately attached.

[0283] Step 2:

[0284] The terminal transmits the acquired image data to the server via a secure communication protocol. As input, it uses the captured image and the added metadata. The terminal performs data compression, optimizes the communication, and communicates with the server in a form that minimizes the delay. After transmission, the transmission log is saved in the terminal.

[0285] Step 3:

[0286] The server takes in the image data received from the terminal for analysis processing. The input is the image group transmitted from the terminal and its metadata. The server uses the generated AI model to detect surface defects of the infrastructure through the image analysis module. The AI model performs pixel-level analysis on the image data, identifies the defective locations, and outputs the analysis data marked thereon.

[0287] Step 4:

[0288] The server evaluates the deterioration status of the infrastructure based on the analysis results. The input data consists of analyzed image data, past inspection records, and environmental data. The server integrates these and performs calculations to evaluate the health of the infrastructure, generating a list of prioritized areas requiring repair as an evaluation result.

[0289] Step 5:

[0290] The server generates a detailed report based on the degradation assessment results. The inputs used are the degradation assessment results and priority information. The server organizes the information, creates a report in a format easily understood by stakeholders, and electronically notifies the user. As a result, the user receives a detailed report as output, which allows them to consider future countermeasures.

[0291] Step 6:

[0292] Users receive reports sent from the server and use them to develop repair plans. They refer to the report information as input and collaborate with the repair team to proceed with specific actions. At the same time, users can check the next inspection schedule set by the server and obtain output to plan ongoing infrastructure inspections.

[0293] (Application Example 1)

[0294] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0295] Early detection and repair of deterioration and damage to equipment and machinery in manufacturing is crucial for improving quality and maintaining productivity. However, conventional inspection methods rely on human verification, which limits the frequency and accuracy of inspections, making early problem detection difficult. In particular, large factories need to efficiently monitor multiple machines and pieces of equipment, but managing them is extremely labor-intensive and inefficient.

[0296] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0297] In this invention, the server includes a means for capturing images, an artificial intelligence image analysis means, a means for evaluating deterioration, a means for determining priority, a means for generating and notifying reports, and a means for scheduling inspections. This makes it possible to quickly and accurately identify wear and damage to multiple pieces of equipment and machinery, determine the priority of necessary repairs, and automatically create a periodic inspection schedule.

[0298] "Equipment and machinery" refers to the collective term for production equipment and manufacturing machinery used in factories and manufacturing industries, and these are the items whose deterioration and damage are monitored.

[0299] "Photography means" refers to devices used to acquire images of equipment or machinery, mainly cameras and sensors, and also has the function of adding metadata such as location information and time of shooting.

[0300] "Artificial intelligence image analysis means" refers to an analysis function that uses AI technology to analyze acquired images and identify wear and damage to equipment and machinery contained therein.

[0301] A "deterioration evaluation method" is a function that evaluates the deterioration state of equipment and machinery based on wear and damage information identified by artificial intelligence image analysis methods.

[0302] The "priority determination means" is a function for quantitatively calculating the priority of areas requiring repair based on the deterioration evaluation means.

[0303] The "report generation and notification means" is a function for creating a report based on the results of the deterioration assessment and notifying administrators and responsible persons of it.

[0304] The "inspection scheduling means" is a function that automatically generates a schedule for periodic inspections based on the condition of the equipment and machinery.

[0305] This system aims to quickly and accurately identify wear and damage of equipment and machinery in the factory and support an appropriate repair plan. The following details its embodiments.

[0306] First, the terminal is equipped with a high-resolution camera, which is used to periodically acquire image data of equipment and machinery. The camera has a function of adding position information and shooting time to the image as metadata. This terminal is implemented on devices suitable for the factory environment, such as robots and smart glasses. Next, using the Yawata Tianya communication technology, these image data are securely transmitted to the server.

[0307] The server analyzes the collected image data using a high-efficiency data processing system. For the analysis, an AI image analysis model is utilized, and libraries such as TensorFlow and PyTorch are used. This AI model identifies defects such as wear and damage in the image. Based on the analysis results, the deterioration state is evaluated, and further, the priority of repair is determined. This process is performed by an algorithm that takes into account past data and weather conditions, etc.

[0308] After that, a deterioration evaluation report is automatically generated. The report includes the identified problem areas, priorities, and specific countermeasures if repair is necessary, and is notified to the administrator and relevant responsible persons.

[0309] In addition, the server generates an inspection schedule and automatically sets the next inspection time. This schedule is designed to be directly synchronized with the administrator's digital calendar.

[0310] As a specific example, consider the case where the belt conveyor in the factory is periodically photographed and wear is detected by AI analysis. This information is summarized in the report, and a notice is sent to prompt the responsible person to quickly replace the parts.

[0311] An example of a prompt message for the generated AI model is: "Based on the following image, identify wear and damage to the machinery and equipment to be inspected. Output detailed information and priority, and if repairs are needed, indicate the recommended actions."

[0312] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0313] Step 1:

[0314] The terminal captures high-resolution images of equipment and machinery within the factory. During this process, location and timestamp information are added to the image data as metadata. The input is physical factory equipment, and the output is image data with location and timestamp information added. Specifically, the camera captures the deterioration status of conveyor belts and press machines, and then adds GPS information and timestamps.

[0315] Step 2:

[0316] The device transmits captured image data to the server based on a secure communication protocol. This transmission involves encryption to maintain data integrity. The input is the image data collected by the device, and the output is the encrypted image data received by the server. Specifically, the SSL / TLS protocol is used to transmit the image data and prevent unauthorized access.

[0317] Step 3:

[0318] The server analyzes the received image data using an AI model. The AI ​​model includes a generative AI model that identifies defects in the input image. The input is encrypted image data received by the server, which is decrypted and then analyzed. The output is labeled data indicating areas of wear and damage. Specifically, an AI trained with the TensorFlow library identifies worn areas of the belt in the image on a pixel-by-pixel basis.

[0319] Step 4:

[0320] The server evaluates the equipment's deterioration status and determines repair priorities based on defect information identified by AI. The input is labeled analysis data, and the output is repair items categorized by priority. Specifically, certain thresholds are set, and priority ranks are assigned according to the degree of deterioration.

[0321] Step 5:

[0322] The server generates a detailed report based on the evaluation results and notifies administrators and users. This report includes information on areas requiring repair and recommended actions based on priority. The input is repair item data with assigned priorities, and the output is a structured report. Specifically, the report is generated in PDF format and delivered to users via email or a dedicated application.

[0323] Step 6:

[0324] The server calculates the next inspection schedule and automatically registers it in the user's digital calendar. Inputs are equipment degradation assessment data and existing maintenance schedules, and output is an updated calendar event. Specifically, it uses the Google Calendar API to automatically add appointments and facilitate the next inspection.

[0325] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0326] This invention provides a system that streamlines inspection work while taking into account the user's emotional state by combining an emotion engine with an infrastructure inspection system. The system mainly consists of three aspects: physical infrastructure inspection, AI-based image analysis and deterioration assessment, and recognition of the user's emotional state.

[0327] First, the device captures images of the infrastructure on-site and adds location information and the date and time of capture as metadata. This data is compressed and sent to the server via a secure communication protocol.

[0328] Next, the server processes the received images using an AI image analysis module to identify defects. Based on the analysis data, it evaluates the degree of deterioration and determines the repair priority. The detailed evaluation report generated in this process is notified to the administrator, and the next inspection schedule is automatically set.

[0329] Furthermore, this invention incorporates an emotion engine that evaluates the user's emotional state in real time. Emotional data is acquired through built-in sensors (e.g., camera and microphone) of the terminal used by the user during inspections, and the user's stress and fatigue levels are analyzed. This information is used to adjust the priority of inspection tasks and is linked with report generation and notification means to provide appropriate feedback.

[0330] As a concrete example, when inspecting a bridge, the user takes images using a tablet equipped with emotion recognition capabilities. If the emotion engine detects the user's stress level from their facial expressions and voice, the server automatically adjusts the next task and schedule to prevent the work from becoming excessively burdensome. This allows the user to perform inspection tasks efficiently while reducing stress.

[0331] In this way, the present invention provides an advanced infrastructure inspection system that combines high-precision inspection using AI with flexible work management that takes into account the user's emotional state.

[0332] The following describes the processing flow.

[0333] Step 1:

[0334] The device captures images of the infrastructure on-site. The device has GPS and timestamp capabilities to add location information and the date and time of capture, and this metadata is embedded in the image data.

[0335] Step 2:

[0336] The device compresses the image data it acquires and sends it to the server using a secure communication protocol. Encryption is applied during data transmission to maintain communication security.

[0337] Step 3:

[0338] The server passes the received images to the AI ​​image analysis module for preprocessing. This preprocessing includes noise reduction and image resolution adjustment, which improves the accuracy of the image analysis.

[0339] Step 4:

[0340] An AI image analysis module processes images to identify defects such as cracks and damage on the infrastructure. Information on detected defects, including location, size, and shape, is recorded.

[0341] Step 5:

[0342] The server evaluates the deterioration status of structures based on defect information. The evaluation takes into account past inspection data and local environmental information, and the deterioration status is scored.

[0343] Step 6:

[0344] The server uses the degradation assessment results to determine repair priorities. These priorities are ranked according to the need for repair, and sections requiring immediate attention are identified.

[0345] Step 7:

[0346] The server generates an evaluation report and notifies the administrator. The report includes the deterioration assessment results, repair priorities, and recommended plans for the next inspection. Notifications are sent via email or a dedicated dashboard.

[0347] Step 8:

[0348] The emotion engine evaluates the user's emotional state in real time. It uses the device's camera and microphone to analyze the user's facial expressions and voice tone, and calculates stress and fatigue levels.

[0349] Step 9:

[0350] The server dynamically adjusts the assignment of inspection tasks based on data from the emotion engine. If a user's stress level is high, it automatically reduces the workload or distributes tasks to other workers.

[0351] Step 10:

[0352] The server creates the next inspection schedule and notifies the user. The inspection date, time, and frequency are adjusted considering the user's emotional state, optimizing the user's workload.

[0353] (Example 2)

[0354] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0355] While conventional infrastructure inspection systems could assess the physical deterioration of structures, they did not adequately consider the psychological burden on inspectors or the efficiency of their work. In particular, increased stress and fatigue among workers can negatively impact the quality of work and their health. Therefore, there is a need to conduct highly accurate infrastructure inspections while simultaneously reducing the psychological burden on workers and improving work efficiency.

[0356] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0357] In this invention, the server includes artificial intelligence image analysis means, degradation evaluation means, priority determination means, emotion evaluation means, and feedback provision means. This enables precise defect detection and degradation evaluation of infrastructure, as well as real-time adjustment of work schedules and feedback based on the emotional state of workers. This reduces the psychological burden on workers and improves work efficiency.

[0358] "Infrastructure" refers to social infrastructure facilities such as bridges, roads, and tunnels, and inspections are necessary to maintain the safety and functionality of these structures.

[0359] "Shooting equipment" refers to a device that has the function of acquiring image data at infrastructure sites and adds metadata including location information and the date and time of shooting.

[0360] "Artificial intelligence image analysis means" refers to artificial intelligence technology used to identify and analyze defects in infrastructure using captured image data.

[0361] "Deterioration evaluation means" refers to a technology that determines and evaluates the degree of deterioration of a structure based on defect information identified by artificial intelligence image analysis means.

[0362] A "priority determination method" refers to a technology that calculates the necessity and urgency of infrastructure repairs based on the results of deterioration assessment methods, and then determines their priority.

[0363] "Report generation and notification means" refers to technology that, in conjunction with priority determination means, aggregates deterioration assessment results and priority information and provides them to administrators as reports.

[0364] "Inspection scheduling means" refers to technology that automatically creates schedules to efficiently plan and manage periodic inspections of infrastructure.

[0365] "Emotional assessment tools" refer to technologies used to monitor workers' emotional states in real time and to assess their stress and fatigue levels.

[0366] "Feedback provision means" refers to technology that has the function of adjusting workload and suggesting improvements based on the worker's state obtained through emotional evaluation means.

[0367] This invention is a system for efficiently performing infrastructure inspection work and aims to reduce the psychological burden on workers. The system consists of the following three main components.

[0368] First, the terminal is used in the field and has the function of capturing images of infrastructure. Specifically, it uses a mobile device equipped with a camera and GPS function to add location information and the date and time of capture as metadata to the image. This data is then efficiently transmitted to the server using data compression technology after capture.

[0369] Next, the server plays a central role in processing the received image data. The server includes image analysis modules utilizing AI technologies such as TensorFlow to perform detailed image analysis. This analysis identifies minor defects and uses that information to perform a degradation assessment. The assessment results are generated as a report in JSON format and notified to the administrator. In addition, a schedule for the next inspection is automatically set.

[0370] Furthermore, the system incorporates an emotion assessment mechanism that monitors the user's emotional state in real time. The user's device is equipped with a camera and microphone, and these sensors collect the user's facial expressions and voice information. This information is then used with analysis libraries such as Librosa to assess stress and fatigue levels. The assessment results are passed to a feedback system, which provides appropriate work instructions and break suggestions based on the stress level.

[0371] As a concrete example, if a user takes an image using a tablet device while inspecting a bridge, and the emotion engine detects high stress from the user's facial expression, the server will automatically revise the schedule and suggest ways to reduce the workload. In this case, an example of a prompt message would be, "If the user is in a high-stress state and a high-temperature environment while inspecting the bridge, please indicate how the work schedule should be changed."

[0372] Thus, the system of the present invention combines advanced image analysis technology and emotion analysis technology, making it possible to perform infrastructure inspection work efficiently and safely.

[0373] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0374] Step 1:

[0375] The device captures images of infrastructure on-site and acquires image data. The input is video from the device's camera sensor, and the output is an image file in JPEG format. During this process, location information is obtained using the built-in GPS, and the date and time of capture are recorded using the system clock, and these are added to the image metadata.

[0376] Step 2:

[0377] The device compresses the acquired image data using a compression algorithm to reduce its size and sends it to the server using the HTTPS protocol to ensure security. Input consists of JPEG images and metadata, while output is a compressed data packet. A checksum is generated before transmission to guarantee data integrity.

[0378] Step 3:

[0379] The server passes the received image data to an AI image analysis module. The input is a compressed JPEG image, and the output is analyzed defect information. The AI ​​model, using TensorFlow, identifies problematic defects in the image at the pixel level and highlights areas that require particular attention.

[0380] Step 4:

[0381] The server performs a deterioration assessment using the analysis results from the AI ​​image analysis module. The input is defect information, and the output is a deterioration assessment report. For example, it quantifies the width of cracks and the spread of rust to measure the degree of deterioration. Based on these results, it calculates a repair priority score.

[0382] Step 5:

[0383] The server evaluates the user's emotional state in real time using emotion assessment tools. Input is sensor information from the terminal's camera and microphone, and output is stress and fatigue scores. Voice and facial expression data are analyzed using Librosa and OpenCV to determine the user's psychological state.

[0384] Step 6:

[0385] The server automatically adjusts the schedule for the next task based on the analysis results and emotional assessment. The input is the deterioration assessment score and emotional state score, and the output is the updated task schedule. It suggests reducing the workload or adding breaks as needed.

[0386] Step 7:

[0387] The server ultimately provides all results to the user as feedback. Inputs are updated schedules and sentiment feedback information, while outputs are specific instructions and suggestions displayed on the user's tablet. This ensures that the work is not overly burdensome and allows for efficient work execution.

[0388] (Application Example 2)

[0389] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0390] Infrastructure inspections require considerable effort and time, and efficiency can decrease depending on the psychological state of the workers. Furthermore, accurately assessing deterioration and defects can be difficult, potentially leading to errors in prioritizing repairs. In this context, there is a need to accurately understand the condition of the infrastructure while reducing the psychological burden on workers and improving work efficiency.

[0391] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0392] In this invention, the server includes an imaging device for infrastructure, a machine learning image analysis device that processes the captured images to identify defects, and an adjustment device that uses an emotion recognition engine to analyze the operator's psychological state in real time and adjusts the work based on this information. This enables precise evaluation of the infrastructure's condition, reduces the operator's psychological burden, and improves work efficiency.

[0393] A "photography device" is a device that acquires images of infrastructure and generates visual data necessary for detailed defect identification and deterioration assessment.

[0394] A "machine learning image analysis device" is a device that uses AI technology to automatically identify defects based on captured images.

[0395] A "condition evaluation device" is a device that evaluates the deterioration state of a structure based on defect information obtained by a machine learning image analysis device.

[0396] A "priority determination device" is a device that determines the priority of repairs based on the evaluated deterioration status.

[0397] A "report generation and notification device" is a device that automatically generates reports based on evaluation results and priorities, and notifies relevant parties.

[0398] A "inspection planning device" is a device that automatically creates a schedule for periodic inspections.

[0399] An "emotion recognition engine" is a technology that analyzes the psychological state of workers in real time and adjusts their work content and schedule as needed.

[0400] The system for realizing this application primarily includes hardware such as an imaging device, image analysis device, condition evaluation device, priority determination device, report generation and notification device, inspection planning device, and emotion recognition engine. These devices work together to efficiently perform infrastructure inspections and adjust work according to the psychological state of the workers.

[0401] The server receives image data acquired from the imaging device using edge AI devices such as NVIDIA Jetson and Intel Movidius. Image analysis is performed using machine learning models with TensorFlow or PyTorch to identify defects in the structure. Subsequently, a condition evaluation device analyzes the defect information and evaluates the deterioration state of the structure. Next, a priority determination device calculates the repair priority based on this evaluation result.

[0402] Furthermore, the report generation and notification device generates a report summarizing the results and notifies the administrator. The inspection planning device automatically sets the schedule for the next periodic inspection. In addition, the emotion recognition engine evaluates the worker's psychological state in real time and adjusts the work content and schedule as needed.

[0403] As a concrete example, if a robot in a factory takes images of equipment and detects an anomaly, that information is immediately reported to the manager. At the same time, an emotion recognition engine measures the fatigue level of workers and suggests breaks to alleviate excessive strain. This entire process comprehensively supports improved work efficiency and the maintenance of workers' physical and mental health.

[0404] Examples of prompts to input into a generative AI model:

[0405] "Use AI to analyze images taken by factory equipment inspection robots to detect deterioration and abnormalities. Also, evaluate employee stress levels based on emotional data and provide appropriate feedback."

[0406] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0407] Step 1:

[0408] The terminal acquires images using a camera at the infrastructure site. At this time, location information and time of capture are added to the image as metadata, and the data is compressed. The input consists of infrastructure image data, location information, and time data, while the output is compressed image data.

[0409] Step 2:

[0410] The terminal transmits compressed image data to the server using a secure communication protocol. The input is compressed image data, and the output is the accurate transmission of data to the server.

[0411] Step 3:

[0412] The server processes the received image data using a machine learning image analysis system. It utilizes frameworks such as TensorFlow and PyTorch to perform image analysis to identify defects. The input is the received image data, and the output is the identified defect information.

[0413] Step 4:

[0414] The server evaluates the degradation state using a condition evaluation device based on defect information obtained from image analysis. The input is defect information, and the output is the evaluated degradation state data.

[0415] Step 5:

[0416] The server calculates repair priorities using a priority determination device based on condition evaluation data. The input is deterioration status data, and the output is the repair priority.

[0417] Step 6:

[0418] The server generates a report using a report generation and notification device based on the evaluation results and priorities, and notifies the administrator. The input is the evaluation results and priority data, and the output is a notification of the generated report to the administrator.

[0419] Step 7:

[0420] The server automatically generates the next scheduled inspection schedule using the inspection planning device. The input is the evaluation results and priority data, and the output is the automatically generated schedule.

[0421] Step 8:

[0422] The server uses an emotion recognition engine to monitor the user's psychological state in real time and evaluate stress and fatigue. The input is the user's emotional data, and the output is the evaluated stress level.

[0423] Step 9:

[0424] The server provides feedback that adjusts the work schedule and content based on the user's psychological state. The input is the assessed stress level, and the output is the adjusted work feedback.

[0425] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0426] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0427] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0428] [Third Embodiment]

[0429] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0430] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0431] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0432] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0433] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0434] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0435] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0436] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0437] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0438] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0439] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0440] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0441] This invention provides a system that integrates several key elements to enable efficient and accurate inspection of infrastructure. This system functions through the collaboration of three parties: a terminal, a server, and a user.

[0442] First, the terminal acquires images of the infrastructure at the site. The terminal is equipped with a camera, and metadata such as location information and time of capture is added to the captured images. This allows for precise identification of where and when the images were taken. These image data are transmitted to the server via secure communication.

[0443] Next, the server processes the received images. First, an image analysis module feeds the image data into an AI model to identify defects on the infrastructure surface. This AI model is trained on a vast amount of training data and can detect cracks and damage quickly and with high accuracy. As a result, the locations where defects exist are marked at the pixel level, and detailed information such as the size and shape of the defects is extracted.

[0444] Subsequently, the server evaluates the state of deterioration based on these analysis results. Past inspection data and environmental conditions (e.g., weather information) are also taken into consideration to assess the overall health of the infrastructure. This evaluation determines the priority of areas that require repair.

[0445] In the report generation step, the server generates a detailed degradation assessment report. The report includes detected defects, their priority, and recommended repair actions, providing administrators with information to gain an overall understanding of the structural health. Users receive this report from the server and use it to consider countermeasures and plan future maintenance.

[0446] Furthermore, the server automatically sets the next inspection schedule. This allows for continuous monitoring of the infrastructure's status and enables safe and efficient management.

[0447] Specific example

[0448] For example, in the case of bridge inspections, the terminal takes pictures of each section of the bridge, and the acquired images are sent to the server. The server analyzes the images and identifies the location and severity of any cracks. This information is compiled into a deterioration assessment report, and if it is determined that repairs are of high priority, prompt action is required. Upon receiving the report, the user works quickly in cooperation with the repair team to carry out the work. In addition, the next inspection schedule set by the server is automatically added to the user's digital calendar, promoting planned inspection activities.

[0449] The following describes the processing flow.

[0450] Step 1:

[0451] The device captures images of the infrastructure. By adding metadata such as location information and the date and time of capture, the source of the image is clearly identified.

[0452] Step 2:

[0453] The device sends the captured image to the server using a secure communication protocol. Here, the data is compressed during transmission, improving transmission speed and reducing network load.

[0454] Step 3:

[0455] The server passes the received images to the AI ​​image analysis module. First, image preprocessing is performed, such as adjusting the resolution and removing noise. This improves the accuracy of the analysis.

[0456] Step 4:

[0457] The AI ​​image analysis module analyzes pre-processed images to identify defects appearing on the infrastructure surface. Specifically, it detects areas of cracks and damage and records their location, size, and shape in detail.

[0458] Step 5:

[0459] The server evaluates the deterioration status of the structure based on the analysis results. This evaluation also takes into account past inspection data and environmental factors. The degree of deterioration is scored, and an evaluation report is created that lists the scores.

[0460] Step 6:

[0461] The server determines repair priorities based on degradation assessments. Priorities are tiered from those requiring immediate attention to those that only require routine maintenance, which helps administrators make informed decisions.

[0462] Step 7:

[0463] The server generates an evaluation report and notifies the user in PDF or digital dashboard format. The report includes information such as the structural integrity, repair recommendations, and priority lists.

[0464] Step 8:

[0465] The server automatically sets the next maintenance schedule and notifies the user. This schedule is optimized based on the infrastructure status and synchronized with the user's digital calendar.

[0466] (Example 1)

[0467] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0468] Infrastructure inspection work is inefficient due to its reliance on manual processes, leading to high risks of overlooking defects and delayed assessments. Furthermore, the manual management of inspection data makes accurate assessment of deterioration and the development of appropriate repair plans difficult. To address these challenges and improve infrastructure safety, an efficient and accurate automated inspection system is required.

[0469] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0470] In this invention, the server includes an artificial intelligence analysis means for analyzing images to identify infrastructure defects at the pixel level, a deterioration evaluation means for comprehensively evaluating the deterioration state of the infrastructure based on the identified defect information, and a report generation and notification means for generating a deterioration evaluation report and providing it to the relevant parties. This automates infrastructure inspection work, enabling efficient and accurate detection and evaluation of defects.

[0471] "Image acquisition means" refers to a device that has the function of taking images of infrastructure and acquiring related location information and time information of the images taken.

[0472] "Data transmission means" refers to a device or system that has the function of transmitting acquired image data to a server via a secure communication protocol.

[0473] An "artificial intelligence analysis means" is a system that has the function of analyzing images using an AI model to identify infrastructure defects contained in the acquired image at the pixel level.

[0474] A "deterioration evaluation method" is a system that has the function of evaluating the deterioration state of the entire infrastructure based on defect information identified by artificial intelligence analysis methods.

[0475] A "priority calculation means" is a system that has the function of determining the priority of repairs based on a deterioration evaluation means.

[0476] A "report generation and notification means" is a device or system that has the function of generating a report based on the results of a deterioration assessment and providing it to the relevant parties.

[0477] A "scheduling method" is a system that has the function of automatically planning and setting the date for the next inspection.

[0478] As an embodiment of the present invention, an information processing system for efficiently inspecting infrastructure is provided. This system consists of three main components: a terminal, a server, and a user.

[0479] The device consists of hardware equipped with a high-resolution camera and takes images in the infrastructure field. Location and time information are automatically added to the captured images. The device transmits this image data to the server via a secure protocol (e.g., HTTPS).

[0480] The server is a high-performance computer system that analyzes received image data. The server uses a generative AI model to perform image analysis to identify defects within the images. This AI model is trained on a large amount of training data and can quickly and accurately identify cracks and damage at the pixel level. Based on the analyzed data, the server further considers past inspection records and external environmental data to assess the deterioration status of the infrastructure. Based on the assessment results, it calculates repair priorities and generates a detailed deterioration assessment report. The server has the function to notify the user of this report and the next inspection schedule.

[0481] Users receive these notifications and plan repairs and maintenance based on the information provided. Users refer to reports and instruct repair work if necessary. Users also conduct planned inspections according to the automatically set next inspection schedule. This process makes infrastructure inspections more efficient and effective than before.

[0482] As a concrete example, in bridge inspections, the terminal acquires images of the bridge from various angles, and these images are sent to the server. The server analyzes the images and identifies defects such as cracks. A report based on this information is promptly notified to the user, allowing for necessary actions to be taken quickly. An example of a prompt message would be, "Input the section images of this bridge into the AI ​​model to identify the location and severity of defects and generate a report."

[0483] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0484] Step 1:

[0485] The terminal acquires images of the infrastructure on-site. The input consists of visual data captured by the terminal's camera, as well as GPS location and timestamp information. Based on this information, metadata is added to the images, preparing image data with precisely identified location and timestamps.

[0486] Step 2:

[0487] The device transmits the acquired image data to the server via a secure communication protocol. The input consists of the captured image and its associated metadata. The device compresses the data, optimizes communication, and minimizes latency before sending it to the server. After transmission, a transmission log is stored on the device.

[0488] Step 3:

[0489] The server takes in image data received from the terminal for analysis. The input consists of a set of images and their metadata sent from the terminal. The server uses a generative AI model to detect surface defects in the infrastructure through an image analysis module. The AI ​​model performs pixel-level analysis on the image data, identifies the defective areas, and outputs analysis data with those areas marked.

[0490] Step 4:

[0491] The server evaluates the deterioration status of the infrastructure based on the analysis results. The input data consists of analyzed image data, past inspection records, and environmental data. The server integrates these and performs calculations to evaluate the health of the infrastructure, generating a list of prioritized areas requiring repair as an evaluation result.

[0492] Step 5:

[0493] The server generates a detailed report based on the degradation assessment results. The inputs used are the degradation assessment results and priority information. The server organizes the information, creates a report in a format easily understood by stakeholders, and electronically notifies the user. As a result, the user receives a detailed report as output, which allows them to consider future countermeasures.

[0494] Step 6:

[0495] Users receive reports sent from the server and use them to develop repair plans. They refer to the report information as input and collaborate with the repair team to proceed with specific actions. At the same time, users can check the next inspection schedule set by the server and obtain output to plan ongoing infrastructure inspections.

[0496] (Application Example 1)

[0497] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0498] Early detection and repair of deterioration and damage to equipment and machinery in manufacturing is crucial for improving quality and maintaining productivity. However, conventional inspection methods rely on human verification, which limits the frequency and accuracy of inspections, making early problem detection difficult. In particular, large factories need to efficiently monitor multiple machines and pieces of equipment, but managing them is extremely labor-intensive and inefficient.

[0499] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0500] In this invention, the server includes a means for capturing images, an artificial intelligence image analysis means, a means for evaluating deterioration, a means for determining priority, a means for generating and notifying reports, and a means for scheduling inspections. This makes it possible to quickly and accurately identify wear and damage to multiple pieces of equipment and machinery, determine the priority of necessary repairs, and automatically create a periodic inspection schedule.

[0501] "Equipment and machinery" refers to the collective term for production equipment and manufacturing machinery used in factories and manufacturing industries, and these are the items whose deterioration and damage are monitored.

[0502] "Photography means" refers to devices used to acquire images of equipment or machinery, mainly cameras and sensors, and also has the function of adding metadata such as location information and time of shooting.

[0503] "Artificial intelligence image analysis means" refers to an analysis function that uses AI technology to analyze acquired images and identify wear and damage to equipment and machinery contained therein.

[0504] A "deterioration evaluation method" is a function that evaluates the deterioration state of equipment and machinery based on wear and damage information identified by artificial intelligence image analysis methods.

[0505] The "priority determination means" is a function for quantitatively calculating the priority of areas requiring repair based on the deterioration evaluation means.

[0506] The "report generation and notification means" is a function for creating a report based on the results of the deterioration assessment and notifying administrators and responsible persons of it.

[0507] The "inspection scheduling means" is a function that automatically generates a schedule for periodic inspections based on the condition of the equipment and machinery.

[0508] This system aims to quickly and accurately identify wear and tear and damage to equipment and machinery within a factory, and to support appropriate repair planning. Its embodiments are described in detail below.

[0509] First, the terminal is equipped with a high-resolution camera, which is used to periodically acquire image data of equipment and machinery. The camera has the function of adding location information and time of capture as metadata to the image. This terminal will be implemented in devices suitable for the factory environment, such as robots and smart glasses. Next, this image data is securely transmitted to a server using tamper-evident communication technology.

[0510] The server analyzes the collected image data using a highly efficient data processing system. This analysis utilizes AI image analysis models and libraries such as TensorFlow and PyTorch. These AI models identify defects such as wear and damage within the images. Based on the analysis results, the degree of deterioration is evaluated, and repair priorities are determined. This process is carried out using algorithms that take into account historical data and weather conditions.

[0511] Subsequently, a deterioration assessment report is automatically generated. The report includes identified problem areas, priorities, and specific measures to be taken if repairs are necessary, and is notified to managers and relevant personnel.

[0512] The server also generates an inspection schedule and automatically sets the next inspection date. This schedule is directly synchronized with the administrator's digital calendar.

[0513] As a concrete example, consider a scenario where a conveyor belt in a factory is regularly photographed, and wear is detected through AI analysis. This information is compiled into a report, and a notification is sent to the person in charge promptly urging them to replace the parts.

[0514] An example of a prompt message for the generated AI model is: "Based on the following image, identify wear and damage to the machinery and equipment to be inspected. Output detailed information and priority, and if repairs are needed, indicate the recommended actions."

[0515] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0516] Step 1:

[0517] The terminal captures high-resolution images of equipment and machinery within the factory. During this process, location and timestamp information are added to the image data as metadata. The input is physical factory equipment, and the output is image data with location and timestamp information added. Specifically, the camera captures the deterioration status of conveyor belts and press machines, and then adds GPS information and timestamps.

[0518] Step 2:

[0519] The device transmits captured image data to the server based on a secure communication protocol. This transmission involves encryption to maintain data integrity. The input is the image data collected by the device, and the output is the encrypted image data received by the server. Specifically, the SSL / TLS protocol is used to transmit the image data and prevent unauthorized access.

[0520] Step 3:

[0521] The server analyzes the received image data using an AI model. The AI ​​model includes a generative AI model that identifies defects in the input image. The input is encrypted image data received by the server, which is decrypted and then analyzed. The output is labeled data indicating areas of wear and damage. Specifically, an AI trained with the TensorFlow library identifies worn areas of the belt in the image on a pixel-by-pixel basis.

[0522] Step 4:

[0523] The server evaluates the equipment's deterioration status and determines repair priorities based on defect information identified by AI. The input is labeled analysis data, and the output is repair items categorized by priority. Specifically, certain thresholds are set, and priority ranks are assigned according to the degree of deterioration.

[0524] Step 5:

[0525] The server generates a detailed report based on the evaluation results and notifies administrators and users. This report includes information on areas requiring repair and recommended actions based on priority. The input is repair item data with assigned priorities, and the output is a structured report. Specifically, the report is generated in PDF format and delivered to users via email or a dedicated application.

[0526] Step 6:

[0527] The server calculates the next inspection schedule and automatically registers it in the user's digital calendar. Inputs are equipment degradation assessment data and existing maintenance schedules, and output is an updated calendar event. Specifically, it uses the Google Calendar API to automatically add appointments and facilitate the next inspection.

[0528] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0529] This invention provides a system that streamlines inspection work while taking into account the user's emotional state by combining an emotion engine with an infrastructure inspection system. The system mainly consists of three aspects: physical infrastructure inspection, AI-based image analysis and deterioration assessment, and recognition of the user's emotional state.

[0530] First, the device captures images of the infrastructure on-site and adds location information and the date and time of capture as metadata. This data is compressed and sent to the server via a secure communication protocol.

[0531] Next, the server processes the received images using an AI image analysis module to identify defects. Based on the analysis data, it evaluates the degree of deterioration and determines the repair priority. The detailed evaluation report generated in this process is notified to the administrator, and the next inspection schedule is automatically set.

[0532] Furthermore, this invention incorporates an emotion engine that evaluates the user's emotional state in real time. Emotional data is acquired through built-in sensors (e.g., camera and microphone) of the terminal used by the user during inspections, and the user's stress and fatigue levels are analyzed. This information is used to adjust the priority of inspection tasks and is linked with report generation and notification means to provide appropriate feedback.

[0533] As a concrete example, when inspecting a bridge, the user takes images using a tablet equipped with emotion recognition capabilities. If the emotion engine detects the user's stress level from their facial expressions and voice, the server automatically adjusts the next task and schedule to prevent the work from becoming excessively burdensome. This allows the user to perform inspection tasks efficiently while reducing stress.

[0534] In this way, the present invention provides an advanced infrastructure inspection system that combines high-precision inspection using AI with flexible work management that takes into account the user's emotional state.

[0535] The following describes the processing flow.

[0536] Step 1:

[0537] The device captures images of the infrastructure on-site. The device has GPS and timestamp capabilities to add location information and the date and time of capture, and this metadata is embedded in the image data.

[0538] Step 2:

[0539] The device compresses the image data it acquires and sends it to the server using a secure communication protocol. Encryption is applied during data transmission to maintain communication security.

[0540] Step 3:

[0541] The server passes the received images to the AI ​​image analysis module for preprocessing. This preprocessing includes noise reduction and image resolution adjustment, which improves the accuracy of the image analysis.

[0542] Step 4:

[0543] An AI image analysis module processes images to identify defects such as cracks and damage on the infrastructure. Information on detected defects, including location, size, and shape, is recorded.

[0544] Step 5:

[0545] The server evaluates the deterioration status of structures based on defect information. The evaluation takes into account past inspection data and local environmental information, and the deterioration status is scored.

[0546] Step 6:

[0547] The server uses the degradation assessment results to determine repair priorities. These priorities are ranked according to the need for repair, and sections requiring immediate attention are identified.

[0548] Step 7:

[0549] The server generates an evaluation report and notifies the administrator. The report includes the deterioration assessment results, repair priorities, and recommended plans for the next inspection. Notifications are sent via email or a dedicated dashboard.

[0550] Step 8:

[0551] The emotion engine evaluates the user's emotional state in real time. It uses the device's camera and microphone to analyze the user's facial expressions and voice tone, and calculates stress and fatigue levels.

[0552] Step 9:

[0553] The server dynamically adjusts the assignment of inspection tasks based on data from the emotion engine. If a user's stress level is high, it automatically reduces the workload or distributes tasks to other workers.

[0554] Step 10:

[0555] The server creates the next inspection schedule and notifies the user. The inspection date, time, and frequency are adjusted considering the user's emotional state, optimizing the user's workload.

[0556] (Example 2)

[0557] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0558] While conventional infrastructure inspection systems could assess the physical deterioration of structures, they did not adequately consider the psychological burden on inspectors or the efficiency of their work. In particular, increased stress and fatigue among workers can negatively impact the quality of work and their health. Therefore, there is a need to conduct highly accurate infrastructure inspections while simultaneously reducing the psychological burden on workers and improving work efficiency.

[0559] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0560] In this invention, the server includes artificial intelligence image analysis means, degradation evaluation means, priority determination means, emotion evaluation means, and feedback provision means. This enables precise defect detection and degradation evaluation of infrastructure, as well as real-time adjustment of work schedules and feedback based on the emotional state of workers. This reduces the psychological burden on workers and improves work efficiency.

[0561] "Infrastructure" refers to social infrastructure facilities such as bridges, roads, and tunnels, and inspections are necessary to maintain the safety and functionality of these structures.

[0562] "Shooting equipment" refers to a device that has the function of acquiring image data at infrastructure sites and adds metadata including location information and the date and time of shooting.

[0563] "Artificial intelligence image analysis means" refers to artificial intelligence technology used to identify and analyze defects in infrastructure using captured image data.

[0564] "Deterioration evaluation means" refers to a technology that determines and evaluates the degree of deterioration of a structure based on defect information identified by artificial intelligence image analysis means.

[0565] A "priority determination method" refers to a technology that calculates the necessity and urgency of infrastructure repairs based on the results of deterioration assessment methods, and then determines their priority.

[0566] "Report generation and notification means" refers to technology that, in conjunction with priority determination means, aggregates deterioration assessment results and priority information and provides them to administrators as reports.

[0567] "Inspection scheduling means" refers to technology that automatically creates schedules to efficiently plan and manage periodic inspections of infrastructure.

[0568] "Emotional assessment tools" refer to technologies used to monitor workers' emotional states in real time and to assess their stress and fatigue levels.

[0569] "Feedback provision means" refers to technology that has the function of adjusting workload and suggesting improvements based on the worker's state obtained through emotional evaluation means.

[0570] This invention is a system for efficiently performing infrastructure inspection work and aims to reduce the psychological burden on workers. The system consists of the following three main components.

[0571] First, the terminal is used in the field and has the function of capturing images of infrastructure. Specifically, it uses a mobile device equipped with a camera and GPS function to add location information and the date and time of capture as metadata to the image. This data is then efficiently transmitted to the server using data compression technology after capture.

[0572] Next, the server plays a central role in processing the received image data. The server includes image analysis modules utilizing AI technologies such as TensorFlow to perform detailed image analysis. This analysis identifies minor defects and uses that information to perform a degradation assessment. The assessment results are generated as a report in JSON format and notified to the administrator. In addition, a schedule for the next inspection is automatically set.

[0573] Furthermore, the system incorporates an emotion assessment mechanism that monitors the user's emotional state in real time. The user's device is equipped with a camera and microphone, and these sensors collect the user's facial expressions and voice information. This information is then used with analysis libraries such as Librosa to assess stress and fatigue levels. The assessment results are passed to a feedback system, which provides appropriate work instructions and break suggestions based on the stress level.

[0574] As a concrete example, if a user takes an image using a tablet device while inspecting a bridge, and the emotion engine detects high stress from the user's facial expression, the server will automatically revise the schedule and suggest ways to reduce the workload. In this case, an example of a prompt message would be, "If the user is in a high-stress state and a high-temperature environment while inspecting the bridge, please indicate how the work schedule should be changed."

[0575] Thus, the system of the present invention combines advanced image analysis technology and emotion analysis technology, making it possible to perform infrastructure inspection work efficiently and safely.

[0576] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0577] Step 1:

[0578] The device captures images of infrastructure on-site and acquires image data. The input is video from the device's camera sensor, and the output is an image file in JPEG format. During this process, location information is obtained using the built-in GPS, and the date and time of capture are recorded using the system clock, and these are added to the image metadata.

[0579] Step 2:

[0580] The device compresses the acquired image data using a compression algorithm to reduce its size and sends it to the server using the HTTPS protocol to ensure security. Input consists of JPEG images and metadata, while output is a compressed data packet. A checksum is generated before transmission to guarantee data integrity.

[0581] Step 3:

[0582] The server passes the received image data to an AI image analysis module. The input is a compressed JPEG image, and the output is analyzed defect information. The AI ​​model, using TensorFlow, identifies problematic defects in the image at the pixel level and highlights areas that require particular attention.

[0583] Step 4:

[0584] The server performs a deterioration assessment using the analysis results from the AI ​​image analysis module. The input is defect information, and the output is a deterioration assessment report. For example, it quantifies the width of cracks and the spread of rust to measure the degree of deterioration. Based on these results, it calculates a repair priority score.

[0585] Step 5:

[0586] The server evaluates the user's emotional state in real time using emotion assessment tools. Input is sensor information from the terminal's camera and microphone, and output is stress and fatigue scores. Voice and facial expression data are analyzed using Librosa and OpenCV to determine the user's psychological state.

[0587] Step 6:

[0588] The server automatically adjusts the schedule for the next task based on the analysis results and emotional assessment. The input is the deterioration assessment score and emotional state score, and the output is the updated task schedule. It suggests reducing the workload or adding breaks as needed.

[0589] Step 7:

[0590] The server ultimately provides all results to the user as feedback. Inputs are updated schedules and sentiment feedback information, while outputs are specific instructions and suggestions displayed on the user's tablet. This ensures that the work is not overly burdensome and allows for efficient work execution.

[0591] (Application Example 2)

[0592] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0593] Infrastructure inspections require considerable effort and time, and efficiency can decrease depending on the psychological state of the workers. Furthermore, accurately assessing deterioration and defects can be difficult, potentially leading to errors in prioritizing repairs. In this context, there is a need to accurately understand the condition of the infrastructure while reducing the psychological burden on workers and improving work efficiency.

[0594] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0595] In this invention, the server includes an imaging device for infrastructure, a machine learning image analysis device that processes the captured images to identify defects, and an adjustment device that uses an emotion recognition engine to analyze the operator's psychological state in real time and adjusts the work based on this information. This enables precise evaluation of the infrastructure's condition, reduces the operator's psychological burden, and improves work efficiency.

[0596] A "photography device" is a device that acquires images of infrastructure and generates visual data necessary for detailed defect identification and deterioration assessment.

[0597] A "machine learning image analysis device" is a device that uses AI technology to automatically identify defects based on captured images.

[0598] A "condition evaluation device" is a device that evaluates the deterioration state of a structure based on defect information obtained by a machine learning image analysis device.

[0599] A "priority determination device" is a device that determines the priority of repairs based on the evaluated deterioration status.

[0600] A "report generation and notification device" is a device that automatically generates reports based on evaluation results and priorities, and notifies relevant parties.

[0601] A "inspection planning device" is a device that automatically creates a schedule for periodic inspections.

[0602] An "emotion recognition engine" is a technology that analyzes the psychological state of workers in real time and adjusts their work content and schedule as needed.

[0603] The system for realizing this application primarily includes hardware such as an imaging device, image analysis device, condition evaluation device, priority determination device, report generation and notification device, inspection planning device, and emotion recognition engine. These devices work together to efficiently perform infrastructure inspections and adjust work according to the psychological state of the workers.

[0604] The server receives image data acquired from the imaging device using edge AI devices such as NVIDIA Jetson and Intel Movidius. Image analysis is performed using machine learning models with TensorFlow or PyTorch to identify defects in the structure. Subsequently, a condition evaluation device analyzes the defect information and evaluates the deterioration state of the structure. Next, a priority determination device calculates the repair priority based on this evaluation result.

[0605] Furthermore, the report generation and notification device generates a report summarizing the results and notifies the administrator. The inspection planning device automatically sets the schedule for the next periodic inspection. In addition, the emotion recognition engine evaluates the worker's psychological state in real time and adjusts the work content and schedule as needed.

[0606] As a concrete example, if a robot in a factory takes images of equipment and detects an anomaly, that information is immediately reported to the manager. At the same time, an emotion recognition engine measures the fatigue level of workers and suggests breaks to alleviate excessive strain. This entire process comprehensively supports improved work efficiency and the maintenance of workers' physical and mental health.

[0607] Examples of prompts to input into a generative AI model:

[0608] "Use AI to analyze images taken by factory equipment inspection robots to detect deterioration and abnormalities. Also, evaluate employee stress levels based on emotional data and provide appropriate feedback."

[0609] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0610] Step 1:

[0611] The terminal acquires images using a camera at the infrastructure site. At this time, location information and time of capture are added to the image as metadata, and the data is compressed. The input consists of infrastructure image data, location information, and time data, while the output is compressed image data.

[0612] Step 2:

[0613] The terminal transmits compressed image data to the server using a secure communication protocol. The input is compressed image data, and the output is the accurate transmission of data to the server.

[0614] Step 3:

[0615] The server processes the received image data using a machine learning image analysis system. It utilizes frameworks such as TensorFlow and PyTorch to perform image analysis to identify defects. The input is the received image data, and the output is the identified defect information.

[0616] Step 4:

[0617] The server evaluates the degradation state using a condition evaluation device based on defect information obtained from image analysis. The input is defect information, and the output is the evaluated degradation state data.

[0618] Step 5:

[0619] The server calculates repair priorities using a priority determination device based on condition evaluation data. The input is deterioration status data, and the output is the repair priority.

[0620] Step 6:

[0621] The server generates a report using a report generation and notification device based on the evaluation results and priorities, and notifies the administrator. The input is the evaluation results and priority data, and the output is a notification of the generated report to the administrator.

[0622] Step 7:

[0623] The server automatically generates the next scheduled inspection schedule using the inspection planning device. The input is the evaluation results and priority data, and the output is the automatically generated schedule.

[0624] Step 8:

[0625] The server uses an emotion recognition engine to monitor the user's psychological state in real time and evaluate stress and fatigue. The input is the user's emotional data, and the output is the evaluated stress level.

[0626] Step 9:

[0627] The server provides feedback that adjusts the work schedule and content based on the user's psychological state. The input is the assessed stress level, and the output is the adjusted work feedback.

[0628] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0629] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0630] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0631] [Fourth Embodiment]

[0632] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0633] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0634] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0635] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0636] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0637] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0638] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0639] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0640] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0641] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0642] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0643] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0644] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0645] This invention provides a system that integrates several key elements to enable efficient and accurate inspection of infrastructure. This system functions through the collaboration of three parties: a terminal, a server, and a user.

[0646] First, the terminal acquires images of the infrastructure at the site. The terminal is equipped with a camera, and metadata such as location information and time of capture is added to the captured images. This allows for precise identification of where and when the images were taken. These image data are transmitted to the server via secure communication.

[0647] Next, the server processes the received images. First, an image analysis module feeds the image data into an AI model to identify defects on the infrastructure surface. This AI model is trained on a vast amount of training data and can detect cracks and damage quickly and with high accuracy. As a result, the locations where defects exist are marked at the pixel level, and detailed information such as the size and shape of the defects is extracted.

[0648] Subsequently, the server evaluates the state of deterioration based on these analysis results. Past inspection data and environmental conditions (e.g., weather information) are also taken into consideration to assess the overall health of the infrastructure. This evaluation determines the priority of areas that require repair.

[0649] In the report generation step, the server generates a detailed degradation assessment report. The report includes detected defects, their priority, and recommended repair actions, providing administrators with information to gain an overall understanding of the structural health. Users receive this report from the server and use it to consider countermeasures and plan future maintenance.

[0650] Furthermore, the server automatically sets the next inspection schedule. This allows for continuous monitoring of the infrastructure's status and enables safe and efficient management.

[0651] Specific example

[0652] For example, in the case of bridge inspections, the terminal takes pictures of each section of the bridge, and the acquired images are sent to the server. The server analyzes the images and identifies the location and severity of any cracks. This information is compiled into a deterioration assessment report, and if it is determined that repairs are of high priority, prompt action is required. Upon receiving the report, the user works quickly in cooperation with the repair team to carry out the work. In addition, the next inspection schedule set by the server is automatically added to the user's digital calendar, promoting planned inspection activities.

[0653] The following describes the processing flow.

[0654] Step 1:

[0655] The device captures images of the infrastructure. By adding metadata such as location information and the date and time of capture, the source of the image is clearly identified.

[0656] Step 2:

[0657] The device sends the captured image to the server using a secure communication protocol. Here, the data is compressed during transmission, improving transmission speed and reducing network load.

[0658] Step 3:

[0659] The server passes the received images to the AI ​​image analysis module. First, image preprocessing is performed, such as adjusting the resolution and removing noise. This improves the accuracy of the analysis.

[0660] Step 4:

[0661] The AI ​​image analysis module analyzes pre-processed images to identify defects appearing on the infrastructure surface. Specifically, it detects areas of cracks and damage and records their location, size, and shape in detail.

[0662] Step 5:

[0663] The server evaluates the deterioration status of the structure based on the analysis results. This evaluation also takes into account past inspection data and environmental factors. The degree of deterioration is scored, and an evaluation report is created that lists the scores.

[0664] Step 6:

[0665] The server determines repair priorities based on degradation assessments. Priorities are tiered from those requiring immediate attention to those that only require routine maintenance, which helps administrators make informed decisions.

[0666] Step 7:

[0667] The server generates an evaluation report and notifies the user in PDF or digital dashboard format. The report includes information such as the structural integrity, repair recommendations, and priority lists.

[0668] Step 8:

[0669] The server automatically sets the next maintenance schedule and notifies the user. This schedule is optimized based on the infrastructure status and synchronized with the user's digital calendar.

[0670] (Example 1)

[0671] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0672] Infrastructure inspection work is inefficient due to its reliance on manual processes, leading to high risks of overlooking defects and delayed assessments. Furthermore, the manual management of inspection data makes accurate assessment of deterioration and the development of appropriate repair plans difficult. To address these challenges and improve infrastructure safety, an efficient and accurate automated inspection system is required.

[0673] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0674] In this invention, the server includes an artificial intelligence analysis means for analyzing images to identify infrastructure defects at the pixel level, a deterioration evaluation means for comprehensively evaluating the deterioration state of the infrastructure based on the identified defect information, and a report generation and notification means for generating a deterioration evaluation report and providing it to the relevant parties. This automates infrastructure inspection work, enabling efficient and accurate detection and evaluation of defects.

[0675] "Image acquisition means" refers to a device that has the function of taking images of infrastructure and acquiring related location information and time information of the images taken.

[0676] "Data transmission means" refers to a device or system that has the function of transmitting acquired image data to a server via a secure communication protocol.

[0677] An "artificial intelligence analysis means" is a system that has the function of analyzing images using an AI model to identify infrastructure defects contained in the acquired image at the pixel level.

[0678] A "deterioration evaluation method" is a system that has the function of evaluating the deterioration state of the entire infrastructure based on defect information identified by artificial intelligence analysis methods.

[0679] A "priority calculation means" is a system that has the function of determining the priority of repairs based on a deterioration evaluation means.

[0680] A "report generation and notification means" is a device or system that has the function of generating a report based on the results of a deterioration assessment and providing it to the relevant parties.

[0681] A "scheduling method" is a system that has the function of automatically planning and setting the date for the next inspection.

[0682] As an embodiment of the present invention, an information processing system for efficiently inspecting infrastructure is provided. This system consists of three main components: a terminal, a server, and a user.

[0683] The device consists of hardware equipped with a high-resolution camera and takes images in the infrastructure field. Location and time information are automatically added to the captured images. The device transmits this image data to the server via a secure protocol (e.g., HTTPS).

[0684] The server is a high-performance computer system that analyzes received image data. The server uses a generative AI model to perform image analysis to identify defects within the images. This AI model is trained on a large amount of training data and can quickly and accurately identify cracks and damage at the pixel level. Based on the analyzed data, the server further considers past inspection records and external environmental data to assess the deterioration status of the infrastructure. Based on the assessment results, it calculates repair priorities and generates a detailed deterioration assessment report. The server has the function to notify the user of this report and the next inspection schedule.

[0685] Users receive these notifications and plan repairs and maintenance based on the information provided. Users refer to reports and instruct repair work if necessary. Users also conduct planned inspections according to the automatically set next inspection schedule. This process makes infrastructure inspections more efficient and effective than before.

[0686] As a concrete example, in bridge inspections, the terminal acquires images of the bridge from various angles, and these images are sent to the server. The server analyzes the images and identifies defects such as cracks. A report based on this information is promptly notified to the user, allowing for necessary actions to be taken quickly. An example of a prompt message would be, "Input the section images of this bridge into the AI ​​model to identify the location and severity of defects and generate a report."

[0687] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0688] Step 1:

[0689] The terminal acquires images of the infrastructure on-site. The input consists of visual data captured by the terminal's camera, as well as GPS location and timestamp information. Based on this information, metadata is added to the images, preparing image data with precisely identified location and timestamps.

[0690] Step 2:

[0691] The device transmits the acquired image data to the server via a secure communication protocol. The input consists of the captured image and its associated metadata. The device compresses the data, optimizes communication, and minimizes latency before sending it to the server. After transmission, a transmission log is stored on the device.

[0692] Step 3:

[0693] The server takes in image data received from the terminal for analysis. The input consists of a set of images and their metadata sent from the terminal. The server uses a generative AI model to detect surface defects in the infrastructure through an image analysis module. The AI ​​model performs pixel-level analysis on the image data, identifies the defective areas, and outputs analysis data with those areas marked.

[0694] Step 4:

[0695] The server evaluates the deterioration status of the infrastructure based on the analysis results. The input data consists of analyzed image data, past inspection records, and environmental data. The server integrates these and performs calculations to evaluate the health of the infrastructure, generating a list of prioritized areas requiring repair as an evaluation result.

[0696] Step 5:

[0697] The server generates a detailed report based on the degradation assessment results. The inputs used are the degradation assessment results and priority information. The server organizes the information, creates a report in a format easily understood by stakeholders, and electronically notifies the user. As a result, the user receives a detailed report as output, which allows them to consider future countermeasures.

[0698] Step 6:

[0699] Users receive reports sent from the server and use them to develop repair plans. They refer to the report information as input and collaborate with the repair team to proceed with specific actions. At the same time, users can check the next inspection schedule set by the server and obtain output to plan ongoing infrastructure inspections.

[0700] (Application Example 1)

[0701] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0702] Early detection and repair of deterioration and damage to equipment and machinery in manufacturing is crucial for improving quality and maintaining productivity. However, conventional inspection methods rely on human verification, which limits the frequency and accuracy of inspections, making early problem detection difficult. In particular, large factories need to efficiently monitor multiple machines and pieces of equipment, but managing them is extremely labor-intensive and inefficient.

[0703] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0704] In this invention, the server includes a means for capturing images, an artificial intelligence image analysis means, a means for evaluating deterioration, a means for determining priority, a means for generating and notifying reports, and a means for scheduling inspections. This makes it possible to quickly and accurately identify wear and damage to multiple pieces of equipment and machinery, determine the priority of necessary repairs, and automatically create a periodic inspection schedule.

[0705] "Equipment and machinery" refers to the collective term for production equipment and manufacturing machinery used in factories and manufacturing industries, and these are the items whose deterioration and damage are monitored.

[0706] "Photography means" refers to devices used to acquire images of equipment or machinery, mainly cameras and sensors, and also has the function of adding metadata such as location information and time of shooting.

[0707] "Artificial intelligence image analysis means" refers to an analysis function that uses AI technology to analyze acquired images and identify wear and damage to equipment and machinery contained therein.

[0708] A "deterioration evaluation method" is a function that evaluates the deterioration state of equipment and machinery based on wear and damage information identified by artificial intelligence image analysis methods.

[0709] The "priority determination means" is a function for quantitatively calculating the priority of areas requiring repair based on the deterioration evaluation means.

[0710] The "report generation and notification means" is a function for creating a report based on the results of the deterioration assessment and notifying administrators and responsible persons of it.

[0711] The "inspection scheduling means" is a function that automatically generates a schedule for periodic inspections based on the condition of the equipment and machinery.

[0712] This system aims to quickly and accurately identify wear and tear and damage to equipment and machinery within a factory, and to support appropriate repair planning. Its embodiments are described in detail below.

[0713] First, the terminal is equipped with a high-resolution camera, which is used to periodically acquire image data of equipment and machinery. The camera has the function of adding location information and time of capture as metadata to the image. This terminal will be implemented in devices suitable for the factory environment, such as robots and smart glasses. Next, this image data is securely transmitted to a server using tamper-evident communication technology.

[0714] The server analyzes the collected image data using a highly efficient data processing system. This analysis utilizes AI image analysis models and libraries such as TensorFlow and PyTorch. These AI models identify defects such as wear and damage within the images. Based on the analysis results, the degree of deterioration is evaluated, and repair priorities are determined. This process is carried out using algorithms that take into account historical data and weather conditions.

[0715] Subsequently, a deterioration assessment report is automatically generated. The report includes identified problem areas, priorities, and specific measures to be taken if repairs are necessary, and is notified to managers and relevant personnel.

[0716] The server also generates an inspection schedule and automatically sets the next inspection date. This schedule is directly synchronized with the administrator's digital calendar.

[0717] As a concrete example, consider a scenario where a conveyor belt in a factory is regularly photographed, and wear is detected through AI analysis. This information is compiled into a report, and a notification is sent to the person in charge promptly urging them to replace the parts.

[0718] An example of a prompt message for the generated AI model is: "Based on the following image, identify wear and damage to the machinery and equipment to be inspected. Output detailed information and priority, and if repairs are needed, indicate the recommended actions."

[0719] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0720] Step 1:

[0721] The terminal captures high-resolution images of equipment and machinery within the factory. During this process, location and timestamp information are added to the image data as metadata. The input is physical factory equipment, and the output is image data with location and timestamp information added. Specifically, the camera captures the deterioration status of conveyor belts and press machines, and then adds GPS information and timestamps.

[0722] Step 2:

[0723] The device transmits captured image data to the server based on a secure communication protocol. This transmission involves encryption to maintain data integrity. The input is the image data collected by the device, and the output is the encrypted image data received by the server. Specifically, the SSL / TLS protocol is used to transmit the image data and prevent unauthorized access.

[0724] Step 3:

[0725] The server analyzes the received image data using an AI model. The AI ​​model includes a generative AI model that identifies defects in the input image. The input is encrypted image data received by the server, which is decrypted and then analyzed. The output is labeled data indicating areas of wear and damage. Specifically, an AI trained with the TensorFlow library identifies worn areas of the belt in the image on a pixel-by-pixel basis.

[0726] Step 4:

[0727] The server evaluates the equipment's deterioration status and determines repair priorities based on defect information identified by AI. The input is labeled analysis data, and the output is repair items categorized by priority. Specifically, certain thresholds are set, and priority ranks are assigned according to the degree of deterioration.

[0728] Step 5:

[0729] The server generates a detailed report based on the evaluation results and notifies administrators and users. This report includes information on areas requiring repair and recommended actions based on priority. The input is repair item data with assigned priorities, and the output is a structured report. Specifically, the report is generated in PDF format and delivered to users via email or a dedicated application.

[0730] Step 6:

[0731] The server calculates the next inspection schedule and automatically registers it in the user's digital calendar. Inputs are equipment degradation assessment data and existing maintenance schedules, and output is an updated calendar event. Specifically, it uses the Google Calendar API to automatically add appointments and facilitate the next inspection.

[0732] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0733] This invention provides a system that streamlines inspection work while taking into account the user's emotional state by combining an emotion engine with an infrastructure inspection system. The system mainly consists of three aspects: physical infrastructure inspection, AI-based image analysis and deterioration assessment, and recognition of the user's emotional state.

[0734] First, the device captures images of the infrastructure on-site and adds location information and the date and time of capture as metadata. This data is compressed and sent to the server via a secure communication protocol.

[0735] Next, the server processes the received images using an AI image analysis module to identify defects. Based on the analysis data, it evaluates the degree of deterioration and determines the repair priority. The detailed evaluation report generated in this process is notified to the administrator, and the next inspection schedule is automatically set.

[0736] Furthermore, this invention incorporates an emotion engine that evaluates the user's emotional state in real time. Emotional data is acquired through built-in sensors (e.g., camera and microphone) of the terminal used by the user during inspections, and the user's stress and fatigue levels are analyzed. This information is used to adjust the priority of inspection tasks and is linked with report generation and notification means to provide appropriate feedback.

[0737] As a concrete example, when inspecting a bridge, the user takes images using a tablet equipped with emotion recognition capabilities. If the emotion engine detects the user's stress level from their facial expressions and voice, the server automatically adjusts the next task and schedule to prevent the work from becoming excessively burdensome. This allows the user to perform inspection tasks efficiently while reducing stress.

[0738] In this way, the present invention provides an advanced infrastructure inspection system that combines high-precision inspection using AI with flexible work management that takes into account the user's emotional state.

[0739] The following describes the processing flow.

[0740] Step 1:

[0741] The device captures images of the infrastructure on-site. The device has GPS and timestamp capabilities to add location information and the date and time of capture, and this metadata is embedded in the image data.

[0742] Step 2:

[0743] The device compresses the image data it acquires and sends it to the server using a secure communication protocol. Encryption is applied during data transmission to maintain communication security.

[0744] Step 3:

[0745] The server passes the received images to the AI ​​image analysis module for preprocessing. This preprocessing includes noise reduction and image resolution adjustment, which improves the accuracy of the image analysis.

[0746] Step 4:

[0747] An AI image analysis module processes images to identify defects such as cracks and damage on the infrastructure. Information on detected defects, including location, size, and shape, is recorded.

[0748] Step 5:

[0749] The server evaluates the deterioration status of structures based on defect information. The evaluation takes into account past inspection data and local environmental information, and the deterioration status is scored.

[0750] Step 6:

[0751] The server uses the degradation assessment results to determine repair priorities. These priorities are ranked according to the need for repair, and sections requiring immediate attention are identified.

[0752] Step 7:

[0753] The server generates an evaluation report and notifies the administrator. The report includes the deterioration assessment results, repair priorities, and recommended plans for the next inspection. Notifications are sent via email or a dedicated dashboard.

[0754] Step 8:

[0755] The emotion engine evaluates the user's emotional state in real time. It uses the device's camera and microphone to analyze the user's facial expressions and voice tone, and calculates stress and fatigue levels.

[0756] Step 9:

[0757] The server dynamically adjusts the assignment of inspection tasks based on data from the emotion engine. If a user's stress level is high, it automatically reduces the workload or distributes tasks to other workers.

[0758] Step 10:

[0759] The server creates the next inspection schedule and notifies the user. The inspection date, time, and frequency are adjusted considering the user's emotional state, optimizing the user's workload.

[0760] (Example 2)

[0761] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0762] While conventional infrastructure inspection systems could assess the physical deterioration of structures, they did not adequately consider the psychological burden on inspectors or the efficiency of their work. In particular, increased stress and fatigue among workers can negatively impact the quality of work and their health. Therefore, there is a need to conduct highly accurate infrastructure inspections while simultaneously reducing the psychological burden on workers and improving work efficiency.

[0763] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0764] In this invention, the server includes artificial intelligence image analysis means, degradation evaluation means, priority determination means, emotion evaluation means, and feedback provision means. This enables precise defect detection and degradation evaluation of infrastructure, as well as real-time adjustment of work schedules and feedback based on the emotional state of workers. This reduces the psychological burden on workers and improves work efficiency.

[0765] "Infrastructure" refers to social infrastructure facilities such as bridges, roads, and tunnels, and inspections are necessary to maintain the safety and functionality of these structures.

[0766] "Shooting equipment" refers to a device that has the function of acquiring image data at infrastructure sites and adds metadata including location information and the date and time of shooting.

[0767] "Artificial intelligence image analysis means" refers to artificial intelligence technology used to identify and analyze defects in infrastructure using captured image data.

[0768] "Deterioration evaluation means" refers to a technology that determines and evaluates the degree of deterioration of a structure based on defect information identified by artificial intelligence image analysis means.

[0769] A "priority determination method" refers to a technology that calculates the necessity and urgency of infrastructure repairs based on the results of deterioration assessment methods, and then determines their priority.

[0770] "Report generation and notification means" refers to technology that, in conjunction with priority determination means, aggregates deterioration assessment results and priority information and provides them to administrators as reports.

[0771] "Inspection scheduling means" refers to technology that automatically creates schedules to efficiently plan and manage periodic inspections of infrastructure.

[0772] "Emotional assessment tools" refer to technologies used to monitor workers' emotional states in real time and to assess their stress and fatigue levels.

[0773] "Feedback provision means" refers to technology that has the function of adjusting workload and suggesting improvements based on the worker's state obtained through emotional evaluation means.

[0774] This invention is a system for efficiently performing infrastructure inspection work and aims to reduce the psychological burden on workers. The system consists of the following three main components.

[0775] First, the terminal is used in the field and has the function of capturing images of infrastructure. Specifically, it uses a mobile device equipped with a camera and GPS function to add location information and the date and time of capture as metadata to the image. This data is then efficiently transmitted to the server using data compression technology after capture.

[0776] Next, the server plays a central role in processing the received image data. The server includes image analysis modules utilizing AI technologies such as TensorFlow to perform detailed image analysis. This analysis identifies minor defects and uses that information to perform a degradation assessment. The assessment results are generated as a report in JSON format and notified to the administrator. In addition, a schedule for the next inspection is automatically set.

[0777] Furthermore, the system incorporates an emotion assessment mechanism that monitors the user's emotional state in real time. The user's device is equipped with a camera and microphone, and these sensors collect the user's facial expressions and voice information. This information is then used with analysis libraries such as Librosa to assess stress and fatigue levels. The assessment results are passed to a feedback system, which provides appropriate work instructions and break suggestions based on the stress level.

[0778] As a concrete example, if a user takes an image using a tablet device while inspecting a bridge, and the emotion engine detects high stress from the user's facial expression, the server will automatically revise the schedule and suggest ways to reduce the workload. In this case, an example of a prompt message would be, "If the user is in a high-stress state and a high-temperature environment while inspecting the bridge, please indicate how the work schedule should be changed."

[0779] Thus, the system of the present invention combines advanced image analysis technology and emotion analysis technology, making it possible to perform infrastructure inspection work efficiently and safely.

[0780] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0781] Step 1:

[0782] The device captures images of infrastructure on-site and acquires image data. The input is video from the device's camera sensor, and the output is an image file in JPEG format. During this process, location information is obtained using the built-in GPS, and the date and time of capture are recorded using the system clock, and these are added to the image metadata.

[0783] Step 2:

[0784] The device compresses the acquired image data using a compression algorithm to reduce its size and sends it to the server using the HTTPS protocol to ensure security. Input consists of JPEG images and metadata, while output is a compressed data packet. A checksum is generated before transmission to guarantee data integrity.

[0785] Step 3:

[0786] The server passes the received image data to an AI image analysis module. The input is a compressed JPEG image, and the output is analyzed defect information. The AI ​​model, using TensorFlow, identifies problematic defects in the image at the pixel level and highlights areas that require particular attention.

[0787] Step 4:

[0788] The server performs a deterioration assessment using the analysis results from the AI ​​image analysis module. The input is defect information, and the output is a deterioration assessment report. For example, it quantifies the width of cracks and the spread of rust to measure the degree of deterioration. Based on these results, it calculates a repair priority score.

[0789] Step 5:

[0790] The server evaluates the user's emotional state in real time using emotion assessment tools. Input is sensor information from the terminal's camera and microphone, and output is stress and fatigue scores. Voice and facial expression data are analyzed using Librosa and OpenCV to determine the user's psychological state.

[0791] Step 6:

[0792] The server automatically adjusts the schedule for the next task based on the analysis results and emotional assessment. The input is the deterioration assessment score and emotional state score, and the output is the updated task schedule. It suggests reducing the workload or adding breaks as needed.

[0793] Step 7:

[0794] The server ultimately provides all results to the user as feedback. Inputs are updated schedules and sentiment feedback information, while outputs are specific instructions and suggestions displayed on the user's tablet. This ensures that the work is not overly burdensome and allows for efficient work execution.

[0795] (Application Example 2)

[0796] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0797] Infrastructure inspections require considerable effort and time, and efficiency can decrease depending on the psychological state of the workers. Furthermore, accurately assessing deterioration and defects can be difficult, potentially leading to errors in prioritizing repairs. In this context, there is a need to accurately understand the condition of the infrastructure while reducing the psychological burden on workers and improving work efficiency.

[0798] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0799] In this invention, the server includes an imaging device for infrastructure, a machine learning image analysis device that processes the captured images to identify defects, and an adjustment device that uses an emotion recognition engine to analyze the operator's psychological state in real time and adjusts the work based on this information. This enables precise evaluation of the infrastructure's condition, reduces the operator's psychological burden, and improves work efficiency.

[0800] A "photography device" is a device that acquires images of infrastructure and generates visual data necessary for detailed defect identification and deterioration assessment.

[0801] A "machine learning image analysis device" is a device that uses AI technology to automatically identify defects based on captured images.

[0802] A "condition evaluation device" is a device that evaluates the deterioration state of a structure based on defect information obtained by a machine learning image analysis device.

[0803] A "priority determination device" is a device that determines the priority of repairs based on the evaluated deterioration status.

[0804] A "report generation and notification device" is a device that automatically generates reports based on evaluation results and priorities, and notifies relevant parties.

[0805] A "inspection planning device" is a device that automatically creates a schedule for periodic inspections.

[0806] An "emotion recognition engine" is a technology that analyzes the psychological state of workers in real time and adjusts their work content and schedule as needed.

[0807] The system for realizing this application primarily includes hardware such as an imaging device, image analysis device, condition evaluation device, priority determination device, report generation and notification device, inspection planning device, and emotion recognition engine. These devices work together to efficiently perform infrastructure inspections and adjust work according to the psychological state of the workers.

[0808] The server receives image data acquired from the imaging device using edge AI devices such as NVIDIA Jetson and Intel Movidius. Image analysis is performed using machine learning models with TensorFlow or PyTorch to identify defects in the structure. Subsequently, a condition evaluation device analyzes the defect information and evaluates the deterioration state of the structure. Next, a priority determination device calculates the repair priority based on this evaluation result.

[0809] Furthermore, the report generation and notification device generates a report summarizing the results and notifies the administrator. The inspection planning device automatically sets the schedule for the next periodic inspection. In addition, the emotion recognition engine evaluates the worker's psychological state in real time and adjusts the work content and schedule as needed.

[0810] As a concrete example, if a robot in a factory takes images of equipment and detects an anomaly, that information is immediately reported to the manager. At the same time, an emotion recognition engine measures the fatigue level of workers and suggests breaks to alleviate excessive strain. This entire process comprehensively supports improved work efficiency and the maintenance of workers' physical and mental health.

[0811] Examples of prompts to input into a generative AI model:

[0812] "Use AI to analyze images taken by factory equipment inspection robots to detect deterioration and abnormalities. Also, evaluate employee stress levels based on emotional data and provide appropriate feedback."

[0813] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0814] Step 1:

[0815] The terminal acquires images using a camera at the infrastructure site. At this time, location information and time of capture are added to the image as metadata, and the data is compressed. The input consists of infrastructure image data, location information, and time data, while the output is compressed image data.

[0816] Step 2:

[0817] The terminal transmits compressed image data to the server using a secure communication protocol. The input is compressed image data, and the output is the accurate transmission of data to the server.

[0818] Step 3:

[0819] The server processes the received image data using a machine learning image analysis system. It utilizes frameworks such as TensorFlow and PyTorch to perform image analysis to identify defects. The input is the received image data, and the output is the identified defect information.

[0820] Step 4:

[0821] The server evaluates the degradation state using a condition evaluation device based on defect information obtained from image analysis. The input is defect information, and the output is the evaluated degradation state data.

[0822] Step 5:

[0823] The server calculates repair priorities using a priority determination device based on condition evaluation data. The input is deterioration status data, and the output is the repair priority.

[0824] Step 6:

[0825] The server generates a report using a report generation and notification device based on the evaluation results and priorities, and notifies the administrator. The input is the evaluation results and priority data, and the output is a notification of the generated report to the administrator.

[0826] Step 7:

[0827] The server automatically generates the next scheduled inspection schedule using the inspection planning device. The input is the evaluation results and priority data, and the output is the automatically generated schedule.

[0828] Step 8:

[0829] The server uses an emotion recognition engine to monitor the user's psychological state in real time and evaluate stress and fatigue. The input is the user's emotional data, and the output is the evaluated stress level.

[0830] Step 9:

[0831] The server provides feedback that adjusts the work schedule and content based on the user's psychological state. The input is the assessed stress level, and the output is the adjusted work feedback.

[0832] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0833] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0834] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0835] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0836] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0837] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0838] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0839] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0840] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0841] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0842] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0843] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0844] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0845] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0846] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0847] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0848] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0849] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0850] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0851] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0852] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0853] The following is further disclosed regarding the embodiments described above.

[0854] (Claim 1)

[0855] Photography methods targeting infrastructure,

[0856] An artificial intelligence image analysis method that analyzes captured images to identify defects,

[0857] A deterioration evaluation means for evaluating the deterioration state of a structure based on defect information identified by the artificial intelligence image analysis means,

[0858] A priority determination means for calculating the priority of repairs based on the deterioration evaluation means,

[0859] A report generation and notification system that generates an evaluation report and notifies the administrator,

[0860] A means for automatically creating a schedule for periodic inspections,

[0861] A system that includes this.

[0862] (Claim 2)

[0863] The system according to claim 1, characterized in that the shooting means has a function of adding location information and shooting time information to the captured image.

[0864] (Claim 3)

[0865] The system according to claim 1, characterized in that the artificial intelligence image analysis means has the function of processing multiple images to identify infrastructure defects on a pixel-by-pixel basis.

[0866] "Example 1"

[0867] (Claim 1)

[0868] Image acquisition method,

[0869] A data transmission means for transmitting acquired images via a secure communication protocol,

[0870] An artificial intelligence analysis method that analyzes images to identify infrastructure defects at the pixel level,

[0871] A deterioration assessment method that comprehensively evaluates the deterioration state of infrastructure based on identified defect information,

[0872] A priority calculation means for determining the priority of repairs based on the evaluation results,

[0873] A means for generating and notifying reports that generate deterioration assessment reports and provide them to relevant parties,

[0874] A scheduling method that automatically creates the schedule for the next inspection,

[0875] A system that includes this.

[0876] (Claim 2)

[0877] The system according to claim 1, characterized in that the image acquisition means has a function to acquire location information and shooting time information.

[0878] (Claim 3)

[0879] The system according to claim 1, characterized in that the artificial intelligence analysis means has the function of efficiently identifying infrastructure defects using an AI model.

[0880] "Application Example 1"

[0881] (Claim 1)

[0882] Photography methods for equipment and machinery,

[0883] An artificial intelligence image analysis method that analyzes captured images to identify wear and damage,

[0884] A deterioration evaluation means for evaluating the deterioration state of the equipment based on wear information identified by the artificial intelligence image analysis means,

[0885] A priority determination means for calculating the priority of repairs based on the deterioration evaluation means,

[0886] A report generation and notification mechanism that generates an evaluation report and notifies the administrator,

[0887] A means for automatically creating a schedule for periodic inspections,

[0888] A system that includes this.

[0889] (Claim 2)

[0890] The system according to claim 1, characterized in that the shooting means has a function of adding location information and shooting time information to the captured image.

[0891] (Claim 3)

[0892] The system according to claim 1, characterized in that the artificial intelligence image analysis means has the function of processing multiple images to identify wear on equipment and machinery on a pixel-by-pixel basis.

[0893] "Example 2 of combining an emotion engine"

[0894] (Claim 1)

[0895] Photography methods targeting infrastructure,

[0896] An artificial intelligence image analysis method that analyzes captured images to identify defects,

[0897] A deterioration evaluation means for evaluating the deterioration state of a structure based on defect information identified by the artificial intelligence image analysis means,

[0898] A priority determination means for calculating the priority of repairs based on the deterioration evaluation means,

[0899] A report generation and notification system that generates an evaluation report and notifies the administrator,

[0900] A means for automatically creating a schedule for periodic inspections,

[0901] A means of evaluating the user's emotional state in real time and adjusting the work schedule accordingly.

[0902] A system including a feedback provision means that provides the user with adjustments to their workload and feedback based on the aforementioned emotion evaluation means.

[0903] (Claim 2)

[0904] The system according to claim 1, characterized in that the shooting means has a function of adding location information and shooting time information to the captured image.

[0905] (Claim 3)

[0906] The system according to claim 1, characterized in that the artificial intelligence image analysis means has the function of processing multiple images to identify infrastructure defects on a pixel-by-pixel basis.

[0907] "Application example 2 when combining with an emotional engine"

[0908] (Claim 1)

[0909] A photographic device for infrastructure,

[0910] A machine learning image analysis device that processes captured images to identify defects,

[0911] A condition evaluation device that evaluates the deterioration state of a structure based on defect information identified by the aforementioned machine learning image analysis device,

[0912] A priority determination device that calculates the repair priority based on the condition evaluation device,

[0913] A report generation and notification device that generates evaluation reports and notifies users,

[0914] An inspection planning device that automatically creates a schedule for periodic inspections,

[0915] An adjustment device that uses an emotion recognition engine to analyze the operator's psychological state in real time and adjusts the work based on this information,

[0916] A system that includes this.

[0917] (Claim 2)

[0918] The system according to claim 1, characterized in that the imaging device has a function of adding location identification information and imaging time information to the captured image.

[0919] (Claim 3)

[0920] The system according to claim 1, characterized in that the machine learning image analysis device has the function of processing multiple images to identify infrastructure defects on a pixel-by-pixel basis. [Explanation of Symbols]

[0921] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Photography methods targeting infrastructure, An artificial intelligence image analysis method that analyzes captured images to identify defects, A deterioration evaluation means for evaluating the deterioration state of a structure based on defect information identified by the artificial intelligence image analysis means, A priority determination means for calculating the priority of repairs based on the deterioration evaluation means, A report generation and notification system that generates an evaluation report and notifies the administrator, A means for automatically creating a schedule for periodic inspections, A system that includes this.

2. The system according to claim 1, characterized in that the shooting means has a function of adding location information and shooting time information to the captured image.

3. The system according to claim 1, characterized in that the artificial intelligence image analysis means has the function of processing multiple images to identify infrastructure defects on a pixel-by-pixel basis.

Images