system

Abstract

Provide a system. [Solution] A means for receiving captured image data; means for preprocessing the received image data; A means for analyzing the preprocessed image data and extracting information about the equipment and wiring; A means to generate optimal construction plans and repair procedures based on the analysis results, A means for transmitting the generated construction plan and procedure manual to a user terminal; means for receiving and analyzing post-construction feedback data; A means to retrain the generative AI based on feedback data, and A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional telecommunications construction work, information on facilities and wiring had to be checked manually, and construction plans created based on the results, requiring a great deal of time and effort. Furthermore, human error could lead to a decline in quality, or inappropriate construction plans could result in rework. To solve these problems, a system combining image recognition functions and generative AI was needed to improve construction efficiency and quality. [Means for solving the problem]

[0005] The present invention is a system that includes a means for receiving captured image data, a means for preprocessing the received image data, a means for analyzing the preprocessed image data and extracting information about facilities and wiring, a means for generating optimal construction plans and repair procedures based on the analysis results, a means for transmitting the generated construction plans and procedures to a user terminal, a means for receiving and analyzing feedback data after construction, and a means for relearning the generating AI based on the feedback data. This system can improve the quality and reduce the labor required for telecommunications construction work.

[0006] "Means for receiving image data" refers to a function that enables the server to receive image data sent from the user terminal.

[0007] "Preprocessing means" refers to a function that performs processing to improve the accuracy of image recognition, such as removing noise from received image data and adjusting resolution.

[0008] "Means for analyzing" refers to the ability to analyze pre-processed image data and execute image recognition algorithms to extract information about equipment and wiring.

[0009] "Means of generation" refers to the function of using generative AI to generate optimal construction plans and repair procedures based on the analysis results.

[0010] "Transmission means" refers to a function for transmitting the generated construction plan and procedure manual to the user terminal.

[0011] The "means for receiving feedback data" refers to a function for receiving image data and work reports showing the on-site conditions after construction.

[0012] "Means for analysis (feedback data)" refers to the function for analyzing received feedback data and evaluating the accuracy of the construction plan and areas for improvement.

[0013] "Means for relearning" refers to a function that allows the generation AI to relearn based on the analysis results of feedback data, thereby improving the accuracy of the next plan generation. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention is a system that aims to improve the quality of telecommunications construction and reduce the man-hours required, and combines image recognition functions with generative AI. This system performs the following program processing.

[0036] System Program Processing

[0037] 1. Take and upload an image

[0038] Device: The user takes pictures of the equipment and wiring being worked on at the telecommunications construction site using a smartphone or tablet device and uploads the image data to the server. This image data includes the inside of the distribution board, cable connection points, etc.

[0039] 2. Image Reception and Preprocessing

[0040] Server: Receives the uploaded image data and pre-processes it. Pre-processing includes image noise reduction and resolution adjustment, which improves the accuracy of the image recognition algorithm.

[0041] 3. Image Recognition Processing

[0042] Server: Analyzes the pre-processed image data and extracts information about the equipment and wiring. The server uses image recognition algorithms to identify the type, location, and defects of the wiring. For example, it automatically identifies the color code and connection location of the wiring.

[0043] 4. Generate construction plans

[0044] Server: Generates optimal construction plans and repair procedures based on the image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans that detail areas where wiring changes are required and repair procedures.

[0045] 5. Submit and confirm your plan

[0046] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[0047] Terminal: The user checks the construction plan sent to them and carries out the work according to the instructions.

[0048] 6. Construction implementation

[0049] User: The user carries out the actual construction work based on the construction plan provided. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[0050] 7. Uploading Feedback Data

[0051] Terminal: After the work is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[0052] 8. Receiving and Analyzing Feedback

[0053] Server: Receives feedback data after construction and analyzes it again. The feedback data includes information on changes in condition before and after construction and the appropriateness of the construction plan.

[0054] 9. Retraining generative AI

[0055] Server: Based on the analysis results of the feedback data, the generation AI is retrained. This improves the accuracy of the generation AI, which is then used to generate the next construction plan.

[0056] Specific examples

[0057] Case: Adding new wiring

[0058] 1. Take and upload an image

[0059] Terminal: A user takes a picture of an equipment rack in the office with their smartphone and uploads it to the server.

[0060] 2. Image Reception and Preprocessing

[0061] Server: Preprocesses the received image data by removing noise and adjusting the resolution.

[0062] 3. Image Recognition Processing

[0063] Server: Analyzes the preprocessed image and identifies the current wiring status and available ports.

[0064] 4. Generate construction plans

[0065] Server: Based on the image recognition results, the optimal layout of new wiring is generated and specific wiring procedures are compiled into a PDF.

[0066] 5. Submit and confirm your plan

[0067] Server: Sends the completed instructions to the device.

[0068] Terminal: The user checks the installation plan sent to them and carries out the new wiring work.

[0069] 6. Construction implementation

[0070] User: Create new wiring according to a user-generated plan.

[0071] 7. Uploading Feedback Data

[0072] Terminal: After the installation is complete, take a picture of the completed wiring and upload it to the server.

[0073] 8. Receiving and Analyzing Feedback

[0074] Server: Receives and analyzes the uploaded feedback data.

[0075] 9. Retraining generative AI

[0076] Server: The generation AI is retrained based on the results of the feedback analysis, which helps improve the accuracy of the next plan generation.

[0077] This system will enable improved quality and reduced labor costs in telecommunications construction work.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[0081] Step 2:

[0082] Terminal: The captured image data is uploaded from the terminal to the server via wireless communication or wired connection.

[0083] Step 3:

[0084] Server: The server stores the received image data and prepares it for further processing.

[0085] Step 4:

[0086] Server: Performs pre-processing of image data. Pre-processing includes image noise reduction, contrast adjustment, and resolution adjustment, which improves the accuracy of image recognition algorithms.

[0087] Step 5:

[0088] Server: The preprocessed image data is input into an image recognition algorithm to extract information about the equipment and wiring. Identified elements include the device model number, wiring type, connected port location, and defective parts.

[0089] Step 6:

[0090] Server: Analyzes image recognition results and generates optimal construction plans and repair procedures based on the current situation. Generative AI optimizes this process using past data and best practices.

[0091] Step 7:

[0092] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[0093] Step 8:

[0094] Terminal: The user checks the construction plan and procedure manual received and carries out the work according to the displayed procedure.

[0095] Step 9:

[0096] Terminal: After the construction is completed, the user takes another image of the site and uploads it to the server along with a construction completion report.

[0097] Step 10:

[0098] Server: Receives post-construction feedback data, pre-processes and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[0099] Step 11:

[0100] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[0101] This series of steps will improve the quality and reduce the amount of work required in telecommunications construction.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] Telecommunications construction projects require improved quality and reduced man-hours. However, manual on-site inspections and construction plan creation are time-consuming and labor-intensive, and there is a high possibility of human error. Furthermore, quality checks and follow-ups after construction are often insufficient, making them ineffective at improving quality. Furthermore, there is a lack of systems with re-learning capabilities, making it difficult to accumulate experience and knowledge.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information, means for generating optimal plans and procedures based on the analysis results, means for transmitting the generated plans and procedures to a user terminal, means for receiving and analyzing feedback data after construction, and means for relearning the generation algorithm based on the feedback data. This makes it possible to improve the quality and reduce the labor required for telecommunications construction work.

[0107] The "receiving means" is a function for receiving image data transmitted from a user terminal.

[0108] The "preprocessing means" is a function for removing noise and adjusting resolution from received image data.

[0109] The "means for analyzing" is a function that applies an algorithm to extract specific information from the pre-processed image data.

[0110] "Means for generation" refers to a function that automatically generates optimal plans and procedures based on the analysis results.

[0111] "Transmission means" is a function for sending the generated plan or procedure manual to the user's terminal.

[0112] The "means for receiving and analyzing" is a function for receiving feedback data sent by the user after implementation and analyzing it again.

[0113] The "means for relearning" is a function that improves the generation algorithm based on the analysis results and increases the accuracy of the next generation.

[0114] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[0115] "Server" means a computer system for processing and storing data.

[0116] An "image recognition algorithm" is an analytical method for extracting specific information from image data.

[0117] A "generative algorithm" is an algorithm for generating new plans and procedures based on extracted information.

[0118] "Feedback data" refers to data on the condition of the site and the results of construction provided by the user after construction.

[0119] This invention is a system that aims to improve the quality and reduce the man-hours required for telecommunications construction work, and combines image recognition functions with generative AI. This system realizes efficient construction plan generation and feedback at construction sites through collaboration between users, terminals, and servers.

[0120] First, a user uses a device such as a smartphone or tablet at a telecommunications construction site. The user takes pictures of important locations at the construction site, such as distribution boards and cable connection points. Specific examples of devices used include smartphones and tablets. This makes it possible to accurately record the site conditions. The user then uploads the captured image data to a server using a dedicated application.

[0121] The server then receives the uploaded image data. Specific software used on the server includes the OpenCV library for image preprocessing and Tensorflow (registered trademark) and PyTorch for image recognition algorithms. The received image data undergoes preprocessing such as noise removal and resolution adjustment. This preprocessing step improves the quality of the image and increases the accuracy of subsequent analysis.

[0122] Once preprocessing is complete, the server uses deep learning to analyze the image data. The goal of the analysis is to identify the type, location, and defect of the wiring. For example, it can automatically identify the color code and connection location of the wiring.

[0123] Furthermore, the server uses generative AI (specifically, the GPT-4 (registered trademark) model, etc.) based on the image recognition results to generate optimal construction plans and repair procedures. These plans include efficient work procedures and lists of necessary tools and materials. The server converts the generated construction plans and procedures into PDF or image format and sends them to the user's device. The user then checks them and carries out the work according to the instructions.

[0124] After the construction is completed, the user takes another image of the wiring status and uploads it to the server again. The server receives the re-uploaded feedback data and re-analyzes it. By re-training the generation AI based on the analysis results, it is expected that the accuracy of the next construction plan generation will improve.

[0125] Specific examples

[0126] For example, consider the case of adding new wiring to an equipment rack in an office. A user takes a picture of the equipment rack with their smartphone and uploads it to a server. The server receives the image and performs preprocessing. It then uses an image recognition algorithm to identify the wiring status and available ports. Based on the image recognition results, the server uses generative AI to propose the optimal layout for the new wiring, generates a PDF with specific wiring procedures, and sends it to the user. The user performs the wiring work according to the received plan, and after completing the work, takes another picture and uploads the feedback data. The server reanalyzes this data and retrains the generative AI to improve the accuracy of the next plan generation.

[0127] Prompt Sentence Examples

[0128] "Propose the optimal placement for new cabling. Upload an image of your office equipment rack. Identify the current cabling situation and open ports, and generate an installation plan with efficient cabling procedures. Analyze the image below."

[0129] In this way, the present invention provides a concrete method for improving quality and reducing labor costs in telecommunications construction. By utilizing generative AI models and image recognition technology, efficient and accurate construction becomes possible.

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1:

[0132] Taking and uploading images

[0133] Device: The user takes pictures of wiring and equipment at a telecommunications construction site using a smartphone or tablet. When taking pictures, they record important details such as the inside of the wiring board and cable connection points. Specifically, the device uses the built-in camera app to take photos, and then uploads the captured image data to a server using a dedicated app. The input is the captured image data, and the output is an image file sent to the server.

[0134] Step 2:

[0135] Image reception and preprocessing

[0136] Server: The server receives the uploaded image data. It uses OpenCV and other tools to remove noise and adjust the resolution of the received image data to improve the image quality. Specific operations include smoothing and sharpening the image. The input is the uploaded image data, and the output is high-quality image data that has undergone preprocessing.

[0137] Step 3:

[0138] Image Recognition Processing

[0139] Server: Analyzes preprocessed image data using a deep learning model (for example, TensorFlow or PyTorch). Identifies the type, location, and defective areas of the wiring from the image. Specific operations include color code recognition and object detection. The input is preprocessed image data, and the output is the type and location of the wiring based on the analysis results, as well as the identification of defective areas.

[0140] Step 4:

[0141] Construction plan generation

[0142] Server: Based on the image recognition results, a generative AI (e.g., a GPT-4 model) is used to generate optimal construction plans and repair procedures. Specifically, it creates construction procedures based on the recognized wiring information. The input is the image recognition results data, and the output is the generation of specific construction plans and procedures. The plans include efficient work procedures, necessary tools, and material lists.

[0143] Step 5:

[0144] Submit and confirm your plan

[0145] Server: Converts the generated construction plans and procedures into PDF or image format and sends them to the user's device. Specific operations include converting the file format and sending it via email or a dedicated application. The input is the construction plan or procedure data, and the output is a file sent to the user's device.

[0146] Terminal: The user checks the construction plan received on the terminal. Specifically, the user opens the received PDF file and checks the construction procedures. At this time, the user prepares the tools and materials necessary for construction. The input is the document data sent from the server, and the output is that construction preparations are complete.

[0147] Step 6:

[0148] Construction implementation

[0149] User: The user carries out the work based on the presented construction plan. Specific actions include connecting cables as instructed and repairing any defects. Once the work is complete, the user takes another photo of the completed equipment and wiring. The input is the construction plan information, and the output is image data of the completed wiring.

[0150] Step 7:

[0151] Feedback data upload

[0152] Terminal: After the construction is completed, the user uploads the retaken image data and implementation report to the server using a dedicated application. Specific operations include selecting and sending the retaken image file. The input is the image data after the construction is completed and the implementation report, and the output is the feedback data uploaded to the server.

[0153] Step 8:

[0154] Receiving and analyzing feedback

[0155] Server: The server receives the uploaded feedback data and performs the analysis again. Specific operations include comparing images before and after construction and evaluating the appropriateness of the construction plan. The input is the feedback data, and the output is the analysis results.

[0156] Step 9:

[0157] Retraining generative AI

[0158] Server: The server retrains the generative AI based on the analysis results of the feedback data. Specific operations include adjusting the parameters of the generative model and adding new data sets. The input is the feedback analysis results, and the output is a retrained generative AI model. This improves the accuracy of the next construction plan generation.

[0159] (Application example 1)

[0160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] Providing efficient and appropriate maintenance procedures for robots in operation in factories is difficult. In particular, it is necessary for on-site technicians to accurately assess the robot's condition and quickly perform necessary maintenance work. Another challenge is that there is no established method for utilizing feedback after maintenance to improve future maintenance plans.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0163] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about the facility or equipment, means for generating an optimal maintenance plan or repair procedure based on the analysis results, means for transmitting the generated maintenance plan or procedure manual to a user terminal, means for receiving and analyzing feedback data after construction, and means for relearning the generating AI based on the feedback data. This enables the provision of efficient maintenance procedures for robots in factories and continuous improvement based on subsequent feedback.

[0164] "Image data" refers to photographed image information that records the status of robots and equipment in the factory.

[0165] "Preprocessing" refers to data processing operations such as noise removal and resolution adjustment that are performed to make the received image data easier to analyze.

[0166] "Analysis" refers to the process of extracting specific information from preprocessed image data and using that information to evaluate the condition of facilities and equipment.

[0167] "Facility" refers to all the devices and systems used within a factory.

[0168] "Equipment" refers to specific devices and tools in a factory, including robots.

[0169] A "maintenance plan" refers to a procedure manual for repairing and maintaining facilities and equipment, created based on extracted information.

[0170] "Repair procedure" refers to the specific repair methods and processes used for equipment and facilities.

[0171] "User terminal" refers to a mobile terminal such as a smartphone or tablet used by a field technician.

[0172] "Feedback data" refers to information such as images taken after maintenance work and work reports.

[0173] "Analysis results" refers to the analysis results of information and data extracted by image recognition algorithms.

[0174] "Generative AI" refers to artificial intelligence that automatically generates optimal maintenance procedures and plans based on assumptions and data.

[0175] This invention is a system that provides efficient maintenance procedures for robots operating in factories. Specifically, it generates an optimal maintenance plan based on captured image data, and then uses subsequent feedback data to improve the accuracy of the generating AI.

[0176] System Hardware

[0177] This system uses the following hardware:

[0178] 1. Smartphone or tablet device: The field technician takes a photo of the robot's status and uploads the image data to the server.

[0179] 2. Server: Receives image data, pre-processes, analyzes, generates maintenance plans, and retrains feedback data.

[0180] System Software

[0181] The following software is used:

[0182] 1. Image recognition algorithms: (TensorFlow, OpenCV) are used to analyze image data and identify the status of facilities and equipment.

[0183] 2. Generative AI: Uses (GPT-4) to generate an optimal maintenance plan based on the analysis results.

[0184] 3. Database: (MySQL (registered trademark), PostgreSQL) is used to manage image data and feedback information.

[0185] System operation explanation

[0186] 1. Take and upload an image

[0187] An engineer uses a smartphone to take a photo of the robot, for example, the joints of the robot arm, and uploads the image to a server using a dedicated app.

[0188] 2. Receiving and preprocessing image data

[0189] The server performs noise reduction and resolution adjustment on the received image data, using TensorFlow to improve the image quality.

[0190] 3. Image Recognition Processing

[0191] The preprocessed image data is analyzed using OpenCV to identify robot parts and defects, for example, to identify the degree of wear on joints.

[0192] 4. Generate a maintenance plan

[0193] Using GPT-4, a generative AI, it generates a detailed maintenance plan based on the recognition results, including specific steps and a list of required tools.

[0194] 5. Submitting and implementing the plan

[0195] The server sends the generated maintenance plan in PDF format to the technician's terminal, where the technician checks the plan and performs the actual maintenance work.

[0196] 6. Uploading feedback data

[0197] After completing the maintenance, the technician takes another photo of the robot's condition and uploads it to the server as feedback data, allowing the technician to check the results of the maintenance.

[0198] 7. Feedback analysis and retraining of the generative AI

[0199] The server re-analyzes the received feedback data and extracts useful information, which allows the generating AI to re-train and improve the accuracy of the next maintenance plan.

[0200] Specific examples

[0201] Case: Robot arm joint maintenance

[0202] 1. Take and upload an image

[0203] Terminal: A technician takes pictures of the robot arm's joints and uploads them to the server.

[0204] 2. Receiving and preprocessing image data

[0205] Server: Perform noise removal and resolution adjustment using TensorFlow.

[0206] 3. Image Recognition Processing

[0207] Server: Uses OpenCV to identify the degree of wear on the joints.

[0208] 4. Generate a maintenance plan

[0209] Server: Uses GPT-4 to generate replacement procedures for worn parts and a list of required tools.

[0210] 5. Submitting and implementing the plan

[0211] Server: The generated maintenance plan is sent to the technician in PDF format, who then performs the work according to the plan.

[0212] 6. Uploading feedback data

[0213] Terminal: After maintenance is completed, an image of the robot joints is taken and uploaded to the server.

[0214] 7. Feedback analysis and retraining of the generative AI

[0215] Server: Analyzes the feedback data sent and retrains GPT-4.

[0216] Example of input prompt for generative AI model:

[0217] Please generate detailed instructions for maintaining the joints of the robot arm below.

[0218] Prerequisites:

[0219] Wear on the joints was observed.

[0220] The tools available are a wrench, screwdriver, and lubricant.

[0221] Replacement parts are available in advance.

[0222] Instructions:

[0223] 1. Safety measures procedures

[0224] 2. Specific replacement procedures for worn parts

[0225] 3. List of tools and materials needed

[0226] 4. Final check step

[0227] This enables proper and efficient maintenance of robots in factories.

[0228] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0229] Step 1:

[0230] Taking and uploading images

[0231] The terminal provides a means for users to take images of specific parts of the robot, such as joints. The captured image data is uploaded to a server via a dedicated app. The input is the captured image data, and the output is the upload of the image data to the server.

[0232] Step 2:

[0233] Receiving and preprocessing image data

[0234] The server performs noise reduction and resolution adjustment on the received image data. Specifically, it uses TensorFlow to improve the image quality. The input is the image data sent from the user device, and the output is the preprocessed image data.

[0235] Step 3:

[0236] Image Recognition Processing

[0237] The server analyzes the preprocessed image data using OpenCV to identify robot parts and defects. For example, it identifies the degree of wear on joints. The input is the preprocessed image data, and the output is the analyzed specific information (e.g., the location and degree of wear).

[0238] Step 4:

[0239] Generate a maintenance plan

[0240] The server uses GPT-4 as a generation AI to generate a detailed maintenance plan based on the analysis results. The generated plan includes specific procedures and a list of required tools. The input is the analyzed specific information, and the output is the maintenance plan.

[0241] Step 5:

[0242] Submitting and implementing the plan

[0243] The server sends the generated maintenance plan in PDF format to the user's terminal. The user checks the received plan and performs maintenance work based on it. The input is the maintenance plan, and the output is the procedure checked by the user.

[0244] Step 6:

[0245] Feedback data upload

[0246] After the maintenance is completed, the terminal takes another photograph of the robot's condition and uploads it to the server as feedback data. The input is the image data after maintenance, and the output is the feedback data uploaded to the server.

[0247] Step 7:

[0248] Feedback analysis and retraining of the generation AI

[0249] The server re-analyzes the received feedback data and extracts useful information, which re-trains the generative AI and improves the accuracy of the next maintenance plan. The input is the feedback data, and the output is the re-trained generative AI model.

[0250] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0251] This invention is a system that aims to improve the quality and reduce the man-hours of telecommunications construction work, and combines image recognition functions with generative AI. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more appropriate method for presenting construction plans is realized. This system performs the following program processing.

[0252] System Program Processing

[0253] 1. Take and upload an image

[0254] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[0255] 2. Image Reception and Preprocessing

[0256] Server: Receives the uploaded image data and pre-processes it. Pre-processing includes image noise reduction and resolution adjustment, which improves the accuracy of the image recognition algorithm.

[0257] 3. Image Recognition Processing

[0258] Server: The pre-processed image data is input into an image recognition algorithm to extract information about the equipment and wiring. The server uses the image recognition algorithm to identify the type, location, and defects of the wiring. For example, it automatically identifies the color code and connection location of the wiring.

[0259] 4. Generate construction plans

[0260] Server: Generates optimal construction plans and repair procedures based on the image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans that detail areas where wiring changes are required and repair procedures.

[0261] 5. User Emotion Recognition

[0262] Device: While the user is reviewing the construction plan or procedure manual, the built-in camera captures the user's facial expression, and the captured facial image is sent to the emotion engine.

[0263] Emotion engine: Analyzes facial images to identify the user's emotional state, for example, detecting when the user is confused or frustrated.

[0264] 6. Adjusting the way plans are presented

[0265] Server: Based on the user's emotional state, adjust the presentation of the generated construction plans and procedures, for example by providing more detailed explanations or additional visual aids to facilitate user understanding.

[0266] 7. Construction implementation

[0267] User: The user carries out the actual construction work based on the adjusted construction plan. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[0268] 8. Uploading Feedback Data

[0269] Terminal: After the work is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[0270] 9. Receiving and Analyzing Feedback

[0271] Server: Receives post-construction feedback data, pre-processes it, and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[0272] 10. Retraining generative AI

[0273] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[0274] Specific examples

[0275] Case: Adding new wiring

[0276] 1. Take and upload an image

[0277] Terminal: A user takes a picture of an equipment rack in the office with their smartphone and uploads it to the server.

[0278] 2. Image Reception and Preprocessing

[0279] Server: Preprocesses the received image data by removing noise and adjusting the resolution.

[0280] 3. Image Recognition Processing

[0281] Server: Analyzes the preprocessed image and identifies the current wiring status and available ports.

[0282] 4. Generate construction plans

[0283] Server: Based on the image recognition results, the optimal layout of new wiring is generated and specific wiring procedures are compiled into a PDF.

[0284] 5. User Emotion Recognition

[0285] Device: As the user reviews the construction plan, the built-in camera captures the user's facial expression.

[0286] Emotion engine: Analyzes captured facial expressions to identify when the user is confused.

[0287] 6. Adjusting the way plans are presented

[0288] Server: Provide construction plans with more detailed instructions and additional visual aids to resolve user confusion.

[0289] 7. Construction implementation

[0290] User: Carry out wiring work based on the coordinated plan.

[0291] 8. Uploading Feedback Data

[0292] Terminal: After the installation is complete, take a picture of the completed wiring and upload it to the server.

[0293] 9. Receiving and Analyzing Feedback

[0294] Server: Analyzes the uploaded feedback data and identifies issues and areas for improvement.

[0295] 10. Retraining generative AI

[0296] Server: Retrains the generation AI based on the feedback to improve the accuracy of the next construction plan generation.

[0297] This series of steps utilizes an emotion engine to deepen user understanding and ensure high-quality construction.

[0298] The processing flow will be explained below.

[0299] Step 1:

[0300] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[0301] Step 2:

[0302] Terminal: The captured image data is uploaded from the terminal to the server via wireless communication or wired connection.

[0303] Step 3:

[0304] Server: The server stores the received image data and prepares it for further processing.

[0305] Step 4:

[0306] Server: Performs pre-processing of image data. Pre-processing includes image noise reduction, contrast adjustment, and resolution adjustment, which improves the accuracy of image recognition algorithms.

[0307] Step 5:

[0308] Server: The preprocessed image data is input into an image recognition algorithm to extract information about the equipment and wiring. Identified elements include the type, location, and defect of the wiring. For example, the color code and connection location of the wiring can be automatically identified.

[0309] Step 6:

[0310] Server: Generates optimal construction plans and repair procedures based on image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans detailing areas where wiring changes are required and repair procedures.

[0311] Step 7:

[0312] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[0313] Step 8:

[0314] Device: When the user checks the construction plan or procedure manual they received, the built-in camera captures their facial expression.

[0315] Step 9:

[0316] Emotion engine: Analyzes captured facial images to identify the user's emotional state, for example, detecting when the user is confused or dissatisfied.

[0317] Step 10:

[0318] Server: Based on the user's emotional state, adjust the presentation of the generated construction plans and procedures, for example by providing more detailed explanations or additional visual aids to facilitate user understanding.

[0319] Step 11:

[0320] User: The user carries out the actual construction work based on the adjusted construction plan. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[0321] Step 12:

[0322] Terminal: After the construction is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[0323] Step 13:

[0324] Server: Receives post-construction feedback data, pre-processes and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[0325] Step 14:

[0326] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[0327] This series of steps utilizes an emotion engine to deepen user understanding and ensure high-quality construction.

[0328] Example 2

[0329] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0330] In conventional telecommunications construction, ensuring the quality and efficiency of construction required a great deal of time and effort. In particular, there were many uncertainties in the appropriateness of construction plans and the presentation of repair procedures according to the on-site situation, and the progress of construction was affected by the user's own level of understanding and emotional state. This resulted in issues such as inconsistent construction quality, increased man-hours, and reduced user satisfaction.

[0331] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0332] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about facilities and wiring, means for generating an optimal construction plan and repair procedure based on the analysis results, means for transmitting the generated construction plan and procedure manual to a user terminal, means for recognizing the user's emotions during construction, means for adjusting the presentation method of the construction plan and procedure manual based on the recognized emotions, means for receiving and analyzing feedback data after construction, and means for relearning the generation AI based on the feedback data. This makes it possible to automatically generate an optimal construction plan based on image data of the construction site and flexibly adjust it according to the user's emotional state, thereby improving the quality and efficiency of construction.

[0333] "Photographed image data" refers to an image file photographed by a user at a construction site, and contains information including the state of wiring and equipment at the construction site.

[0334] The "receiving means" is a mechanism by which the server retrieves image data sent from the user terminal via the Internet.

[0335] The "preprocessing means" is a step in which the received image data is subjected to processes such as noise removal and resolution adjustment to improve the accuracy of the image recognition algorithm.

[0336] "Means for extracting information" refers to algorithms or techniques that analyze the pre-processed image data and identify relevant information about wiring and equipment.

[0337] The "means of generation" is a system that designs optimal construction plans and repair procedures based on the analysis results and presents them in document or digital format.

[0338] "Transmission means" refers to a method for transferring the construction plan and procedure manual generated by the server to the user's terminal.

[0339] The "means for recognizing emotions" is a technology that captures the user's facial expressions during construction and identifies the user's emotional state from the facial expression data.

[0340] The "means for adjusting the presentation method" is a mechanism for changing the content and presentation method of construction plans and procedures based on the emotion recognition results, thereby improving the user's understanding and satisfaction.

[0341] "Feedback data" refers to information on the results of construction and usage experience provided by users after construction.

[0342] "Means for receiving and analyzing" refers to a technology in which the server receives feedback data and uses that data to evaluate the performance of the construction plan and system.

[0343] "Relearning means" refers to a method for updating the generative AI model based on analyzed feedback data to improve the accuracy and efficiency of the next construction plan generation.

[0344] A "generative AI model" is an artificial intelligence model used to automatically generate construction plans and repair procedures, and includes natural language generation models such as GPT-4.

[0345] This invention is a system aimed at improving the quality and reducing the man-hours required for telecommunications construction. This system combines image recognition, generative AI, and an emotion engine that recognizes the user's emotions.

[0346] First, the user takes images of the construction site using a device (e.g., a smartphone or tablet). The images taken include the inside of a wiring board and cable connection points. The user then uses a dedicated application to upload the captured image data to a server. This upload requires an internet connection (Wi-Fi or mobile data communication).

[0347] The server uses the OpenCV library to preprocess the received image data, which includes image noise reduction (e.g., applying a Gaussian filter) and resolution adjustment (e.g., resizing). This preprocessing improves the accuracy of the image recognition algorithms used later.

[0348] The preprocessed image data is then fed into an image recognition algorithm using the TensorFlow library, which the server uses to extract information about the equipment and wiring, such as color codes, connection locations, and fault locations.

[0349] Based on the extracted information, the server uses generative AI (e.g., a natural language generation model such as GPT-4) to generate optimal construction plans and repair procedures. The generated plans are output in PDF format and transferred to the user's device. The user can refer to these plans to proceed with the construction work.

[0350] Additionally, while the user is reviewing the construction plan, the device's built-in camera captures the user's facial expression. This facial expression data is sent to an emotion engine (e.g., Emotion API), which then analyzes the user's emotional state on the server. Based on the analysis results, for example, if the user is confused, the server adjusts the way the construction plan is presented. Specifically, it provides detailed explanations or additional visual guides.

[0351] After the construction is completed, the user takes another image of the construction site and uploads it to the server as feedback data. The server receives and analyzes this feedback data. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[0352] Finally, the server retrains the generative AI model based on this feedback data, helping to improve the accuracy of the next construction plan generation.

[0353] As a concrete example, consider the task of adding new wiring. A user takes a picture of an equipment rack in their office with their smartphone and uploads it to the server. The server preprocesses and analyzes the image data to identify the current wiring status and available ports. It then compiles the generated construction plan into a PDF and provides it to the user. If the server determines that the user's facial expression is confused while reviewing the construction plan, it re-provides the plan with detailed explanations and additional visual materials. The user then carries out the wiring work based on the adjusted plan and uploads an image of the completed work. The server analyzes the feedback data and retrains the generating AI to improve the accuracy of the next construction plan.

[0354] Example prompt sentence:

[0355] "Server: Upload images of your telecommunications construction site. We will use generative AI to provide you with the optimal construction plan."

[0356] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0357] Step 1:

[0358] Taking and uploading images

[0359] User: The user uses a smartphone or tablet to take detailed images of the construction site, such as the inside of a wiring cabinet or cable connection points, carefully capturing them from multiple angles.

[0360] Device: Upload the captured image data to the server using a dedicated application (e.g., a mobile app). When you press the "Upload Image" button in the application, the image data is sent to the server via an internet connection.

[0361] Input: Construction site image data

[0362] Output: Image data uploaded to the server

[0363] Step 2:

[0364] Image reception and preprocessing

[0365] Server: The server receives the image data sent by the user. For example, a REST API using the Flask framework is used to receive the data.

[0366] Server: Preprocesses the received image data using the OpenCV library, applying a Gaussian filter to remove noise and resizing the image to the desired resolution.

[0367] Input: Uploaded image data

[0368] Output: Preprocessed image data

[0369] Step 3:

[0370] Image Recognition Processing

[0371] Server: The preprocessed image data is input to the image recognition algorithm using the TensorFlow library.

[0372] Server: Image recognition algorithms automatically extract information about wiring and equipment, such as the type of wiring, its location, color code, and any defects.

[0373] Input: Preprocessed image data

[0374] Output: Extracted information on facilities and wiring

[0375] Step 4:

[0376] Construction plan generation

[0377] Server: Based on the extracted information, a generative AI model (e.g., GPT-4) is used to generate optimal construction plans and repair procedures.

[0378] Server: The generated construction plans and repair procedures are documented in PDF format and provided to users in an easy-to-understand format.

[0379] Input: Image recognition results (extracted information on facilities and wiring)

[0380] Output: PDF document of construction plan

[0381] Step 5:

[0382] User Emotion Recognition

[0383] Device: While the user is viewing the construction plan, the built-in camera captures the user's facial expression.

[0384] Emotion Engine: The captured facial image is sent to the emotion engine, which analyzes it to identify the user's emotional state (e.g., confusion, frustration, relief).

[0385] Input: User's face image

[0386] Output: User's emotional state

[0387] Step 6:

[0388] Adjusting the way plans are presented

[0389] Server: Based on the emotion recognition results, it adjusts the way the construction plan is presented, for example, by providing detailed explanations or additional visual guides if the user is confused.

[0390] Server: The adjusted construction plan is compiled into a PDF again and resent to the user's device.

[0391] Input: User's emotional state

[0392] Output: PDF document of the coordinated construction plan

[0393] Step 7:

[0394] Construction implementation

[0395] User: Carries out work on-site based on the coordinated construction plan, for example installing new wiring or carrying out repairs as instructed.

[0396] User: Once the construction is complete, take another image of the completed state and upload it to the server as feedback data.

[0397] Input: Coordinated construction plan PDF document

[0398] Output: Completed image data and feedback

[0399] Step 8:

[0400] Feedback data upload

[0401] Terminal: After the construction is completed, the user uploads the image data and work report to the server.

[0402] Terminal: Press the "Report Completion" button in the dedicated application to send the data.

[0403] Input: Completed image data and feedback

[0404] Output: Feedback data uploaded to the server

[0405] Step 9:

[0406] Receiving and analyzing feedback

[0407] Server: Receives the feedback data and preprocesses it again.

[0408] Server: Analyzes the feedback and identifies the accuracy of the construction plan, the suitability of the execution, and areas for improvement.

[0409] Input: Uploaded feedback data

[0410] Output: Feedback analysis results

[0411] Step 10:

[0412] Retraining generative AI

[0413] Server: Based on the analysis results, the generative AI model is retrained to improve the accuracy of the next construction plan generation. For retraining, new training data including feedback data is used.

[0414] Input: Feedback analysis results

[0415] Output: An updated generative AI model

[0416] (Application example 2)

[0417] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0418] Equipment repair and maintenance work within factories requires accurate understanding of on-site conditions and the ability to provide repair plans quickly and efficiently. To improve worker understanding and efficiency, it is important not only to provide technical instructions but also to respond flexibly while taking into account the worker's emotional state. While conventional systems use image recognition for automation and generative AI for planning, they lack the ability to adjust based on the worker's emotions, making it difficult to achieve both improved work efficiency and quality.

[0419] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0420] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about the equipment and structure, means for generating an optimal construction plan and repair procedure based on the analysis results, means for transmitting the generated construction plan and procedure manual to a user terminal, means for recognizing the user's emotional state, means for adjusting the presentation method of the construction plan based on the emotional state, means for receiving and analyzing feedback data after construction, and means for relearning the generation AI based on the feedback data. This makes it possible to improve the worker's understanding, increase the efficiency of repair work, and improve the quality of the entire construction work.

[0421] "Photographed image data" refers to digital data of on-site images of facilities and structures photographed by a device such as a user terminal or a robot.

[0422] "Means for receiving" refers to a function or device for receiving image data sent from a user terminal or robot via a network.

[0423] "Preprocessing means" refers to software and processes for performing preprocessing such as noise removal and resolution adjustment on received image data.

[0424] "Means for analyzing" refers to image recognition algorithms and processes for extracting facility or structural information from pre-processed image data.

[0425] "Means of generation" refers to the generative AI model and process for creating optimal construction plans and repair procedures based on the analysis results.

[0426] "Transmission means" refers to a function or device for transmitting the generated construction plan or procedure manual to a user terminal.

[0427] "Means for recognizing the user's emotional state" refers to an emotion engine and its process for analyzing the user's facial expression image captured by the built-in camera and identifying the user's emotional state.

[0428] "Adjustment means" refers to a function or device for changing the presentation method of the generated construction plan based on the user's emotional state.

[0429] "Feedback data" refers to data related to work results, such as on-site images taken after construction and implementation reports.

[0430] "Means for re-learning" refers to the process of re-learning the generative AI model based on feedback data to improve the accuracy of the next construction plan generation.

[0431] To implement the present invention, the following system configuration is required.

[0432] Hardware Configuration

[0433] Robot body: A mobile robot used in factories, equipped with built-in cameras and sensors.

[0434] User device: A smartphone or tablet device with a built-in camera that communicates with the robot.

[0435] Server: A central server that processes and stores all data.

[0436] Software Configuration

[0437] Image recognition algorithms: YOLO and OpenCV are used to preprocess the received image data and extract information about the facilities and structures.

[0438] Generative AI model: Uses OpenAI® GPT-4 and Google® BERT. Generates optimal construction plans and repair procedures based on analysis results.

[0439] Emotion Engine: Using Emotion API and Affectiva, it recognizes the user's emotional state and adjusts how construction plans are presented.

[0440] Cloud server: Amazon Web Services (AWS(registered trademark)) or Google Cloud is used to process and store data.

[0441] System processing flow

[0442] The server mainly performs the following processes:

[0443] 1. Image Reception and Preprocessing

[0444] The server receives image data sent from user terminals and robots, and performs preprocessing such as noise removal and resolution adjustment on the received image data.

[0445] 2. Image Recognition Processing

[0446] The pre-processed image data is then fed into an image recognition algorithm to extract information about the equipment or structure, such as identifying abnormalities or broken areas on a conveyor belt.

[0447] 3. Creation of construction plans

[0448] Based on the extracted information, a generative AI model is used to generate optimal repair procedures, including detailed descriptions of work procedures when using related labor-saving devices.

[0449] 4. Emotion recognition and presentation adjustment

[0450] While the user is reviewing the construction plan, the built-in camera on the user's device captures the user's facial expression and sends it to the emotion engine. Based on the emotion recognition results, the way the construction plan is presented is adjusted to make it easier for the user to understand.

[0451] 5. Receiving and Analyzing Feedback

[0452] After construction is completed, feedback data (images and implementation reports) sent from the user device and robot is received and analyzed to determine the accuracy of the construction plan and identify areas for improvement.

[0453] 6. Retraining generative AI

[0454] The generative AI model will be retrained based on the feedback data to improve the accuracy of the next construction plan generation.

[0455] Specific examples

[0456] A specific example is a robot detecting an abnormality in a conveyor belt and repairing it.

[0457] First, the robot photographs any abnormalities on the conveyor belt and uploads the image data to a server. The server then uses an image recognition algorithm to identify the abnormalities and generates optimal repair procedures using a generative AI model. It then uses its built-in camera to recognize the user's emotional state and adjusts the way the procedure manual is presented as needed. After the repair is complete, the robot takes another image and uploads it to the server as feedback data. This data is used to retrain the generative AI model, improving its accuracy the next time.

[0458] Prompt Sentence Examples

[0459] For example, you might input the following prompt to a generative AI model:

[0460] Prompt: "Generate the optimal procedure for repairing the conveyor belt malfunction. Use the image data below as a starting point. [Image data link]. Please add detailed explanations of any parts that may be difficult for the worker."

[0461] In this way, by combining image recognition algorithms, generative AI models, and emotion engines, the present invention achieves improved efficiency and quality in equipment repair and maintenance work within factories.

[0462] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0463] Step 1:

[0464] Take and upload images:

[0465] The user terminal and robot take images of the on-site equipment and upload the image data to the server. At this time, the user terminal takes a picture of the problem area with a camera and transfers the captured image file to the server via the network. The input is the on-site image data, and the output is the saving of the image file on the server.

[0466] Step 2:

[0467] Image reception and pre-processing:

[0468] The server receives image data sent from the user device and the robot. After receiving the image data, it performs preprocessing such as noise removal and resolution adjustment on the image data. Specifically, it cleans up the images using an image processing library such as OpenCV. The input is the image file from the user device and the robot, and the output is the preprocessed image data.

[0469] Step 3:

[0470] Image Recognition Processing:

[0471] The server inputs the preprocessed image data into an image recognition algorithm to extract information about the equipment and its structure. For example, YOLO can be used to identify abnormalities in the equipment. The input is the preprocessed image data, and the output is a list of identified abnormalities and equipment information.

[0472] Step 4:

[0473] Generate construction plans:

[0474] The server uses a generative AI model based on the image recognition results to generate optimal construction plans and repair procedures. The plan generation includes specific work procedures and a list of required tools. The input is the image recognition results, and the output is the generated construction plan and procedure manual.

[0475] Step 5:

[0476] Submit your plan:

[0477] The server sends the generated construction plans and procedures to the user's terminal. At this time, the plans and procedures are provided in PDF file or text format. The construction plans and procedures are input, and the data is sent to the user's terminal as output.

[0478] Step 6:

[0479] User Emotion Recognition:

[0480] While the user is checking the construction plan, the device's built-in camera captures the user's facial expression. The captured facial image is sent to the server, which then uses an emotion engine to analyze the user's emotional state. The input is the user's facial image data, and the output is the user's emotional state (confusion, dissatisfaction, etc.).

[0481] Step 7:

[0482] Adjustments to how plans are presented:

[0483] The server adjusts the presentation of the generated construction plans and procedures based on the user's emotional state. For example, if the user is confused, it provides additional visual aids or detailed explanations. The user's emotional state is the input, and the adjusted construction plans and procedures are delivered as the output.

[0484] Step 8:

[0485] Construction implementation:

[0486] The user performs the actual repair work based on the adjusted construction plan. Specifically, the user repairs the equipment using the necessary tools and materials while referring to the generated procedure manual. The input is the adjusted construction plan, and the output is the completed repaired equipment.

[0487] Step 9:

[0488] Feedback data upload:

[0489] After the construction is completed, the user terminal and the robot take images of the site again and upload them to the server as feedback data. The input is the image data after construction, and the output is the feedback data, which is saved on the server.

[0490] Step 10:

[0491] Receiving and analyzing feedback:

[0492] The server receives the feedback data and uses image recognition algorithms to verify and analyze the construction results. Areas for improvement are identified and reflected in the generation of the next construction plan. The input is the feedback data, and the output is the analysis results and a list of areas for improvement.

[0493] Step 11:

[0494] Retraining generative AI:

[0495] The server retrains the generative AI model based on the analysis results to improve the accuracy of the next construction plan generation. The input is the analysis results and points for improvement, and the output is an improved generative AI model.

[0496] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0497] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0498] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0499] [Second embodiment]

[0500] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0501] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0502] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0503] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0504] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0505] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0506] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0507] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0508] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0509] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0510] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0511] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0512] This invention is a system that aims to improve the quality of telecommunications construction and reduce the man-hours required, and combines image recognition functions with generative AI. This system performs the following program processing.

[0513] System Program Processing

[0514] 1. Take and upload an image

[0515] Device: The user takes pictures of the equipment and wiring being worked on at the telecommunications construction site using a smartphone or tablet device and uploads the image data to the server. This image data includes the inside of the distribution board, cable connection points, etc.

[0516] 2. Image Reception and Preprocessing

[0517] Server: Receives the uploaded image data and pre-processes it. Pre-processing includes image noise reduction and resolution adjustment, which improves the accuracy of the image recognition algorithm.

[0518] 3. Image Recognition Processing

[0519] Server: Analyzes the pre-processed image data and extracts information about the equipment and wiring. The server uses image recognition algorithms to identify the type, location, and defects of the wiring. For example, it automatically identifies the color code and connection location of the wiring.

[0520] 4. Generate construction plans

[0521] Server: Generates optimal construction plans and repair procedures based on the image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans that detail areas where wiring changes are required and repair procedures.

[0522] 5. Submit and confirm your plan

[0523] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[0524] Terminal: The user checks the construction plan sent to them and carries out the work according to the instructions.

[0525] 6. Construction implementation

[0526] User: The user carries out the actual construction work based on the construction plan provided. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[0527] 7. Uploading Feedback Data

[0528] Terminal: After the work is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[0529] 8. Receiving and Analyzing Feedback

[0530] Server: Receives feedback data after construction and analyzes it again. The feedback data includes information on changes in condition before and after construction and the appropriateness of the construction plan.

[0531] 9. Retraining generative AI

[0532] Server: Based on the analysis results of the feedback data, the generation AI is retrained. This improves the accuracy of the generation AI, which is then used to generate the next construction plan.

[0533] Specific examples

[0534] Case: Adding new wiring

[0535] 1. Take and upload an image

[0536] Terminal: A user takes a picture of an equipment rack in the office with their smartphone and uploads it to the server.

[0537] 2. Image Reception and Preprocessing

[0538] Server: Preprocesses the received image data by removing noise and adjusting the resolution.

[0539] 3. Image Recognition Processing

[0540] Server: Analyzes the preprocessed image and identifies the current wiring status and available ports.

[0541] 4. Generate construction plans

[0542] Server: Based on the image recognition results, the optimal layout of new wiring is generated and specific wiring procedures are compiled into a PDF.

[0543] 5. Submit and confirm your plan

[0544] Server: Sends the completed instructions to the device.

[0545] Terminal: The user checks the installation plan sent to them and carries out the new wiring work.

[0546] 6. Construction implementation

[0547] User: Create new wiring according to a user-generated plan.

[0548] 7. Uploading Feedback Data

[0549] Terminal: After the installation is complete, take a picture of the completed wiring and upload it to the server.

[0550] 8. Receiving and Analyzing Feedback

[0551] Server: Receives and analyzes the uploaded feedback data.

[0552] 9. Retraining generative AI

[0553] Server: The generation AI is retrained based on the results of the feedback analysis, which helps improve the accuracy of the next plan generation.

[0554] This system will enable improved quality and reduced labor costs in telecommunications construction work.

[0555] The processing flow will be explained below.

[0556] Step 1:

[0557] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[0558] Step 2:

[0559] Terminal: The captured image data is uploaded from the terminal to the server via wireless communication or wired connection.

[0560] Step 3:

[0561] Server: The server stores the received image data and prepares it for further processing.

[0562] Step 4:

[0563] Server: Performs pre-processing of image data. Pre-processing includes image noise reduction, contrast adjustment, and resolution adjustment, which improves the accuracy of image recognition algorithms.

[0564] Step 5:

[0565] Server: The preprocessed image data is input into an image recognition algorithm to extract information about the equipment and wiring. Identified elements include the device model number, wiring type, connected port location, and defective parts.

[0566] Step 6:

[0567] Server: Analyzes image recognition results and generates optimal construction plans and repair procedures based on the current situation. Generative AI optimizes this process using past data and best practices.

[0568] Step 7:

[0569] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[0570] Step 8:

[0571] Terminal: The user checks the construction plan and procedure manual received and carries out the work according to the displayed procedure.

[0572] Step 9:

[0573] Terminal: After the construction is completed, the user takes another image of the site and uploads it to the server along with a construction completion report.

[0574] Step 10:

[0575] Server: Receives post-construction feedback data, pre-processes and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[0576] Step 11:

[0577] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[0578] This series of steps will improve the quality and reduce the amount of work required in telecommunications construction.

[0579] Example 1

[0580] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0581] Telecommunications construction projects require improved quality and reduced man-hours. However, manual on-site inspections and construction plan creation are time-consuming and labor-intensive, and there is a high possibility of human error. Furthermore, quality checks and follow-ups after construction are often insufficient, making them ineffective at improving quality. Furthermore, there is a lack of systems with re-learning capabilities, making it difficult to accumulate experience and knowledge.

[0582] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0583] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information, means for generating optimal plans and procedures based on the analysis results, means for transmitting the generated plans and procedures to a user terminal, means for receiving and analyzing feedback data after construction, and means for relearning the generation algorithm based on the feedback data. This makes it possible to improve the quality and reduce the labor required for telecommunications construction work.

[0584] The "receiving means" is a function for receiving image data transmitted from a user terminal.

[0585] The "preprocessing means" is a function for removing noise and adjusting resolution from received image data.

[0586] The "means for analyzing" is a function that applies an algorithm to extract specific information from the pre-processed image data.

[0587] "Means for generation" refers to a function that automatically generates optimal plans and procedures based on the analysis results.

[0588] "Transmission means" is a function for sending the generated plan or procedure manual to the user's terminal.

[0589] The "means for receiving and analyzing" is a function for receiving feedback data sent by the user after implementation and analyzing it again.

[0590] The "means for relearning" is a function that improves the generation algorithm based on the analysis results and increases the accuracy of the next generation.

[0591] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[0592] "Server" means a computer system for processing and storing data.

[0593] An "image recognition algorithm" is an analytical method for extracting specific information from image data.

[0594] A "generative algorithm" is an algorithm for generating new plans and procedures based on extracted information.

[0595] "Feedback data" refers to data on the condition of the site and the results of construction provided by the user after construction.

[0596] This invention is a system that aims to improve the quality and reduce the man-hours required for telecommunications construction work, and combines image recognition functions with generative AI. This system realizes efficient construction plan generation and feedback at construction sites through collaboration between users, terminals, and servers.

[0597] First, a user uses a device such as a smartphone or tablet at a telecommunications construction site. The user takes pictures of important locations at the construction site, such as distribution boards and cable connection points. Specific examples of devices used include smartphones and tablets. This makes it possible to accurately record the site conditions. The user then uploads the captured image data to a server using a dedicated application.

[0598] The server then receives the uploaded image data. Specific software used on the server includes the OpenCV library for image preprocessing and TensorFlow and PyTorch for image recognition algorithms. The received image data undergoes preprocessing such as noise removal and resolution adjustment. This preprocessing step improves the quality of the image and increases the accuracy of subsequent analysis.

[0599] Once preprocessing is complete, the server uses deep learning to analyze the image data. The goal of the analysis is to identify the type, location, and defect of the wiring. For example, it can automatically identify the color code and connection location of the wiring.

[0600] Furthermore, the server uses generative AI (specifically, a GPT-4 model) to generate optimal construction plans and repair procedures based on the image recognition results. These plans include efficient work procedures and lists of necessary tools and materials. The server converts the generated construction plans and procedures into PDF or image format and sends them to the user's device. The user then checks them and carries out the work according to the instructions.

[0601] After the construction is completed, the user takes another image of the wiring status and uploads it to the server again. The server receives the re-uploaded feedback data and re-analyzes it. By re-training the generation AI based on the analysis results, it is expected that the accuracy of the next construction plan generation will improve.

[0602] Specific examples

[0603] For example, consider the case of adding new wiring to an equipment rack in an office. A user takes a picture of the equipment rack with their smartphone and uploads it to a server. The server receives the image and performs preprocessing. It then uses an image recognition algorithm to identify the wiring status and available ports. Based on the image recognition results, the server uses generative AI to propose the optimal layout for the new wiring, generates a PDF with specific wiring procedures, and sends it to the user. The user performs the wiring work according to the received plan, and after completing the work, takes another picture and uploads the feedback data. The server reanalyzes this data and retrains the generative AI to improve the accuracy of the next plan generation.

[0604] Prompt Sentence Examples

[0605] "Propose the optimal placement for new cabling. Upload an image of your office equipment rack. Identify the current cabling situation and open ports, and generate an installation plan with efficient cabling procedures. Analyze the image below."

[0606] In this way, the present invention provides a concrete method for improving quality and reducing labor costs in telecommunications construction. By utilizing generative AI models and image recognition technology, efficient and accurate construction becomes possible.

[0607] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0608] Step 1:

[0609] Taking and uploading images

[0610] Device: The user takes pictures of wiring and equipment at a telecommunications construction site using a smartphone or tablet. When taking pictures, they record important details such as the inside of the wiring board and cable connection points. Specifically, the device uses the built-in camera app to take photos, and then uploads the captured image data to a server using a dedicated app. The input is the captured image data, and the output is an image file sent to the server.

[0611] Step 2:

[0612] Image reception and preprocessing

[0613] Server: The server receives the uploaded image data. It uses OpenCV and other tools to remove noise and adjust the resolution of the received image data to improve the image quality. Specific operations include smoothing and sharpening the image. The input is the uploaded image data, and the output is high-quality image data that has undergone preprocessing.

[0614] Step 3:

[0615] Image Recognition Processing

[0616] Server: Analyzes preprocessed image data using a deep learning model (for example, TensorFlow or PyTorch). Identifies the type, location, and defective areas of the wiring from the image. Specific operations include color code recognition and object detection. The input is preprocessed image data, and the output is the type and location of the wiring based on the analysis results, as well as the identification of defective areas.

[0617] Step 4:

[0618] Construction plan generation

[0619] Server: Based on the image recognition results, a generative AI (e.g., a GPT-4 model) is used to generate optimal construction plans and repair procedures. Specifically, it creates construction procedures based on the recognized wiring information. The input is the image recognition results data, and the output is the generation of specific construction plans and procedures. The plans include efficient work procedures, necessary tools, and material lists.

[0620] Step 5:

[0621] Submit and confirm your plan

[0622] Server: Converts the generated construction plans and procedures into PDF or image format and sends them to the user's device. Specific operations include converting the file format and sending it via email or a dedicated application. The input is the construction plan or procedure data, and the output is a file sent to the user's device.

[0623] Terminal: The user checks the construction plan received on the terminal. Specifically, the user opens the received PDF file and checks the construction procedures. At this time, the user prepares the tools and materials necessary for construction. The input is the document data sent from the server, and the output is that construction preparations are complete.

[0624] Step 6:

[0625] Construction implementation

[0626] User: The user carries out the work based on the presented construction plan. Specific actions include connecting cables as instructed and repairing any defects. Once the work is complete, the user takes another photo of the completed equipment and wiring. The input is the construction plan information, and the output is image data of the completed wiring.

[0627] Step 7:

[0628] Feedback data upload

[0629] Terminal: After the construction is completed, the user uploads the retaken image data and implementation report to the server using a dedicated application. Specific operations include selecting and sending the retaken image file. The input is the image data after the construction is completed and the implementation report, and the output is the feedback data uploaded to the server.

[0630] Step 8:

[0631] Receiving and analyzing feedback

[0632] Server: The server receives the uploaded feedback data and performs the analysis again. Specific operations include comparing images before and after construction and evaluating the appropriateness of the construction plan. The input is the feedback data, and the output is the analysis results.

[0633] Step 9:

[0634] Retraining generative AI

[0635] Server: The server retrains the generative AI based on the analysis results of the feedback data. Specific operations include adjusting the parameters of the generative model and adding new data sets. The input is the feedback analysis results, and the output is a retrained generative AI model. This improves the accuracy of the next construction plan generation.

[0636] (Application example 1)

[0637] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0638] Providing efficient and appropriate maintenance procedures for robots in operation in factories is difficult. In particular, it is necessary for on-site technicians to accurately assess the robot's condition and quickly perform necessary maintenance work. Another challenge is that there is no established method for utilizing feedback after maintenance to improve future maintenance plans.

[0639] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0640] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about the facility or equipment, means for generating an optimal maintenance plan or repair procedure based on the analysis results, means for transmitting the generated maintenance plan or procedure manual to a user terminal, means for receiving and analyzing feedback data after construction, and means for relearning the generating AI based on the feedback data. This enables the provision of efficient maintenance procedures for robots in factories and continuous improvement based on subsequent feedback.

[0641] "Image data" refers to photographed image information that records the status of robots and equipment in the factory.

[0642] "Preprocessing" refers to data processing operations such as noise removal and resolution adjustment that are performed to make the received image data easier to analyze.

[0643] "Analysis" refers to the process of extracting specific information from preprocessed image data and using that information to evaluate the condition of facilities and equipment.

[0644] "Facility" refers to all the devices and systems used within a factory.

[0645] "Equipment" refers to specific devices and tools in a factory, including robots.

[0646] A "maintenance plan" refers to a procedure manual for repairing and maintaining facilities and equipment, created based on extracted information.

[0647] "Repair procedure" refers to the specific repair methods and processes used for equipment and facilities.

[0648] "User terminal" refers to a mobile terminal such as a smartphone or tablet used by a field technician.

[0649] "Feedback data" refers to information such as images taken after maintenance work and work reports.

[0650] "Analysis results" refers to the analysis results of information and data extracted by image recognition algorithms.

[0651] "Generative AI" refers to artificial intelligence that automatically generates optimal maintenance procedures and plans based on assumptions and data.

[0652] This invention is a system that provides efficient maintenance procedures for robots operating in factories. Specifically, it generates an optimal maintenance plan based on captured image data, and then uses subsequent feedback data to improve the accuracy of the generating AI.

[0653] System Hardware

[0654] This system uses the following hardware:

[0655] 1. Smartphone or tablet device: The field technician takes a photo of the robot's status and uploads the image data to the server.

[0656] 2. Server: Receives image data, pre-processes, analyzes, generates maintenance plans, and retrains feedback data.

[0657] System Software

[0658] The following software is used:

[0659] 1. Image recognition algorithms: (TensorFlow, OpenCV) are used to analyze image data and identify the status of facilities and equipment.

[0660] 2. Generative AI: Uses (GPT-4) to generate an optimal maintenance plan based on the analysis results.

[0661] 3. Database: (MySQL, PostgreSQL) is used to manage image data and feedback information.

[0662] System operation explanation

[0663] 1. Take and upload an image

[0664] An engineer uses a smartphone to take a photo of the robot, for example, the joints of the robot arm, and uploads the image to a server using a dedicated app.

[0665] 2. Receiving and preprocessing image data

[0666] The server performs noise reduction and resolution adjustment on the received image data, using TensorFlow to improve the image quality.

[0667] 3. Image Recognition Processing

[0668] The preprocessed image data is analyzed using OpenCV to identify robot parts and defects, for example, to identify the degree of wear on joints.

[0669] 4. Generate a maintenance plan

[0670] Using GPT-4, a generative AI, it generates a detailed maintenance plan based on the recognition results, including specific steps and a list of required tools.

[0671] 5. Submitting and implementing the plan

[0672] The server sends the generated maintenance plan in PDF format to the technician's terminal, where the technician checks the plan and performs the actual maintenance work.

[0673] 6. Uploading feedback data

[0674] After completing the maintenance, the technician takes another photo of the robot's condition and uploads it to the server as feedback data, allowing the technician to check the results of the maintenance.

[0675] 7. Feedback analysis and retraining of the generative AI

[0676] The server re-analyzes the received feedback data and extracts useful information, which allows the generating AI to re-train and improve the accuracy of the next maintenance plan.

[0677] Specific examples

[0678] Case: Robot arm joint maintenance

[0679] 1. Take and upload an image

[0680] Terminal: A technician takes pictures of the robot arm's joints and uploads them to the server.

[0681] 2. Receiving and preprocessing image data

[0682] Server: Perform noise removal and resolution adjustment using TensorFlow.

[0683] 3. Image Recognition Processing

[0684] Server: Uses OpenCV to identify the degree of wear on the joints.

[0685] 4. Generate a maintenance plan

[0686] Server: Uses GPT-4 to generate replacement procedures for worn parts and a list of required tools.

[0687] 5. Submitting and implementing the plan

[0688] Server: The generated maintenance plan is sent to the technician in PDF format, who then performs the work according to the plan.

[0689] 6. Uploading feedback data

[0690] Terminal: After maintenance is completed, an image of the robot joints is taken and uploaded to the server.

[0691] 7. Feedback analysis and retraining of the generative AI

[0692] Server: Analyzes the feedback data sent and retrains GPT-4.

[0693] Example of input prompt for generative AI model:

[0694] Please generate detailed instructions for maintaining the joints of the robot arm below.

[0695] Prerequisites:

[0696] Wear on the joints was observed.

[0697] The tools available are a wrench, screwdriver, and lubricant.

[0698] Replacement parts are available in advance.

[0699] Instructions:

[0700] 1. Safety measures procedures

[0701] 2. Specific replacement procedures for worn parts

[0702] 3. List of tools and materials needed

[0703] 4. Final check step

[0704] This enables proper and efficient maintenance of robots in factories.

[0705] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0706] Step 1:

[0707] Taking and uploading images

[0708] The terminal provides a means for users to take images of specific parts of the robot, such as joints. The captured image data is uploaded to a server via a dedicated app. The input is the captured image data, and the output is the upload of the image data to the server.

[0709] Step 2:

[0710] Receiving and preprocessing image data

[0711] The server performs noise reduction and resolution adjustment on the received image data. Specifically, it uses TensorFlow to improve the image quality. The input is the image data sent from the user device, and the output is the preprocessed image data.

[0712] Step 3:

[0713] Image Recognition Processing

[0714] The server analyzes the preprocessed image data using OpenCV to identify robot parts and defects. For example, it identifies the degree of wear on joints. The input is the preprocessed image data, and the output is the analyzed specific information (e.g., the location and degree of wear).

[0715] Step 4:

[0716] Generate a maintenance plan

[0717] The server uses GPT-4 as a generation AI to generate a detailed maintenance plan based on the analysis results. The generated plan includes specific procedures and a list of required tools. The input is the analyzed specific information, and the output is the maintenance plan.

[0718] Step 5:

[0719] Submitting and implementing the plan

[0720] The server sends the generated maintenance plan in PDF format to the user's terminal. The user checks the received plan and performs maintenance work based on it. The input is the maintenance plan, and the output is the procedure checked by the user.

[0721] Step 6:

[0722] Feedback data upload

[0723] After the maintenance is completed, the terminal takes another photograph of the robot's condition and uploads it to the server as feedback data. The input is the image data after maintenance, and the output is the feedback data uploaded to the server.

[0724] Step 7:

[0725] Feedback analysis and retraining of the generation AI

[0726] The server re-analyzes the received feedback data and extracts useful information, which re-trains the generative AI and improves the accuracy of the next maintenance plan. The input is the feedback data, and the output is the re-trained generative AI model.

[0727] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0728] This invention is a system that aims to improve the quality and reduce the man-hours of telecommunications construction work, and combines image recognition functions with generative AI. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more appropriate method for presenting construction plans is realized. This system performs the following program processing.

[0729] System Program Processing

[0730] 1. Take and upload an image

[0731] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[0732] 2. Image Reception and Preprocessing

[0733] Server: Receives the uploaded image data and pre-processes it. Pre-processing includes image noise reduction and resolution adjustment, which improves the accuracy of the image recognition algorithm.

[0734] 3. Image Recognition Processing

[0735] Server: The pre-processed image data is input into an image recognition algorithm to extract information about the equipment and wiring. The server uses the image recognition algorithm to identify the type, location, and defects of the wiring. For example, it automatically identifies the color code and connection location of the wiring.

[0736] 4. Generate construction plans

[0737] Server: Generates optimal construction plans and repair procedures based on the image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans that detail areas where wiring changes are required and repair procedures.

[0738] 5. User Emotion Recognition

[0739] Device: While the user is reviewing the construction plan or procedure manual, the built-in camera captures the user's facial expression, and the captured facial image is sent to the emotion engine.

[0740] Emotion engine: Analyzes facial images to identify the user's emotional state, for example, detecting when the user is confused or frustrated.

[0741] 6. Adjusting the way plans are presented

[0742] Server: Based on the user's emotional state, adjust the presentation of the generated construction plans and procedures, for example by providing more detailed explanations or additional visual aids to facilitate user understanding.

[0743] 7. Construction implementation

[0744] User: The user carries out the actual construction work based on the adjusted construction plan. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[0745] 8. Uploading Feedback Data

[0746] Terminal: After the work is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[0747] 9. Receiving and Analyzing Feedback

[0748] Server: Receives post-construction feedback data, pre-processes it, and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[0749] 10. Retraining generative AI

[0750] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[0751] Specific examples

[0752] Case: Adding new wiring

[0753] 1. Take and upload an image

[0754] Terminal: A user takes a picture of an equipment rack in the office with their smartphone and uploads it to the server.

[0755] 2. Image Reception and Preprocessing

[0756] Server: Preprocesses the received image data by removing noise and adjusting the resolution.

[0757] 3. Image Recognition Processing

[0758] Server: Analyzes the preprocessed image and identifies the current wiring status and available ports.

[0759] 4. Generate construction plans

[0760] Server: Based on the image recognition results, the optimal layout of new wiring is generated and specific wiring procedures are compiled into a PDF.

[0761] 5. User Emotion Recognition

[0762] Device: As the user reviews the construction plan, the built-in camera captures the user's facial expression.

[0763] Emotion engine: Analyzes captured facial expressions to identify when the user is confused.

[0764] 6. Adjusting the way plans are presented

[0765] Server: Provide construction plans with more detailed instructions and additional visual aids to resolve user confusion.

[0766] 7. Construction implementation

[0767] User: Carry out wiring work based on the coordinated plan.

[0768] 8. Uploading Feedback Data

[0769] Terminal: After the installation is complete, take a picture of the completed wiring and upload it to the server.

[0770] 9. Receiving and Analyzing Feedback

[0771] Server: Analyzes the uploaded feedback data and identifies issues and areas for improvement.

[0772] 10. Retraining generative AI

[0773] Server: Retrains the generation AI based on the feedback to improve the accuracy of the next construction plan generation.

[0774] This series of steps utilizes an emotion engine to deepen user understanding and ensure high-quality construction.

[0775] The processing flow will be explained below.

[0776] Step 1:

[0777] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[0778] Step 2:

[0779] Terminal: The captured image data is uploaded from the terminal to the server via wireless communication or wired connection.

[0780] Step 3:

[0781] Server: The server stores the received image data and prepares it for further processing.

[0782] Step 4:

[0783] Server: Performs pre-processing of image data. Pre-processing includes image noise reduction, contrast adjustment, and resolution adjustment, which improves the accuracy of image recognition algorithms.

[0784] Step 5:

[0785] Server: The preprocessed image data is input into an image recognition algorithm to extract information about the equipment and wiring. Identified elements include the type, location, and defect of the wiring. For example, the color code and connection location of the wiring can be automatically identified.

[0786] Step 6:

[0787] Server: Generates optimal construction plans and repair procedures based on image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans detailing areas where wiring changes are required and repair procedures.

[0788] Step 7:

[0789] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[0790] Step 8:

[0791] Device: When the user checks the construction plan or procedure manual they received, the built-in camera captures their facial expression.

[0792] Step 9:

[0793] Emotion engine: Analyzes captured facial images to identify the user's emotional state, for example, detecting when the user is confused or dissatisfied.

[0794] Step 10:

[0795] Server: Based on the user's emotional state, adjust the presentation of the generated construction plans and procedures, for example by providing more detailed explanations or additional visual aids to facilitate user understanding.

[0796] Step 11:

[0797] User: The user carries out the actual construction work based on the adjusted construction plan. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[0798] Step 12:

[0799] Terminal: After the construction is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[0800] Step 13:

[0801] Server: Receives post-construction feedback data, pre-processes and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[0802] Step 14:

[0803] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[0804] This series of steps utilizes an emotion engine to deepen user understanding and ensure high-quality construction.

[0805] Example 2

[0806] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0807] In conventional telecommunications construction, ensuring the quality and efficiency of construction required a great deal of time and effort. In particular, there were many uncertainties in the appropriateness of construction plans and the presentation of repair procedures according to the on-site situation, and the progress of construction was affected by the user's own level of understanding and emotional state. This resulted in issues such as inconsistent construction quality, increased man-hours, and reduced user satisfaction.

[0808] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0809] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about facilities and wiring, means for generating an optimal construction plan and repair procedure based on the analysis results, means for transmitting the generated construction plan and procedure manual to a user terminal, means for recognizing the user's emotions during construction, means for adjusting the presentation method of the construction plan and procedure manual based on the recognized emotions, means for receiving and analyzing feedback data after construction, and means for relearning the generation AI based on the feedback data. This makes it possible to automatically generate an optimal construction plan based on image data of the construction site and flexibly adjust it according to the user's emotional state, thereby improving the quality and efficiency of construction.

[0810] "Photographed image data" refers to an image file photographed by a user at a construction site, and contains information including the state of wiring and equipment at the construction site.

[0811] The "receiving means" is a mechanism by which the server retrieves image data sent from the user terminal via the Internet.

[0812] The "preprocessing means" is a step in which the received image data is subjected to processes such as noise removal and resolution adjustment to improve the accuracy of the image recognition algorithm.

[0813] "Means for extracting information" refers to algorithms or techniques that analyze the pre-processed image data and identify relevant information about wiring and equipment.

[0814] The "means of generation" is a system that designs optimal construction plans and repair procedures based on the analysis results and presents them in document or digital format.

[0815] "Transmission means" refers to a method for transferring the construction plan and procedure manual generated by the server to the user's terminal.

[0816] The "means for recognizing emotions" is a technology that captures the user's facial expressions during construction and identifies the user's emotional state from the facial expression data.

[0817] The "means for adjusting the presentation method" is a mechanism for changing the content and presentation method of construction plans and procedures based on the emotion recognition results, thereby improving the user's understanding and satisfaction.

[0818] "Feedback data" refers to information on the results of construction and usage experience provided by users after construction.

[0819] "Means for receiving and analyzing" refers to a technology in which the server receives feedback data and uses that data to evaluate the performance of the construction plan and system.

[0820] "Relearning means" refers to a method for updating the generative AI model based on analyzed feedback data to improve the accuracy and efficiency of the next construction plan generation.

[0821] A "generative AI model" is an artificial intelligence model used to automatically generate construction plans and repair procedures, and includes natural language generation models such as GPT-4.

[0822] This invention is a system aimed at improving the quality and reducing the man-hours required for telecommunications construction. This system combines image recognition, generative AI, and an emotion engine that recognizes the user's emotions.

[0823] First, the user takes images of the construction site using a device (e.g., a smartphone or tablet). The images taken include the inside of a wiring board and cable connection points. The user then uses a dedicated application to upload the captured image data to a server. This upload requires an internet connection (Wi-Fi or mobile data communication).

[0824] The server uses the OpenCV library to preprocess the received image data, which includes image noise reduction (e.g., applying a Gaussian filter) and resolution adjustment (e.g., resizing). This preprocessing improves the accuracy of the image recognition algorithms used later.

[0825] The preprocessed image data is then fed into an image recognition algorithm using the TensorFlow library, which the server uses to extract information about the equipment and wiring, such as color codes, connection locations, and fault locations.

[0826] Based on the extracted information, the server uses generative AI (e.g., a natural language generation model such as GPT-4) to generate optimal construction plans and repair procedures. The generated plans are output in PDF format and transferred to the user's device. The user can refer to these plans to proceed with the construction work.

[0827] Additionally, while the user is reviewing the construction plan, the device's built-in camera captures the user's facial expression. This facial expression data is sent to an emotion engine (e.g., Emotion API), which then analyzes the user's emotional state on the server. Based on the analysis results, for example, if the user is confused, the server adjusts the way the construction plan is presented. Specifically, it provides detailed explanations or additional visual guides.

[0828] After the construction is completed, the user takes another image of the construction site and uploads it to the server as feedback data. The server receives and analyzes this feedback data. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[0829] Finally, the server retrains the generative AI model based on this feedback data, helping to improve the accuracy of the next construction plan generation.

[0830] As a concrete example, consider the task of adding new wiring. A user takes a picture of an equipment rack in their office with their smartphone and uploads it to the server. The server preprocesses and analyzes the image data to identify the current wiring status and available ports. It then compiles the generated construction plan into a PDF and provides it to the user. If the server determines that the user's facial expression is confused while reviewing the construction plan, it re-provides the plan with detailed explanations and additional visual materials. The user then carries out the wiring work based on the adjusted plan and uploads an image of the completed work. The server analyzes the feedback data and retrains the generating AI to improve the accuracy of the next construction plan.

[0831] Example prompt sentence:

[0832] "Server: Upload images of your telecommunications construction site. We will use generative AI to provide you with the optimal construction plan."

[0833] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0834] Step 1:

[0835] Taking and uploading images

[0836] User: The user uses a smartphone or tablet to take detailed images of the construction site, such as the inside of a wiring cabinet or cable connection points, carefully capturing them from multiple angles.

[0837] Device: Upload the captured image data to the server using a dedicated application (e.g., a mobile app). When you press the "Upload Image" button in the application, the image data is sent to the server via an internet connection.

[0838] Input: Construction site image data

[0839] Output: Image data uploaded to the server

[0840] Step 2:

[0841] Image reception and preprocessing

[0842] Server: The server receives the image data sent by the user. For example, a REST API using the Flask framework is used to receive the data.

[0843] Server: Preprocesses the received image data using the OpenCV library, applying a Gaussian filter to remove noise and resizing the image to the desired resolution.

[0844] Input: Uploaded image data

[0845] Output: Preprocessed image data

[0846] Step 3:

[0847] Image Recognition Processing

[0848] Server: The preprocessed image data is input to the image recognition algorithm using the TensorFlow library.

[0849] Server: Image recognition algorithms automatically extract information about wiring and equipment, such as the type of wiring, its location, color code, and any defects.

[0850] Input: Preprocessed image data

[0851] Output: Extracted information on facilities and wiring

[0852] Step 4:

[0853] Construction plan generation

[0854] Server: Based on the extracted information, a generative AI model (e.g., GPT-4) is used to generate optimal construction plans and repair procedures.

[0855] Server: The generated construction plans and repair procedures are documented in PDF format and provided to users in an easy-to-understand format.

[0856] Input: Image recognition results (extracted information on facilities and wiring)

[0857] Output: PDF document of construction plan

[0858] Step 5:

[0859] User Emotion Recognition

[0860] Device: While the user is viewing the construction plan, the built-in camera captures the user's facial expression.

[0861] Emotion Engine: The captured facial image is sent to the emotion engine, which analyzes it to identify the user's emotional state (e.g., confusion, frustration, relief).

[0862] Input: User's face image

[0863] Output: User's emotional state

[0864] Step 6:

[0865] Adjusting the way plans are presented

[0866] Server: Based on the emotion recognition results, it adjusts the way the construction plan is presented, for example, by providing detailed explanations or additional visual guides if the user is confused.

[0867] Server: The adjusted construction plan is compiled into a PDF again and resent to the user's device.

[0868] Input: User's emotional state

[0869] Output: PDF document of the coordinated construction plan

[0870] Step 7:

[0871] Construction implementation

[0872] User: Carries out work on-site based on the coordinated construction plan, for example installing new wiring or carrying out repairs as instructed.

[0873] User: Once the construction is complete, take another image of the completed state and upload it to the server as feedback data.

[0874] Input: Coordinated construction plan PDF document

[0875] Output: Completed image data and feedback

[0876] Step 8:

[0877] Feedback data upload

[0878] Terminal: After the construction is completed, the user uploads the image data and work report to the server.

[0879] Terminal: Press the "Report Completion" button in the dedicated application to send the data.

[0880] Input: Completed image data and feedback

[0881] Output: Feedback data uploaded to the server

[0882] Step 9:

[0883] Receiving and analyzing feedback

[0884] Server: Receives the feedback data and preprocesses it again.

[0885] Server: Analyzes the feedback and identifies the accuracy of the construction plan, the suitability of the execution, and areas for improvement.

[0886] Input: Uploaded feedback data

[0887] Output: Feedback analysis results

[0888] Step 10:

[0889] Retraining generative AI

[0890] Server: Based on the analysis results, the generative AI model is retrained to improve the accuracy of the next construction plan generation. For retraining, new training data including feedback data is used.

[0891] Input: Feedback analysis results

[0892] Output: An updated generative AI model

[0893] (Application example 2)

[0894] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0895] Equipment repair and maintenance work within factories requires accurate understanding of on-site conditions and the ability to provide repair plans quickly and efficiently. To improve worker understanding and efficiency, it is important not only to provide technical instructions but also to respond flexibly while taking into account the worker's emotional state. While conventional systems use image recognition for automation and generative AI for planning, they lack the ability to adjust based on the worker's emotions, making it difficult to achieve both improved work efficiency and quality.

[0896] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0897] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about the equipment and structure, means for generating an optimal construction plan and repair procedure based on the analysis results, means for transmitting the generated construction plan and procedure manual to a user terminal, means for recognizing the user's emotional state, means for adjusting the presentation method of the construction plan based on the emotional state, means for receiving and analyzing feedback data after construction, and means for relearning the generation AI based on the feedback data. This makes it possible to improve the worker's understanding, increase the efficiency of repair work, and improve the quality of the entire construction work.

[0898] "Photographed image data" refers to digital data of on-site images of facilities and structures photographed by a device such as a user terminal or a robot.

[0899] "Means for receiving" refers to a function or device for receiving image data sent from a user terminal or robot via a network.

[0900] "Preprocessing means" refers to software and processes for performing preprocessing such as noise removal and resolution adjustment on received image data.

[0901] "Means for analyzing" refers to image recognition algorithms and processes for extracting facility or structural information from pre-processed image data.

[0902] "Means of generation" refers to the generative AI model and process for creating optimal construction plans and repair procedures based on the analysis results.

[0903] "Transmission means" refers to a function or device for transmitting the generated construction plan or procedure manual to a user terminal.

[0904] "Means for recognizing the user's emotional state" refers to an emotion engine and its process for analyzing the user's facial expression image captured by the built-in camera and identifying the user's emotional state.

[0905] "Adjustment means" refers to a function or device for changing the presentation method of the generated construction plan based on the user's emotional state.

[0906] "Feedback data" refers to data related to work results, such as on-site images taken after construction and implementation reports.

[0907] "Means for re-learning" refers to the process of re-learning the generative AI model based on feedback data to improve the accuracy of the next construction plan generation.

[0908] To implement the present invention, the following system configuration is required.

[0909] Hardware Configuration

[0910] Robot body: A mobile robot used in factories, equipped with built-in cameras and sensors.

[0911] User device: A smartphone or tablet device with a built-in camera that communicates with the robot.

[0912] Server: A central server that processes and stores all data.

[0913] Software Configuration

[0914] Image recognition algorithms: YOLO and OpenCV are used to preprocess the received image data and extract information about the facilities and structures.

[0915] Generative AI model: Uses OpenAI GPT-4 and Google BERT. Generates optimal construction plans and repair procedures based on analysis results.

[0916] Emotion Engine: Using Emotion API and Affectiva, it recognizes the user's emotional state and adjusts how construction plans are presented.

[0917] Cloud servers: Amazon Web Services (AWS) and Google Cloud are used to process and store data.

[0918] System processing flow

[0919] The server mainly performs the following processes:

[0920] 1. Image Reception and Preprocessing

[0921] The server receives image data sent from user terminals and robots, and performs preprocessing such as noise removal and resolution adjustment on the received image data.

[0922] 2. Image Recognition Processing

[0923] The pre-processed image data is then fed into an image recognition algorithm to extract information about the equipment or structure, such as identifying abnormalities or broken areas on a conveyor belt.

[0924] 3. Creation of construction plans

[0925] Based on the extracted information, a generative AI model is used to generate optimal repair procedures, including detailed descriptions of work procedures when using related labor-saving devices.

[0926] 4. Emotion recognition and presentation adjustment

[0927] While the user is reviewing the construction plan, the built-in camera on the user's device captures the user's facial expression and sends it to the emotion engine. Based on the emotion recognition results, the way the construction plan is presented is adjusted to make it easier for the user to understand.

[0928] 5. Receiving and Analyzing Feedback

[0929] After construction is completed, feedback data (images and implementation reports) sent from the user device and robot is received and analyzed to determine the accuracy of the construction plan and identify areas for improvement.

[0930] 6. Retraining generative AI

[0931] The generative AI model will be retrained based on the feedback data to improve the accuracy of the next construction plan generation.

[0932] Specific examples

[0933] A specific example is a robot detecting an abnormality in a conveyor belt and repairing it.

[0934] First, the robot photographs any abnormalities on the conveyor belt and uploads the image data to a server. The server then uses an image recognition algorithm to identify the abnormalities and generates optimal repair procedures using a generative AI model. It then uses its built-in camera to recognize the user's emotional state and adjusts the way the procedure manual is presented as needed. After the repair is complete, the robot takes another image and uploads it to the server as feedback data. This data is used to retrain the generative AI model, improving its accuracy the next time.

[0935] Prompt Sentence Examples

[0936] For example, you might input the following prompt to a generative AI model:

[0937] Prompt: "Generate the optimal procedure for repairing the conveyor belt malfunction. Use the image data below as a starting point. [Image data link]. Please add detailed explanations of any parts that may be difficult for the worker."

[0938] In this way, by combining image recognition algorithms, generative AI models, and emotion engines, the present invention achieves improved efficiency and quality in equipment repair and maintenance work within factories.

[0939] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0940] Step 1:

[0941] Take and upload images:

[0942] The user terminal and robot take images of the on-site equipment and upload the image data to the server. At this time, the user terminal takes a picture of the problem area with a camera and transfers the captured image file to the server via the network. The input is the on-site image data, and the output is the saving of the image file on the server.

[0943] Step 2:

[0944] Image reception and pre-processing:

[0945] The server receives image data sent from the user device and the robot. After receiving the image data, it performs preprocessing such as noise removal and resolution adjustment on the image data. Specifically, it cleans up the images using an image processing library such as OpenCV. The input is the image file from the user device and the robot, and the output is the preprocessed image data.

[0946] Step 3:

[0947] Image Recognition Processing:

[0948] The server inputs the preprocessed image data into an image recognition algorithm to extract information about the equipment and its structure. For example, YOLO can be used to identify abnormalities in the equipment. The input is the preprocessed image data, and the output is a list of identified abnormalities and equipment information.

[0949] Step 4:

[0950] Generate construction plans:

[0951] The server uses a generative AI model based on the image recognition results to generate optimal construction plans and repair procedures. The plan generation includes specific work procedures and a list of required tools. The input is the image recognition results, and the output is the generated construction plan and procedure manual.

[0952] Step 5:

[0953] Submit your plan:

[0954] The server sends the generated construction plans and procedures to the user's terminal. At this time, the plans and procedures are provided in PDF file or text format. The construction plans and procedures are input, and the data is sent to the user's terminal as output.

[0955] Step 6:

[0956] User Emotion Recognition:

[0957] While the user is checking the construction plan, the device's built-in camera captures the user's facial expression. The captured facial image is sent to the server, which then uses an emotion engine to analyze the user's emotional state. The input is the user's facial image data, and the output is the user's emotional state (confusion, dissatisfaction, etc.).

[0958] Step 7:

[0959] Adjustments to how plans are presented:

[0960] The server adjusts the presentation of the generated construction plans and procedures based on the user's emotional state. For example, if the user is confused, it provides additional visual aids or detailed explanations. The user's emotional state is the input, and the adjusted construction plans and procedures are delivered as the output.

[0961] Step 8:

[0962] Construction implementation:

[0963] The user performs the actual repair work based on the adjusted construction plan. Specifically, the user repairs the equipment using the necessary tools and materials while referring to the generated procedure manual. The input is the adjusted construction plan, and the output is the completed repaired equipment.

[0964] Step 9:

[0965] Feedback data upload:

[0966] After the construction is completed, the user terminal and the robot take images of the site again and upload them to the server as feedback data. The input is the image data after construction, and the output is the feedback data, which is saved on the server.

[0967] Step 10:

[0968] Receiving and analyzing feedback:

[0969] The server receives the feedback data and uses image recognition algorithms to verify and analyze the construction results. Areas for improvement are identified and reflected in the generation of the next construction plan. The input is the feedback data, and the output is the analysis results and a list of areas for improvement.

[0970] Step 11:

[0971] Retraining generative AI:

[0972] The server retrains the generative AI model based on the analysis results to improve the accuracy of the next construction plan generation. The input is the analysis results and points for improvement, and the output is an improved generative AI model.

[0973] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0974] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0975] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0976] [Third embodiment]

[0977] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0978] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0979] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0980] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0981] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0982] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0983] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0984] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0985] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0986] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0987] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0988] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0989] This invention is a system that aims to improve the quality of telecommunications construction and reduce the man-hours required, and combines image recognition functions with generative AI. This system performs the following program processing.

[0990] System Program Processing

[0991] 1. Take and upload an image

[0992] Device: The user takes pictures of the equipment and wiring being worked on at the telecommunications construction site using a smartphone or tablet device and uploads the image data to the server. This image data includes the inside of the distribution board, cable connection points, etc.

[0993] 2. Image Reception and Preprocessing

[0994] Server: Receives the uploaded image data and pre-processes it. Pre-processing includes image noise reduction and resolution adjustment, which improves the accuracy of the image recognition algorithm.

[0995] 3. Image Recognition Processing

[0996] Server: Analyzes the pre-processed image data and extracts information about the equipment and wiring. The server uses image recognition algorithms to identify the type, location, and defects of the wiring. For example, it automatically identifies the color code and connection location of the wiring.

[0997] 4. Generate construction plans

[0998] Server: Generates optimal construction plans and repair procedures based on the image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans that detail areas where wiring changes are required and repair procedures.

[0999] 5. Submit and confirm your plan

[1000] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[1001] Terminal: The user checks the construction plan sent to them and carries out the work according to the instructions.

[1002] 6. Construction implementation

[1003] User: The user carries out the actual construction work based on the construction plan provided. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[1004] 7. Uploading Feedback Data

[1005] Terminal: After the work is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[1006] 8. Receiving and Analyzing Feedback

[1007] Server: Receives feedback data after construction and analyzes it again. The feedback data includes information on changes in condition before and after construction and the appropriateness of the construction plan.

[1008] 9. Retraining generative AI

[1009] Server: Based on the analysis results of the feedback data, the generation AI is retrained. This improves the accuracy of the generation AI, which is then used to generate the next construction plan.

[1010] Specific examples

[1011] Case: Adding new wiring

[1012] 1. Take and upload an image

[1013] Terminal: A user takes a picture of an equipment rack in the office with their smartphone and uploads it to the server.

[1014] 2. Image Reception and Preprocessing

[1015] Server: Preprocesses the received image data by removing noise and adjusting the resolution.

[1016] 3. Image Recognition Processing

[1017] Server: Analyzes the preprocessed image and identifies the current wiring status and available ports.

[1018] 4. Generate construction plans

[1019] Server: Based on the image recognition results, the optimal layout of new wiring is generated and specific wiring procedures are compiled into a PDF.

[1020] 5. Submit and confirm your plan

[1021] Server: Sends the completed instructions to the device.

[1022] Terminal: The user checks the installation plan sent to them and carries out the new wiring work.

[1023] 6. Construction implementation

[1024] User: Create new wiring according to a user-generated plan.

[1025] 7. Uploading Feedback Data

[1026] Terminal: After the installation is complete, take a picture of the completed wiring and upload it to the server.

[1027] 8. Receiving and Analyzing Feedback

[1028] Server: Receives and analyzes the uploaded feedback data.

[1029] 9. Retraining generative AI

[1030] Server: The generation AI is retrained based on the results of the feedback analysis, which helps improve the accuracy of the next plan generation.

[1031] This system will enable improved quality and reduced labor costs in telecommunications construction work.

[1032] The processing flow will be explained below.

[1033] Step 1:

[1034] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[1035] Step 2:

[1036] Terminal: The captured image data is uploaded from the terminal to the server via wireless communication or wired connection.

[1037] Step 3:

[1038] Server: The server stores the received image data and prepares it for further processing.

[1039] Step 4:

[1040] Server: Performs pre-processing of image data. Pre-processing includes image noise reduction, contrast adjustment, and resolution adjustment, which improves the accuracy of image recognition algorithms.

[1041] Step 5:

[1042] Server: The preprocessed image data is input into an image recognition algorithm to extract information about the equipment and wiring. Identified elements include the device model number, wiring type, connected port location, and defective parts.

[1043] Step 6:

[1044] Server: Analyzes image recognition results and generates optimal construction plans and repair procedures based on the current situation. Generative AI optimizes this process using past data and best practices.

[1045] Step 7:

[1046] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[1047] Step 8:

[1048] Terminal: The user checks the construction plan and procedure manual received and carries out the work according to the displayed procedure.

[1049] Step 9:

[1050] Terminal: After the construction is completed, the user takes another image of the site and uploads it to the server along with a construction completion report.

[1051] Step 10:

[1052] Server: Receives post-construction feedback data, pre-processes and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[1053] Step 11:

[1054] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[1055] This series of steps will improve the quality and reduce the amount of work required in telecommunications construction.

[1056] Example 1

[1057] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1058] Telecommunications construction projects require improved quality and reduced man-hours. However, manual on-site inspections and construction plan creation are time-consuming and labor-intensive, and there is a high possibility of human error. Furthermore, quality checks and follow-ups after construction are often insufficient, making them ineffective at improving quality. Furthermore, there is a lack of systems with re-learning capabilities, making it difficult to accumulate experience and knowledge.

[1059] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1060] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information, means for generating optimal plans and procedures based on the analysis results, means for transmitting the generated plans and procedures to a user terminal, means for receiving and analyzing feedback data after construction, and means for relearning the generation algorithm based on the feedback data. This makes it possible to improve the quality and reduce the labor required for telecommunications construction work.

[1061] The "receiving means" is a function for receiving image data transmitted from a user terminal.

[1062] The "preprocessing means" is a function for removing noise and adjusting resolution from received image data.

[1063] The "means for analyzing" is a function that applies an algorithm to extract specific information from the pre-processed image data.

[1064] "Means for generation" refers to a function that automatically generates optimal plans and procedures based on the analysis results.

[1065] "Transmission means" is a function for sending the generated plan or procedure manual to the user's terminal.

[1066] The "means for receiving and analyzing" is a function for receiving feedback data sent by the user after implementation and analyzing it again.

[1067] The "means for relearning" is a function that improves the generation algorithm based on the analysis results and increases the accuracy of the next generation.

[1068] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[1069] "Server" means a computer system for processing and storing data.

[1070] An "image recognition algorithm" is an analytical method for extracting specific information from image data.

[1071] A "generative algorithm" is an algorithm for generating new plans and procedures based on extracted information.

[1072] "Feedback data" refers to data on the condition of the site and the results of construction provided by the user after construction.

[1073] This invention is a system that aims to improve the quality and reduce the man-hours required for telecommunications construction work, and combines image recognition functions with generative AI. This system realizes efficient construction plan generation and feedback at construction sites through collaboration between users, terminals, and servers.

[1074] First, a user uses a device such as a smartphone or tablet at a telecommunications construction site. The user takes pictures of important locations at the construction site, such as distribution boards and cable connection points. Specific examples of devices used include smartphones and tablets. This makes it possible to accurately record the site conditions. The user then uploads the captured image data to a server using a dedicated application.

[1075] The server then receives the uploaded image data. Specific software used on the server includes the OpenCV library for image preprocessing and TensorFlow and PyTorch for image recognition algorithms. The received image data undergoes preprocessing such as noise removal and resolution adjustment. This preprocessing step improves the quality of the image and increases the accuracy of subsequent analysis.

[1076] Once preprocessing is complete, the server uses deep learning to analyze the image data. The goal of the analysis is to identify the type, location, and defect of the wiring. For example, it can automatically identify the color code and connection location of the wiring.

[1077] Furthermore, the server uses generative AI (specifically, a GPT-4 model) to generate optimal construction plans and repair procedures based on the image recognition results. These plans include efficient work procedures and lists of necessary tools and materials. The server converts the generated construction plans and procedures into PDF or image format and sends them to the user's device. The user then checks them and carries out the work according to the instructions.

[1078] After the construction is completed, the user takes another image of the wiring status and uploads it to the server again. The server receives the re-uploaded feedback data and re-analyzes it. By re-training the generation AI based on the analysis results, it is expected that the accuracy of the next construction plan generation will improve.

[1079] Specific examples

[1080] For example, consider the case of adding new wiring to an equipment rack in an office. A user takes a picture of the equipment rack with their smartphone and uploads it to a server. The server receives the image and performs preprocessing. It then uses an image recognition algorithm to identify the wiring status and available ports. Based on the image recognition results, the server uses generative AI to propose the optimal layout for the new wiring, generates a PDF with specific wiring procedures, and sends it to the user. The user performs the wiring work according to the received plan, and after completing the work, takes another picture and uploads the feedback data. The server reanalyzes this data and retrains the generative AI to improve the accuracy of the next plan generation.

[1081] Prompt Sentence Examples

[1082] "Propose the optimal placement for new cabling. Upload an image of your office equipment rack. Identify the current cabling situation and open ports, and generate an installation plan with efficient cabling procedures. Analyze the image below."

[1083] In this way, the present invention provides a concrete method for improving quality and reducing labor costs in telecommunications construction. By utilizing generative AI models and image recognition technology, efficient and accurate construction becomes possible.

[1084] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1085] Step 1:

[1086] Taking and uploading images

[1087] Device: The user takes pictures of wiring and equipment at a telecommunications construction site using a smartphone or tablet. When taking pictures, they record important details such as the inside of the wiring board and cable connection points. Specifically, the device uses the built-in camera app to take photos, and then uploads the captured image data to a server using a dedicated app. The input is the captured image data, and the output is an image file sent to the server.

[1088] Step 2:

[1089] Image reception and preprocessing

[1090] Server: The server receives the uploaded image data. It uses OpenCV and other tools to remove noise and adjust the resolution of the received image data to improve the image quality. Specific operations include smoothing and sharpening the image. The input is the uploaded image data, and the output is high-quality image data that has undergone preprocessing.

[1091] Step 3:

[1092] Image Recognition Processing

[1093] Server: Analyzes preprocessed image data using a deep learning model (for example, TensorFlow or PyTorch). Identifies the type, location, and defective areas of the wiring from the image. Specific operations include color code recognition and object detection. The input is preprocessed image data, and the output is the type and location of the wiring based on the analysis results, as well as the identification of defective areas.

[1094] Step 4:

[1095] Construction plan generation

[1096] Server: Based on the image recognition results, a generative AI (e.g., a GPT-4 model) is used to generate optimal construction plans and repair procedures. Specifically, it creates construction procedures based on the recognized wiring information. The input is the image recognition results data, and the output is the generation of specific construction plans and procedures. The plans include efficient work procedures, necessary tools, and material lists.

[1097] Step 5:

[1098] Submit and confirm your plan

[1099] Server: Converts the generated construction plans and procedures into PDF or image format and sends them to the user's device. Specific operations include converting the file format and sending it via email or a dedicated application. The input is the construction plan or procedure data, and the output is a file sent to the user's device.

[1100] Terminal: The user checks the construction plan received on the terminal. Specifically, the user opens the received PDF file and checks the construction procedures. At this time, the user prepares the tools and materials necessary for construction. The input is the document data sent from the server, and the output is that construction preparations are complete.

[1101] Step 6:

[1102] Construction implementation

[1103] User: The user carries out the work based on the presented construction plan. Specific actions include connecting cables as instructed and repairing any defects. Once the work is complete, the user takes another photo of the completed equipment and wiring. The input is the construction plan information, and the output is image data of the completed wiring.

[1104] Step 7:

[1105] Feedback data upload

[1106] Terminal: After the construction is completed, the user uploads the retaken image data and implementation report to the server using a dedicated application. Specific operations include selecting and sending the retaken image file. The input is the image data after the construction is completed and the implementation report, and the output is the feedback data uploaded to the server.

[1107] Step 8:

[1108] Receiving and analyzing feedback

[1109] Server: The server receives the uploaded feedback data and performs the analysis again. Specific operations include comparing images before and after construction and evaluating the appropriateness of the construction plan. The input is the feedback data, and the output is the analysis results.

[1110] Step 9:

[1111] Retraining generative AI

[1112] Server: The server retrains the generative AI based on the analysis results of the feedback data. Specific operations include adjusting the parameters of the generative model and adding new data sets. The input is the feedback analysis results, and the output is a retrained generative AI model. This improves the accuracy of the next construction plan generation.

[1113] (Application example 1)

[1114] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1115] Providing efficient and appropriate maintenance procedures for robots in operation in factories is difficult. In particular, it is necessary for on-site technicians to accurately assess the robot's condition and quickly perform necessary maintenance work. Another challenge is that there is no established method for utilizing feedback after maintenance to improve future maintenance plans.

[1116] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1117] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about the facility or equipment, means for generating an optimal maintenance plan or repair procedure based on the analysis results, means for transmitting the generated maintenance plan or procedure manual to a user terminal, means for receiving and analyzing feedback data after construction, and means for relearning the generating AI based on the feedback data. This enables the provision of efficient maintenance procedures for robots in factories and continuous improvement based on subsequent feedback.

[1118] "Image data" refers to photographed image information that records the status of robots and equipment in the factory.

[1119] "Preprocessing" refers to data processing operations such as noise removal and resolution adjustment that are performed to make the received image data easier to analyze.

[1120] "Analysis" refers to the process of extracting specific information from preprocessed image data and using that information to evaluate the condition of facilities and equipment.

[1121] "Facility" refers to all the devices and systems used within a factory.

[1122] "Equipment" refers to specific devices and tools in a factory, including robots.

[1123] A "maintenance plan" refers to a procedure manual for repairing and maintaining facilities and equipment, created based on extracted information.

[1124] "Repair procedure" refers to the specific repair methods and processes used for equipment and facilities.

[1125] "User terminal" refers to a mobile terminal such as a smartphone or tablet used by a field technician.

[1126] "Feedback data" refers to information such as images taken after maintenance work and work reports.

[1127] "Analysis results" refers to the analysis results of information and data extracted by image recognition algorithms.

[1128] "Generative AI" refers to artificial intelligence that automatically generates optimal maintenance procedures and plans based on assumptions and data.

[1129] This invention is a system that provides efficient maintenance procedures for robots operating in factories. Specifically, it generates an optimal maintenance plan based on captured image data, and then uses subsequent feedback data to improve the accuracy of the generating AI.

[1130] System Hardware

[1131] This system uses the following hardware:

[1132] 1. Smartphone or tablet device: The field technician takes a photo of the robot's status and uploads the image data to the server.

[1133] 2. Server: Receives image data, pre-processes, analyzes, generates maintenance plans, and retrains feedback data.

[1134] System Software

[1135] The following software is used:

[1136] 1. Image recognition algorithms: (TensorFlow, OpenCV) are used to analyze image data and identify the status of facilities and equipment.

[1137] 2. Generative AI: Uses (GPT-4) to generate an optimal maintenance plan based on the analysis results.

[1138] 3. Database: (MySQL, PostgreSQL) is used to manage image data and feedback information.

[1139] System operation explanation

[1140] 1. Take and upload an image

[1141] An engineer uses a smartphone to take a photo of the robot, for example, the joints of the robot arm, and uploads the image to a server using a dedicated app.

[1142] 2. Receiving and preprocessing image data

[1143] The server performs noise reduction and resolution adjustment on the received image data, using TensorFlow to improve the image quality.

[1144] 3. Image Recognition Processing

[1145] The preprocessed image data is analyzed using OpenCV to identify robot parts and defects, for example, to identify the degree of wear on joints.

[1146] 4. Generate a maintenance plan

[1147] Using GPT-4, a generative AI, it generates a detailed maintenance plan based on the recognition results, including specific steps and a list of required tools.

[1148] 5. Submitting and implementing the plan

[1149] The server sends the generated maintenance plan in PDF format to the technician's terminal, where the technician checks the plan and performs the actual maintenance work.

[1150] 6. Uploading feedback data

[1151] After completing the maintenance, the technician takes another photo of the robot's condition and uploads it to the server as feedback data, allowing the technician to check the results of the maintenance.

[1152] 7. Feedback analysis and retraining of the generative AI

[1153] The server re-analyzes the received feedback data and extracts useful information, which allows the generating AI to re-train and improve the accuracy of the next maintenance plan.

[1154] Specific examples

[1155] Case: Robot arm joint maintenance

[1156] 1. Take and upload an image

[1157] Terminal: A technician takes pictures of the robot arm's joints and uploads them to the server.

[1158] 2. Receiving and preprocessing image data

[1159] Server: Perform noise removal and resolution adjustment using TensorFlow.

[1160] 3. Image Recognition Processing

[1161] Server: Uses OpenCV to identify the degree of wear on the joints.

[1162] 4. Generate a maintenance plan

[1163] Server: Uses GPT-4 to generate replacement procedures for worn parts and a list of required tools.

[1164] 5. Submitting and implementing the plan

[1165] Server: The generated maintenance plan is sent to the technician in PDF format, who then performs the work according to the plan.

[1166] 6. Uploading feedback data

[1167] Terminal: After maintenance is completed, an image of the robot joints is taken and uploaded to the server.

[1168] 7. Feedback analysis and retraining of the generative AI

[1169] Server: Analyzes the feedback data sent and retrains GPT-4.

[1170] Example of input prompt for generative AI model:

[1171] Please generate detailed instructions for maintaining the joints of the robot arm below.

[1172] Prerequisites:

[1173] Wear on the joints was observed.

[1174] The tools available are a wrench, screwdriver, and lubricant.

[1175] Replacement parts are available in advance.

[1176] Instructions:

[1177] 1. Safety measures procedures

[1178] 2. Specific replacement procedures for worn parts

[1179] 3. List of tools and materials needed

[1180] 4. Final check step

[1181] This enables proper and efficient maintenance of robots in factories.

[1182] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1183] Step 1:

[1184] Taking and uploading images

[1185] The terminal provides a means for users to take images of specific parts of the robot, such as joints. The captured image data is uploaded to a server via a dedicated app. The input is the captured image data, and the output is the upload of the image data to the server.

[1186] Step 2:

[1187] Receiving and preprocessing image data

[1188] The server performs noise reduction and resolution adjustment on the received image data. Specifically, it uses TensorFlow to improve the image quality. The input is the image data sent from the user device, and the output is the preprocessed image data.

[1189] Step 3:

[1190] Image Recognition Processing

[1191] The server analyzes the preprocessed image data using OpenCV to identify robot parts and defects. For example, it identifies the degree of wear on joints. The input is the preprocessed image data, and the output is the analyzed specific information (e.g., the location and degree of wear).

[1192] Step 4:

[1193] Generate a maintenance plan

[1194] The server uses GPT-4 as a generation AI to generate a detailed maintenance plan based on the analysis results. The generated plan includes specific procedures and a list of required tools. The input is the analyzed specific information, and the output is the maintenance plan.

[1195] Step 5:

[1196] Submitting and implementing the plan

[1197] The server sends the generated maintenance plan in PDF format to the user's terminal. The user checks the received plan and performs maintenance work based on it. The input is the maintenance plan, and the output is the procedure checked by the user.

[1198] Step 6:

[1199] Feedback data upload

[1200] After the maintenance is completed, the terminal takes another photograph of the robot's condition and uploads it to the server as feedback data. The input is the image data after maintenance, and the output is the feedback data uploaded to the server.

[1201] Step 7:

[1202] Feedback analysis and retraining of the generation AI

[1203] The server re-analyzes the received feedback data and extracts useful information, which re-trains the generative AI and improves the accuracy of the next maintenance plan. The input is the feedback data, and the output is the re-trained generative AI model.

[1204] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1205] This invention is a system that aims to improve the quality and reduce the man-hours of telecommunications construction work, and combines image recognition functions with generative AI. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more appropriate method for presenting construction plans is realized. This system performs the following program processing.

[1206] System Program Processing

[1207] 1. Take and upload an image

[1208] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[1209] 2. Image Reception and Preprocessing

[1210] Server: Receives the uploaded image data and pre-processes it. Pre-processing includes image noise reduction and resolution adjustment, which improves the accuracy of the image recognition algorithm.

[1211] 3. Image Recognition Processing

[1212] Server: The pre-processed image data is input into an image recognition algorithm to extract information about the equipment and wiring. The server uses the image recognition algorithm to identify the type, location, and defects of the wiring. For example, it automatically identifies the color code and connection location of the wiring.

[1213] 4. Generate construction plans

[1214] Server: Generates optimal construction plans and repair procedures based on the image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans that detail areas where wiring changes are required and repair procedures.

[1215] 5. User Emotion Recognition

[1216] Device: While the user is reviewing the construction plan or procedure manual, the built-in camera captures the user's facial expression, and the captured facial image is sent to the emotion engine.

[1217] Emotion engine: Analyzes facial images to identify the user's emotional state, for example, detecting when the user is confused or frustrated.

[1218] 6. Adjusting the way plans are presented

[1219] Server: Based on the user's emotional state, adjust the presentation of the generated construction plans and procedures, for example by providing more detailed explanations or additional visual aids to facilitate user understanding.

[1220] 7. Construction implementation

[1221] User: The user carries out the actual construction work based on the adjusted construction plan. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[1222] 8. Uploading Feedback Data

[1223] Terminal: After the work is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[1224] 9. Receiving and Analyzing Feedback

[1225] Server: Receives post-construction feedback data, pre-processes it, and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[1226] 10. Retraining generative AI

[1227] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[1228] Specific examples

[1229] Case: Adding new wiring

[1230] 1. Take and upload an image

[1231] Terminal: A user takes a picture of an equipment rack in the office with their smartphone and uploads it to the server.

[1232] 2. Image Reception and Preprocessing

[1233] Server: Preprocesses the received image data by removing noise and adjusting the resolution.

[1234] 3. Image Recognition Processing

[1235] Server: Analyzes the preprocessed image and identifies the current wiring status and available ports.

[1236] 4. Generate construction plans

[1237] Server: Based on the image recognition results, the optimal layout of new wiring is generated and specific wiring procedures are compiled into a PDF.

[1238] 5. User Emotion Recognition

[1239] Device: As the user reviews the construction plan, the built-in camera captures the user's facial expression.

[1240] Emotion engine: Analyzes captured facial expressions to identify when the user is confused.

[1241] 6. Adjusting the way plans are presented

[1242] Server: Provide construction plans with more detailed instructions and additional visual aids to resolve user confusion.

[1243] 7. Construction implementation

[1244] User: Carry out wiring work based on the coordinated plan.

[1245] 8. Uploading Feedback Data

[1246] Terminal: After the installation is complete, take a picture of the completed wiring and upload it to the server.

[1247] 9. Receiving and Analyzing Feedback

[1248] Server: Analyzes the uploaded feedback data and identifies issues and areas for improvement.

[1249] 10. Retraining generative AI

[1250] Server: Retrains the generation AI based on the feedback to improve the accuracy of the next construction plan generation.

[1251] This series of steps utilizes an emotion engine to deepen user understanding and ensure high-quality construction.

[1252] The processing flow will be explained below.

[1253] Step 1:

[1254] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[1255] Step 2:

[1256] Terminal: The captured image data is uploaded from the terminal to the server via wireless communication or wired connection.

[1257] Step 3:

[1258] Server: The server stores the received image data and prepares it for further processing.

[1259] Step 4:

[1260] Server: Performs pre-processing of image data. Pre-processing includes image noise reduction, contrast adjustment, and resolution adjustment, which improves the accuracy of image recognition algorithms.

[1261] Step 5:

[1262] Server: The preprocessed image data is input into an image recognition algorithm to extract information about the equipment and wiring. Identified elements include the type, location, and defect of the wiring. For example, the color code and connection location of the wiring can be automatically identified.

[1263] Step 6:

[1264] Server: Generates optimal construction plans and repair procedures based on image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans detailing areas where wiring changes are required and repair procedures.

[1265] Step 7:

[1266] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[1267] Step 8:

[1268] Device: When the user checks the construction plan or procedure manual they received, the built-in camera captures their facial expression.

[1269] Step 9:

[1270] Emotion engine: Analyzes captured facial images to identify the user's emotional state, for example, detecting when the user is confused or dissatisfied.

[1271] Step 10:

[1272] Server: Based on the user's emotional state, adjust the presentation of the generated construction plans and procedures, for example by providing more detailed explanations or additional visual aids to facilitate user understanding.

[1273] Step 11:

[1274] User: The user carries out the actual construction work based on the adjusted construction plan. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[1275] Step 12:

[1276] Terminal: After the construction is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[1277] Step 13:

[1278] Server: Receives post-construction feedback data, pre-processes and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[1279] Step 14:

[1280] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[1281] This series of steps utilizes an emotion engine to deepen user understanding and ensure high-quality construction.

[1282] Example 2

[1283] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1284] In conventional telecommunications construction, ensuring the quality and efficiency of construction required a great deal of time and effort. In particular, there were many uncertainties in the appropriateness of construction plans and the presentation of repair procedures according to the on-site situation, and the progress of construction was affected by the user's own level of understanding and emotional state. This resulted in issues such as inconsistent construction quality, increased man-hours, and reduced user satisfaction.

[1285] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1286] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about facilities and wiring, means for generating an optimal construction plan and repair procedure based on the analysis results, means for transmitting the generated construction plan and procedure manual to a user terminal, means for recognizing the user's emotions during construction, means for adjusting the presentation method of the construction plan and procedure manual based on the recognized emotions, means for receiving and analyzing feedback data after construction, and means for relearning the generation AI based on the feedback data. This makes it possible to automatically generate an optimal construction plan based on image data of the construction site and flexibly adjust it according to the user's emotional state, thereby improving the quality and efficiency of construction.

[1287] "Photographed image data" refers to an image file photographed by a user at a construction site, and contains information including the state of wiring and equipment at the construction site.

[1288] The "receiving means" is a mechanism by which the server retrieves image data sent from the user terminal via the Internet.

[1289] The "preprocessing means" is a step in which the received image data is subjected to processes such as noise removal and resolution adjustment to improve the accuracy of the image recognition algorithm.

[1290] "Means for extracting information" refers to algorithms or techniques that analyze the pre-processed image data and identify relevant information about wiring and equipment.

[1291] The "means of generation" is a system that designs optimal construction plans and repair procedures based on the analysis results and presents them in document or digital format.

[1292] "Transmission means" refers to a method for transferring the construction plan and procedure manual generated by the server to the user's terminal.

[1293] The "means for recognizing emotions" is a technology that captures the user's facial expressions during construction and identifies the user's emotional state from the facial expression data.

[1294] The "means for adjusting the presentation method" is a mechanism for changing the content and presentation method of construction plans and procedures based on the emotion recognition results, thereby improving the user's understanding and satisfaction.

[1295] "Feedback data" refers to information on the results of construction and usage experience provided by users after construction.

[1296] "Means for receiving and analyzing" refers to a technology in which the server receives feedback data and uses that data to evaluate the performance of the construction plan and system.

[1297] "Relearning means" refers to a method for updating the generative AI model based on analyzed feedback data to improve the accuracy and efficiency of the next construction plan generation.

[1298] A "generative AI model" is an artificial intelligence model used to automatically generate construction plans and repair procedures, and includes natural language generation models such as GPT-4.

[1299] This invention is a system aimed at improving the quality and reducing the man-hours required for telecommunications construction. This system combines image recognition, generative AI, and an emotion engine that recognizes the user's emotions.

[1300] First, the user takes images of the construction site using a device (e.g., a smartphone or tablet). The images taken include the inside of a wiring board and cable connection points. The user then uses a dedicated application to upload the captured image data to a server. This upload requires an internet connection (Wi-Fi or mobile data communication).

[1301] The server uses the OpenCV library to preprocess the received image data, which includes image noise reduction (e.g., applying a Gaussian filter) and resolution adjustment (e.g., resizing). This preprocessing improves the accuracy of the image recognition algorithms used later.

[1302] The preprocessed image data is then fed into an image recognition algorithm using the TensorFlow library, which the server uses to extract information about the equipment and wiring, such as color codes, connection locations, and fault locations.

[1303] Based on the extracted information, the server uses generative AI (e.g., a natural language generation model such as GPT-4) to generate optimal construction plans and repair procedures. The generated plans are output in PDF format and transferred to the user's device. The user can refer to these plans to proceed with the construction work.

[1304] Additionally, while the user is reviewing the construction plan, the device's built-in camera captures the user's facial expression. This facial expression data is sent to an emotion engine (e.g., Emotion API), which then analyzes the user's emotional state on the server. Based on the analysis results, for example, if the user is confused, the server adjusts the way the construction plan is presented. Specifically, it provides detailed explanations or additional visual guides.

[1305] After the construction is completed, the user takes another image of the construction site and uploads it to the server as feedback data. The server receives and analyzes this feedback data. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[1306] Finally, the server retrains the generative AI model based on this feedback data, helping to improve the accuracy of the next construction plan generation.

[1307] As a concrete example, consider the task of adding new wiring. A user takes a picture of an equipment rack in their office with their smartphone and uploads it to the server. The server preprocesses and analyzes the image data to identify the current wiring status and available ports. It then compiles the generated construction plan into a PDF and provides it to the user. If the server determines that the user's facial expression is confused while reviewing the construction plan, it re-provides the plan with detailed explanations and additional visual materials. The user then carries out the wiring work based on the adjusted plan and uploads an image of the completed work. The server analyzes the feedback data and retrains the generating AI to improve the accuracy of the next construction plan.

[1308] Example prompt sentence:

[1309] "Server: Upload images of your telecommunications construction site. We will use generative AI to provide you with the optimal construction plan."

[1310] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1311] Step 1:

[1312] Taking and uploading images

[1313] User: The user uses a smartphone or tablet to take detailed images of the construction site, such as the inside of a wiring cabinet or cable connection points, carefully capturing them from multiple angles.

[1314] Device: Upload the captured image data to the server using a dedicated application (e.g., a mobile app). When you press the "Upload Image" button in the application, the image data is sent to the server via an internet connection.

[1315] Input: Construction site image data

[1316] Output: Image data uploaded to the server

[1317] Step 2:

[1318] Image reception and preprocessing

[1319] Server: The server receives the image data sent by the user. For example, a REST API using the Flask framework is used to receive the data.

[1320] Server: Preprocesses the received image data using the OpenCV library, applying a Gaussian filter to remove noise and resizing the image to the desired resolution.

[1321] Input: Uploaded image data

[1322] Output: Preprocessed image data

[1323] Step 3:

[1324] Image Recognition Processing

[1325] Server: The preprocessed image data is input to the image recognition algorithm using the TensorFlow library.

[1326] Server: Image recognition algorithms automatically extract information about wiring and equipment, such as the type of wiring, its location, color code, and any defects.

[1327] Input: Preprocessed image data

[1328] Output: Extracted information on facilities and wiring

[1329] Step 4:

[1330] Construction plan generation

[1331] Server: Based on the extracted information, a generative AI model (e.g., GPT-4) is used to generate optimal construction plans and repair procedures.

[1332] Server: The generated construction plans and repair procedures are documented in PDF format and provided to users in an easy-to-understand format.

[1333] Input: Image recognition results (extracted information on facilities and wiring)

[1334] Output: PDF document of construction plan

[1335] Step 5:

[1336] User Emotion Recognition

[1337] Device: While the user is viewing the construction plan, the built-in camera captures the user's facial expression.

[1338] Emotion Engine: The captured facial image is sent to the emotion engine, which analyzes it to identify the user's emotional state (e.g., confusion, frustration, relief).

[1339] Input: User's face image

[1340] Output: User's emotional state

[1341] Step 6:

[1342] Adjusting the way plans are presented

[1343] Server: Based on the emotion recognition results, it adjusts the way the construction plan is presented, for example, by providing detailed explanations or additional visual guides if the user is confused.

[1344] Server: The adjusted construction plan is compiled into a PDF again and resent to the user's device.

[1345] Input: User's emotional state

[1346] Output: PDF document of the coordinated construction plan

[1347] Step 7:

[1348] Construction implementation

[1349] User: Carries out work on-site based on the coordinated construction plan, for example installing new wiring or carrying out repairs as instructed.

[1350] User: Once the construction is complete, take another image of the completed state and upload it to the server as feedback data.

[1351] Input: Coordinated construction plan PDF document

[1352] Output: Completed image data and feedback

[1353] Step 8:

[1354] Feedback data upload

[1355] Terminal: After the construction is completed, the user uploads the image data and work report to the server.

[1356] Terminal: Press the "Report Completion" button in the dedicated application to send the data.

[1357] Input: Completed image data and feedback

[1358] Output: Feedback data uploaded to the server

[1359] Step 9:

[1360] Receiving and analyzing feedback

[1361] Server: Receives the feedback data and preprocesses it again.

[1362] Server: Analyzes the feedback and identifies the accuracy of the construction plan, the suitability of the execution, and areas for improvement.

[1363] Input: Uploaded feedback data

[1364] Output: Feedback analysis results

[1365] Step 10:

[1366] Retraining generative AI

[1367] Server: Based on the analysis results, the generative AI model is retrained to improve the accuracy of the next construction plan generation. For retraining, new training data including feedback data is used.

[1368] Input: Feedback analysis results

[1369] Output: An updated generative AI model

[1370] (Application example 2)

[1371] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1372] Equipment repair and maintenance work within factories requires accurate understanding of on-site conditions and the ability to provide repair plans quickly and efficiently. To improve worker understanding and efficiency, it is important not only to provide technical instructions but also to respond flexibly while taking into account the worker's emotional state. While conventional systems use image recognition for automation and generative AI for planning, they lack the ability to adjust based on the worker's emotions, making it difficult to achieve both improved work efficiency and quality.

[1373] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1374] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about the equipment and structure, means for generating an optimal construction plan and repair procedure based on the analysis results, means for transmitting the generated construction plan and procedure manual to a user terminal, means for recognizing the user's emotional state, means for adjusting the presentation method of the construction plan based on the emotional state, means for receiving and analyzing feedback data after construction, and means for relearning the generation AI based on the feedback data. This makes it possible to improve the worker's understanding, increase the efficiency of repair work, and improve the quality of the entire construction work.

[1375] "Photographed image data" refers to digital data of on-site images of facilities and structures photographed by a device such as a user terminal or a robot.

[1376] "Means for receiving" refers to a function or device for receiving image data sent from a user terminal or robot via a network.

[1377] "Preprocessing means" refers to software and processes for performing preprocessing such as noise removal and resolution adjustment on received image data.

[1378] "Means for analyzing" refers to image recognition algorithms and processes for extracting facility or structural information from pre-processed image data.

[1379] "Means of generation" refers to the generative AI model and process for creating optimal construction plans and repair procedures based on the analysis results.

[1380] "Transmission means" refers to a function or device for transmitting the generated construction plan or procedure manual to a user terminal.

[1381] "Means for recognizing the user's emotional state" refers to an emotion engine and its process for analyzing the user's facial expression image captured by the built-in camera and identifying the user's emotional state.

[1382] "Adjustment means" refers to a function or device for changing the presentation method of the generated construction plan based on the user's emotional state.

[1383] "Feedback data" refers to data related to work results, such as on-site images taken after construction and implementation reports.

[1384] "Means for re-learning" refers to the process of re-learning the generative AI model based on feedback data to improve the accuracy of the next construction plan generation.

[1385] To implement the present invention, the following system configuration is required.

[1386] Hardware Configuration

[1387] Robot body: A mobile robot used in factories, equipped with built-in cameras and sensors.

[1388] User device: A smartphone or tablet device with a built-in camera that communicates with the robot.

[1389] Server: A central server that processes and stores all data.

[1390] Software Configuration

[1391] Image recognition algorithms: YOLO and OpenCV are used to preprocess the received image data and extract information about the facilities and structures.

[1392] Generative AI model: Uses OpenAI GPT-4 and Google BERT. Generates optimal construction plans and repair procedures based on analysis results.

[1393] Emotion Engine: Using Emotion API and Affectiva, it recognizes the user's emotional state and adjusts how construction plans are presented.

[1394] Cloud servers: Amazon Web Services (AWS) and Google Cloud are used to process and store data.

[1395] System processing flow

[1396] The server mainly performs the following processes:

[1397] 1. Image Reception and Preprocessing

[1398] The server receives image data sent from user terminals and robots, and performs preprocessing such as noise removal and resolution adjustment on the received image data.

[1399] 2. Image Recognition Processing

[1400] The pre-processed image data is then fed into an image recognition algorithm to extract information about the equipment or structure, such as identifying abnormalities or broken areas on a conveyor belt.

[1401] 3. Creation of construction plans

[1402] Based on the extracted information, a generative AI model is used to generate optimal repair procedures, including detailed descriptions of work procedures when using related labor-saving devices.

[1403] 4. Emotion recognition and presentation adjustment

[1404] While the user is reviewing the construction plan, the built-in camera on the user's device captures the user's facial expression and sends it to the emotion engine. Based on the emotion recognition results, the way the construction plan is presented is adjusted to make it easier for the user to understand.

[1405] 5. Receiving and Analyzing Feedback

[1406] After construction is completed, feedback data (images and implementation reports) sent from the user device and robot is received and analyzed to determine the accuracy of the construction plan and identify areas for improvement.

[1407] 6. Retraining generative AI

[1408] The generative AI model will be retrained based on the feedback data to improve the accuracy of the next construction plan generation.

[1409] Specific examples

[1410] A specific example is a robot detecting an abnormality in a conveyor belt and repairing it.

[1411] First, the robot photographs any abnormalities on the conveyor belt and uploads the image data to a server. The server then uses an image recognition algorithm to identify the abnormalities and generates optimal repair procedures using a generative AI model. It then uses its built-in camera to recognize the user's emotional state and adjusts the way the procedure manual is presented as needed. After the repair is complete, the robot takes another image and uploads it to the server as feedback data. This data is used to retrain the generative AI model, improving its accuracy the next time.

[1412] Prompt Sentence Examples

[1413] For example, you might input the following prompt to a generative AI model:

[1414] Prompt: "Generate the optimal procedure for repairing the conveyor belt malfunction. Use the image data below as a starting point. [Image data link]. Please add detailed explanations of any parts that may be difficult for the worker."

[1415] In this way, by combining image recognition algorithms, generative AI models, and emotion engines, the present invention achieves improved efficiency and quality in equipment repair and maintenance work within factories.

[1416] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1417] Step 1:

[1418] Take and upload images:

[1419] The user terminal and robot take images of the on-site equipment and upload the image data to the server. At this time, the user terminal takes a picture of the problem area with a camera and transfers the captured image file to the server via the network. The input is the on-site image data, and the output is the saving of the image file on the server.

[1420] Step 2:

[1421] Image reception and pre-processing:

[1422] The server receives image data sent from the user device and the robot. After receiving the image data, it performs preprocessing such as noise removal and resolution adjustment on the image data. Specifically, it cleans up the images using an image processing library such as OpenCV. The input is the image file from the user device and the robot, and the output is the preprocessed image data.

[1423] Step 3:

[1424] Image Recognition Processing:

[1425] The server inputs the preprocessed image data into an image recognition algorithm to extract information about the equipment and its structure. For example, YOLO can be used to identify abnormalities in the equipment. The input is the preprocessed image data, and the output is a list of identified abnormalities and equipment information.

[1426] Step 4:

[1427] Generate construction plans:

[1428] The server uses a generative AI model based on the image recognition results to generate optimal construction plans and repair procedures. The plan generation includes specific work procedures and a list of required tools. The input is the image recognition results, and the output is the generated construction plan and procedure manual.

[1429] Step 5:

[1430] Submit your plan:

[1431] The server sends the generated construction plans and procedures to the user's terminal. At this time, the plans and procedures are provided in PDF file or text format. The construction plans and procedures are input, and the data is sent to the user's terminal as output.

[1432] Step 6:

[1433] User Emotion Recognition:

[1434] While the user is checking the construction plan, the device's built-in camera captures the user's facial expression. The captured facial image is sent to the server, which then uses an emotion engine to analyze the user's emotional state. The input is the user's facial image data, and the output is the user's emotional state (confusion, dissatisfaction, etc.).

[1435] Step 7:

[1436] Adjustments to how plans are presented:

[1437] The server adjusts the presentation of the generated construction plans and procedures based on the user's emotional state. For example, if the user is confused, it provides additional visual aids or detailed explanations. The user's emotional state is the input, and the adjusted construction plans and procedures are delivered as the output.

[1438] Step 8:

[1439] Construction implementation:

[1440] The user performs the actual repair work based on the adjusted construction plan. Specifically, the user repairs the equipment using the necessary tools and materials while referring to the generated procedure manual. The input is the adjusted construction plan, and the output is the completed repaired equipment.

[1441] Step 9:

[1442] Feedback data upload:

[1443] After the construction is completed, the user terminal and the robot take images of the site again and upload them to the server as feedback data. The input is the image data after construction, and the output is the feedback data, which is saved on the server.

[1444] Step 10:

[1445] Receiving and analyzing feedback:

[1446] The server receives the feedback data and uses image recognition algorithms to verify and analyze the construction results. Areas for improvement are identified and reflected in the generation of the next construction plan. The input is the feedback data, and the output is the analysis results and a list of areas for improvement.

[1447] Step 11:

[1448] Retraining generative AI:

[1449] The server retrains the generative AI model based on the analysis results to improve the accuracy of the next construction plan generation. The input is the analysis results and points for improvement, and the output is an improved generative AI model.

[1450] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1451] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1452] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1453] [Fourth embodiment]

[1454] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1455] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1456] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1457] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1458] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1459] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1460] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1461] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1462] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1463] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1464] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1465] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1466] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1467] This invention is a system that aims to improve the quality of telecommunications construction and reduce the man-hours required, and combines image recognition functions with generative AI. This system performs the following program processing.

[1468] System Program Processing

[1469] 1. Take and upload an image

[1470] Device: The user takes pictures of the equipment and wiring being worked on at the telecommunications construction site using a smartphone or tablet device and uploads the image data to the server. This image data includes the inside of the distribution board, cable connection points, etc.

[1471] 2. Image Reception and Preprocessing

[1472] Server: Receives the uploaded image data and pre-processes it. Pre-processing includes image noise reduction and resolution adjustment, which improves the accuracy of the image recognition algorithm.

[1473] 3. Image Recognition Processing

[1474] Server: Analyzes the pre-processed image data and extracts information about the equipment and wiring. The server uses image recognition algorithms to identify the type, location, and defects of the wiring. For example, it automatically identifies the color code and connection location of the wiring.

[1475] 4. Generate construction plans

[1476] Server: Generates optimal construction plans and repair procedures based on the image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans that detail areas where wiring changes are required and repair procedures.

[1477] 5. Submit and confirm your plan

[1478] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[1479] Terminal: The user checks the construction plan sent to them and carries out the work according to the instructions.

[1480] 6. Construction implementation

[1481] User: The user carries out the actual construction work based on the construction plan provided. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[1482] 7. Uploading Feedback Data

[1483] Terminal: After the work is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[1484] 8. Receiving and Analyzing Feedback

[1485] Server: Receives feedback data after construction and analyzes it again. The feedback data includes information on changes in condition before and after construction and the appropriateness of the construction plan.

[1486] 9. Retraining generative AI

[1487] Server: Based on the analysis results of the feedback data, the generation AI is retrained. This improves the accuracy of the generation AI, which is then used to generate the next construction plan.

[1488] Specific examples

[1489] Case: Adding new wiring

[1490] 1. Take and upload an image

[1491] Terminal: A user takes a picture of an equipment rack in the office with their smartphone and uploads it to the server.

[1492] 2. Image Reception and Preprocessing

[1493] Server: Preprocesses the received image data by removing noise and adjusting the resolution.

[1494] 3. Image Recognition Processing

[1495] Server: Analyzes the preprocessed image and identifies the current wiring status and available ports.

[1496] 4. Generate construction plans

[1497] Server: Based on the image recognition results, the optimal layout of new wiring is generated and specific wiring procedures are compiled into a PDF.

[1498] 5. Submit and confirm your plan

[1499] Server: Sends the completed instructions to the device.

[1500] Terminal: The user checks the installation plan sent to them and carries out the new wiring work.

[1501] 6. Construction implementation

[1502] User: Create new wiring according to a user-generated plan.

[1503] 7. Uploading Feedback Data

[1504] Terminal: After the installation is complete, take a picture of the completed wiring and upload it to the server.

[1505] 8. Receiving and Analyzing Feedback

[1506] Server: Receives and analyzes the uploaded feedback data.

[1507] 9. Retraining generative AI

[1508] Server: The generation AI is retrained based on the results of the feedback analysis, which helps improve the accuracy of the next plan generation.

[1509] This system will enable improved quality and reduced labor costs in telecommunications construction work.

[1510] The processing flow will be explained below.

[1511] Step 1:

[1512] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[1513] Step 2:

[1514] Terminal: The captured image data is uploaded from the terminal to the server via wireless communication or wired connection.

[1515] Step 3:

[1516] Server: The server stores the received image data and prepares it for further processing.

[1517] Step 4:

[1518] Server: Performs pre-processing of image data. Pre-processing includes image noise reduction, contrast adjustment, and resolution adjustment, which improves the accuracy of image recognition algorithms.

[1519] Step 5:

[1520] Server: The preprocessed image data is input into an image recognition algorithm to extract information about the equipment and wiring. Identified elements include the device model number, wiring type, connected port location, and defective parts.

[1521] Step 6:

[1522] Server: Analyzes image recognition results and generates optimal construction plans and repair procedures based on the current situation. Generative AI optimizes this process using past data and best practices.

[1523] Step 7:

[1524] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[1525] Step 8:

[1526] Terminal: The user checks the construction plan and procedure manual received and carries out the work according to the displayed procedure.

[1527] Step 9:

[1528] Terminal: After the construction is completed, the user takes another image of the site and uploads it to the server along with a construction completion report.

[1529] Step 10:

[1530] Server: Receives post-construction feedback data, pre-processes and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[1531] Step 11:

[1532] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[1533] This series of steps will improve the quality and reduce the amount of work required in telecommunications construction.

[1534] Example 1

[1535] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1536] Telecommunications construction projects require improved quality and reduced man-hours. However, manual on-site inspections and construction plan creation are time-consuming and labor-intensive, and there is a high possibility of human error. Furthermore, quality checks and follow-ups after construction are often insufficient, making them ineffective at improving quality. Furthermore, there is a lack of systems with re-learning capabilities, making it difficult to accumulate experience and knowledge.

[1537] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1538] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information, means for generating optimal plans and procedures based on the analysis results, means for transmitting the generated plans and procedures to a user terminal, means for receiving and analyzing feedback data after construction, and means for relearning the generation algorithm based on the feedback data. This makes it possible to improve the quality and reduce the labor required for telecommunications construction work.

[1539] The "receiving means" is a function for receiving image data transmitted from a user terminal.

[1540] The "preprocessing means" is a function for removing noise and adjusting resolution from received image data.

[1541] The "means for analyzing" is a function that applies an algorithm to extract specific information from the pre-processed image data.

[1542] "Means for generation" refers to a function that automatically generates optimal plans and procedures based on the analysis results.

[1543] "Transmission means" is a function for sending the generated plan or procedure manual to the user's terminal.

[1544] The "means for receiving and analyzing" is a function for receiving feedback data sent by the user after implementation and analyzing it again.

[1545] The "means for relearning" is a function that improves the generation algorithm based on the analysis results and increases the accuracy of the next generation.

[1546] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[1547] "Server" means a computer system for processing and storing data.

[1548] An "image recognition algorithm" is an analytical method for extracting specific information from image data.

[1549] A "generative algorithm" is an algorithm for generating new plans and procedures based on extracted information.

[1550] "Feedback data" refers to data on the condition of the site and the results of construction provided by the user after construction.

[1551] This invention is a system that aims to improve the quality and reduce the man-hours required for telecommunications construction work, and combines image recognition functions with generative AI. This system realizes efficient construction plan generation and feedback at construction sites through collaboration between users, terminals, and servers.

[1552] First, a user uses a device such as a smartphone or tablet at a telecommunications construction site. The user takes pictures of important locations at the construction site, such as distribution boards and cable connection points. Specific examples of devices used include smartphones and tablets. This makes it possible to accurately record the site conditions. The user then uploads the captured image data to a server using a dedicated application.

[1553] The server then receives the uploaded image data. Specific software used on the server includes the OpenCV library for image preprocessing and TensorFlow and PyTorch for image recognition algorithms. The received image data undergoes preprocessing such as noise removal and resolution adjustment. This preprocessing step improves the quality of the image and increases the accuracy of subsequent analysis.

[1554] Once preprocessing is complete, the server uses deep learning to analyze the image data. The goal of the analysis is to identify the type, location, and defect of the wiring. For example, it can automatically identify the color code and connection location of the wiring.

[1555] Furthermore, the server uses generative AI (specifically, a GPT-4 model) to generate optimal construction plans and repair procedures based on the image recognition results. These plans include efficient work procedures and lists of necessary tools and materials. The server converts the generated construction plans and procedures into PDF or image format and sends them to the user's device. The user then checks them and carries out the work according to the instructions.

[1556] After the construction is completed, the user takes another image of the wiring status and uploads it to the server again. The server receives the re-uploaded feedback data and re-analyzes it. By re-training the generation AI based on the analysis results, it is expected that the accuracy of the next construction plan generation will improve.

[1557] Specific examples

[1558] For example, consider the case of adding new wiring to an equipment rack in an office. A user takes a picture of the equipment rack with their smartphone and uploads it to a server. The server receives the image and performs preprocessing. It then uses an image recognition algorithm to identify the wiring status and available ports. Based on the image recognition results, the server uses generative AI to propose the optimal layout for the new wiring, generates a PDF with specific wiring procedures, and sends it to the user. The user performs the wiring work according to the received plan, and after completing the work, takes another picture and uploads the feedback data. The server reanalyzes this data and retrains the generative AI to improve the accuracy of the next plan generation.

[1559] Prompt Sentence Examples

[1560] "Propose the optimal placement for new cabling. Upload an image of your office equipment rack. Identify the current cabling situation and open ports, and generate an installation plan with efficient cabling procedures. Analyze the image below."

[1561] In this way, the present invention provides a concrete method for improving quality and reducing labor costs in telecommunications construction. By utilizing generative AI models and image recognition technology, efficient and accurate construction becomes possible.

[1562] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1563] Step 1:

[1564] Taking and uploading images

[1565] Device: The user takes pictures of wiring and equipment at a telecommunications construction site using a smartphone or tablet. When taking pictures, they record important details such as the inside of the wiring board and cable connection points. Specifically, the device uses the built-in camera app to take photos, and then uploads the captured image data to a server using a dedicated app. The input is the captured image data, and the output is an image file sent to the server.

[1566] Step 2:

[1567] Image reception and preprocessing

[1568] Server: The server receives the uploaded image data. It uses OpenCV and other tools to remove noise and adjust the resolution of the received image data to improve the image quality. Specific operations include smoothing and sharpening the image. The input is the uploaded image data, and the output is high-quality image data that has undergone preprocessing.

[1569] Step 3:

[1570] Image Recognition Processing

[1571] Server: Analyzes preprocessed image data using a deep learning model (for example, TensorFlow or PyTorch). Identifies the type, location, and defective areas of the wiring from the image. Specific operations include color code recognition and object detection. The input is preprocessed image data, and the output is the type and location of the wiring based on the analysis results, as well as the identification of defective areas.

[1572] Step 4:

[1573] Construction plan generation

[1574] Server: Based on the image recognition results, a generative AI (e.g., a GPT-4 model) is used to generate optimal construction plans and repair procedures. Specifically, it creates construction procedures based on the recognized wiring information. The input is the image recognition results data, and the output is the generation of specific construction plans and procedures. The plans include efficient work procedures, necessary tools, and material lists.

[1575] Step 5:

[1576] Submit and confirm your plan

[1577] Server: Converts the generated construction plans and procedures into PDF or image format and sends them to the user's device. Specific operations include converting the file format and sending it via email or a dedicated application. The input is the construction plan or procedure data, and the output is a file sent to the user's device.

[1578] Terminal: The user checks the construction plan received on the terminal. Specifically, the user opens the received PDF file and checks the construction procedures. At this time, the user prepares the tools and materials necessary for construction. The input is the document data sent from the server, and the output is that construction preparations are complete.

[1579] Step 6:

[1580] Construction implementation

[1581] User: The user carries out the work based on the presented construction plan. Specific actions include connecting cables as instructed and repairing any defects. Once the work is complete, the user takes another photo of the completed equipment and wiring. The input is the construction plan information, and the output is image data of the completed wiring.

[1582] Step 7:

[1583] Feedback data upload

[1584] Terminal: After the construction is completed, the user uploads the retaken image data and implementation report to the server using a dedicated application. Specific operations include selecting and sending the retaken image file. The input is the image data after the construction is completed and the implementation report, and the output is the feedback data uploaded to the server.

[1585] Step 8:

[1586] Receiving and analyzing feedback

[1587] Server: The server receives the uploaded feedback data and performs the analysis again. Specific operations include comparing images before and after construction and evaluating the appropriateness of the construction plan. The input is the feedback data, and the output is the analysis results.

[1588] Step 9:

[1589] Retraining generative AI

[1590] Server: The server retrains the generative AI based on the analysis results of the feedback data. Specific operations include adjusting the parameters of the generative model and adding new data sets. The input is the feedback analysis results, and the output is a retrained generative AI model. This improves the accuracy of the next construction plan generation.

[1591] (Application example 1)

[1592] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1593] Providing efficient and appropriate maintenance procedures for robots in operation in factories is difficult. In particular, it is necessary for on-site technicians to accurately assess the robot's condition and quickly perform necessary maintenance work. Another challenge is that there is no established method for utilizing feedback after maintenance to improve future maintenance plans.

[1594] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1595] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about the facility or equipment, means for generating an optimal maintenance plan or repair procedure based on the analysis results, means for transmitting the generated maintenance plan or procedure manual to a user terminal, means for receiving and analyzing feedback data after construction, and means for relearning the generating AI based on the feedback data. This enables the provision of efficient maintenance procedures for robots in factories and continuous improvement based on subsequent feedback.

[1596] "Image data" refers to photographed image information that records the status of robots and equipment in the factory.

[1597] "Preprocessing" refers to data processing operations such as noise removal and resolution adjustment that are performed to make the received image data easier to analyze.

[1598] "Analysis" refers to the process of extracting specific information from preprocessed image data and using that information to evaluate the condition of facilities and equipment.

[1599] "Facility" refers to all the devices and systems used within a factory.

[1600] "Equipment" refers to specific devices and tools in a factory, including robots.

[1601] A "maintenance plan" refers to a procedure manual for repairing and maintaining facilities and equipment, created based on extracted information.

[1602] "Repair procedure" refers to the specific repair methods and processes used for equipment and facilities.

[1603] "User terminal" refers to a mobile terminal such as a smartphone or tablet used by a field technician.

[1604] "Feedback data" refers to information such as images taken after maintenance work and work reports.

[1605] "Analysis results" refers to the analysis results of information and data extracted by image recognition algorithms.

[1606] "Generative AI" refers to artificial intelligence that automatically generates optimal maintenance procedures and plans based on assumptions and data.

[1607] This invention is a system that provides efficient maintenance procedures for robots operating in factories. Specifically, it generates an optimal maintenance plan based on captured image data, and then uses subsequent feedback data to improve the accuracy of the generating AI.

[1608] System Hardware

[1609] This system uses the following hardware:

[1610] 1. Smartphone or tablet device: The field technician takes a photo of the robot's status and uploads the image data to the server.

[1611] 2. Server: Receives image data, pre-processes, analyzes, generates maintenance plans, and retrains feedback data.

[1612] System Software

[1613] The following software is used:

[1614] 1. Image recognition algorithms: (TensorFlow, OpenCV) are used to analyze image data and identify the status of facilities and equipment.

[1615] 2. Generative AI: Uses (GPT-4) to generate an optimal maintenance plan based on the analysis results.

[1616] 3. Database: (MySQL, PostgreSQL) is used to manage image data and feedback information.

[1617] System operation explanation

[1618] 1. Take and upload an image

[1619] An engineer uses a smartphone to take a photo of the robot, for example, the joints of the robot arm, and uploads the image to a server using a dedicated app.

[1620] 2. Receiving and preprocessing image data

[1621] The server performs noise reduction and resolution adjustment on the received image data, using TensorFlow to improve the image quality.

[1622] 3. Image Recognition Processing

[1623] The preprocessed image data is analyzed using OpenCV to identify robot parts and defects, for example, to identify the degree of wear on joints.

[1624] 4. Generate a maintenance plan

[1625] Using GPT-4, a generative AI, it generates a detailed maintenance plan based on the recognition results, including specific steps and a list of required tools.

[1626] 5. Submitting and implementing the plan

[1627] The server sends the generated maintenance plan in PDF format to the technician's terminal, where the technician checks the plan and performs the actual maintenance work.

[1628] 6. Uploading feedback data

[1629] After completing the maintenance, the technician takes another photo of the robot's condition and uploads it to the server as feedback data, allowing the technician to check the results of the maintenance.

[1630] 7. Feedback analysis and retraining of the generative AI

[1631] The server re-analyzes the received feedback data and extracts useful information, which allows the generating AI to re-train and improve the accuracy of the next maintenance plan.

[1632] Specific examples

[1633] Case: Robot arm joint maintenance

[1634] 1. Take and upload an image

[1635] Terminal: A technician takes pictures of the robot arm's joints and uploads them to the server.

[1636] 2. Receiving and preprocessing image data

[1637] Server: Perform noise removal and resolution adjustment using TensorFlow.

[1638] 3. Image Recognition Processing

[1639] Server: Uses OpenCV to identify the degree of wear on the joints.

[1640] 4. Generate a maintenance plan

[1641] Server: Uses GPT-4 to generate replacement procedures for worn parts and a list of required tools.

[1642] 5. Submitting and implementing the plan

[1643] Server: The generated maintenance plan is sent to the technician in PDF format, who then performs the work according to the plan.

[1644] 6. Uploading feedback data

[1645] Terminal: After maintenance is completed, an image of the robot joints is taken and uploaded to the server.

[1646] 7. Feedback analysis and retraining of the generative AI

[1647] Server: Analyzes the feedback data sent and retrains GPT-4.

[1648] Example of input prompt for generative AI model:

[1649] Please generate detailed instructions for maintaining the joints of the robot arm below.

[1650] Prerequisites:

[1651] Wear on the joints was observed.

[1652] The tools available are a wrench, screwdriver, and lubricant.

[1653] Replacement parts are available in advance.

[1654] Instructions:

[1655] 1. Safety measures procedures

[1656] 2. Specific replacement procedures for worn parts

[1657] 3. List of tools and materials needed

[1658] 4. Final check step

[1659] This enables proper and efficient maintenance of robots in factories.

[1660] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1661] Step 1:

[1662] Taking and uploading images

[1663] The terminal provides a means for users to take images of specific parts of the robot, such as joints. The captured image data is uploaded to a server via a dedicated app. The input is the captured image data, and the output is the upload of the image data to the server.

[1664] Step 2:

[1665] Receiving and preprocessing image data

[1666] The server performs noise reduction and resolution adjustment on the received image data. Specifically, it uses TensorFlow to improve the image quality. The input is the image data sent from the user device, and the output is the preprocessed image data.

[1667] Step 3:

[1668] Image Recognition Processing

[1669] The server analyzes the preprocessed image data using OpenCV to identify robot parts and defects. For example, it identifies the degree of wear on joints. The input is the preprocessed image data, and the output is the analyzed specific information (e.g., the location and degree of wear).

[1670] Step 4:

[1671] Generate a maintenance plan

[1672] The server uses GPT-4 as a generation AI to generate a detailed maintenance plan based on the analysis results. The generated plan includes specific procedures and a list of required tools. The input is the analyzed specific information, and the output is the maintenance plan.

[1673] Step 5:

[1674] Submitting and implementing the plan

[1675] The server sends the generated maintenance plan in PDF format to the user's terminal. The user checks the received plan and performs maintenance work based on it. The input is the maintenance plan, and the output is the procedure checked by the user.

[1676] Step 6:

[1677] Feedback data upload

[1678] After the maintenance is completed, the terminal takes another photograph of the robot's condition and uploads it to the server as feedback data. The input is the image data after maintenance, and the output is the feedback data uploaded to the server.

[1679] Step 7:

[1680] Feedback analysis and retraining of the generation AI

[1681] The server re-analyzes the received feedback data and extracts useful information, which re-trains the generative AI and improves the accuracy of the next maintenance plan. The input is the feedback data, and the output is the re-trained generative AI model.

[1682] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1683] This invention is a system that aims to improve the quality and reduce the man-hours of telecommunications construction work, and combines image recognition functions with generative AI. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more appropriate method for presenting construction plans is realized. This system performs the following program processing.

[1684] System Program Processing

[1685] 1. Take and upload an image

[1686] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[1687] 2. Image Reception and Preprocessing

[1688] Server: Receives the uploaded image data and pre-processes it. Pre-processing includes image noise reduction and resolution adjustment, which improves the accuracy of the image recognition algorithm.

[1689] 3. Image Recognition Processing

[1690] Server: The pre-processed image data is input into an image recognition algorithm to extract information about the equipment and wiring. The server uses the image recognition algorithm to identify the type, location, and defects of the wiring. For example, it automatically identifies the color code and connection location of the wiring.

[1691] 4. Generate construction plans

[1692] Server: Generates optimal construction plans and repair procedures based on the image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans that detail areas where wiring changes are required and repair procedures.

[1693] 5. User Emotion Recognition

[1694] Device: While the user is reviewing the construction plan or procedure manual, the built-in camera captures the user's facial expression, and the captured facial image is sent to the emotion engine.

[1695] Emotion engine: Analyzes facial images to identify the user's emotional state, for example, detecting when the user is confused or frustrated.

[1696] 6. Adjusting the way plans are presented

[1697] Server: Based on the user's emotional state, adjust the presentation of the generated construction plans and procedures, for example by providing more detailed explanations or additional visual aids to facilitate user understanding.

[1698] 7. Construction implementation

[1699] User: The user carries out the actual construction work based on the adjusted construction plan. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[1700] 8. Uploading Feedback Data

[1701] Terminal: After the work is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[1702] 9. Receiving and Analyzing Feedback

[1703] Server: Receives post-construction feedback data, pre-processes it, and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[1704] 10. Retraining generative AI

[1705] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[1706] Specific examples

[1707] Case: Adding new wiring

[1708] 1. Take and upload an image

[1709] Terminal: A user takes a picture of an equipment rack in the office with their smartphone and uploads it to the server.

[1710] 2. Image Reception and Preprocessing

[1711] Server: Preprocesses the received image data by removing noise and adjusting the resolution.

[1712] 3. Image Recognition Processing

[1713] Server: Analyzes the preprocessed image and identifies the current wiring status and available ports.

[1714] 4. Generate construction plans

[1715] Server: Based on the image recognition results, the optimal layout of new wiring is generated and specific wiring procedures are compiled into a PDF.

[1716] 5. User Emotion Recognition

[1717] Device: As the user reviews the construction plan, the built-in camera captures the user's facial expression.

[1718] Emotion engine: Analyzes captured facial expressions to identify when the user is confused.

[1719] 6. Adjusting the way plans are presented

[1720] Server: Provide construction plans with more detailed instructions and additional visual aids to resolve user confusion.

[1721] 7. Construction implementation

[1722] User: Carry out wiring work based on the coordinated plan.

[1723] 8. Uploading Feedback Data

[1724] Terminal: After the installation is complete, take a picture of the completed wiring and upload it to the server.

[1725] 9. Receiving and Analyzing Feedback

[1726] Server: Analyzes the uploaded feedback data and identifies issues and areas for improvement.

[1727] 10. Retraining generative AI

[1728] Server: Retrains the generation AI based on the feedback to improve the accuracy of the next construction plan generation.

[1729] This series of steps utilizes an emotion engine to deepen user understanding and ensure high-quality construction.

[1730] The processing flow will be explained below.

[1731] Step 1:

[1732] Device: Users use smartphones or tablet devices at telecommunications construction sites to take images of the site, including the inside of distribution panels and cable connection points.

[1733] Step 2:

[1734] Terminal: The captured image data is uploaded from the terminal to the server via wireless communication or wired connection.

[1735] Step 3:

[1736] Server: The server stores the received image data and prepares it for further processing.

[1737] Step 4:

[1738] Server: Performs pre-processing of image data. Pre-processing includes image noise reduction, contrast adjustment, and resolution adjustment, which improves the accuracy of image recognition algorithms.

[1739] Step 5:

[1740] Server: The preprocessed image data is input into an image recognition algorithm to extract information about the equipment and wiring. Identified elements include the type, location, and defect of the wiring. For example, the color code and connection location of the wiring can be automatically identified.

[1741] Step 6:

[1742] Server: Generates optimal construction plans and repair procedures based on image recognition results. The server uses generative AI to automatically create efficient construction plans from the analysis results and provide them to the user. For example, it generates plans detailing areas where wiring changes are required and repair procedures.

[1743] Step 7:

[1744] Server: Creates the generated construction plans and procedures in PDF or image format and sends them to the user's device.

[1745] Step 8:

[1746] Device: When the user checks the construction plan or procedure manual they received, the built-in camera captures their facial expression.

[1747] Step 9:

[1748] Emotion engine: Analyzes captured facial images to identify the user's emotional state, for example, detecting when the user is confused or dissatisfied.

[1749] Step 10:

[1750] Server: Based on the user's emotional state, adjust the presentation of the generated construction plans and procedures, for example by providing more detailed explanations or additional visual aids to facilitate user understanding.

[1751] Step 11:

[1752] User: The user carries out the actual construction work based on the adjusted construction plan. After the construction work is completed, the user takes photos of the completed equipment and wiring.

[1753] Step 12:

[1754] Terminal: After the construction is completed, the user takes another photo and uploads the image data and the implementation report to the server.

[1755] Step 13:

[1756] Server: Receives post-construction feedback data, pre-processes and analyzes it. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[1757] Step 14:

[1758] Server: Based on the analysis results, the generation AI is retrained to improve the accuracy of the next construction plan generation. This creates a feedback loop, continuously improving the performance of the entire system.

[1759] This series of steps utilizes an emotion engine to deepen user understanding and ensure high-quality construction.

[1760] Example 2

[1761] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1762] In conventional telecommunications construction, ensuring the quality and efficiency of construction required a great deal of time and effort. In particular, there were many uncertainties in the appropriateness of construction plans and the presentation of repair procedures according to the on-site situation, and the progress of construction was affected by the user's own level of understanding and emotional state. This resulted in issues such as inconsistent construction quality, increased man-hours, and reduced user satisfaction.

[1763] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1764] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about facilities and wiring, means for generating an optimal construction plan and repair procedure based on the analysis results, means for transmitting the generated construction plan and procedure manual to a user terminal, means for recognizing the user's emotions during construction, means for adjusting the presentation method of the construction plan and procedure manual based on the recognized emotions, means for receiving and analyzing feedback data after construction, and means for relearning the generation AI based on the feedback data. This makes it possible to automatically generate an optimal construction plan based on image data of the construction site and flexibly adjust it according to the user's emotional state, thereby improving the quality and efficiency of construction.

[1765] "Photographed image data" refers to an image file photographed by a user at a construction site, and contains information including the state of wiring and equipment at the construction site.

[1766] The "receiving means" is a mechanism by which the server retrieves image data sent from the user terminal via the Internet.

[1767] The "preprocessing means" is a step in which the received image data is subjected to processes such as noise removal and resolution adjustment to improve the accuracy of the image recognition algorithm.

[1768] "Means for extracting information" refers to algorithms or techniques that analyze the pre-processed image data and identify relevant information about wiring and equipment.

[1769] The "means of generation" is a system that designs optimal construction plans and repair procedures based on the analysis results and presents them in document or digital format.

[1770] "Transmission means" refers to a method for transferring the construction plan and procedure manual generated by the server to the user's terminal.

[1771] The "means for recognizing emotions" is a technology that captures the user's facial expressions during construction and identifies the user's emotional state from the facial expression data.

[1772] The "means for adjusting the presentation method" is a mechanism for changing the content and presentation method of construction plans and procedures based on the emotion recognition results, thereby improving the user's understanding and satisfaction.

[1773] "Feedback data" refers to information on the results of construction and usage experience provided by users after construction.

[1774] "Means for receiving and analyzing" refers to a technology in which the server receives feedback data and uses that data to evaluate the performance of the construction plan and system.

[1775] "Relearning means" refers to a method for updating the generative AI model based on analyzed feedback data to improve the accuracy and efficiency of the next construction plan generation.

[1776] A "generative AI model" is an artificial intelligence model used to automatically generate construction plans and repair procedures, and includes natural language generation models such as GPT-4.

[1777] This invention is a system aimed at improving the quality and reducing the man-hours required for telecommunications construction. This system combines image recognition, generative AI, and an emotion engine that recognizes the user's emotions.

[1778] First, the user takes images of the construction site using a device (e.g., a smartphone or tablet). The images taken include the inside of a wiring board and cable connection points. The user then uses a dedicated application to upload the captured image data to a server. This upload requires an internet connection (Wi-Fi or mobile data communication).

[1779] The server uses the OpenCV library to preprocess the received image data, which includes image noise reduction (e.g., applying a Gaussian filter) and resolution adjustment (e.g., resizing). This preprocessing improves the accuracy of the image recognition algorithms used later.

[1780] The preprocessed image data is then fed into an image recognition algorithm using the TensorFlow library, which the server uses to extract information about the equipment and wiring, such as color codes, connection locations, and fault locations.

[1781] Based on the extracted information, the server uses generative AI (e.g., a natural language generation model such as GPT-4) to generate optimal construction plans and repair procedures. The generated plans are output in PDF format and transferred to the user's device. The user can refer to these plans to proceed with the construction work.

[1782] Additionally, while the user is reviewing the construction plan, the device's built-in camera captures the user's facial expression. This facial expression data is sent to an emotion engine (e.g., Emotion API), which then analyzes the user's emotional state on the server. Based on the analysis results, for example, if the user is confused, the server adjusts the way the construction plan is presented. Specifically, it provides detailed explanations or additional visual guides.

[1783] After the construction is completed, the user takes another image of the construction site and uploads it to the server as feedback data. The server receives and analyzes this feedback data. The analysis results include the accuracy of the construction plan, the suitability of the implementation, and areas for improvement.

[1784] Finally, the server retrains the generative AI model based on this feedback data, helping to improve the accuracy of the next construction plan generation.

[1785] As a concrete example, consider the task of adding new wiring. A user takes a picture of an equipment rack in their office with their smartphone and uploads it to the server. The server preprocesses and analyzes the image data to identify the current wiring status and available ports. It then compiles the generated construction plan into a PDF and provides it to the user. If the server determines that the user's facial expression is confused while reviewing the construction plan, it re-provides the plan with detailed explanations and additional visual materials. The user then carries out the wiring work based on the adjusted plan and uploads an image of the completed work. The server analyzes the feedback data and retrains the generating AI to improve the accuracy of the next construction plan.

[1786] Example prompt sentence:

[1787] "Server: Upload images of your telecommunications construction site. We will use generative AI to provide you with the optimal construction plan."

[1788] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1789] Step 1:

[1790] Taking and uploading images

[1791] User: The user uses a smartphone or tablet to take detailed images of the construction site, such as the inside of a wiring cabinet or cable connection points, carefully capturing them from multiple angles.

[1792] Device: Upload the captured image data to the server using a dedicated application (e.g., a mobile app). When you press the "Upload Image" button in the application, the image data is sent to the server via an internet connection.

[1793] Input: Construction site image data

[1794] Output: Image data uploaded to the server

[1795] Step 2:

[1796] Image reception and preprocessing

[1797] Server: The server receives the image data sent by the user. For example, a REST API using the Flask framework is used to receive the data.

[1798] Server: Preprocesses the received image data using the OpenCV library, applying a Gaussian filter to remove noise and resizing the image to the desired resolution.

[1799] Input: Uploaded image data

[1800] Output: Preprocessed image data

[1801] Step 3:

[1802] Image Recognition Processing

[1803] Server: The preprocessed image data is input to the image recognition algorithm using the TensorFlow library.

[1804] Server: Image recognition algorithms automatically extract information about wiring and equipment, such as the type of wiring, its location, color code, and any defects.

[1805] Input: Preprocessed image data

[1806] Output: Extracted information on facilities and wiring

[1807] Step 4:

[1808] Construction plan generation

[1809] Server: Based on the extracted information, a generative AI model (e.g., GPT-4) is used to generate optimal construction plans and repair procedures.

[1810] Server: The generated construction plans and repair procedures are documented in PDF format and provided to users in an easy-to-understand format.

[1811] Input: Image recognition results (extracted information on facilities and wiring)

[1812] Output: PDF document of construction plan

[1813] Step 5:

[1814] User Emotion Recognition

[1815] Device: While the user is viewing the construction plan, the built-in camera captures the user's facial expression.

[1816] Emotion Engine: The captured facial image is sent to the emotion engine, which analyzes it to identify the user's emotional state (e.g., confusion, frustration, relief).

[1817] Input: User's face image

[1818] Output: User's emotional state

[1819] Step 6:

[1820] Adjusting the way plans are presented

[1821] Server: Based on the emotion recognition results, it adjusts the way the construction plan is presented, for example, by providing detailed explanations or additional visual guides if the user is confused.

[1822] Server: The adjusted construction plan is compiled into a PDF again and resent to the user's device.

[1823] Input: User's emotional state

[1824] Output: PDF document of the coordinated construction plan

[1825] Step 7:

[1826] Construction implementation

[1827] User: Carries out work on-site based on the coordinated construction plan, for example installing new wiring or carrying out repairs as instructed.

[1828] User: Once the construction is complete, take another image of the completed state and upload it to the server as feedback data.

[1829] Input: Coordinated construction plan PDF document

[1830] Output: Completed image data and feedback

[1831] Step 8:

[1832] Feedback data upload

[1833] Terminal: After the construction is completed, the user uploads the image data and work report to the server.

[1834] Terminal: Press the "Report Completion" button in the dedicated application to send the data.

[1835] Input: Completed image data and feedback

[1836] Output: Feedback data uploaded to the server

[1837] Step 9:

[1838] Receiving and analyzing feedback

[1839] Server: Receives the feedback data and preprocesses it again.

[1840] Server: Analyzes the feedback and identifies the accuracy of the construction plan, the suitability of the execution, and areas for improvement.

[1841] Input: Uploaded feedback data

[1842] Output: Feedback analysis results

[1843] Step 10:

[1844] Retraining generative AI

[1845] Server: Based on the analysis results, the generative AI model is retrained to improve the accuracy of the next construction plan generation. For retraining, new training data including feedback data is used.

[1846] Input: Feedback analysis results

[1847] Output: An updated generative AI model

[1848] (Application example 2)

[1849] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1850] Equipment repair and maintenance work within factories requires accurate understanding of on-site conditions and the ability to provide repair plans quickly and efficiently. To improve worker understanding and efficiency, it is important not only to provide technical instructions but also to respond flexibly while taking into account the worker's emotional state. While conventional systems use image recognition for automation and generative AI for planning, they lack the ability to adjust based on the worker's emotions, making it difficult to achieve both improved work efficiency and quality.

[1851] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1852] In this invention, the server includes means for receiving captured image data, means for preprocessing the received image data, means for analyzing the preprocessed image data and extracting information about the equipment and structure, means for generating an optimal construction plan and repair procedure based on the analysis results, means for transmitting the generated construction plan and procedure manual to a user terminal, means for recognizing the user's emotional state, means for adjusting the presentation method of the construction plan based on the emotional state, means for receiving and analyzing feedback data after construction, and means for relearning the generation AI based on the feedback data. This makes it possible to improve the worker's understanding, increase the efficiency of repair work, and improve the quality of the entire construction work.

[1853] "Photographed image data" refers to digital data of on-site images of facilities and structures photographed by a device such as a user terminal or a robot.

[1854] "Means for receiving" refers to a function or device for receiving image data sent from a user terminal or robot via a network.

[1855] "Preprocessing means" refers to software and processes for performing preprocessing such as noise removal and resolution adjustment on received image data.

[1856] "Means for analyzing" refers to image recognition algorithms and processes for extracting facility or structural information from pre-processed image data.

[1857] "Means of generation" refers to the generative AI model and process for creating optimal construction plans and repair procedures based on the analysis results.

[1858] "Transmission means" refers to a function or device for transmitting the generated construction plan or procedure manual to a user terminal.

[1859] "Means for recognizing the user's emotional state" refers to an emotion engine and its process for analyzing the user's facial expression image captured by the built-in camera and identifying the user's emotional state.

[1860] "Adjustment means" refers to a function or device for changing the presentation method of the generated construction plan based on the user's emotional state.

[1861] "Feedback data" refers to data related to work results, such as on-site images taken after construction and implementation reports.

[1862] "Means for re-learning" refers to the process of re-learning the generative AI model based on feedback data to improve the accuracy of the next construction plan generation.

[1863] To implement the present invention, the following system configuration is required.

[1864] Hardware Configuration

[1865] Robot body: A mobile robot used in factories, equipped with built-in cameras and sensors.

[1866] User device: A smartphone or tablet device with a built-in camera that communicates with the robot.

[1867] Server: A central server that processes and stores all data.

[1868] Software Configuration

[1869] Image recognition algorithms: YOLO and OpenCV are used to preprocess the received image data and extract information about the facilities and structures.

[1870] Generative AI model: Uses OpenAI GPT-4 and Google BERT. Generates optimal construction plans and repair procedures based on analysis results.

[1871] Emotion Engine: Using Emotion API and Affectiva, it recognizes the user's emotional state and adjusts how construction plans are presented.

[1872] Cloud servers: Amazon Web Services (AWS) and Google Cloud are used to process and store data.

[1873] System processing flow

[1874] The server mainly performs the following processes:

[1875] 1. Image Reception and Preprocessing

[1876] The server receives image data sent from user terminals and robots, and performs preprocessing such as noise removal and resolution adjustment on the received image data.

[1877] 2. Image Recognition Processing

[1878] The pre-processed image data is then fed into an image recognition algorithm to extract information about the equipment or structure, such as identifying abnormalities or broken areas on a conveyor belt.

[1879] 3. Creation of construction plans

[1880] Based on the extracted information, a generative AI model is used to generate optimal repair procedures, including detailed descriptions of work procedures when using related labor-saving devices.

[1881] 4. Emotion recognition and presentation adjustment

[1882] While the user is reviewing the construction plan, the built-in camera on the user's device captures the user's facial expression and sends it to the emotion engine. Based on the emotion recognition results, the way the construction plan is presented is adjusted to make it easier for the user to understand.

[1883] 5. Receiving and Analyzing Feedback

[1884] After construction is completed, feedback data (images and implementation reports) sent from the user device and robot is received and analyzed to determine the accuracy of the construction plan and identify areas for improvement.

[1885] 6. Retraining generative AI

[1886] The generative AI model will be retrained based on the feedback data to improve the accuracy of the next construction plan generation.

[1887] Specific examples

[1888] A specific example is a robot detecting an abnormality in a conveyor belt and repairing it.

[1889] First, the robot photographs any abnormalities on the conveyor belt and uploads the image data to a server. The server then uses an image recognition algorithm to identify the abnormalities and generates optimal repair procedures using a generative AI model. It then uses its built-in camera to recognize the user's emotional state and adjusts the way the procedure manual is presented as needed. After the repair is complete, the robot takes another image and uploads it to the server as feedback data. This data is used to retrain the generative AI model, improving its accuracy the next time.

[1890] Prompt Sentence Examples

[1891] For example, you might input the following prompt to a generative AI model:

[1892] Prompt: "Generate the optimal procedure for repairing the conveyor belt malfunction. Use the image data below as a starting point. [Image data link]. Please add detailed explanations of any parts that may be difficult for the worker."

[1893] In this way, by combining image recognition algorithms, generative AI models, and emotion engines, the present invention achieves improved efficiency and quality in equipment repair and maintenance work within factories.

[1894] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1895] Step 1:

[1896] Take and upload images:

[1897] The user terminal and robot take images of the on-site equipment and upload the image data to the server. At this time, the user terminal takes a picture of the problem area with a camera and transfers the captured image file to the server via the network. The input is the on-site image data, and the output is the saving of the image file on the server.

[1898] Step 2:

[1899] Image reception and pre-processing:

[1900] The server receives image data sent from the user device and the robot. After receiving the image data, it performs preprocessing such as noise removal and resolution adjustment on the image data. Specifically, it cleans up the images using an image processing library such as OpenCV. The input is the image file from the user device and the robot, and the output is the preprocessed image data.

[1901] Step 3:

[1902] Image Recognition Processing:

[1903] The server inputs the preprocessed image data into an image recognition algorithm to extract information about the equipment and its structure. For example, YOLO can be used to identify abnormalities in the equipment. The input is the preprocessed image data, and the output is a list of identified abnormalities and equipment information.

[1904] Step 4:

[1905] Generate construction plans:

[1906] The server uses a generative AI model based on the image recognition results to generate optimal construction plans and repair procedures. The plan generation includes specific work procedures and a list of required tools. The input is the image recognition results, and the output is the generated construction plan and procedure manual.

[1907] Step 5:

[1908] Submit your plan:

[1909] The server sends the generated construction plans and procedures to the user's terminal. At this time, the plans and procedures are provided in PDF file or text format. The construction plans and procedures are input, and the data is sent to the user's terminal as output.

[1910] Step 6:

[1911] User Emotion Recognition:

[1912] While the user is checking the construction plan, the device's built-in camera captures the user's facial expression. The captured facial image is sent to the server, which then uses an emotion engine to analyze the user's emotional state. The input is the user's facial image data, and the output is the user's emotional state (confusion, dissatisfaction, etc.).

[1913] Step 7:

[1914] Adjustments to how plans are presented:

[1915] The server adjusts the presentation of the generated construction plans and procedures based on the user's emotional state. For example, if the user is confused, it provides additional visual aids or detailed explanations. The user's emotional state is the input, and the adjusted construction plans and procedures are delivered as the output.

[1916] Step 8:

[1917] Construction implementation:

[1918] The user performs the actual repair work based on the adjusted construction plan. Specifically, the user repairs the equipment using the necessary tools and materials while referring to the generated procedure manual. The input is the adjusted construction plan, and the output is the completed repaired equipment.

[1919] Step 9:

[1920] Feedback data upload:

[1921] After the construction is completed, the user terminal and the robot take images of the site again and upload them to the server as feedback data. The input is the image data after construction, and the output is the feedback data, which is saved on the server.

[1922] Step 10:

[1923] Receiving and analyzing feedback:

[1924] The server receives the feedback data and uses image recognition algorithms to verify and analyze the construction results. Areas for improvement are identified and reflected in the generation of the next construction plan. The input is the feedback data, and the output is the analysis results and a list of areas for improvement.

[1925] Step 11:

[1926] Retraining generative AI:

[1927] The server retrains the generative AI model based on the analysis results to improve the accuracy of the next construction plan generation. The input is the analysis results and points for improvement, and the output is an improved generative AI model.

[1928] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1929] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1930] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1931] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1932] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1933] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1934] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1935] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1936] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1937] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1938] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1939] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1940] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1941] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1942] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1943] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1944] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1945] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1946] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1947] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1948] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1949] The following is further disclosed regarding the above embodiment.

[1950] (Claim 1)

[1951] A means for receiving captured image data;

[1952] means for preprocessing the received image data;

[1953] A means for analyzing the preprocessed image data and extracting information about the equipment and wiring;

[1954] A means to generate optimal construction plans and repair procedures based on the analysis results,

[1955] A means for transmitting the generated construction plan and procedure manual to a user terminal;

[1956] means for receiving and analyzing post-construction feedback data;

[1957] A means to retrain the generative AI based on feedback data, and

[1958] A system including:

[1959] (Claim 2)

[1960] 2. The system according to claim 1, wherein the user terminal uploads the captured image data to the server.

[1961] (Claim 3)

[1962] 10. The system of claim 1, wherein the server analyzes the image data using an image recognition algorithm.

[1963] "Example 1"

[1964] (Claim 1)

[1965] A means for receiving captured image data;

[1966] means for preprocessing the received image data;

[1967] means for analyzing the preprocessed image data and extracting information;

[1968] A means of generating optimal plans and procedures based on the analysis results,

[1969] means for transmitting the generated plan and procedure manual to a user terminal;

[1970] means for receiving and analyzing post-implementation feedback data;

[1971] a means for retraining the generative algorithm based on the feedback data;

[1972] A system including:

[1973] (Claim 2)

[1974] 2. The system according to claim 1, wherein the user terminal uploads the captured image data to the server.

[1975] (Claim 3)

[1976] 10. The system of claim 1, wherein the server analyzes the image data using an image recognition algorithm.

[1977] "Application Example 1"

[1978] (Claim 1)

[1979] A means for receiving captured image data;

[1980] means for preprocessing the received image data;

[1981] means for analyzing the preprocessed image data and extracting facility and equipment information;

[1982] A means to generate optimal maintenance plans and repair procedures based on the a...

Claims

1. A means for receiving captured image data; means for preprocessing the received image data; A means for analyzing the preprocessed image data and extracting information about the equipment and wiring; A means to generate optimal construction plans and repair procedures based on the analysis results, A means for transmitting the generated construction plan and procedure manual to a user terminal; means for receiving and analyzing post-construction feedback data; A means to retrain the generative AI based on feedback data, and A system including:

2. 2. The system according to claim 1, wherein the user terminal uploads the captured image data to the server.

3. 10. The system of claim 1, wherein the server analyzes the image data using an image recognition algorithm.

Images