system

Abstract

Provide a system. A means for inputting equipment status information; means for transmitting equipment status information to a server; A means for analyzing the equipment status information received by the server; A means for determining whether simple repair is possible based on the analyzed information; a means for presenting a repair manual if simple repair is possible; A means of contacting the equipment manufacturer if simple repairs are not possible; A means of coordinating repair schedules with equipment manufacturers; means for notifying the user of the repair schedule; A means of recording the history of troubleshooting; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When a malfunction occurs in equipment (air conditioning, lighting, network equipment, etc.) in large buildings such as offices and hotels, users have to endure an uncomfortable environment, and building managers are unable to determine the cause of the malfunction and are forced to contact the equipment manufacturer. Equipment manufacturers also face the problem of having to rush to the site even when the problem can be solved with simple work. Furthermore, the building maintenance industry is facing increasingly serious issues such as an aging workforce and a labor shortage, creating a need for a new system that can quickly and efficiently respond to malfunctions. [Means for solving the problem]

[0005] This invention is a system that includes a means for inputting equipment status information and sending it to a server, a means for analyzing the equipment status information received by the server and determining whether simple repair is possible based on the analyzed information, a means for presenting a repair manual if simple repair is possible, and a means for contacting the equipment manufacturer if simple repair is not possible.Furthermore, by providing a means for arranging a repair schedule with the equipment manufacturer and notifying the user, and a means for recording the trouble response history, the system aims to improve the efficiency of failure response and reduce maintenance costs.

[0006] "Equipment status information" refers to information that indicates the state of the equipment, such as a photo of the equipment or an error message.

[0007] "Server" refers to a central processing unit that receives and transmits data over a network and performs analysis and processing.

[0008] "Analysis" refers to data processing to identify the cause of a failure based on equipment status information and determine whether simple repairs are possible.

[0009] "Simple repairs" refers to basic repairs that users can perform themselves, such as turning the power on and off or cleaning the filter.

[0010] "Repair Manual" refers to a document or guideline that details procedures and methods for carrying out simple repairs.

[0011] "Equipment manufacturers" refer to companies or contractors that manufacture or maintain facilities for buildings and hotels (such as air conditioning, lighting, and network equipment).

[0012] "Repair Schedule" refers to the date and time set for the equipment manufacturer to visit the site and repair the equipment.

[0013] "Trouble response history" refers to data that compiles records of responses to equipment failures and problems. This data is useful for predicting future failures and for preventive maintenance. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention is a building maintenance system for responding to equipment failures quickly and efficiently. The system supports responding to equipment failures by having the user input equipment status information, analyzing it via a server, and providing a simple repair manual or contacting the equipment manufacturer as necessary.

[0036] User Input

[0037] Users (building managers and general users) use their smartphones or PCs to input equipment status information into the "Building Trouble SOS" platform. This status information includes photos of the equipment and text of error messages. The device converts the input information into JSON format and sends it to the server.

[0038] Initial diagnosis by the server

[0039] The server analyzes the received JSON data and converts the photo and text data into the appropriate format. It then sends the data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The generated AI model predicts the cause of the failure based on the received information. The server determines whether a simple repair is possible based on the diagnosis results returned by the AI ​​model.

[0040] Simple repair manual provided

[0041] If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual and sends it to the user's device. The user then attempts the simple repair by following the repair manual displayed on their smartphone or PC. The manual contains detailed instructions on specific repair procedures, allowing the user to follow the steps to resolve the problem.

[0042] Contacting equipment manufacturers and arranging repair schedules

[0043] If the problem is not resolved by simple repairs, the user reports the situation on the application. The device then sends this report back to the server. The server records the information that the simple repairs failed and immediately and automatically contacts the equipment manufacturer. The server then automatically coordinates a repair schedule with the equipment manufacturer and determines the earliest possible date and time for repairs. The server then notifies the user's device of the confirmed schedule. The user then checks the notified repair schedule and waits for on-site repairs.

[0044] Recording and using troubleshooting history

[0045] The server records each troubleshooting step in a database. This data includes information entered by the user, the AI ​​diagnosis results, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. This data can be used to predict future failures and for preventive maintenance.

[0046] Specific examples

[0047] For example, if a building's air conditioning stops working, the building manager takes a photo of the air conditioning unit with their smartphone and uploads it to the app. If an error message is displayed, they enter the details. The device then sends this information to the server. The server analyzes the photo and text based on a generative AI model and determines that the filter is likely clogged. Based on the diagnosis, the server presents filter cleaning instructions and sends them to the user. The user then attempts to clean the filter by following the instructions displayed on their smartphone. If cleaning does not improve the situation, the user reports it again through the application, and the server automatically contacts the equipment manufacturer and schedules a repair. All of this information is recorded in a database and will be used for future analysis.

[0048] In this way, the system supports a series of processes for quickly and efficiently resolving equipment problems in building maintenance.

[0049] The processing flow will be explained below.

[0050] Step 1:

[0051] When a user notices that a building facility has broken down, they access the "Building Trouble SOS" platform using their smartphone or PC, take a photo of the facility, and enter a description of the current situation (e.g., "The air conditioning is not working" or "An error message is displayed") in text.

[0052] Step 2:

[0053] The device receives the input photo and text data and converts it into JSON format, including the type of equipment (air conditioning, lighting, network equipment, etc.).

[0054] Step 3:

[0055] The device sends the generated JSON data to the server, which includes the user's input information and photo data.

[0056] Step 4:

[0057] The server parses the received JSON data and converts the photo and text data into the appropriate format, which prepares it for passing to the AI ​​model.

[0058] Step 5:

[0059] The server sends the converted data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The AI ​​model predicts the cause of the failure.

[0060] Step 6:

[0061] The AI ​​model analyzes the photo and text data to identify the cause of the malfunction, such as a clogged filter or a blown fuse, and returns the diagnosis results to the server.

[0062] Step 7:

[0063] The server receives the diagnosis results from the AI ​​model and determines whether a simple repair is possible. If a simple repair is possible, it searches for the corresponding repair manual.

[0064] Step 8:

[0065] If the server determines that simple repair is possible, it retrieves the corresponding repair manual and sends it to the user's device. The repair manual contains specific repair procedures.

[0066] Step 9:

[0067] The device displays the received repair manual on its screen. The user attempts simple repairs by following the instructions in the manual. For example, the manual contains instructions for cleaning the filter and turning the power on and off.

[0068] Step 10:

[0069] After the user performs the simple repair, they report the results in the application. If the repair is successful, the application simply ends.

[0070] Step 11:

[0071] If the problem is not resolved by the simple repair, the user reports it as "not resolved" on the application. The terminal then sends this report to the server.

[0072] Step 12:

[0073] The server records the information about the failure of the simple repair and immediately contacts the equipment manufacturer automatically, sending a repair request to the equipment manufacturer.

[0074] Step 13:

[0075] The server receives the response from the equipment manufacturer and arranges the earliest possible date for repair. It also acquires the schedule information received from the equipment manufacturer.

[0076] Step 14:

[0077] The server transmits the confirmed repair schedule to the user's terminal and notifies the user.

[0078] Step 15:

[0079] The terminal notifies the user of the received repair schedule, and the user confirms the notified repair schedule and waits for the equipment manufacturer to visit.

[0080] Step 16:

[0081] The server records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the repair manual provided, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmation of repair dates.

[0082] Step 17:

[0083] The server analyzes the recorded data and updates a data model that can be used for future failure prediction and preventive maintenance, thereby improving the accuracy and speed of future failure responses.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] Conventional building maintenance systems often have delays in responding to equipment failures and unclear repair procedures, making it difficult to resolve problems. Even after simple repairs are attempted, feedback on their success or lack of prompt response time is often lacking, and additional specialized support is often not provided. Furthermore, data for failure prediction and preventive maintenance is not accumulated or analyzed, making it difficult to prevent future problems.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes a means for analyzing the equipment status information, a means for determining whether a simple repair is possible based on the analyzed information, and a means for contacting the manufacturer if a simple repair is not possible, thereby enabling the cause of the equipment failure to be quickly identified and an appropriate response to be proposed.

[0089] "Equipment status information" refers to data such as the operating status of machines and equipment within a building and error messages.

[0090] A "web server" refers to a computer system that transmits and receives data over the Internet or an intranet.

[0091] "Analysis" refers to the process of processing received data to understand its contents and identify the cause of the problem.

[0092] "Simple repair" refers to repair work that can be done by the user themselves, without the need for a specialized technician.

[0093] A "repair manual" is a document that provides instructions and methods for repairing a specific problem.

[0094] "Manufacturer" refers to a company that designs, manufactures, and sells equipment.

[0095] "Repair Schedule" means the date and time for carrying out repair work.

[0096] "Trouble response history" refers to data that records the troubles that occurred, how they were resolved, and the results.

[0097] "Equipment images" refers to photographs and video data of facilities and equipment.

[0098] "Fault determination" refers to the operation of identifying the cause and location of a failure in equipment or devices based on analyzed data.

[0099] "Preventive maintenance" refers to maintenance activities such as inspections and repairs carried out before equipment or machinery breaks down.

[0100] "Failure prediction" refers to the operation of predicting the possibility of future failure of facilities or equipment based on accumulated data.

[0101] This invention provides a building maintenance system that allows for quick and efficient response to equipment failures. Specifically, users, terminals, and servers work together to collect and analyze equipment status information and take action as needed.

[0102] Users access the "Building Trouble SOS" platform using their smartphones or PCs. They enter information about the equipment's status and, if necessary, take and upload photos. This information includes specific equipment malfunctions, error messages, and images of the equipment. The user's device converts this information into JSON format and sends it to the server.

[0103] The server analyzes the received JSON data and separates the photo data from the text data. The photo data is processed as an image file (e.g., JPEG format), and the text data is extracted as string information. The server then sends the photo data to an image recognition AI model (e.g., using TENSORFLOW (registered trademark)) and the text data to a natural language processing model (e.g., a model based on GPT-3 (registered trademark) or BERT). This allows the server to analyze the cause of the failure and, based on that information, determine whether a simple repair is possible.

[0104] If a simple repair is determined to be possible, the server retrieves the corresponding repair manual from the database and sends it to the user's device. The repair manual contains detailed instructions for the repair, so the user can follow them to attempt the simple repair. For example, if the air conditioner's filter is diagnosed as clogged, a manual including instructions for cleaning the filter will be provided.

[0105] If the simple repair is unsuccessful, the user reports this on the application. The device again sends the information to the server, which then automatically contacts the manufacturer. The server then coordinates a repair date with the manufacturer, determines the earliest possible date and time for repair, and notifies the user's device of this information. The user can then confirm the notified repair date and wait for on-site repair.

[0106] The server also records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the proposed repair manual, the success or failure of simple repairs, communication with the manufacturer, and confirmation of repair dates. This data is used to predict future failures and for preventive maintenance.

[0107] As a specific example of how it works, let's say a user takes a photo of an air conditioning unit with their smartphone and uploads it to the "Building Trouble SOS" app. They also enter the contents of any error messages. The device converts this information into JSON format and sends it to the server. The server analyzes the photo and text based on an AI model and diagnoses that "the filter is likely clogged." Based on this diagnosis, it presents filter cleaning instructions and sends them to the user. The user attempts to clean the filter by following the instructions displayed on their smartphone, but if there is no improvement, they report it again. The server automatically contacts the manufacturer and arranges a repair date. All of this information is recorded in a database and will be used for later analysis.

[0108] An example of a prompt sentence might be:

[0109] "If the air conditioning in a building is not working, it is likely that the filter is clogged. Enter the filter cleaning procedure as a simple repair manual in this case."

[0110] In this way, the system provides a series of processes for quickly and efficiently resolving equipment problems in building maintenance.

[0111] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0112] Step 1:

[0113] Users access the "Building Trouble SOS" platform using their smartphones or PCs. They input information about the equipment's status (for example, a report that the air conditioning is not working or an error message), and take and upload photos of the equipment if necessary. This input process provides specific information about the equipment's malfunction.

[0114] Step 2:

[0115] The terminal converts the information entered by the user and the uploaded photo into a single JSON data, which contains text information and image data separately, and sends the converted JSON data to the server.

[0116] Step 3:

[0117] The server parses the received JSON data and separates the photo data from the text data. The server saves the photo data as an image file (e.g., JPEG format) and the text data as string information. This analysis prepares the data needed for the next step.

[0118] Step 4:

[0119] The server sends the photo data to an image recognition AI model (e.g., using TensorFlow) and the text data to a natural language processing model (e.g., a GPT-3 or BERT-based model). The server receives the diagnosis results returned by these models. For example, an image recognition model may return a diagnosis that "the filter is likely clogged." The input at this stage is image data and text data, and the output is the diagnosis result.

[0120] Step 5:

[0121] The server determines whether a simple repair is possible based on the diagnosis results obtained from the AI ​​model. If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual from the database and sends it to the user's device. For example, a repair procedure such as "filter cleaning procedure" is provided to the user. In this step, the diagnosis results of the AI ​​model are the input, and the simple repair manual is the output.

[0122] Step 6:

[0123] The user attempts to perform a simple repair according to the repair manual displayed on the terminal, for example, removing and cleaning the filter of the air conditioning unit, and checks the results of the simple repair (for example, whether the air conditioning works properly).

[0124] Step 7:

[0125] If the simple repair is not successful, the user reports this in the application. The device then sends the data containing the user's report back to the server. This report includes the repair procedure that was attempted and the results.

[0126] Step 8:

[0127] The server records the failure of the simple repair based on the data resent by the user and automatically contacts the manufacturer. The server then automatically coordinates a repair schedule with the manufacturer and determines the earliest possible date and time. This output is the repair schedule information, which is then notified to the user's terminal.

[0128] Step 9:

[0129] The server records each step of the troubleshooting process in a database, including information entered by the user, the diagnosis results from the AI ​​model, the repair manual provided, the success or failure of simple repairs, communication with the manufacturer, and confirmation of repair dates. This data is used to predict future failures and for preventive maintenance.

[0130] (Application example 1)

[0131] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] When a robot operating in a factory breaks down or malfunctions, a high level of specialized knowledge and experience is required to identify the cause and quickly repair it. This makes it difficult for on-site operators and managers to respond quickly, leading to production line shutdowns and unexpected increases in costs. Furthermore, if appropriate repair procedures and communication and coordination with equipment manufacturers are not carried out efficiently, repair delays may occur. There is a need for a system that can solve these problems and perform robot maintenance efficiently and effectively.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0134] In this invention, the server includes a means for users to input equipment status information using a smart device, a means for using a generative AI model to analyze equipment photos and error messages, and a means for optimizing diagnostic and repair processes specific to the application category. This allows on-site operators and managers to easily input status information when a robot operating in a factory breaks down or malfunctions, enabling quick and accurate fault diagnosis based on that information. Furthermore, if a simple repair is possible, the system presents specific repair procedures and, if necessary, automatically contacts the equipment manufacturer and arranges a repair schedule, thereby minimizing repair delays and shortening production line downtime.

[0135] The "means for inputting equipment status information" refers to a means by which a user inputs information such as a photograph of the equipment or an error message into a terminal.

[0136] The "means for transmitting equipment status information to the server" is a communication means for transferring input equipment status information to the server.

[0137] The "means for analyzing the equipment status information received by the server" is an analysis function for processing the status information received by the server and identifying the cause of an abnormality or failure.

[0138] The "means for determining whether simple repair is possible based on the analyzed information" is a means for determining whether simple repair is possible based on the analyzed information.

[0139] The "means for presenting a repair manual when simple repair is possible" is a means for providing a user with a manual describing the repair method when simple repair is possible.

[0140] "Means for contacting the equipment manufacturer when simple repairs are not possible" refers to a means for automatically contacting the equipment manufacturer when repairs are difficult.

[0141] "Means for coordinating repair schedules with equipment manufacturers" refers to means for coordinating and deciding repair schedules with equipment manufacturers.

[0142] The "means for notifying the user of the repair schedule" is a means for notifying the user of the confirmed repair schedule.

[0143] "Means for recording the history of troubleshooting" refers to a means for saving the steps and results of each troubleshooting in a database.

[0144] "Means for a user to input equipment status information using a smart device" refers to means for a user to input equipment status information using a device such as a smartphone or smart glasses.

[0145] "Means using a generative AI model to analyze equipment photos and error messages" means means using a generative AI model to analyze equipment photos and error messages.

[0146] "Measures to optimize diagnostic and repair processes specific to the application category" are measures to streamline diagnostic and repair processes to suit the specific requirements and environment of the application.

[0147] This invention describes in detail a method for realizing a maintenance management system for robots operating in a factory. This system begins when a user inputs equipment status information using a smart device and sends that information to a server. Each step is described in detail below.

[0148] 1. User Input

[0149] Users (factory managers and operators) use their smartphones or smart glasses to input status information about robots that have malfunctioned or are experiencing an abnormality into the application. This status information includes a photo of the robot and an error message. The photographed and input information is converted into JSON format and sent to the server.

[0150] 2. Initial diagnosis by the server

[0151] The server receives JSON data sent from the user device. The received data is analyzed using a generative AI model (e.g., TensorFlow or PyTorch). The photo and text data are converted into an appropriate format and sent to the AI ​​model to predict the cause of the failure. During this analysis, anomalies are identified from the photo and the problem is extracted from the error message.

[0152] 3. Simple repair manual provided

[0153] If the server determines that simple repairs are possible based on the analysis results, it provides a repair manual to the user's terminal. The repair manual contains detailed instructions on specific procedures, and the user can attempt to repair the device by following the instructions displayed on the smart device.

[0154] 4. Contact the equipment manufacturer and arrange repair schedules

[0155] If the simple repair fails or if it is determined that a more complex repair is required, the user can report it again through the application. Based on the report, the server will automatically contact the equipment manufacturer and arrange a repair date. Once the arrangement is complete, the server will notify the user of the confirmed repair date.

[0156] 5. Recording and using troubleshooting history

[0157] The server records the history of each troubleshooting in a database. This data includes information entered by the user, the results of the AI ​​diagnosis, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. The accumulated data is used to predict future failures and for preventive maintenance.

[0158] Hardware and software used

[0159] Hardware: Smartphones, smart glasses, servers

[0160] Software: Generative AI models (TensorFlow, PyTorch), applications (for smart devices)

[0161] Specific examples

[0162] For example, if a robot arm in operation in a factory stops working properly, the operator takes a photo of the robot arm with the smart glasses and enters an error message. The information is then sent to the server via the app. The server analyzes the data using a generative AI model and diagnoses the malfunction as "malfunction due to motor overload." Based on the results, the user is presented with specific repair procedures. If the repair is unsuccessful, the operator contacts the equipment manufacturer again via the app to arrange a repair date.

[0163] Prompt Sentence Examples

[0164] "Please take a photo of any abnormalities in the robotic arms operating in the factory and enter the error message."

[0165] This system makes it possible to respond quickly and efficiently to malfunctions or abnormalities in robots operating in factories.

[0166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0167] Step 1:

[0168] The user uses a smart device to input equipment status information. Specifically, they take a photo of the abnormal part of the robot and enter the error message into the application. The input information is converted into JSON format.

[0169] Input: Photo data, text data (error message)

[0170] Output: JSON format data

[0171] Step 2:

[0172] The user's device (smartphone or smart glasses) sends equipment status information converted into JSON format to the server.

[0173] Input: JSON format data

[0174] Output: Data transfer to the server

[0175] Step 3:

[0176] The server analyzes the received JSON data. Specifically, it converts the photo data and text data into an appropriate format and sends it to a generative AI model (TensorFlow or PyTorch). The AI ​​model performs image analysis, identifies abnormalities from the photo, and performs text analysis to extract the cause of the failure from the error message.

[0177] Input: JSON format data

[0178] Output: Image analysis results, text analysis results

[0179] Step 4:

[0180] The server determines whether a simple repair is possible based on the analysis results. Based on the diagnosis results returned by the AI ​​model, it determines whether a simple repair is possible. This process refers to past repair history and common failure patterns in the database.

[0181] Input: Image analysis results, text analysis results

[0182] Output: Simple repair possible / impossible result

[0183] Step 5:

[0184] If it is determined that simple repair is possible, the server sends the corresponding repair manual to the user's terminal. The repair manual contains detailed instructions on the specific repair procedures, and the user attempts to repair the device by following the instructions displayed on the smart device.

[0185] Input: Simple repair possibility determination result

[0186] Output: Repair manual

[0187] Step 6:

[0188] If the simple repair fails or if it is determined that a more complex repair is required, the user can report it again through the application. Based on the report, the server will automatically contact the equipment manufacturer and arrange a repair date. Once the arrangement is complete, the server will notify the user of the confirmed repair date.

[0189] Input: User report data (simple repair failure report)

[0190] Output: Contact with equipment manufacturer, results of repair schedule adjustment, notification to user

[0191] Step 7:

[0192] The server records the history of each troubleshooting in a database. The recorded data includes information entered by the user, the results of the AI ​​diagnosis, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. This data is used to predict future failures and for preventive maintenance.

[0193] Input: Trouble response data (user input information, AI diagnosis results, repair results, communication logs, etc.)

[0194] Output: Records in the database

[0195] Prompt Sentence Examples

[0196] "Please take a photo of any abnormalities in the robotic arms operating in the factory and enter the error message."

[0197] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0198] This invention is a building maintenance system for responding quickly and efficiently to equipment breakdowns. The system supports responding to equipment breakdowns by having the user input equipment status information, analyzing it via a server, and providing a simple repair manual or contacting the equipment manufacturer as necessary. Furthermore, the system improves the user experience by combining it with an emotion engine that recognizes the user's emotions and flexibly adjusts its response depending on the situation.

[0199] User Input

[0200] Users (building managers and general users) use their smartphones or PCs to input equipment status information into the "Building Trouble SOS" platform. This status information includes photos of the equipment and text of error messages. The device converts the input information into JSON format and sends it to the server.

[0201] Initial diagnosis by the server

[0202] The server analyzes the received JSON data and converts the photo and text data into the appropriate format. It then sends the data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The generated AI model predicts the cause of the failure based on the received information. The server determines whether a simple repair is possible based on the diagnosis results returned by the AI ​​model.

[0203] Simple repair manual provided

[0204] If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual and sends it to the user's device. The user then attempts the simple repair by following the repair manual displayed on their smartphone or PC. The manual contains detailed instructions on specific repair procedures, allowing the user to follow the steps to resolve the problem.

[0205] Contacting equipment manufacturers and arranging repair schedules

[0206] If the problem is not resolved by simple repairs, the user reports the situation on the application. The device then sends this report back to the server. The server records the information that the simple repairs failed and immediately and automatically contacts the equipment manufacturer. The server then automatically coordinates a repair schedule with the equipment manufacturer and determines the earliest possible date and time for repairs. The server then notifies the user's device of the confirmed schedule. The user then checks the notified repair schedule and waits for on-site repairs.

[0207] User support with emotion engine

[0208] Furthermore, this system has a function that allows the emotion engine to recognize the user's emotions based on the user's input information and behavior. For example, if the user frequently inputs error messages or shows behavior that suggests frustration with repair procedures, the emotion engine will detect the user's stress or dissatisfaction.

[0209] Flexible response based on emotions

[0210] Based on the user's emotions recognized by the emotion engine, the server can flexibly change the messages and repair manuals it presents. For example, if the user is feeling anxious, it can provide a message containing words of encouragement and support. If the user is confused by a complicated procedure, it can present a simpler explanation or a video tutorial.

[0211] Expert Notification and Additional Support

[0212] If the emotion engine detects a high level of stress or dissatisfaction in the user, the server will automatically notify the equipment manufacturer's experts and request additional support. This feature allows users to receive professional assistance quickly and without having to struggle alone, improving the speed and accuracy of problem resolution.

[0213] Recording and using troubleshooting history

[0214] The server records each troubleshooting step in a database. This data includes information entered by the user, the AI ​​diagnosis results, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, confirmed repair schedules, and emotional information recognized by the emotion engine. This data can be used to predict future failures and for preventive maintenance.

[0215] Specific examples

[0216] For example, if a building's air conditioning stops working, the building manager takes a photo of the air conditioning unit with their smartphone and uploads it to the app. If an error message is displayed, they enter the details. The device then sends this information to the server. The server analyzes the photo and text based on a generative AI model and determines that the filter is likely clogged. Based on the diagnosis, the server presents filter cleaning instructions and sends them to the user. The user then attempts to clean the filter by following the instructions displayed on their smartphone. If cleaning does not improve the situation, the user reports it again through the application, and the server automatically contacts the equipment manufacturer and schedules a repair. All of this information is recorded in a database and will be used for future analysis.

[0217] Furthermore, if the user indicates a high level of stress in the emotion engine, the server notifies experts and prepares a means to provide additional support. For example, if a situation arises in which the user is "very troubled by frequent error messages," the emotion engine will recognize the stress and the server will immediately contact experts to request a prompt response. In this way, the system quickly and efficiently resolves equipment problems in building maintenance while simultaneously reducing the user's psychological burden.

[0218] The processing flow will be explained below.

[0219] Step 1:

[0220] When a user notices that a building facility has broken down, they access the "Building Trouble SOS" platform using their smartphone or PC, take a photo of the facility, and enter a description of the current situation (e.g., "The air conditioning is not working" or "An error message is displayed") in text.

[0221] Step 2:

[0222] The device receives the input photo and text data and converts it into JSON format, including the type of equipment (air conditioning, lighting, network equipment, etc.).

[0223] Step 3:

[0224] The device sends the generated JSON data to the server, which includes the user's input information and photo data.

[0225] Step 4:

[0226] The server parses the received JSON data and converts the photo and text data into the appropriate format, which prepares it for passing to the AI ​​model.

[0227] Step 5:

[0228] The server sends the converted data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The AI ​​model predicts the cause of the failure.

[0229] Step 6:

[0230] The AI ​​model analyzes the photo and text data to identify the cause of the malfunction, such as a clogged filter or a blown fuse, and returns the diagnosis results to the server.

[0231] Step 7:

[0232] The server receives the diagnosis results from the AI ​​model and determines whether a simple repair is possible. If a simple repair is possible, it searches for the corresponding repair manual.

[0233] Step 8:

[0234] If the server determines that simple repair is possible, it retrieves the corresponding repair manual and sends it to the user's device. The repair manual contains specific repair procedures.

[0235] Step 9:

[0236] The device displays the received repair manual on its screen. The user attempts simple repairs by following the instructions in the manual. For example, the manual contains instructions for cleaning the filter and turning the power on and off.

[0237] Step 10:

[0238] After the user performs the simple repair, they report the results in the application. If the repair is successful, the application simply ends.

[0239] Step 11:

[0240] If the problem is not resolved by the simple repair, the user reports it as "not resolved" on the application. The terminal then sends this report to the server.

[0241] Step 12:

[0242] The server records the information about the failure of the simple repair and immediately contacts the equipment manufacturer automatically, sending a repair request to the equipment manufacturer.

[0243] Step 13:

[0244] The server receives the response from the equipment manufacturer and arranges the earliest possible date for repair. It also acquires the schedule information received from the equipment manufacturer.

[0245] Step 14:

[0246] The server transmits the confirmed repair schedule to the user's terminal and notifies the user.

[0247] Step 15:

[0248] The terminal notifies the user of the received repair schedule, and the user confirms the notified repair schedule and waits for the equipment manufacturer to visit.

[0249] Step 16:

[0250] The server records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the repair manual provided, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmation of repair dates.

[0251] Step 17:

[0252] The emotion engine analyzes the user's input information and behavior to recognize the user's emotional state. For example, it can detect stress or frustration based on frequent error messages or delayed responses to operations.

[0253] Step 18:

[0254] The server flexibly changes the messages and repair manuals it presents based on the user's emotions recognized by the emotion engine. For example, if the user feels anxious, it displays encouraging messages, and if the user is confused by a complicated procedure, it provides simpler explanations.

[0255] Step 19:

[0256] If the emotion engine detects high levels of stress or dissatisfaction, the server automatically notifies the equipment manufacturer's experts and requests additional support, allowing users to receive fast, professional assistance.

[0257] Step 20:

[0258] The server records the trouble response history, including emotion recognition information, in a database and updates the data model used for future failure prediction and preventive maintenance.

[0259] By taking the above specific steps, the system can quickly and efficiently resolve equipment problems while reducing the psychological burden on users and providing a better user experience.

[0260] Example 2

[0261] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0262] Conventional building maintenance systems tend to respond slowly when equipment malfunctions occur, resulting in the user having to spend time and effort to deal with the issue. Furthermore, the system only provides a uniform response without considering the user's emotions or stress level, which can lead to a decline in user satisfaction. Furthermore, the accumulation and analysis of trouble response data was insufficient, meaning that the data could not be used to predict future malfunctions or for preventive maintenance.

[0263] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0264] In this invention, the server includes means for a user to input equipment status information, means for a terminal to send the equipment status information to the server, means for analyzing the equipment status information received by the server, means for identifying the cause of the failure using a generative AI model based on the analyzed information and determining whether a simple repair is possible, means for presenting a repair manual to the user if a simple repair is possible, means for contacting the equipment manufacturer if a simple repair is not possible, means for coordinating a repair schedule with the equipment manufacturer, means for notifying the user of the adjusted repair schedule, means for analyzing the user's emotions using an emotion engine and coordinating a response based on the analysis, and means for recording the trouble response history. This not only enables equipment failures to be responded to quickly and efficiently, but also reduces the psychological burden on the user and enables the use of data for failure prediction and preventive maintenance.

[0265] "User" refers to a person who uses the system to input equipment status information and receive repairs and support.

[0266] "Terminal" refers to a device such as a computer or smartphone that a user uses to input equipment status information.

[0267] "Server" refers to a computer system that receives equipment status information sent from a terminal and performs functions such as analysis, providing repair manuals, and contacting the equipment manufacturer.

[0268] "Generative AI model" refers to an artificial intelligence model used to analyze photographs and text data and identify the cause of equipment failure.

[0269] "Cause of failure" refers to the analysis results used to identify the cause of equipment abnormalities or malfunctions.

[0270] "Simple repair" refers to a simple repair method that the user can perform themselves.

[0271] "Repair manual" refers to a written or digital file that details steps and methods for performing simple repairs.

[0272] "Equipment manufacturer" refers to the company or professional that manufactured or sold the faulty equipment.

[0273] "Repair Schedule" means the date and time set by the equipment manufacturer for on-site repair work.

[0274] An "emotion engine" refers to software that analyzes emotions from user input and behavior, and then takes appropriate action based on the results.

[0275] "Trouble response history" refers to a record of a series of responses made within the system, such as information on equipment failures, analysis results, repair manuals, repair status, and emotion analysis results from the emotion engine.

[0276] "Failure prediction" refers to predicting future equipment failures by analyzing accumulated trouble response history.

[0277] "Preventive maintenance" refers to maintenance activities to prevent equipment failures before they occur, based on the results of failure prediction.

[0278] This invention is a building maintenance system that supports responding to equipment failures by having users input equipment status information, analyzing it via a server, and providing simple repair manuals or contacting the equipment manufacturer as necessary. Furthermore, the user experience is improved by combining it with an emotion engine that recognizes the user's emotions and flexibly adjusts its response depending on the situation.

[0279] Hardware and software used

[0280] Users access the system using a smartphone or PC. The system uses the following hardware and software:

[0281] User device: Smartphone or PC

[0282] Server: Cloud server or on-premise server

[0283] Generative AI models: ResNet (image analysis), BERT (natural language processing)

[0284] Emotion engine: IBM Watson (registered trademark)

[0285] Data processing and calculation

[0286] 1. User Input

[0287] Users log in to the Building Trouble SOS platform and enter equipment status information, including photos of the equipment and the text of any error messages.

[0288] Specific examples

[0289] For example, if the air conditioning in a building stops working, a user can take a photo of the air conditioning unit with their smartphone, upload it to the app, and enter the details of any error messages that may be displayed.

[0290] 2. Data Transmission

[0291] The terminal converts the input information into JSON format and sends it to the server.

[0292] Specific examples

[0293] The device encodes the photo data into Base64 format, retrieves the error message from the text field, generates a JSON containing these, and sends it to the server via an HTTP POST request.

[0294] 3. Initial diagnosis

[0295] The server parses the received JSON data, converts the photo data and text data into an appropriate format, and then sends the data to an AI model (e.g., ResNet) for photo analysis and a natural language processing model (e.g., BERT) for text analysis, requesting a diagnosis.

[0296] Specific examples

[0297] The server analyzes the photo and text based on a generative AI model and diagnoses whether the filter is likely clogged.

[0298] Repair manual provided

[0299] The server determines whether a simple repair is possible based on the diagnosis results, and if so, obtains the appropriate repair manual and sends it to the user's device.

[0300] Specific examples

[0301] The server retrieves the filter cleaning instructions from the database and sends the PDF file to the user's device.

[0302] 4. Use of Emotion Engine

[0303] The server sends the entered error message and the user's behavior log to an emotion engine (e.g., IBM Watson), which recognizes the user's emotional state (stress, frustration, anxiety, etc.) and provides messages of support and encouragement.

[0304] Specific examples

[0305] If the emotion engine detects a high level of stress in the user, the server generates a cheering message and sends it to the user's terminal.

[0306] Expert Notice

[0307] If simple repairs fail or if the user shows high stress, the server automatically contacts the equipment manufacturer and schedules a repair.

[0308] Specific examples

[0309] If the user reports again that the simple repair has failed, the server will contact the equipment manufacturer, set the earliest possible repair date, and notify the user.

[0310] Record of troubleshooting history

[0311] The server records all troubleshooting steps in a database, including information entered by the user, the diagnosis results of the AI ​​model, the proposed repair manual, the success or failure of the repair, communication with the equipment manufacturer, the confirmed repair schedule, and the emotional information recognized by the emotion engine.

[0312] Prompt Sentence Examples

[0313] Here is an example prompt:

[0314] Prompt statement

[0315] "Please identify the cause of the building equipment failure based on the error message and photo below. Error message: 'Air conditioner not working', Photo: [URL of photo of air conditioner]"

[0316] The above is a specific description of an embodiment of the present invention. The system aims to enable quick and efficient response to equipment failures and reduce the psychological burden on users.

[0317] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0318] Step 1:

[0319] Users input the equipment status information by logging into the Building Trouble SOS platform using a smartphone or PC, taking a photo of the equipment, and entering error messages and a detailed description of the situation.

[0320] Input: Equipment photo, error message, detailed description of the situation

[0321] Output: Equipment status information in JSON format

[0322] Specific actions

[0323] The user takes a photo of the air conditioning unit.

[0324] The user enters the error message "The air conditioner is not working."

[0325] This information is converted to JSON format for transmission to the server.

[0326] Step 2:

[0327] The device sends the status information entered by the user to the server. The data is converted to JSON format and sent to the server as an HTTP POST request.

[0328] Input: Equipment status information in JSON format

[0329] Output: Data sent to the server

[0330] Specific actions

[0331] The device encodes the photo data into Base64 format.

[0332] The terminal retrieves the error message from the text field.

[0333] The device generates JSON data and sends it to the server via an HTTP POST request.

[0334] Step 3:

[0335] The server parses the received JSON data, converts the photo data and text data into an appropriate format, and then sends the data to an AI model (e.g., ResNet) for photo analysis and a natural language processing model (e.g., BERT) for text analysis, requesting a diagnosis.

[0336] Input: Equipment status information in JSON format

[0337] Output: Diagnostic results (e.g., possible clogged filter)

[0338] Specific actions

[0339] The server extracts the photo data from the JSON data and inputs it into an image analysis model (e.g., ResNet).

[0340] The server inputs the error message into a natural language processing model (e.g., BERT).

[0341] The server receives the diagnostic results returned by the AI ​​model and determines whether a simple repair is possible.

[0342] Step 4:

[0343] If the server determines that a simple repair is possible based on the diagnostic results of the generated AI model, it retrieves the corresponding repair manual from the database and sends it to the user's device.

[0344] Input: Diagnostic results

[0345] Output: Repair manual

[0346] Specific actions

[0347] The server retrieves the filter cleaning instructions from the database.

[0348] The server sends a PDF file of the repair manual to the user's device.

[0349] Step 5:

[0350] The user attempts a simple repair by following the repair manual displayed on the terminal.

[0351] Input: Repair manual

[0352] Output: Simple repair success or failure

[0353] Specific actions

[0354] The user opens the PDF manual.

[0355] The user should follow the "filter removal method" and "cleaning procedure" written in the manual.

[0356] Step 6:

[0357] If the problem cannot be resolved by simple repairs, the user reports the situation. The device then sends this information back to the server, which then automatically contacts the equipment manufacturer. The server then automatically schedules a repair and notifies the user.

[0358] Input: Simple repair success / failure, status report

[0359] Output: Contact with equipment manufacturer and repair schedule

[0360] Specific actions

[0361] The user presses a button to report a "repair failed."

[0362] The terminal sends a request to the server to report the status.

[0363] The server automatically contacts the equipment manufacturer via their API and schedules the earliest possible repair date.

[0364] The server notifies the user of the confirmed repair schedule.

[0365] Step 7:

[0366] The server uses an emotion engine to analyze the user's emotions and detect their stress or frustration. Based on the results of the detection, it provides messages of encouragement and support.

[0367] Input: Error message, Action log

[0368] Output: Message of support

[0369] Specific actions

[0370] The server sends the entered error message and the user's action log to the emotion engine.

[0371] The emotion engine analyzes the user's emotions and determines that they are "feeling stressed."

[0372] The server generates a cheering message and sends it to the user's terminal.

[0373] Step 8:

[0374] If the emotion engine detects high levels of stress or dissatisfaction, the server automatically contacts the equipment manufacturer to request additional support.

[0375] Input: Sentiment analysis results

[0376] Output: Liaison with equipment manufacturers and additional support

[0377] Specific actions

[0378] The server sends a "high stress state" notification to the equipment manufacturer.

[0379] The server forwards any additional support instructions or messages from the experts to the user.

[0380] Step 9:

[0381] The server records each troubleshooting step in a database for future analysis and preventative maintenance.

[0382] Input: Troubleshooting data

[0383] Output: Accumulated troubleshooting history

[0384] Specific actions

[0385] The server stores all data in a database, including information entered by the user, the diagnosis results from the AI ​​model, the presented repair manual, and whether the repair was successful or not.

[0386] The stored data is used for analysis to predict failures and plan maintenance.

[0387] (Application example 2)

[0388] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0389] When equipment failure occurs, it is necessary to quickly and efficiently identify the cause of the failure and carry out simple repairs. However, conventional systems take time to identify the cause of the failure, which places a heavy psychological burden on users. In addition, they are unable to respond flexibly and take into account the user's emotions, which can lead to the accumulation of stress and dissatisfaction. To solve these issues, a system is needed that monitors the status of equipment in real time and uses an emotion engine to respond according to the user's emotions.

[0390] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0391] In this invention, the server includes a means for analyzing equipment status information, a means for determining whether simple repairs are possible, a means for presenting a repair manual, and a means for identifying the user's emotions and changing the response accordingly. This allows the cause of equipment failure to be identified quickly and efficiently, and enables flexible response while reducing user stress and frustration. Furthermore, the accumulated trouble response history can be analyzed and used for future failure prediction and preventive maintenance.

[0392] "Equipment status information" refers to data entered by users, such as the equipment's operating status, abnormal conditions, error messages, etc.

[0393] "Server" refers to a central management system that receives, analyzes, processes, and issues various decisions and notifications regarding the status of equipment.

[0394] "Means of analysis" refers to algorithms or programs for identifying the cause of a failure based on equipment status information.

[0395] "Means for determining whether simple repairs are possible" refers to algorithms or mechanisms that determine whether the target equipment can be easily repaired based on the analysis results.

[0396] "Means for presenting a repair manual" refers to a method or system that provides a user with specific repair procedures when simple repairs are possible.

[0397] "Equipment manufacturer" refers to the company or professional engineer who manufactured the equipment in question.

[0398] "Means for arranging repair schedules" refers to a system that communicates with equipment manufacturers and automatically determines the date and time for repairs.

[0399] "User" refers to anyone who operates or manages the facility, including building managers and general users.

[0400] "Means for recording trouble response history" refers to a system that stores the process of each trouble response in a database for later analysis and use.

[0401] "Means for identifying emotions and changing the response according to those emotions" refers to a function that recognizes the user's emotions such as stress or dissatisfaction, and appropriately changes the information provided and the response based on that emotional state.

[0402] "Means for acquiring and analyzing images of equipment" refers to the technology and programs for collecting photographic data of equipment taken with a camera, etc., and analyzing those images.

[0403] "Means for determining abnormality" refers to a function that determines whether the equipment is operating normally or whether an abnormality has occurred based on the acquired data.

[0404] "Failure prediction and preventive maintenance" refers to measures that analyze accumulated data to predict and prevent possible future failures.

[0405] The present invention is a system that analyzes equipment status information, presents repair manuals as needed, and automates communication and coordination with equipment manufacturers. It also has the ability to recognize a user's emotions and respond accordingly. A specific embodiment of the system is described below.

[0406] Hardware and Software Configuration

[0407] The system operates around three main players: the server, the terminal, and the user. The server performs analysis processing and database management, the terminal is responsible for inputting and receiving information, and the user provides equipment status information.

[0408] server:

[0409] Hardware: A server computer equipped with a high-performance CPU and GPU. A network interface card (NIC) that enables high-speed data transfer is used.

[0410] software:

[0411] Analysis module: Uses PhotoAIModel and TextAIModel to analyze equipment photo data and error messages.

[0412] Emotion Recognition Module: Uses the EmotionEngine to recognize emotions based on user input and actions.

[0413] Database: A highly reliable database system such as MySQL (registered trademark) or PostgreSQL is used to store troubleshooting history and equipment status.

[0414] Device:

[0415] Hardware: Smartphone, tablet, smart glasses, head-mounted display (HMD), or PC.

[0416] software:

[0417] Input interface: Provides a dedicated application for users to input equipment status information.

[0418] Display Interface: GUI for displaying repair manuals and messages.

[0419] User:

[0420] Role: Inputs equipment status information (photos, error messages, etc.), refers to repair manuals, and performs simple repairs. Also, uses emotion recognition to reduce user stress and frustration.

[0421] Data processing and calculation

[0422] The terminal sends the equipment status information (photos and error messages) entered by the user to the server, which analyzes the data using the following procedure and takes the necessary action.

[0423] 1. Data reception and analysis:

[0424] The server receives the JSON format data sent from the terminal.

[0425] The received data is input into PhotoAIModel and TextAIModel to analyze the cause of equipment failure.

[0426] 2. Repair decision and manual presentation:

[0427] Based on the analysis results, it is determined whether simple repairs are possible.

[0428] If simple repair is possible, the corresponding repair manual is obtained and sent to the terminal.

[0429] 3. Liaison and coordination with equipment manufacturers:

[0430] If simple repair is not possible, the equipment manufacturer will be contacted automatically.

[0431] We will contact the equipment manufacturer, schedule a repair, and notify the user.

[0432] 4. Emotion recognition and response modification:

[0433] EmotionEngine recognizes the user's emotions based on the user's input information and actions.

[0434] To flexibly change messages to be displayed and contents of repair manuals according to emotional information of a user.

[0435] If it detects high levels of stress or dissatisfaction, it will notify experts and provide additional support.

[0436] Specific examples

[0437] For example, if a robot in a factory stops working, a worker can use smart glasses to take a picture of the robot's condition and send the photo and an error message to the system. The system then analyzes the failure and diagnoses that the sensor needs to be calibrated. The worker is then presented with a calibration procedure, which they follow to attempt calibration. If the worker becomes stressed, the system's emotion engine will recognize this early on and send a message of encouragement or request expert assistance.

[0438] Prompt Sentence Examples

[0439] Please diagnose the cause of the failure based on the following equipment photo and error message. Photo: [Image path], Message: [Error message]

[0440] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0441] Step 1:

[0442] The user uses smart glasses or a head-mounted display (HMD) to capture a photo of the equipment and input an error message, which is then stored on the device.

[0443] Input: Facility photo, error message

[0444] Output: JSON format data saved on the device

[0445] Step 2:

[0446] The device sends the saved state information in JSON format to the server.

[0447] Input: JSON data of facility photos and error messages

[0448] Output: JSON data sent to the server

[0449] Step 3:

[0450] The server analyzes the received JSON data. The photo data is sent to the PhotoAIModel, and the error message is sent to the TextAIModel. Each model analyzes the data and identifies the cause of the error.

[0451] Input: JSON data sent to the server

[0452] Output: Analysis results from PhotoAIModel and TextAIModel

[0453] Step 4:

[0454] The server determines whether simple repairs are possible based on the analysis results obtained from PhotoAIModel and TextAIModel.

[0455] Input: Analysis results of PhotoAIModel and TextAIModel

[0456] Output: Simple repair possible

[0457] Step 5:

[0458] If it is determined that simple repair is possible, the server obtains the corresponding repair manual and sends it to the terminal.

[0459] Input: Judgment result that simple repair is possible

[0460] Output: Repair manual

[0461] Step 6:

[0462] The user attempts to carry out simple repairs according to the repair manual displayed on the terminal, and when the repair is complete, inputs the results into the terminal.

[0463] Input: Repair manual

[0464] Output: Simple repair results

[0465] Step 7:

[0466] If simple repairs are impossible or have failed, the user sends the information back to the server from their terminal, which then automatically contacts the equipment manufacturer and arranges a repair date.

[0467] Input: Simple repair failure information

[0468] Output: Repair schedule adjustment results with equipment manufacturer

[0469] Step 8:

[0470] The server notifies the user of the adjusted repair schedule, and the user confirms the schedule and waits for the repair to be performed.

[0471] Input: Repair schedule adjustment results with equipment manufacturer

[0472] Output: Notification of repair schedule to user

[0473] Step 9:

[0474] The server uses the Emotion Engine to identify emotions based on user input and behavior, and flexibly changes the corresponding messages and repair manuals based on this emotional information.

[0475] Input: User input and behavioral information

[0476] Output: Emotion-based messages and repair manuals

[0477] Step 10:

[0478] If the emotion engine detects high levels of stress or dissatisfaction in the user, the server immediately notifies an expert and provides additional support. Subsequent responses are also recorded in the database.

[0479] Input: Emotional information about high levels of stress and frustration

[0480] Output: Notification to experts, database record of troubleshooting history

[0481] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0482] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0483] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0484] [Second embodiment]

[0485] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0486] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0487] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0488] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0489] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0490] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0491] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0492] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0493] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0494] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0495] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0496] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0497] This invention is a building maintenance system for responding to equipment failures quickly and efficiently. The system supports responding to equipment failures by having the user input equipment status information, analyzing it via a server, and providing a simple repair manual or contacting the equipment manufacturer as necessary.

[0498] User Input

[0499] Users (building managers and general users) use their smartphones or PCs to input equipment status information into the "Building Trouble SOS" platform. This status information includes photos of the equipment and text of error messages. The device converts the input information into JSON format and sends it to the server.

[0500] Initial diagnosis by the server

[0501] The server analyzes the received JSON data and converts the photo and text data into the appropriate format. It then sends the data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The generated AI model predicts the cause of the failure based on the received information. The server determines whether a simple repair is possible based on the diagnosis results returned by the AI ​​model.

[0502] Simple repair manual provided

[0503] If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual and sends it to the user's device. The user then attempts the simple repair by following the repair manual displayed on their smartphone or PC. The manual contains detailed instructions on specific repair procedures, allowing the user to follow the steps to resolve the problem.

[0504] Contacting equipment manufacturers and arranging repair schedules

[0505] If the problem is not resolved by simple repairs, the user reports the situation on the application. The device then sends this report back to the server. The server records the information that the simple repairs failed and immediately and automatically contacts the equipment manufacturer. The server then automatically coordinates a repair schedule with the equipment manufacturer and determines the earliest possible date and time for repairs. The server then notifies the user's device of the confirmed schedule. The user then checks the notified repair schedule and waits for on-site repairs.

[0506] Recording and using troubleshooting history

[0507] The server records each troubleshooting step in a database. This data includes information entered by the user, the AI ​​diagnosis results, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. This data can be used to predict future failures and for preventive maintenance.

[0508] Specific examples

[0509] For example, if a building's air conditioning stops working, the building manager takes a photo of the air conditioning unit with their smartphone and uploads it to the app. If an error message is displayed, they enter the details. The device then sends this information to the server. The server analyzes the photo and text based on a generative AI model and determines that the filter is likely clogged. Based on the diagnosis, the server presents filter cleaning instructions and sends them to the user. The user then attempts to clean the filter by following the instructions displayed on their smartphone. If cleaning does not improve the situation, the user reports it again through the application, and the server automatically contacts the equipment manufacturer and schedules a repair. All of this information is recorded in a database and will be used for future analysis.

[0510] In this way, the system supports a series of processes for quickly and efficiently resolving equipment problems in building maintenance.

[0511] The processing flow will be explained below.

[0512] Step 1:

[0513] When a user notices that a building facility has broken down, they access the "Building Trouble SOS" platform using their smartphone or PC, take a photo of the facility, and enter a description of the current situation (e.g., "The air conditioning is not working" or "An error message is displayed") in text.

[0514] Step 2:

[0515] The device receives the input photo and text data and converts it into JSON format, including the type of equipment (air conditioning, lighting, network equipment, etc.).

[0516] Step 3:

[0517] The device sends the generated JSON data to the server, which includes the user's input information and photo data.

[0518] Step 4:

[0519] The server parses the received JSON data and converts the photo and text data into the appropriate format, which prepares it for passing to the AI ​​model.

[0520] Step 5:

[0521] The server sends the converted data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The AI ​​model predicts the cause of the failure.

[0522] Step 6:

[0523] The AI ​​model analyzes the photo and text data to identify the cause of the malfunction, such as a clogged filter or a blown fuse, and returns the diagnosis results to the server.

[0524] Step 7:

[0525] The server receives the diagnosis results from the AI ​​model and determines whether a simple repair is possible. If a simple repair is possible, it searches for the corresponding repair manual.

[0526] Step 8:

[0527] If the server determines that simple repair is possible, it retrieves the corresponding repair manual and sends it to the user's device. The repair manual contains specific repair procedures.

[0528] Step 9:

[0529] The device displays the received repair manual on its screen. The user attempts simple repairs by following the instructions in the manual. For example, the manual contains instructions for cleaning the filter and turning the power on and off.

[0530] Step 10:

[0531] After the user performs the simple repair, they report the results in the application. If the repair is successful, the application simply ends.

[0532] Step 11:

[0533] If the problem is not resolved by the simple repair, the user reports it as "not resolved" on the application. The terminal then sends this report to the server.

[0534] Step 12:

[0535] The server records the information about the failure of the simple repair and immediately contacts the equipment manufacturer automatically, sending a repair request to the equipment manufacturer.

[0536] Step 13:

[0537] The server receives the response from the equipment manufacturer and arranges the earliest possible date for repair. It also acquires the schedule information received from the equipment manufacturer.

[0538] Step 14:

[0539] The server transmits the confirmed repair schedule to the user's terminal and notifies the user.

[0540] Step 15:

[0541] The terminal notifies the user of the received repair schedule, and the user confirms the notified repair schedule and waits for the equipment manufacturer to visit.

[0542] Step 16:

[0543] The server records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the repair manual provided, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmation of repair dates.

[0544] Step 17:

[0545] The server analyzes the recorded data and updates a data model that can be used for future failure prediction and preventive maintenance, thereby improving the accuracy and speed of future failure responses.

[0546] Example 1

[0547] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0548] Conventional building maintenance systems often have delays in responding to equipment failures and unclear repair procedures, making it difficult to resolve problems. Even after simple repairs are attempted, feedback on their success or lack of prompt response time is often lacking, and additional specialized support is often not provided. Furthermore, data for failure prediction and preventive maintenance is not accumulated or analyzed, making it difficult to prevent future problems.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0550] In this invention, the server includes a means for analyzing the equipment status information, a means for determining whether a simple repair is possible based on the analyzed information, and a means for contacting the manufacturer if a simple repair is not possible, thereby enabling the cause of the equipment failure to be quickly identified and an appropriate response to be proposed.

[0551] "Equipment status information" refers to data such as the operating status of machines and equipment within a building and error messages.

[0552] A "web server" refers to a computer system that transmits and receives data over the Internet or an intranet.

[0553] "Analysis" refers to the process of processing received data to understand its contents and identify the cause of the problem.

[0554] "Simple repair" refers to repair work that can be done by the user themselves, without the need for a specialized technician.

[0555] A "repair manual" is a document that provides instructions and methods for repairing a specific problem.

[0556] "Manufacturer" refers to a company that designs, manufactures, and sells equipment.

[0557] "Repair Schedule" means the date and time for carrying out repair work.

[0558] "Trouble response history" refers to data that records the troubles that occurred, how they were resolved, and the results.

[0559] "Equipment images" refers to photographs and video data of facilities and equipment.

[0560] "Fault determination" refers to the operation of identifying the cause and location of a failure in equipment or devices based on analyzed data.

[0561] "Preventive maintenance" refers to maintenance activities such as inspections and repairs carried out before equipment or machinery breaks down.

[0562] "Failure prediction" refers to the operation of predicting the possibility of future failure of facilities or equipment based on accumulated data.

[0563] This invention provides a building maintenance system that allows for quick and efficient response to equipment failures. Specifically, users, terminals, and servers work together to collect and analyze equipment status information and take action as needed.

[0564] Users access the "Building Trouble SOS" platform using their smartphones or PCs. They enter information about the equipment's status and, if necessary, take and upload photos. This information includes specific equipment malfunctions, error messages, and images of the equipment. The user's device converts this information into JSON format and sends it to the server.

[0565] The server analyzes the received JSON data and separates the photo data from the text data. The photo data is processed as an image file (e.g., JPEG format), and the text data is extracted as string information. The server then sends the photo data to an image recognition AI model (e.g., using TensorFlow) and the text data to a natural language processing model (e.g., a model based on GPT-3 or BERT). This allows the server to analyze the cause of the failure and, based on that information, determine whether a simple repair is possible.

[0566] If a simple repair is determined to be possible, the server retrieves the corresponding repair manual from the database and sends it to the user's device. The repair manual contains detailed instructions for the repair, so the user can follow them to attempt the simple repair. For example, if the air conditioner's filter is diagnosed as clogged, a manual including instructions for cleaning the filter will be provided.

[0567] If the simple repair is unsuccessful, the user reports this on the application. The device again sends the information to the server, which then automatically contacts the manufacturer. The server then coordinates a repair date with the manufacturer, determines the earliest possible date and time for repair, and notifies the user's device of this information. The user can then confirm the notified repair date and wait for on-site repair.

[0568] The server also records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the proposed repair manual, the success or failure of simple repairs, communication with the manufacturer, and confirmation of repair dates. This data is used to predict future failures and for preventive maintenance.

[0569] As a specific example of how it works, let's say a user takes a photo of an air conditioning unit with their smartphone and uploads it to the "Building Trouble SOS" app. They also enter the contents of any error messages. The device converts this information into JSON format and sends it to the server. The server analyzes the photo and text based on an AI model and diagnoses that "the filter is likely clogged." Based on this diagnosis, it presents filter cleaning instructions and sends them to the user. The user attempts to clean the filter by following the instructions displayed on their smartphone, but if there is no improvement, they report it again. The server automatically contacts the manufacturer and arranges a repair date. All of this information is recorded in a database and will be used for later analysis.

[0570] An example of a prompt sentence might be:

[0571] "If the air conditioning in a building is not working, it is likely that the filter is clogged. Enter the filter cleaning procedure as a simple repair manual in this case."

[0572] In this way, the system provides a series of processes for quickly and efficiently resolving equipment problems in building maintenance.

[0573] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0574] Step 1:

[0575] Users access the "Building Trouble SOS" platform using their smartphones or PCs. They input information about the equipment's status (for example, a report that the air conditioning is not working or an error message), and take and upload photos of the equipment if necessary. This input process provides specific information about the equipment's malfunction.

[0576] Step 2:

[0577] The terminal converts the information entered by the user and the uploaded photo into a single JSON data, which contains text information and image data separately, and sends the converted JSON data to the server.

[0578] Step 3:

[0579] The server parses the received JSON data and separates the photo data from the text data. The server saves the photo data as an image file (e.g., JPEG format) and the text data as string information. This analysis prepares the data needed for the next step.

[0580] Step 4:

[0581] The server sends the photo data to an image recognition AI model (e.g., using TensorFlow) and the text data to a natural language processing model (e.g., a GPT-3 or BERT-based model). The server receives the diagnosis results returned by these models. For example, an image recognition model may return a diagnosis that "the filter is likely clogged." The input at this stage is image data and text data, and the output is the diagnosis result.

[0582] Step 5:

[0583] The server determines whether a simple repair is possible based on the diagnosis results obtained from the AI ​​model. If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual from the database and sends it to the user's device. For example, a repair procedure such as "filter cleaning procedure" is provided to the user. In this step, the diagnosis results of the AI ​​model are the input, and the simple repair manual is the output.

[0584] Step 6:

[0585] The user attempts to perform a simple repair according to the repair manual displayed on the terminal, for example, removing and cleaning the filter of the air conditioning unit, and checks the results of the simple repair (for example, whether the air conditioning works properly).

[0586] Step 7:

[0587] If the simple repair is not successful, the user reports this in the application. The device then sends the data containing the user's report back to the server. This report includes the repair procedure that was attempted and the results.

[0588] Step 8:

[0589] The server records the failure of the simple repair based on the data resent by the user and automatically contacts the manufacturer. The server then automatically coordinates a repair schedule with the manufacturer and determines the earliest possible date and time. This output is the repair schedule information, which is then notified to the user's terminal.

[0590] Step 9:

[0591] The server records each step of the troubleshooting process in a database, including information entered by the user, the diagnosis results from the AI ​​model, the repair manual provided, the success or failure of simple repairs, communication with the manufacturer, and confirmation of repair dates. This data is used to predict future failures and for preventive maintenance.

[0592] (Application example 1)

[0593] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0594] When a robot operating in a factory breaks down or malfunctions, a high level of specialized knowledge and experience is required to identify the cause and quickly repair it. This makes it difficult for on-site operators and managers to respond quickly, leading to production line shutdowns and unexpected increases in costs. Furthermore, if appropriate repair procedures and communication and coordination with equipment manufacturers are not carried out efficiently, repair delays may occur. There is a need for a system that can solve these problems and perform robot maintenance efficiently and effectively.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0596] In this invention, the server includes a means for users to input equipment status information using a smart device, a means for using a generative AI model to analyze equipment photos and error messages, and a means for optimizing diagnostic and repair processes specific to the application category. This allows on-site operators and managers to easily input status information when a robot operating in a factory breaks down or malfunctions, enabling quick and accurate fault diagnosis based on that information. Furthermore, if a simple repair is possible, the system presents specific repair procedures and, if necessary, automatically contacts the equipment manufacturer and arranges a repair schedule, thereby minimizing repair delays and shortening production line downtime.

[0597] The "means for inputting equipment status information" refers to a means by which a user inputs information such as a photograph of the equipment or an error message into a terminal.

[0598] The "means for transmitting equipment status information to the server" is a communication means for transferring input equipment status information to the server.

[0599] The "means for analyzing the equipment status information received by the server" is an analysis function for processing the status information received by the server and identifying the cause of an abnormality or failure.

[0600] The "means for determining whether simple repair is possible based on the analyzed information" is a means for determining whether simple repair is possible based on the analyzed information.

[0601] The "means for presenting a repair manual when simple repair is possible" is a means for providing a user with a manual describing the repair method when simple repair is possible.

[0602] "Means for contacting the equipment manufacturer when simple repairs are not possible" refers to a means for automatically contacting the equipment manufacturer when repairs are difficult.

[0603] "Means for coordinating repair schedules with equipment manufacturers" refers to means for coordinating and deciding repair schedules with equipment manufacturers.

[0604] The "means for notifying the user of the repair schedule" is a means for notifying the user of the confirmed repair schedule.

[0605] "Means for recording the history of troubleshooting" refers to a means for saving the steps and results of each troubleshooting in a database.

[0606] "Means for a user to input equipment status information using a smart device" refers to means for a user to input equipment status information using a device such as a smartphone or smart glasses.

[0607] "Means using a generative AI model to analyze equipment photos and error messages" means means using a generative AI model to analyze equipment photos and error messages.

[0608] "Measures to optimize diagnostic and repair processes specific to the application category" are measures to streamline diagnostic and repair processes to suit the specific requirements and environment of the application.

[0609] This invention describes in detail a method for realizing a maintenance management system for robots operating in a factory. This system begins when a user inputs equipment status information using a smart device and sends that information to a server. Each step is described in detail below.

[0610] 1. User Input

[0611] Users (factory managers and operators) use their smartphones or smart glasses to input status information about robots that have malfunctioned or are experiencing an abnormality into the application. This status information includes a photo of the robot and an error message. The photographed and input information is converted into JSON format and sent to the server.

[0612] 2. Initial diagnosis by the server

[0613] The server receives JSON data sent from the user device. The received data is analyzed using a generative AI model (e.g., TensorFlow or PyTorch). The photo and text data are converted into an appropriate format and sent to the AI ​​model to predict the cause of the failure. During this analysis, anomalies are identified from the photo and the problem is extracted from the error message.

[0614] 3. Simple repair manual provided

[0615] If the server determines that simple repairs are possible based on the analysis results, it provides a repair manual to the user's terminal. The repair manual contains detailed instructions on specific procedures, and the user can attempt to repair the device by following the instructions displayed on the smart device.

[0616] 4. Contact the equipment manufacturer and arrange repair schedules

[0617] If the simple repair fails or if it is determined that a more complex repair is required, the user can report it again through the application. Based on the report, the server will automatically contact the equipment manufacturer and arrange a repair date. Once the arrangement is complete, the server will notify the user of the confirmed repair date.

[0618] 5. Recording and using troubleshooting history

[0619] The server records the history of each troubleshooting in a database. This data includes information entered by the user, the results of the AI ​​diagnosis, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. The accumulated data is used to predict future failures and for preventive maintenance.

[0620] Hardware and software used

[0621] Hardware: Smartphones, smart glasses, servers

[0622] Software: Generative AI models (TensorFlow, PyTorch), applications (for smart devices)

[0623] Specific examples

[0624] For example, if a robot arm in operation in a factory stops working properly, the operator takes a photo of the robot arm with the smart glasses and enters an error message. The information is then sent to the server via the app. The server analyzes the data using a generative AI model and diagnoses the malfunction as "malfunction due to motor overload." Based on the results, the user is presented with specific repair procedures. If the repair is unsuccessful, the operator contacts the equipment manufacturer again via the app to arrange a repair date.

[0625] Prompt Sentence Examples

[0626] "Please take a photo of any abnormalities in the robotic arms operating in the factory and enter the error message."

[0627] This system makes it possible to respond quickly and efficiently to malfunctions or abnormalities in robots operating in factories.

[0628] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0629] Step 1:

[0630] The user uses a smart device to input equipment status information. Specifically, they take a photo of the abnormal part of the robot and enter the error message into the application. The input information is converted into JSON format.

[0631] Input: Photo data, text data (error message)

[0632] Output: JSON format data

[0633] Step 2:

[0634] The user's device (smartphone or smart glasses) sends equipment status information converted into JSON format to the server.

[0635] Input: JSON format data

[0636] Output: Data transfer to the server

[0637] Step 3:

[0638] The server analyzes the received JSON data. Specifically, it converts the photo data and text data into an appropriate format and sends it to a generative AI model (TensorFlow or PyTorch). The AI ​​model performs image analysis, identifies abnormalities from the photo, and performs text analysis to extract the cause of the failure from the error message.

[0639] Input: JSON format data

[0640] Output: Image analysis results, text analysis results

[0641] Step 4:

[0642] The server determines whether a simple repair is possible based on the analysis results. Based on the diagnosis results returned by the AI ​​model, it determines whether a simple repair is possible. This process refers to past repair history and common failure patterns in the database.

[0643] Input: Image analysis results, text analysis results

[0644] Output: Simple repair possible / impossible result

[0645] Step 5:

[0646] If it is determined that simple repair is possible, the server sends the corresponding repair manual to the user's terminal. The repair manual contains detailed instructions on the specific repair procedures, and the user attempts to repair the device by following the instructions displayed on the smart device.

[0647] Input: Simple repair possibility determination result

[0648] Output: Repair manual

[0649] Step 6:

[0650] If the simple repair fails or if it is determined that a more complex repair is required, the user can report it again through the application. Based on the report, the server will automatically contact the equipment manufacturer and arrange a repair date. Once the arrangement is complete, the server will notify the user of the confirmed repair date.

[0651] Input: User report data (simple repair failure report)

[0652] Output: Contact with equipment manufacturer, results of repair schedule adjustment, notification to user

[0653] Step 7:

[0654] The server records the history of each troubleshooting in a database. The recorded data includes information entered by the user, the results of the AI ​​diagnosis, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. This data is used to predict future failures and for preventive maintenance.

[0655] Input: Trouble response data (user input information, AI diagnosis results, repair results, communication logs, etc.)

[0656] Output: Records in the database

[0657] Prompt Sentence Examples

[0658] "Please take a photo of any abnormalities in the robotic arms operating in the factory and enter the error message."

[0659] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0660] This invention is a building maintenance system for responding quickly and efficiently to equipment breakdowns. The system supports responding to equipment breakdowns by having the user input equipment status information, analyzing it via a server, and providing a simple repair manual or contacting the equipment manufacturer as necessary. Furthermore, the system improves the user experience by combining it with an emotion engine that recognizes the user's emotions and flexibly adjusts its response depending on the situation.

[0661] User Input

[0662] Users (building managers and general users) use their smartphones or PCs to input equipment status information into the "Building Trouble SOS" platform. This status information includes photos of the equipment and text of error messages. The device converts the input information into JSON format and sends it to the server.

[0663] Initial diagnosis by the server

[0664] The server analyzes the received JSON data and converts the photo and text data into the appropriate format. It then sends the data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The generated AI model predicts the cause of the failure based on the received information. The server determines whether a simple repair is possible based on the diagnosis results returned by the AI ​​model.

[0665] Simple repair manual provided

[0666] If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual and sends it to the user's device. The user then attempts the simple repair by following the repair manual displayed on their smartphone or PC. The manual contains detailed instructions on specific repair procedures, allowing the user to follow the steps to resolve the problem.

[0667] Contacting equipment manufacturers and arranging repair schedules

[0668] If the problem is not resolved by simple repairs, the user reports the situation on the application. The device then sends this report back to the server. The server records the information that the simple repairs failed and immediately and automatically contacts the equipment manufacturer. The server then automatically coordinates a repair schedule with the equipment manufacturer and determines the earliest possible date and time for repairs. The server then notifies the user's device of the confirmed schedule. The user then checks the notified repair schedule and waits for on-site repairs.

[0669] User support with emotion engine

[0670] Furthermore, this system has a function that allows the emotion engine to recognize the user's emotions based on the user's input information and behavior. For example, if the user frequently inputs error messages or shows behavior that suggests frustration with repair procedures, the emotion engine will detect the user's stress or dissatisfaction.

[0671] Flexible response based on emotions

[0672] Based on the user's emotions recognized by the emotion engine, the server can flexibly change the messages and repair manuals it presents. For example, if the user is feeling anxious, it can provide a message containing words of encouragement and support. If the user is confused by a complicated procedure, it can present a simpler explanation or a video tutorial.

[0673] Expert Notification and Additional Support

[0674] If the emotion engine detects a high level of stress or dissatisfaction in the user, the server will automatically notify the equipment manufacturer's experts and request additional support. This feature allows users to receive professional assistance quickly and without having to struggle alone, improving the speed and accuracy of problem resolution.

[0675] Recording and using troubleshooting history

[0676] The server records each troubleshooting step in a database. This data includes information entered by the user, the AI ​​diagnosis results, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, confirmed repair schedules, and emotional information recognized by the emotion engine. This data can be used to predict future failures and for preventive maintenance.

[0677] Specific examples

[0678] For example, if a building's air conditioning stops working, the building manager takes a photo of the air conditioning unit with their smartphone and uploads it to the app. If an error message is displayed, they enter the details. The device then sends this information to the server. The server analyzes the photo and text based on a generative AI model and determines that the filter is likely clogged. Based on the diagnosis, the server presents filter cleaning instructions and sends them to the user. The user then attempts to clean the filter by following the instructions displayed on their smartphone. If cleaning does not improve the situation, the user reports it again through the application, and the server automatically contacts the equipment manufacturer and schedules a repair. All of this information is recorded in a database and will be used for future analysis.

[0679] Furthermore, if the user indicates a high level of stress in the emotion engine, the server notifies experts and prepares a means to provide additional support. For example, if a situation arises in which the user is "very troubled by frequent error messages," the emotion engine will recognize the stress and the server will immediately contact experts to request a prompt response. In this way, the system quickly and efficiently resolves equipment problems in building maintenance while simultaneously reducing the user's psychological burden.

[0680] The processing flow will be explained below.

[0681] Step 1:

[0682] When a user notices that a building facility has broken down, they access the "Building Trouble SOS" platform using their smartphone or PC, take a photo of the facility, and enter a description of the current situation (e.g., "The air conditioning is not working" or "An error message is displayed") in text.

[0683] Step 2:

[0684] The device receives the input photo and text data and converts it into JSON format, including the type of equipment (air conditioning, lighting, network equipment, etc.).

[0685] Step 3:

[0686] The device sends the generated JSON data to the server, which includes the user's input information and photo data.

[0687] Step 4:

[0688] The server parses the received JSON data and converts the photo and text data into the appropriate format, which prepares it for passing to the AI ​​model.

[0689] Step 5:

[0690] The server sends the converted data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The AI ​​model predicts the cause of the failure.

[0691] Step 6:

[0692] The AI ​​model analyzes the photo and text data to identify the cause of the malfunction, such as a clogged filter or a blown fuse, and returns the diagnosis results to the server.

[0693] Step 7:

[0694] The server receives the diagnosis results from the AI ​​model and determines whether a simple repair is possible. If a simple repair is possible, it searches for the corresponding repair manual.

[0695] Step 8:

[0696] If the server determines that simple repair is possible, it retrieves the corresponding repair manual and sends it to the user's device. The repair manual contains specific repair procedures.

[0697] Step 9:

[0698] The device displays the received repair manual on its screen. The user attempts simple repairs by following the instructions in the manual. For example, the manual contains instructions for cleaning the filter and turning the power on and off.

[0699] Step 10:

[0700] After the user performs the simple repair, they report the results in the application. If the repair is successful, the application simply ends.

[0701] Step 11:

[0702] If the problem is not resolved by the simple repair, the user reports it as "not resolved" on the application. The terminal then sends this report to the server.

[0703] Step 12:

[0704] The server records the information about the failure of the simple repair and immediately contacts the equipment manufacturer automatically, sending a repair request to the equipment manufacturer.

[0705] Step 13:

[0706] The server receives the response from the equipment manufacturer and arranges the earliest possible date for repair. It also acquires the schedule information received from the equipment manufacturer.

[0707] Step 14:

[0708] The server transmits the confirmed repair schedule to the user's terminal and notifies the user.

[0709] Step 15:

[0710] The terminal notifies the user of the received repair schedule, and the user confirms the notified repair schedule and waits for the equipment manufacturer to visit.

[0711] Step 16:

[0712] The server records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the repair manual provided, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmation of repair dates.

[0713] Step 17:

[0714] The emotion engine analyzes the user's input information and behavior to recognize the user's emotional state. For example, it can detect stress or frustration based on frequent error messages or delayed responses to operations.

[0715] Step 18:

[0716] The server flexibly changes the messages and repair manuals it presents based on the user's emotions recognized by the emotion engine. For example, if the user feels anxious, it displays encouraging messages, and if the user is confused by a complicated procedure, it provides simpler explanations.

[0717] Step 19:

[0718] If the emotion engine detects high levels of stress or dissatisfaction, the server automatically notifies the equipment manufacturer's experts and requests additional support, allowing users to receive fast, professional assistance.

[0719] Step 20:

[0720] The server records the trouble response history, including emotion recognition information, in a database and updates the data model used for future failure prediction and preventive maintenance.

[0721] By taking the above specific steps, the system can quickly and efficiently resolve equipment problems while reducing the psychological burden on users and providing a better user experience.

[0722] Example 2

[0723] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] Conventional building maintenance systems tend to respond slowly when equipment malfunctions occur, resulting in the user having to spend time and effort to deal with the issue. Furthermore, the system only provides a uniform response without considering the user's emotions or stress level, which can lead to a decline in user satisfaction. Furthermore, the accumulation and analysis of trouble response data was insufficient, meaning that the data could not be used to predict future malfunctions or for preventive maintenance.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0726] In this invention, the server includes means for a user to input equipment status information, means for a terminal to send the equipment status information to the server, means for analyzing the equipment status information received by the server, means for identifying the cause of the failure using a generative AI model based on the analyzed information and determining whether a simple repair is possible, means for presenting a repair manual to the user if a simple repair is possible, means for contacting the equipment manufacturer if a simple repair is not possible, means for coordinating a repair schedule with the equipment manufacturer, means for notifying the user of the adjusted repair schedule, means for analyzing the user's emotions using an emotion engine and coordinating a response based on the analysis, and means for recording the trouble response history. This not only enables equipment failures to be responded to quickly and efficiently, but also reduces the psychological burden on the user and enables the use of data for failure prediction and preventive maintenance.

[0727] "User" refers to a person who uses the system to input equipment status information and receive repairs and support.

[0728] "Terminal" refers to a device such as a computer or smartphone that a user uses to input equipment status information.

[0729] "Server" refers to a computer system that receives equipment status information sent from a terminal and performs functions such as analysis, providing repair manuals, and contacting the equipment manufacturer.

[0730] "Generative AI model" refers to an artificial intelligence model used to analyze photographs and text data and identify the cause of equipment failure.

[0731] "Cause of failure" refers to the analysis results used to identify the cause of equipment abnormalities or malfunctions.

[0732] "Simple repair" refers to a simple repair method that the user can perform themselves.

[0733] "Repair manual" refers to a written or digital file that details steps and methods for performing simple repairs.

[0734] "Equipment manufacturer" refers to the company or professional that manufactured or sold the faulty equipment.

[0735] "Repair Schedule" means the date and time set by the equipment manufacturer for on-site repair work.

[0736] An "emotion engine" refers to software that analyzes emotions from user input and behavior, and then takes appropriate action based on the results.

[0737] "Trouble response history" refers to a record of a series of responses made within the system, such as information on equipment failures, analysis results, repair manuals, repair status, and emotion analysis results from the emotion engine.

[0738] "Failure prediction" refers to predicting future equipment failures by analyzing accumulated trouble response history.

[0739] "Preventive maintenance" refers to maintenance activities to prevent equipment failures before they occur, based on the results of failure prediction.

[0740] This invention is a building maintenance system that supports responding to equipment failures by having users input equipment status information, analyzing it via a server, and providing simple repair manuals or contacting the equipment manufacturer as necessary. Furthermore, the user experience is improved by combining it with an emotion engine that recognizes the user's emotions and flexibly adjusts its response depending on the situation.

[0741] Hardware and software used

[0742] Users access the system using a smartphone or PC. The system uses the following hardware and software:

[0743] User device: Smartphone or PC

[0744] Server: Cloud server or on-premise server

[0745] Generative AI models: ResNet (image analysis), BERT (natural language processing)

[0746] Emotion engine: IBM Watson

[0747] Data processing and calculation

[0748] 1. User Input

[0749] Users log in to the Building Trouble SOS platform and enter equipment status information, including photos of the equipment and the text of any error messages.

[0750] Specific examples

[0751] For example, if the air conditioning in a building stops working, a user can take a photo of the air conditioning unit with their smartphone, upload it to the app, and enter the details of any error messages that may be displayed.

[0752] 2. Data Transmission

[0753] The terminal converts the input information into JSON format and sends it to the server.

[0754] Specific examples

[0755] The device encodes the photo data into Base64 format, retrieves the error message from the text field, generates a JSON containing these, and sends it to the server via an HTTP POST request.

[0756] 3. Initial diagnosis

[0757] The server parses the received JSON data, converts the photo data and text data into an appropriate format, and then sends the data to an AI model (e.g., ResNet) for photo analysis and a natural language processing model (e.g., BERT) for text analysis, requesting a diagnosis.

[0758] Specific examples

[0759] The server analyzes the photo and text based on a generative AI model and diagnoses whether the filter is likely clogged.

[0760] Repair manual provided

[0761] The server determines whether a simple repair is possible based on the diagnosis results, and if so, obtains the appropriate repair manual and sends it to the user's device.

[0762] Specific examples

[0763] The server retrieves the filter cleaning instructions from the database and sends the PDF file to the user's device.

[0764] 4. Use of Emotion Engine

[0765] The server sends the entered error message and the user's behavior log to an emotion engine (e.g., IBM Watson), which recognizes the user's emotional state (stress, frustration, anxiety, etc.) and provides messages of support and encouragement.

[0766] Specific examples

[0767] If the emotion engine detects a high level of stress in the user, the server generates a cheering message and sends it to the user's terminal.

[0768] Expert Notice

[0769] If simple repairs fail or if the user shows high stress, the server automatically contacts the equipment manufacturer and schedules a repair.

[0770] Specific examples

[0771] If the user reports again that the simple repair has failed, the server will contact the equipment manufacturer, set the earliest possible repair date, and notify the user.

[0772] Record of troubleshooting history

[0773] The server records all troubleshooting steps in a database, including information entered by the user, the diagnosis results of the AI ​​model, the proposed repair manual, the success or failure of the repair, communication with the equipment manufacturer, the confirmed repair schedule, and the emotional information recognized by the emotion engine.

[0774] Prompt Sentence Examples

[0775] Here is an example prompt:

[0776] Prompt statement

[0777] "Please identify the cause of the building equipment failure based on the error message and photo below. Error message: 'Air conditioner not working', Photo: [URL of photo of air conditioner]"

[0778] The above is a specific description of an embodiment of the present invention. The system aims to enable quick and efficient response to equipment failures and reduce the psychological burden on users.

[0779] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0780] Step 1:

[0781] Users input the equipment status information by logging into the Building Trouble SOS platform using a smartphone or PC, taking a photo of the equipment, and entering error messages and a detailed description of the situation.

[0782] Input: Equipment photo, error message, detailed description of the situation

[0783] Output: Equipment status information in JSON format

[0784] Specific actions

[0785] The user takes a photo of the air conditioning unit.

[0786] The user enters the error message "The air conditioner is not working."

[0787] This information is converted to JSON format for transmission to the server.

[0788] Step 2:

[0789] The device sends the status information entered by the user to the server. The data is converted to JSON format and sent to the server as an HTTP POST request.

[0790] Input: Equipment status information in JSON format

[0791] Output: Data sent to the server

[0792] Specific actions

[0793] The device encodes the photo data into Base64 format.

[0794] The terminal retrieves the error message from the text field.

[0795] The device generates JSON data and sends it to the server via an HTTP POST request.

[0796] Step 3:

[0797] The server parses the received JSON data, converts the photo data and text data into an appropriate format, and then sends the data to an AI model (e.g., ResNet) for photo analysis and a natural language processing model (e.g., BERT) for text analysis, requesting a diagnosis.

[0798] Input: Equipment status information in JSON format

[0799] Output: Diagnostic results (e.g., possible clogged filter)

[0800] Specific actions

[0801] The server extracts the photo data from the JSON data and inputs it into an image analysis model (e.g., ResNet).

[0802] The server inputs the error message into a natural language processing model (e.g., BERT).

[0803] The server receives the diagnostic results returned by the AI ​​model and determines whether a simple repair is possible.

[0804] Step 4:

[0805] If the server determines that a simple repair is possible based on the diagnostic results of the generated AI model, it retrieves the corresponding repair manual from the database and sends it to the user's device.

[0806] Input: Diagnostic results

[0807] Output: Repair manual

[0808] Specific actions

[0809] The server retrieves the filter cleaning instructions from the database.

[0810] The server sends a PDF file of the repair manual to the user's device.

[0811] Step 5:

[0812] The user attempts a simple repair by following the repair manual displayed on the terminal.

[0813] Input: Repair manual

[0814] Output: Simple repair success or failure

[0815] Specific actions

[0816] The user opens the PDF manual.

[0817] The user should follow the "filter removal method" and "cleaning procedure" written in the manual.

[0818] Step 6:

[0819] If the problem cannot be resolved by simple repairs, the user reports the situation. The device then sends this information back to the server, which then automatically contacts the equipment manufacturer. The server then automatically schedules a repair and notifies the user.

[0820] Input: Simple repair success / failure, status report

[0821] Output: Contact with equipment manufacturer and repair schedule

[0822] Specific actions

[0823] The user presses a button to report a "repair failed."

[0824] The terminal sends a request to the server to report the status.

[0825] The server automatically contacts the equipment manufacturer via their API and schedules the earliest possible repair date.

[0826] The server notifies the user of the confirmed repair schedule.

[0827] Step 7:

[0828] The server uses an emotion engine to analyze the user's emotions and detect their stress or frustration. Based on the results of the detection, it provides messages of encouragement and support.

[0829] Input: Error message, Action log

[0830] Output: Message of support

[0831] Specific actions

[0832] The server sends the entered error message and the user's action log to the emotion engine.

[0833] The emotion engine analyzes the user's emotions and determines that they are "feeling stressed."

[0834] The server generates a cheering message and sends it to the user's terminal.

[0835] Step 8:

[0836] If the emotion engine detects high levels of stress or dissatisfaction, the server automatically contacts the equipment manufacturer to request additional support.

[0837] Input: Sentiment analysis results

[0838] Output: Liaison with equipment manufacturers and additional support

[0839] Specific actions

[0840] The server sends a "high stress state" notification to the equipment manufacturer.

[0841] The server forwards any additional support instructions or messages from the experts to the user.

[0842] Step 9:

[0843] The server records each troubleshooting step in a database for future analysis and preventative maintenance.

[0844] Input: Troubleshooting data

[0845] Output: Accumulated troubleshooting history

[0846] Specific actions

[0847] The server stores all data in a database, including information entered by the user, the diagnosis results from the AI ​​model, the presented repair manual, and whether the repair was successful or not.

[0848] The stored data is used for analysis to predict failures and plan maintenance.

[0849] (Application example 2)

[0850] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0851] When equipment failure occurs, it is necessary to quickly and efficiently identify the cause of the failure and carry out simple repairs. However, conventional systems take time to identify the cause of the failure, which places a heavy psychological burden on users. In addition, they are unable to respond flexibly and take into account the user's emotions, which can lead to the accumulation of stress and dissatisfaction. To solve these issues, a system is needed that monitors the status of equipment in real time and uses an emotion engine to respond according to the user's emotions.

[0852] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0853] In this invention, the server includes a means for analyzing equipment status information, a means for determining whether simple repairs are possible, a means for presenting a repair manual, and a means for identifying the user's emotions and changing the response accordingly. This allows the cause of equipment failure to be identified quickly and efficiently, and enables flexible response while reducing user stress and frustration. Furthermore, the accumulated trouble response history can be analyzed and used for future failure prediction and preventive maintenance.

[0854] "Equipment status information" refers to data entered by users, such as the equipment's operating status, abnormal conditions, error messages, etc.

[0855] "Server" refers to a central management system that receives, analyzes, processes, and issues various decisions and notifications regarding the status of equipment.

[0856] "Means of analysis" refers to algorithms or programs for identifying the cause of a failure based on equipment status information.

[0857] "Means for determining whether simple repairs are possible" refers to algorithms or mechanisms that determine whether the target equipment can be easily repaired based on the analysis results.

[0858] "Means for presenting a repair manual" refers to a method or system that provides a user with specific repair procedures when simple repairs are possible.

[0859] "Equipment manufacturer" refers to the company or professional engineer who manufactured the equipment in question.

[0860] "Means for arranging repair schedules" refers to a system that communicates with equipment manufacturers and automatically determines the date and time for repairs.

[0861] "User" refers to anyone who operates or manages the facility, including building managers and general users.

[0862] "Means for recording trouble response history" refers to a system that stores the process of each trouble response in a database for later analysis and use.

[0863] "Means for identifying emotions and changing the response according to those emotions" refers to a function that recognizes the user's emotions such as stress or dissatisfaction, and appropriately changes the information provided and the response based on that emotional state.

[0864] "Means for acquiring and analyzing images of equipment" refers to the technology and programs for collecting photographic data of equipment taken with a camera, etc., and analyzing those images.

[0865] "Means for determining abnormality" refers to a function that determines whether the equipment is operating normally or whether an abnormality has occurred based on the acquired data.

[0866] "Failure prediction and preventive maintenance" refers to measures that analyze accumulated data to predict and prevent possible future failures.

[0867] The present invention is a system that analyzes equipment status information, presents repair manuals as needed, and automates communication and coordination with equipment manufacturers. It also has the ability to recognize a user's emotions and respond accordingly. A specific embodiment of the system is described below.

[0868] Hardware and Software Configuration

[0869] The system operates around three main players: the server, the terminal, and the user. The server performs analysis processing and database management, the terminal is responsible for inputting and receiving information, and the user provides equipment status information.

[0870] server:

[0871] Hardware: A server computer equipped with a high-performance CPU and GPU. A network interface card (NIC) that enables high-speed data transfer is used.

[0872] software:

[0873] Analysis module: Uses PhotoAIModel and TextAIModel to analyze equipment photo data and error messages.

[0874] Emotion Recognition Module: Uses the EmotionEngine to recognize emotions based on user input and actions.

[0875] Database: Use a reliable database system such as MySQL or PostgreSQL to store troubleshooting history and equipment status.

[0876] Device:

[0877] Hardware: Smartphone, tablet, smart glasses, head-mounted display (HMD), or PC.

[0878] software:

[0879] Input interface: Provides a dedicated application for users to input equipment status information.

[0880] Display Interface: GUI for displaying repair manuals and messages.

[0881] User:

[0882] Role: Inputs equipment status information (photos, error messages, etc.), refers to repair manuals, and performs simple repairs. Also, uses emotion recognition to reduce user stress and frustration.

[0883] Data processing and calculation

[0884] The terminal sends the equipment status information (photos and error messages) entered by the user to the server, which analyzes the data using the following procedure and takes the necessary action.

[0885] 1. Data reception and analysis:

[0886] The server receives the JSON format data sent from the terminal.

[0887] The received data is input into PhotoAIModel and TextAIModel to analyze the cause of equipment failure.

[0888] 2. Repair decision and manual presentation:

[0889] Based on the analysis results, it is determined whether simple repairs are possible.

[0890] If simple repair is possible, the corresponding repair manual is obtained and sent to the terminal.

[0891] 3. Liaison and coordination with equipment manufacturers:

[0892] If simple repair is not possible, the equipment manufacturer will be contacted automatically.

[0893] We will contact the equipment manufacturer, schedule a repair, and notify the user.

[0894] 4. Emotion recognition and response modification:

[0895] EmotionEngine recognizes the user's emotions based on the user's input information and actions.

[0896] To flexibly change messages to be displayed and contents of repair manuals according to emotional information of a user.

[0897] If it detects high levels of stress or dissatisfaction, it will notify experts and provide additional support.

[0898] Specific examples

[0899] For example, if a robot in a factory stops working, a worker can use smart glasses to take a picture of the robot's condition and send the photo and an error message to the system. The system then analyzes the failure and diagnoses that the sensor needs to be calibrated. The worker is then presented with a calibration procedure, which they follow to attempt calibration. If the worker becomes stressed, the system's emotion engine will recognize this early on and send a message of encouragement or request expert assistance.

[0900] Prompt Sentence Examples

[0901] Please diagnose the cause of the failure based on the following equipment photo and error message. Photo: [Image path], Message: [Error message]

[0902] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0903] Step 1:

[0904] The user uses smart glasses or a head-mounted display (HMD) to capture a photo of the equipment and input an error message, which is then stored on the device.

[0905] Input: Facility photo, error message

[0906] Output: JSON format data saved on the device

[0907] Step 2:

[0908] The device sends the saved state information in JSON format to the server.

[0909] Input: JSON data of facility photos and error messages

[0910] Output: JSON data sent to the server

[0911] Step 3:

[0912] The server analyzes the received JSON data. The photo data is sent to the PhotoAIModel, and the error message is sent to the TextAIModel. Each model analyzes the data and identifies the cause of the error.

[0913] Input: JSON data sent to the server

[0914] Output: Analysis results from PhotoAIModel and TextAIModel

[0915] Step 4:

[0916] The server determines whether simple repairs are possible based on the analysis results obtained from PhotoAIModel and TextAIModel.

[0917] Input: Analysis results of PhotoAIModel and TextAIModel

[0918] Output: Simple repair possible

[0919] Step 5:

[0920] If it is determined that simple repair is possible, the server obtains the corresponding repair manual and sends it to the terminal.

[0921] Input: Judgment result that simple repair is possible

[0922] Output: Repair manual

[0923] Step 6:

[0924] The user attempts to carry out simple repairs according to the repair manual displayed on the terminal, and when the repair is complete, inputs the results into the terminal.

[0925] Input: Repair manual

[0926] Output: Simple repair results

[0927] Step 7:

[0928] If simple repairs are impossible or have failed, the user sends the information back to the server from their terminal, which then automatically contacts the equipment manufacturer and arranges a repair date.

[0929] Input: Simple repair failure information

[0930] Output: Repair schedule adjustment results with equipment manufacturer

[0931] Step 8:

[0932] The server notifies the user of the adjusted repair schedule, and the user confirms the schedule and waits for the repair to be performed.

[0933] Input: Repair schedule adjustment results with equipment manufacturer

[0934] Output: Notification of repair schedule to user

[0935] Step 9:

[0936] The server uses the Emotion Engine to identify emotions based on user input and behavior, and flexibly changes the corresponding messages and repair manuals based on this emotional information.

[0937] Input: User input and behavioral information

[0938] Output: Emotion-based messages and repair manuals

[0939] Step 10:

[0940] If the emotion engine detects high levels of stress or dissatisfaction in the user, the server immediately notifies an expert and provides additional support. Subsequent responses are also recorded in the database.

[0941] Input: Emotional information about high levels of stress and frustration

[0942] Output: Notification to experts, database record of troubleshooting history

[0943] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0944] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0945] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0946] [Third embodiment]

[0947] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0948] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0949] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0950] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0951] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0952] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0953] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0954] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0955] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0956] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0957] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0958] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0959] This invention is a building maintenance system for responding to equipment failures quickly and efficiently. The system supports responding to equipment failures by having the user input equipment status information, analyzing it via a server, and providing a simple repair manual or contacting the equipment manufacturer as necessary.

[0960] User Input

[0961] Users (building managers and general users) use their smartphones or PCs to input equipment status information into the "Building Trouble SOS" platform. This status information includes photos of the equipment and text of error messages. The device converts the input information into JSON format and sends it to the server.

[0962] Initial diagnosis by the server

[0963] The server analyzes the received JSON data and converts the photo and text data into the appropriate format. It then sends the data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The generated AI model predicts the cause of the failure based on the received information. The server determines whether a simple repair is possible based on the diagnosis results returned by the AI ​​model.

[0964] Simple repair manual provided

[0965] If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual and sends it to the user's device. The user then attempts the simple repair by following the repair manual displayed on their smartphone or PC. The manual contains detailed instructions on specific repair procedures, allowing the user to follow the steps to resolve the problem.

[0966] Contacting equipment manufacturers and arranging repair schedules

[0967] If the problem is not resolved by simple repairs, the user reports the situation on the application. The device then sends this report back to the server. The server records the information that the simple repairs failed and immediately and automatically contacts the equipment manufacturer. The server then automatically coordinates a repair schedule with the equipment manufacturer and determines the earliest possible date and time for repairs. The server then notifies the user's device of the confirmed schedule. The user then checks the notified repair schedule and waits for on-site repairs.

[0968] Recording and using troubleshooting history

[0969] The server records each troubleshooting step in a database. This data includes information entered by the user, the AI ​​diagnosis results, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. This data can be used to predict future failures and for preventive maintenance.

[0970] Specific examples

[0971] For example, if a building's air conditioning stops working, the building manager takes a photo of the air conditioning unit with their smartphone and uploads it to the app. If an error message is displayed, they enter the details. The device then sends this information to the server. The server analyzes the photo and text based on a generative AI model and determines that the filter is likely clogged. Based on the diagnosis, the server presents filter cleaning instructions and sends them to the user. The user then attempts to clean the filter by following the instructions displayed on their smartphone. If cleaning does not improve the situation, the user reports it again through the application, and the server automatically contacts the equipment manufacturer and schedules a repair. All of this information is recorded in a database and will be used for future analysis.

[0972] In this way, the system supports a series of processes for quickly and efficiently resolving equipment problems in building maintenance.

[0973] The processing flow will be explained below.

[0974] Step 1:

[0975] When a user notices that a building facility has broken down, they access the "Building Trouble SOS" platform using their smartphone or PC, take a photo of the facility, and enter a description of the current situation (e.g., "The air conditioning is not working" or "An error message is displayed") in text.

[0976] Step 2:

[0977] The device receives the input photo and text data and converts it into JSON format, including the type of equipment (air conditioning, lighting, network equipment, etc.).

[0978] Step 3:

[0979] The device sends the generated JSON data to the server, which includes the user's input information and photo data.

[0980] Step 4:

[0981] The server parses the received JSON data and converts the photo and text data into the appropriate format, which prepares it for passing to the AI ​​model.

[0982] Step 5:

[0983] The server sends the converted data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The AI ​​model predicts the cause of the failure.

[0984] Step 6:

[0985] The AI ​​model analyzes the photo and text data to identify the cause of the malfunction, such as a clogged filter or a blown fuse, and returns the diagnosis results to the server.

[0986] Step 7:

[0987] The server receives the diagnosis results from the AI ​​model and determines whether a simple repair is possible. If a simple repair is possible, it searches for the corresponding repair manual.

[0988] Step 8:

[0989] If the server determines that simple repair is possible, it retrieves the corresponding repair manual and sends it to the user's device. The repair manual contains specific repair procedures.

[0990] Step 9:

[0991] The device displays the received repair manual on its screen. The user attempts simple repairs by following the instructions in the manual. For example, the manual contains instructions for cleaning the filter and turning the power on and off.

[0992] Step 10:

[0993] After the user performs the simple repair, they report the results in the application. If the repair is successful, the application simply ends.

[0994] Step 11:

[0995] If the problem is not resolved by the simple repair, the user reports it as "not resolved" on the application. The terminal then sends this report to the server.

[0996] Step 12:

[0997] The server records the information about the failure of the simple repair and immediately contacts the equipment manufacturer automatically, sending a repair request to the equipment manufacturer.

[0998] Step 13:

[0999] The server receives the response from the equipment manufacturer and arranges the earliest possible date for repair. It also acquires the schedule information received from the equipment manufacturer.

[1000] Step 14:

[1001] The server transmits the confirmed repair schedule to the user's terminal and notifies the user.

[1002] Step 15:

[1003] The terminal notifies the user of the received repair schedule, and the user confirms the notified repair schedule and waits for the equipment manufacturer to visit.

[1004] Step 16:

[1005] The server records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the repair manual provided, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmation of repair dates.

[1006] Step 17:

[1007] The server analyzes the recorded data and updates a data model that can be used for future failure prediction and preventive maintenance, thereby improving the accuracy and speed of future failure responses.

[1008] Example 1

[1009] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1010] Conventional building maintenance systems often have delays in responding to equipment failures and unclear repair procedures, making it difficult to resolve problems. Even after simple repairs are attempted, feedback on their success or lack of prompt response time is often lacking, and additional specialized support is often not provided. Furthermore, data for failure prediction and preventive maintenance is not accumulated or analyzed, making it difficult to prevent future problems.

[1011] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1012] In this invention, the server includes a means for analyzing the equipment status information, a means for determining whether a simple repair is possible based on the analyzed information, and a means for contacting the manufacturer if a simple repair is not possible, thereby enabling the cause of the equipment failure to be quickly identified and an appropriate response to be proposed.

[1013] "Equipment status information" refers to data such as the operating status of machines and equipment within a building and error messages.

[1014] A "web server" refers to a computer system that transmits and receives data over the Internet or an intranet.

[1015] "Analysis" refers to the process of processing received data to understand its contents and identify the cause of the problem.

[1016] "Simple repair" refers to repair work that can be done by the user themselves, without the need for a specialized technician.

[1017] A "repair manual" is a document that provides instructions and methods for repairing a specific problem.

[1018] "Manufacturer" refers to a company that designs, manufactures, and sells equipment.

[1019] "Repair Schedule" means the date and time for carrying out repair work.

[1020] "Trouble response history" refers to data that records the troubles that occurred, how they were resolved, and the results.

[1021] "Equipment images" refers to photographs and video data of facilities and equipment.

[1022] "Fault determination" refers to the operation of identifying the cause and location of a failure in equipment or devices based on analyzed data.

[1023] "Preventive maintenance" refers to maintenance activities such as inspections and repairs carried out before equipment or machinery breaks down.

[1024] "Failure prediction" refers to the operation of predicting the possibility of future failure of facilities or equipment based on accumulated data.

[1025] This invention provides a building maintenance system that allows for quick and efficient response to equipment failures. Specifically, users, terminals, and servers work together to collect and analyze equipment status information and take action as needed.

[1026] Users access the "Building Trouble SOS" platform using their smartphones or PCs. They enter information about the equipment's status and, if necessary, take and upload photos. This information includes specific equipment malfunctions, error messages, and images of the equipment. The user's device converts this information into JSON format and sends it to the server.

[1027] The server analyzes the received JSON data and separates the photo data from the text data. The photo data is processed as an image file (e.g., JPEG format), and the text data is extracted as string information. The server then sends the photo data to an image recognition AI model (e.g., using TensorFlow) and the text data to a natural language processing model (e.g., a model based on GPT-3 or BERT). This allows the server to analyze the cause of the failure and, based on that information, determine whether a simple repair is possible.

[1028] If a simple repair is determined to be possible, the server retrieves the corresponding repair manual from the database and sends it to the user's device. The repair manual contains detailed instructions for the repair, so the user can follow them to attempt the simple repair. For example, if the air conditioner's filter is diagnosed as clogged, a manual including instructions for cleaning the filter will be provided.

[1029] If the simple repair is unsuccessful, the user reports this on the application. The device again sends the information to the server, which then automatically contacts the manufacturer. The server then coordinates a repair date with the manufacturer, determines the earliest possible date and time for repair, and notifies the user's device of this information. The user can then confirm the notified repair date and wait for on-site repair.

[1030] The server also records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the proposed repair manual, the success or failure of simple repairs, communication with the manufacturer, and confirmation of repair dates. This data is used to predict future failures and for preventive maintenance.

[1031] As a specific example of how it works, let's say a user takes a photo of an air conditioning unit with their smartphone and uploads it to the "Building Trouble SOS" app. They also enter the contents of any error messages. The device converts this information into JSON format and sends it to the server. The server analyzes the photo and text based on an AI model and diagnoses that "the filter is likely clogged." Based on this diagnosis, it presents filter cleaning instructions and sends them to the user. The user attempts to clean the filter by following the instructions displayed on their smartphone, but if there is no improvement, they report it again. The server automatically contacts the manufacturer and arranges a repair date. All of this information is recorded in a database and will be used for later analysis.

[1032] An example of a prompt sentence might be:

[1033] "If the air conditioning in a building is not working, it is likely that the filter is clogged. Enter the filter cleaning procedure as a simple repair manual in this case."

[1034] In this way, the system provides a series of processes for quickly and efficiently resolving equipment problems in building maintenance.

[1035] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1036] Step 1:

[1037] Users access the "Building Trouble SOS" platform using their smartphones or PCs. They input information about the equipment's status (for example, a report that the air conditioning is not working or an error message), and take and upload photos of the equipment if necessary. This input process provides specific information about the equipment's malfunction.

[1038] Step 2:

[1039] The terminal converts the information entered by the user and the uploaded photo into a single JSON data, which contains text information and image data separately, and sends the converted JSON data to the server.

[1040] Step 3:

[1041] The server parses the received JSON data and separates the photo data from the text data. The server saves the photo data as an image file (e.g., JPEG format) and the text data as string information. This analysis prepares the data needed for the next step.

[1042] Step 4:

[1043] The server sends the photo data to an image recognition AI model (e.g., using TensorFlow) and the text data to a natural language processing model (e.g., a GPT-3 or BERT-based model). The server receives the diagnosis results returned by these models. For example, an image recognition model may return a diagnosis that "the filter is likely clogged." The input at this stage is image data and text data, and the output is the diagnosis result.

[1044] Step 5:

[1045] The server determines whether a simple repair is possible based on the diagnosis results obtained from the AI ​​model. If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual from the database and sends it to the user's device. For example, a repair procedure such as "filter cleaning procedure" is provided to the user. In this step, the diagnosis results of the AI ​​model are the input, and the simple repair manual is the output.

[1046] Step 6:

[1047] The user attempts to perform a simple repair according to the repair manual displayed on the terminal, for example, removing and cleaning the filter of the air conditioning unit, and checks the results of the simple repair (for example, whether the air conditioning works properly).

[1048] Step 7:

[1049] If the simple repair is not successful, the user reports this in the application. The device then sends the data containing the user's report back to the server. This report includes the repair procedure that was attempted and the results.

[1050] Step 8:

[1051] The server records the failure of the simple repair based on the data resent by the user and automatically contacts the manufacturer. The server then automatically coordinates a repair schedule with the manufacturer and determines the earliest possible date and time. This output is the repair schedule information, which is then notified to the user's terminal.

[1052] Step 9:

[1053] The server records each step of the troubleshooting process in a database, including information entered by the user, the diagnosis results from the AI ​​model, the repair manual provided, the success or failure of simple repairs, communication with the manufacturer, and confirmation of repair dates. This data is used to predict future failures and for preventive maintenance.

[1054] (Application example 1)

[1055] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1056] When a robot operating in a factory breaks down or malfunctions, a high level of specialized knowledge and experience is required to identify the cause and quickly repair it. This makes it difficult for on-site operators and managers to respond quickly, leading to production line shutdowns and unexpected increases in costs. Furthermore, if appropriate repair procedures and communication and coordination with equipment manufacturers are not carried out efficiently, repair delays may occur. There is a need for a system that can solve these problems and perform robot maintenance efficiently and effectively.

[1057] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1058] In this invention, the server includes a means for users to input equipment status information using a smart device, a means for using a generative AI model to analyze equipment photos and error messages, and a means for optimizing diagnostic and repair processes specific to the application category. This allows on-site operators and managers to easily input status information when a robot operating in a factory breaks down or malfunctions, enabling quick and accurate fault diagnosis based on that information. Furthermore, if a simple repair is possible, the system presents specific repair procedures and, if necessary, automatically contacts the equipment manufacturer and arranges a repair schedule, thereby minimizing repair delays and shortening production line downtime.

[1059] The "means for inputting equipment status information" refers to a means by which a user inputs information such as a photograph of the equipment or an error message into a terminal.

[1060] The "means for transmitting equipment status information to the server" is a communication means for transferring input equipment status information to the server.

[1061] The "means for analyzing the equipment status information received by the server" is an analysis function for processing the status information received by the server and identifying the cause of an abnormality or failure.

[1062] The "means for determining whether simple repair is possible based on the analyzed information" is a means for determining whether simple repair is possible based on the analyzed information.

[1063] The "means for presenting a repair manual when simple repair is possible" is a means for providing a user with a manual describing the repair method when simple repair is possible.

[1064] "Means for contacting the equipment manufacturer when simple repairs are not possible" refers to a means for automatically contacting the equipment manufacturer when repairs are difficult.

[1065] "Means for coordinating repair schedules with equipment manufacturers" refers to means for coordinating and deciding repair schedules with equipment manufacturers.

[1066] The "means for notifying the user of the repair schedule" is a means for notifying the user of the confirmed repair schedule.

[1067] "Means for recording the history of troubleshooting" refers to a means for saving the steps and results of each troubleshooting in a database.

[1068] "Means for a user to input equipment status information using a smart device" refers to means for a user to input equipment status information using a device such as a smartphone or smart glasses.

[1069] "Means using a generative AI model to analyze equipment photos and error messages" means means using a generative AI model to analyze equipment photos and error messages.

[1070] "Measures to optimize diagnostic and repair processes specific to the application category" are measures to streamline diagnostic and repair processes to suit the specific requirements and environment of the application.

[1071] This invention describes in detail a method for realizing a maintenance management system for robots operating in a factory. This system begins when a user inputs equipment status information using a smart device and sends that information to a server. Each step is described in detail below.

[1072] 1. User Input

[1073] Users (factory managers and operators) use their smartphones or smart glasses to input status information about robots that have malfunctioned or are experiencing an abnormality into the application. This status information includes a photo of the robot and an error message. The photographed and input information is converted into JSON format and sent to the server.

[1074] 2. Initial diagnosis by the server

[1075] The server receives JSON data sent from the user device. The received data is analyzed using a generative AI model (e.g., TensorFlow or PyTorch). The photo and text data are converted into an appropriate format and sent to the AI ​​model to predict the cause of the failure. During this analysis, anomalies are identified from the photo and the problem is extracted from the error message.

[1076] 3. Simple repair manual provided

[1077] If the server determines that simple repairs are possible based on the analysis results, it provides a repair manual to the user's terminal. The repair manual contains detailed instructions on specific procedures, and the user can attempt to repair the device by following the instructions displayed on the smart device.

[1078] 4. Contact the equipment manufacturer and arrange repair schedules

[1079] If the simple repair fails or if it is determined that a more complex repair is required, the user can report it again through the application. Based on the report, the server will automatically contact the equipment manufacturer and arrange a repair date. Once the arrangement is complete, the server will notify the user of the confirmed repair date.

[1080] 5. Recording and using troubleshooting history

[1081] The server records the history of each troubleshooting in a database. This data includes information entered by the user, the results of the AI ​​diagnosis, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. The accumulated data is used to predict future failures and for preventive maintenance.

[1082] Hardware and software used

[1083] Hardware: Smartphones, smart glasses, servers

[1084] Software: Generative AI models (TensorFlow, PyTorch), applications (for smart devices)

[1085] Specific examples

[1086] For example, if a robot arm in operation in a factory stops working properly, the operator takes a photo of the robot arm with the smart glasses and enters an error message. The information is then sent to the server via the app. The server analyzes the data using a generative AI model and diagnoses the malfunction as "malfunction due to motor overload." Based on the results, the user is presented with specific repair procedures. If the repair is unsuccessful, the operator contacts the equipment manufacturer again via the app to arrange a repair date.

[1087] Prompt Sentence Examples

[1088] "Please take a photo of any abnormalities in the robotic arms operating in the factory and enter the error message."

[1089] This system makes it possible to respond quickly and efficiently to malfunctions or abnormalities in robots operating in factories.

[1090] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1091] Step 1:

[1092] The user uses a smart device to input equipment status information. Specifically, they take a photo of the abnormal part of the robot and enter the error message into the application. The input information is converted into JSON format.

[1093] Input: Photo data, text data (error message)

[1094] Output: JSON format data

[1095] Step 2:

[1096] The user's device (smartphone or smart glasses) sends equipment status information converted into JSON format to the server.

[1097] Input: JSON format data

[1098] Output: Data transfer to the server

[1099] Step 3:

[1100] The server analyzes the received JSON data. Specifically, it converts the photo data and text data into an appropriate format and sends it to a generative AI model (TensorFlow or PyTorch). The AI ​​model performs image analysis, identifies abnormalities from the photo, and performs text analysis to extract the cause of the failure from the error message.

[1101] Input: JSON format data

[1102] Output: Image analysis results, text analysis results

[1103] Step 4:

[1104] The server determines whether a simple repair is possible based on the analysis results. Based on the diagnosis results returned by the AI ​​model, it determines whether a simple repair is possible. This process refers to past repair history and common failure patterns in the database.

[1105] Input: Image analysis results, text analysis results

[1106] Output: Simple repair possible / impossible result

[1107] Step 5:

[1108] If it is determined that simple repair is possible, the server sends the corresponding repair manual to the user's terminal. The repair manual contains detailed instructions on the specific repair procedures, and the user attempts to repair the device by following the instructions displayed on the smart device.

[1109] Input: Simple repair possibility determination result

[1110] Output: Repair manual

[1111] Step 6:

[1112] If the simple repair fails or if it is determined that a more complex repair is required, the user can report it again through the application. Based on the report, the server will automatically contact the equipment manufacturer and arrange a repair date. Once the arrangement is complete, the server will notify the user of the confirmed repair date.

[1113] Input: User report data (simple repair failure report)

[1114] Output: Contact with equipment manufacturer, results of repair schedule adjustment, notification to user

[1115] Step 7:

[1116] The server records the history of each troubleshooting in a database. The recorded data includes information entered by the user, the results of the AI ​​diagnosis, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. This data is used to predict future failures and for preventive maintenance.

[1117] Input: Trouble response data (user input information, AI diagnosis results, repair results, communication logs, etc.)

[1118] Output: Records in the database

[1119] Prompt Sentence Examples

[1120] "Please take a photo of any abnormalities in the robotic arms operating in the factory and enter the error message."

[1121] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1122] This invention is a building maintenance system for responding quickly and efficiently to equipment breakdowns. The system supports responding to equipment breakdowns by having the user input equipment status information, analyzing it via a server, and providing a simple repair manual or contacting the equipment manufacturer as necessary. Furthermore, the system improves the user experience by combining it with an emotion engine that recognizes the user's emotions and flexibly adjusts its response depending on the situation.

[1123] User Input

[1124] Users (building managers and general users) use their smartphones or PCs to input equipment status information into the "Building Trouble SOS" platform. This status information includes photos of the equipment and text of error messages. The device converts the input information into JSON format and sends it to the server.

[1125] Initial diagnosis by the server

[1126] The server analyzes the received JSON data and converts the photo and text data into the appropriate format. It then sends the data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The generated AI model predicts the cause of the failure based on the received information. The server determines whether a simple repair is possible based on the diagnosis results returned by the AI ​​model.

[1127] Simple repair manual provided

[1128] If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual and sends it to the user's device. The user then attempts the simple repair by following the repair manual displayed on their smartphone or PC. The manual contains detailed instructions on specific repair procedures, allowing the user to follow the steps to resolve the problem.

[1129] Contacting equipment manufacturers and arranging repair schedules

[1130] If the problem is not resolved by simple repairs, the user reports the situation on the application. The device then sends this report back to the server. The server records the information that the simple repairs failed and immediately and automatically contacts the equipment manufacturer. The server then automatically coordinates a repair schedule with the equipment manufacturer and determines the earliest possible date and time for repairs. The server then notifies the user's device of the confirmed schedule. The user then checks the notified repair schedule and waits for on-site repairs.

[1131] User support with emotion engine

[1132] Furthermore, this system has a function that allows the emotion engine to recognize the user's emotions based on the user's input information and behavior. For example, if the user frequently inputs error messages or shows behavior that suggests frustration with repair procedures, the emotion engine will detect the user's stress or dissatisfaction.

[1133] Flexible response based on emotions

[1134] Based on the user's emotions recognized by the emotion engine, the server can flexibly change the messages and repair manuals it presents. For example, if the user is feeling anxious, it can provide a message containing words of encouragement and support. If the user is confused by a complicated procedure, it can present a simpler explanation or a video tutorial.

[1135] Expert Notification and Additional Support

[1136] If the emotion engine detects a high level of stress or dissatisfaction in the user, the server will automatically notify the equipment manufacturer's experts and request additional support. This feature allows users to receive professional assistance quickly and without having to struggle alone, improving the speed and accuracy of problem resolution.

[1137] Recording and using troubleshooting history

[1138] The server records each troubleshooting step in a database. This data includes information entered by the user, the AI ​​diagnosis results, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, confirmed repair schedules, and emotional information recognized by the emotion engine. This data can be used to predict future failures and for preventive maintenance.

[1139] Specific examples

[1140] For example, if a building's air conditioning stops working, the building manager takes a photo of the air conditioning unit with their smartphone and uploads it to the app. If an error message is displayed, they enter the details. The device then sends this information to the server. The server analyzes the photo and text based on a generative AI model and determines that the filter is likely clogged. Based on the diagnosis, the server presents filter cleaning instructions and sends them to the user. The user then attempts to clean the filter by following the instructions displayed on their smartphone. If cleaning does not improve the situation, the user reports it again through the application, and the server automatically contacts the equipment manufacturer and schedules a repair. All of this information is recorded in a database and will be used for future analysis.

[1141] Furthermore, if the user indicates a high level of stress in the emotion engine, the server notifies experts and prepares a means to provide additional support. For example, if a situation arises in which the user is "very troubled by frequent error messages," the emotion engine will recognize the stress and the server will immediately contact experts to request a prompt response. In this way, the system quickly and efficiently resolves equipment problems in building maintenance while simultaneously reducing the user's psychological burden.

[1142] The processing flow will be explained below.

[1143] Step 1:

[1144] When a user notices that a building facility has broken down, they access the "Building Trouble SOS" platform using their smartphone or PC, take a photo of the facility, and enter a description of the current situation (e.g., "The air conditioning is not working" or "An error message is displayed") in text.

[1145] Step 2:

[1146] The device receives the input photo and text data and converts it into JSON format, including the type of equipment (air conditioning, lighting, network equipment, etc.).

[1147] Step 3:

[1148] The device sends the generated JSON data to the server, which includes the user's input information and photo data.

[1149] Step 4:

[1150] The server parses the received JSON data and converts the photo and text data into the appropriate format, which prepares it for passing to the AI ​​model.

[1151] Step 5:

[1152] The server sends the converted data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The AI ​​model predicts the cause of the failure.

[1153] Step 6:

[1154] The AI ​​model analyzes the photo and text data to identify the cause of the malfunction, such as a clogged filter or a blown fuse, and returns the diagnosis results to the server.

[1155] Step 7:

[1156] The server receives the diagnosis results from the AI ​​model and determines whether a simple repair is possible. If a simple repair is possible, it searches for the corresponding repair manual.

[1157] Step 8:

[1158] If the server determines that simple repair is possible, it retrieves the corresponding repair manual and sends it to the user's device. The repair manual contains specific repair procedures.

[1159] Step 9:

[1160] The device displays the received repair manual on its screen. The user attempts simple repairs by following the instructions in the manual. For example, the manual contains instructions for cleaning the filter and turning the power on and off.

[1161] Step 10:

[1162] After the user performs the simple repair, they report the results in the application. If the repair is successful, the application simply ends.

[1163] Step 11:

[1164] If the problem is not resolved by the simple repair, the user reports it as "not resolved" on the application. The terminal then sends this report to the server.

[1165] Step 12:

[1166] The server records the information about the failure of the simple repair and immediately contacts the equipment manufacturer automatically, sending a repair request to the equipment manufacturer.

[1167] Step 13:

[1168] The server receives the response from the equipment manufacturer and arranges the earliest possible date for repair. It also acquires the schedule information received from the equipment manufacturer.

[1169] Step 14:

[1170] The server transmits the confirmed repair schedule to the user's terminal and notifies the user.

[1171] Step 15:

[1172] The terminal notifies the user of the received repair schedule, and the user confirms the notified repair schedule and waits for the equipment manufacturer to visit.

[1173] Step 16:

[1174] The server records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the repair manual provided, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmation of repair dates.

[1175] Step 17:

[1176] The emotion engine analyzes the user's input information and behavior to recognize the user's emotional state. For example, it can detect stress or frustration based on frequent error messages or delayed responses to operations.

[1177] Step 18:

[1178] The server flexibly changes the messages and repair manuals it presents based on the user's emotions recognized by the emotion engine. For example, if the user feels anxious, it displays encouraging messages, and if the user is confused by a complicated procedure, it provides simpler explanations.

[1179] Step 19:

[1180] If the emotion engine detects high levels of stress or dissatisfaction, the server automatically notifies the equipment manufacturer's experts and requests additional support, allowing users to receive fast, professional assistance.

[1181] Step 20:

[1182] The server records the trouble response history, including emotion recognition information, in a database and updates the data model used for future failure prediction and preventive maintenance.

[1183] By taking the above specific steps, the system can quickly and efficiently resolve equipment problems while reducing the psychological burden on users and providing a better user experience.

[1184] Example 2

[1185] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1186] Conventional building maintenance systems tend to respond slowly when equipment malfunctions occur, resulting in the user having to spend time and effort to deal with the issue. Furthermore, the system only provides a uniform response without considering the user's emotions or stress level, which can lead to a decline in user satisfaction. Furthermore, the accumulation and analysis of trouble response data was insufficient, meaning that the data could not be used to predict future malfunctions or for preventive maintenance.

[1187] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1188] In this invention, the server includes means for a user to input equipment status information, means for a terminal to send the equipment status information to the server, means for analyzing the equipment status information received by the server, means for identifying the cause of the failure using a generative AI model based on the analyzed information and determining whether a simple repair is possible, means for presenting a repair manual to the user if a simple repair is possible, means for contacting the equipment manufacturer if a simple repair is not possible, means for coordinating a repair schedule with the equipment manufacturer, means for notifying the user of the adjusted repair schedule, means for analyzing the user's emotions using an emotion engine and coordinating a response based on the analysis, and means for recording the trouble response history. This not only enables equipment failures to be responded to quickly and efficiently, but also reduces the psychological burden on the user and enables the use of data for failure prediction and preventive maintenance.

[1189] "User" refers to a person who uses the system to input equipment status information and receive repairs and support.

[1190] "Terminal" refers to a device such as a computer or smartphone that a user uses to input equipment status information.

[1191] "Server" refers to a computer system that receives equipment status information sent from a terminal and performs functions such as analysis, providing repair manuals, and contacting the equipment manufacturer.

[1192] "Generative AI model" refers to an artificial intelligence model used to analyze photographs and text data and identify the cause of equipment failure.

[1193] "Cause of failure" refers to the analysis results used to identify the cause of equipment abnormalities or malfunctions.

[1194] "Simple repair" refers to a simple repair method that the user can perform themselves.

[1195] "Repair manual" refers to a written or digital file that details steps and methods for performing simple repairs.

[1196] "Equipment manufacturer" refers to the company or professional that manufactured or sold the faulty equipment.

[1197] "Repair Schedule" means the date and time set by the equipment manufacturer for on-site repair work.

[1198] An "emotion engine" refers to software that analyzes emotions from user input and behavior, and then takes appropriate action based on the results.

[1199] "Trouble response history" refers to a record of a series of responses made within the system, such as information on equipment failures, analysis results, repair manuals, repair status, and emotion analysis results from the emotion engine.

[1200] "Failure prediction" refers to predicting future equipment failures by analyzing accumulated trouble response history.

[1201] "Preventive maintenance" refers to maintenance activities to prevent equipment failures before they occur, based on the results of failure prediction.

[1202] This invention is a building maintenance system that supports responding to equipment failures by having users input equipment status information, analyzing it via a server, and providing simple repair manuals or contacting the equipment manufacturer as necessary. Furthermore, the user experience is improved by combining it with an emotion engine that recognizes the user's emotions and flexibly adjusts its response depending on the situation.

[1203] Hardware and software used

[1204] Users access the system using a smartphone or PC. The system uses the following hardware and software:

[1205] User device: Smartphone or PC

[1206] Server: Cloud server or on-premise server

[1207] Generative AI models: ResNet (image analysis), BERT (natural language processing)

[1208] Emotion engine: IBM Watson

[1209] Data processing and calculation

[1210] 1. User Input

[1211] Users log in to the Building Trouble SOS platform and enter equipment status information, including photos of the equipment and the text of any error messages.

[1212] Specific examples

[1213] For example, if the air conditioning in a building stops working, a user can take a photo of the air conditioning unit with their smartphone, upload it to the app, and enter the details of any error messages that may be displayed.

[1214] 2. Data Transmission

[1215] The terminal converts the input information into JSON format and sends it to the server.

[1216] Specific examples

[1217] The device encodes the photo data into Base64 format, retrieves the error message from the text field, generates a JSON containing these, and sends it to the server via an HTTP POST request.

[1218] 3. Initial diagnosis

[1219] The server parses the received JSON data, converts the photo data and text data into an appropriate format, and then sends the data to an AI model (e.g., ResNet) for photo analysis and a natural language processing model (e.g., BERT) for text analysis, requesting a diagnosis.

[1220] Specific examples

[1221] The server analyzes the photo and text based on a generative AI model and diagnoses whether the filter is likely clogged.

[1222] Repair manual provided

[1223] The server determines whether a simple repair is possible based on the diagnosis results, and if so, obtains the appropriate repair manual and sends it to the user's device.

[1224] Specific examples

[1225] The server retrieves the filter cleaning instructions from the database and sends the PDF file to the user's device.

[1226] 4. Use of Emotion Engine

[1227] The server sends the entered error message and the user's behavior log to an emotion engine (e.g., IBM Watson), which recognizes the user's emotional state (stress, frustration, anxiety, etc.) and provides messages of support and encouragement.

[1228] Specific examples

[1229] If the emotion engine detects a high level of stress in the user, the server generates a cheering message and sends it to the user's terminal.

[1230] Expert Notice

[1231] If simple repairs fail or if the user shows high stress, the server automatically contacts the equipment manufacturer and schedules a repair.

[1232] Specific examples

[1233] If the user reports again that the simple repair has failed, the server will contact the equipment manufacturer, set the earliest possible repair date, and notify the user.

[1234] Record of troubleshooting history

[1235] The server records all troubleshooting steps in a database, including information entered by the user, the diagnosis results of the AI ​​model, the proposed repair manual, the success or failure of the repair, communication with the equipment manufacturer, the confirmed repair schedule, and the emotional information recognized by the emotion engine.

[1236] Prompt Sentence Examples

[1237] Here is an example prompt:

[1238] Prompt statement

[1239] "Please identify the cause of the building equipment failure based on the error message and photo below. Error message: 'Air conditioner not working', Photo: [URL of photo of air conditioner]"

[1240] The above is a specific description of an embodiment of the present invention. The system aims to enable quick and efficient response to equipment failures and reduce the psychological burden on users.

[1241] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1242] Step 1:

[1243] Users input the equipment status information by logging into the Building Trouble SOS platform using a smartphone or PC, taking a photo of the equipment, and entering error messages and a detailed description of the situation.

[1244] Input: Equipment photo, error message, detailed description of the situation

[1245] Output: Equipment status information in JSON format

[1246] Specific actions

[1247] The user takes a photo of the air conditioning unit.

[1248] The user enters the error message "The air conditioner is not working."

[1249] This information is converted to JSON format for transmission to the server.

[1250] Step 2:

[1251] The device sends the status information entered by the user to the server. The data is converted to JSON format and sent to the server as an HTTP POST request.

[1252] Input: Equipment status information in JSON format

[1253] Output: Data sent to the server

[1254] Specific actions

[1255] The device encodes the photo data into Base64 format.

[1256] The terminal retrieves the error message from the text field.

[1257] The device generates JSON data and sends it to the server via an HTTP POST request.

[1258] Step 3:

[1259] The server parses the received JSON data, converts the photo data and text data into an appropriate format, and then sends the data to an AI model (e.g., ResNet) for photo analysis and a natural language processing model (e.g., BERT) for text analysis, requesting a diagnosis.

[1260] Input: Equipment status information in JSON format

[1261] Output: Diagnostic results (e.g., possible clogged filter)

[1262] Specific actions

[1263] The server extracts the photo data from the JSON data and inputs it into an image analysis model (e.g., ResNet).

[1264] The server inputs the error message into a natural language processing model (e.g., BERT).

[1265] The server receives the diagnostic results returned by the AI ​​model and determines whether a simple repair is possible.

[1266] Step 4:

[1267] If the server determines that a simple repair is possible based on the diagnostic results of the generated AI model, it retrieves the corresponding repair manual from the database and sends it to the user's device.

[1268] Input: Diagnostic results

[1269] Output: Repair manual

[1270] Specific actions

[1271] The server retrieves the filter cleaning instructions from the database.

[1272] The server sends a PDF file of the repair manual to the user's device.

[1273] Step 5:

[1274] The user attempts a simple repair by following the repair manual displayed on the terminal.

[1275] Input: Repair manual

[1276] Output: Simple repair success or failure

[1277] Specific actions

[1278] The user opens the PDF manual.

[1279] The user should follow the "filter removal method" and "cleaning procedure" written in the manual.

[1280] Step 6:

[1281] If the problem cannot be resolved by simple repairs, the user reports the situation. The device then sends this information back to the server, which then automatically contacts the equipment manufacturer. The server then automatically schedules a repair and notifies the user.

[1282] Input: Simple repair success / failure, status report

[1283] Output: Contact with equipment manufacturer and repair schedule

[1284] Specific actions

[1285] The user presses a button to report a "repair failed."

[1286] The terminal sends a request to the server to report the status.

[1287] The server automatically contacts the equipment manufacturer via their API and schedules the earliest possible repair date.

[1288] The server notifies the user of the confirmed repair schedule.

[1289] Step 7:

[1290] The server uses an emotion engine to analyze the user's emotions and detect their stress or frustration. Based on the results of the detection, it provides messages of encouragement and support.

[1291] Input: Error message, Action log

[1292] Output: Message of support

[1293] Specific actions

[1294] The server sends the entered error message and the user's action log to the emotion engine.

[1295] The emotion engine analyzes the user's emotions and determines that they are "feeling stressed."

[1296] The server generates a cheering message and sends it to the user's terminal.

[1297] Step 8:

[1298] If the emotion engine detects high levels of stress or dissatisfaction, the server automatically contacts the equipment manufacturer to request additional support.

[1299] Input: Sentiment analysis results

[1300] Output: Liaison with equipment manufacturers and additional support

[1301] Specific actions

[1302] The server sends a "high stress state" notification to the equipment manufacturer.

[1303] The server forwards any additional support instructions or messages from the experts to the user.

[1304] Step 9:

[1305] The server records each troubleshooting step in a database for future analysis and preventative maintenance.

[1306] Input: Troubleshooting data

[1307] Output: Accumulated troubleshooting history

[1308] Specific actions

[1309] The server stores all data in a database, including information entered by the user, the diagnosis results from the AI ​​model, the presented repair manual, and whether the repair was successful or not.

[1310] The stored data is used for analysis to predict failures and plan maintenance.

[1311] (Application example 2)

[1312] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1313] When equipment failure occurs, it is necessary to quickly and efficiently identify the cause of the failure and carry out simple repairs. However, conventional systems take time to identify the cause of the failure, which places a heavy psychological burden on users. In addition, they are unable to respond flexibly and take into account the user's emotions, which can lead to the accumulation of stress and dissatisfaction. To solve these issues, a system is needed that monitors the status of equipment in real time and uses an emotion engine to respond according to the user's emotions.

[1314] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1315] In this invention, the server includes a means for analyzing equipment status information, a means for determining whether simple repairs are possible, a means for presenting a repair manual, and a means for identifying the user's emotions and changing the response accordingly. This allows the cause of equipment failure to be identified quickly and efficiently, and enables flexible response while reducing user stress and frustration. Furthermore, the accumulated trouble response history can be analyzed and used for future failure prediction and preventive maintenance.

[1316] "Equipment status information" refers to data entered by users, such as the equipment's operating status, abnormal conditions, error messages, etc.

[1317] "Server" refers to a central management system that receives, analyzes, processes, and issues various decisions and notifications regarding the status of equipment.

[1318] "Means of analysis" refers to algorithms or programs for identifying the cause of a failure based on equipment status information.

[1319] "Means for determining whether simple repairs are possible" refers to algorithms or mechanisms that determine whether the target equipment can be easily repaired based on the analysis results.

[1320] "Means for presenting a repair manual" refers to a method or system that provides a user with specific repair procedures when simple repairs are possible.

[1321] "Equipment manufacturer" refers to the company or professional engineer who manufactured the equipment in question.

[1322] "Means for arranging repair schedules" refers to a system that communicates with equipment manufacturers and automatically determines the date and time for repairs.

[1323] "User" refers to anyone who operates or manages the facility, including building managers and general users.

[1324] "Means for recording trouble response history" refers to a system that stores the process of each trouble response in a database for later analysis and use.

[1325] "Means for identifying emotions and changing the response according to those emotions" refers to a function that recognizes the user's emotions such as stress or dissatisfaction, and appropriately changes the information provided and the response based on that emotional state.

[1326] "Means for acquiring and analyzing images of equipment" refers to the technology and programs for collecting photographic data of equipment taken with a camera, etc., and analyzing those images.

[1327] "Means for determining abnormality" refers to a function that determines whether the equipment is operating normally or whether an abnormality has occurred based on the acquired data.

[1328] "Failure prediction and preventive maintenance" refers to measures that analyze accumulated data to predict and prevent possible future failures.

[1329] The present invention is a system that analyzes equipment status information, presents repair manuals as needed, and automates communication and coordination with equipment manufacturers. It also has the ability to recognize a user's emotions and respond accordingly. A specific embodiment of the system is described below.

[1330] Hardware and Software Configuration

[1331] The system operates around three main players: the server, the terminal, and the user. The server performs analysis processing and database management, the terminal is responsible for inputting and receiving information, and the user provides equipment status information.

[1332] server:

[1333] Hardware: A server computer equipped with a high-performance CPU and GPU. A network interface card (NIC) that enables high-speed data transfer is used.

[1334] software:

[1335] Analysis module: Uses PhotoAIModel and TextAIModel to analyze equipment photo data and error messages.

[1336] Emotion Recognition Module: Uses the EmotionEngine to recognize emotions based on user input and actions.

[1337] Database: Use a reliable database system such as MySQL or PostgreSQL to store troubleshooting history and equipment status.

[1338] Device:

[1339] Hardware: Smartphone, tablet, smart glasses, head-mounted display (HMD), or PC.

[1340] software:

[1341] Input interface: Provides a dedicated application for users to input equipment status information.

[1342] Display Interface: GUI for displaying repair manuals and messages.

[1343] User:

[1344] Role: Inputs equipment status information (photos, error messages, etc.), refers to repair manuals, and performs simple repairs. Also, uses emotion recognition to reduce user stress and frustration.

[1345] Data processing and calculation

[1346] The terminal sends the equipment status information (photos and error messages) entered by the user to the server, which analyzes the data using the following procedure and takes the necessary action.

[1347] 1. Data reception and analysis:

[1348] The server receives the JSON format data sent from the terminal.

[1349] The received data is input into PhotoAIModel and TextAIModel to analyze the cause of equipment failure.

[1350] 2. Repair decision and manual presentation:

[1351] Based on the analysis results, it is determined whether simple repairs are possible.

[1352] If simple repair is possible, the corresponding repair manual is obtained and sent to the terminal.

[1353] 3. Liaison and coordination with equipment manufacturers:

[1354] If simple repair is not possible, the equipment manufacturer will be contacted automatically.

[1355] We will contact the equipment manufacturer, schedule a repair, and notify the user.

[1356] 4. Emotion recognition and response modification:

[1357] EmotionEngine recognizes the user's emotions based on the user's input information and actions.

[1358] To flexibly change messages to be displayed and contents of repair manuals according to emotional information of a user.

[1359] If it detects high levels of stress or dissatisfaction, it will notify experts and provide additional support.

[1360] Specific examples

[1361] For example, if a robot in a factory stops working, a worker can use smart glasses to take a picture of the robot's condition and send the photo and an error message to the system. The system then analyzes the failure and diagnoses that the sensor needs to be calibrated. The worker is then presented with a calibration procedure, which they follow to attempt calibration. If the worker becomes stressed, the system's emotion engine will recognize this early on and send a message of encouragement or request expert assistance.

[1362] Prompt Sentence Examples

[1363] Please diagnose the cause of the failure based on the following equipment photo and error message. Photo: [Image path], Message: [Error message]

[1364] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1365] Step 1:

[1366] The user uses smart glasses or a head-mounted display (HMD) to capture a photo of the equipment and input an error message, which is then stored on the device.

[1367] Input: Facility photo, error message

[1368] Output: JSON format data saved on the device

[1369] Step 2:

[1370] The device sends the saved state information in JSON format to the server.

[1371] Input: JSON data of facility photos and error messages

[1372] Output: JSON data sent to the server

[1373] Step 3:

[1374] The server analyzes the received JSON data. The photo data is sent to the PhotoAIModel, and the error message is sent to the TextAIModel. Each model analyzes the data and identifies the cause of the error.

[1375] Input: JSON data sent to the server

[1376] Output: Analysis results from PhotoAIModel and TextAIModel

[1377] Step 4:

[1378] The server determines whether simple repairs are possible based on the analysis results obtained from PhotoAIModel and TextAIModel.

[1379] Input: Analysis results of PhotoAIModel and TextAIModel

[1380] Output: Simple repair possible

[1381] Step 5:

[1382] If it is determined that simple repair is possible, the server obtains the corresponding repair manual and sends it to the terminal.

[1383] Input: Judgment result that simple repair is possible

[1384] Output: Repair manual

[1385] Step 6:

[1386] The user attempts to carry out simple repairs according to the repair manual displayed on the terminal, and when the repair is complete, inputs the results into the terminal.

[1387] Input: Repair manual

[1388] Output: Simple repair results

[1389] Step 7:

[1390] If simple repairs are impossible or have failed, the user sends the information back to the server from their terminal, which then automatically contacts the equipment manufacturer and arranges a repair date.

[1391] Input: Simple repair failure information

[1392] Output: Repair schedule adjustment results with equipment manufacturer

[1393] Step 8:

[1394] The server notifies the user of the adjusted repair schedule, and the user confirms the schedule and waits for the repair to be performed.

[1395] Input: Repair schedule adjustment results with equipment manufacturer

[1396] Output: Notification of repair schedule to user

[1397] Step 9:

[1398] The server uses the Emotion Engine to identify emotions based on user input and behavior, and flexibly changes the corresponding messages and repair manuals based on this emotional information.

[1399] Input: User input and behavioral information

[1400] Output: Emotion-based messages and repair manuals

[1401] Step 10:

[1402] If the emotion engine detects high levels of stress or dissatisfaction in the user, the server immediately notifies an expert and provides additional support. Subsequent responses are also recorded in the database.

[1403] Input: Emotional information about high levels of stress and frustration

[1404] Output: Notification to experts, database record of troubleshooting history

[1405] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1406] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1407] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1408] [Fourth embodiment]

[1409] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1410] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1411] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1412] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1413] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1414] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1415] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1416] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1417] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1418] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1419] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1420] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1421] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1422] This invention is a building maintenance system for responding to equipment failures quickly and efficiently. The system supports responding to equipment failures by having the user input equipment status information, analyzing it via a server, and providing a simple repair manual or contacting the equipment manufacturer as necessary.

[1423] User Input

[1424] Users (building managers and general users) use their smartphones or PCs to input equipment status information into the "Building Trouble SOS" platform. This status information includes photos of the equipment and text of error messages. The device converts the input information into JSON format and sends it to the server.

[1425] Initial diagnosis by the server

[1426] The server analyzes the received JSON data and converts the photo and text data into the appropriate format. It then sends the data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The generated AI model predicts the cause of the failure based on the received information. The server determines whether a simple repair is possible based on the diagnosis results returned by the AI ​​model.

[1427] Simple repair manual provided

[1428] If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual and sends it to the user's device. The user then attempts the simple repair by following the repair manual displayed on their smartphone or PC. The manual contains detailed instructions on specific repair procedures, allowing the user to follow the steps to resolve the problem.

[1429] Contacting equipment manufacturers and arranging repair schedules

[1430] If the problem is not resolved by simple repairs, the user reports the situation on the application. The device then sends this report back to the server. The server records the information that the simple repairs failed and immediately and automatically contacts the equipment manufacturer. The server then automatically coordinates a repair schedule with the equipment manufacturer and determines the earliest possible date and time for repairs. The server then notifies the user's device of the confirmed schedule. The user then checks the notified repair schedule and waits for on-site repairs.

[1431] Recording and using troubleshooting history

[1432] The server records each troubleshooting step in a database. This data includes information entered by the user, the AI ​​diagnosis results, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. This data can be used to predict future failures and for preventive maintenance.

[1433] Specific examples

[1434] For example, if a building's air conditioning stops working, the building manager takes a photo of the air conditioning unit with their smartphone and uploads it to the app. If an error message is displayed, they enter the details. The device then sends this information to the server. The server analyzes the photo and text based on a generative AI model and determines that the filter is likely clogged. Based on the diagnosis, the server presents filter cleaning instructions and sends them to the user. The user then attempts to clean the filter by following the instructions displayed on their smartphone. If cleaning does not improve the situation, the user reports it again through the application, and the server automatically contacts the equipment manufacturer and schedules a repair. All of this information is recorded in a database and will be used for future analysis.

[1435] In this way, the system supports a series of processes for quickly and efficiently resolving equipment problems in building maintenance.

[1436] The processing flow will be explained below.

[1437] Step 1:

[1438] When a user notices that a building facility has broken down, they access the "Building Trouble SOS" platform using their smartphone or PC, take a photo of the facility, and enter a description of the current situation (e.g., "The air conditioning is not working" or "An error message is displayed") in text.

[1439] Step 2:

[1440] The device receives the input photo and text data and converts it into JSON format, including the type of equipment (air conditioning, lighting, network equipment, etc.).

[1441] Step 3:

[1442] The device sends the generated JSON data to the server, which includes the user's input information and photo data.

[1443] Step 4:

[1444] The server parses the received JSON data and converts the photo and text data into the appropriate format, which prepares it for passing to the AI ​​model.

[1445] Step 5:

[1446] The server sends the converted data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The AI ​​model predicts the cause of the failure.

[1447] Step 6:

[1448] The AI ​​model analyzes the photo and text data to identify the cause of the malfunction, such as a clogged filter or a blown fuse, and returns the diagnosis results to the server.

[1449] Step 7:

[1450] The server receives the diagnosis results from the AI ​​model and determines whether a simple repair is possible. If a simple repair is possible, it searches for the corresponding repair manual.

[1451] Step 8:

[1452] If the server determines that simple repair is possible, it retrieves the corresponding repair manual and sends it to the user's device. The repair manual contains specific repair procedures.

[1453] Step 9:

[1454] The device displays the received repair manual on its screen. The user attempts simple repairs by following the instructions in the manual. For example, the manual contains instructions for cleaning the filter and turning the power on and off.

[1455] Step 10:

[1456] After the user performs the simple repair, they report the results in the application. If the repair is successful, the application simply ends.

[1457] Step 11:

[1458] If the problem is not resolved by the simple repair, the user reports it as "not resolved" on the application. The terminal then sends this report to the server.

[1459] Step 12:

[1460] The server records the information about the failure of the simple repair and immediately contacts the equipment manufacturer automatically, sending a repair request to the equipment manufacturer.

[1461] Step 13:

[1462] The server receives the response from the equipment manufacturer and arranges the earliest possible date for repair. It also acquires the schedule information received from the equipment manufacturer.

[1463] Step 14:

[1464] The server transmits the confirmed repair schedule to the user's terminal and notifies the user.

[1465] Step 15:

[1466] The terminal notifies the user of the received repair schedule, and the user confirms the notified repair schedule and waits for the equipment manufacturer to visit.

[1467] Step 16:

[1468] The server records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the repair manual provided, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmation of repair dates.

[1469] Step 17:

[1470] The server analyzes the recorded data and updates a data model that can be used for future failure prediction and preventive maintenance, thereby improving the accuracy and speed of future failure responses.

[1471] Example 1

[1472] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1473] Conventional building maintenance systems often have delays in responding to equipment failures and unclear repair procedures, making it difficult to resolve problems. Even after simple repairs are attempted, feedback on their success or lack of prompt response time is often lacking, and additional specialized support is often not provided. Furthermore, data for failure prediction and preventive maintenance is not accumulated or analyzed, making it difficult to prevent future problems.

[1474] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1475] In this invention, the server includes a means for analyzing the equipment status information, a means for determining whether a simple repair is possible based on the analyzed information, and a means for contacting the manufacturer if a simple repair is not possible, thereby enabling the cause of the equipment failure to be quickly identified and an appropriate response to be proposed.

[1476] "Equipment status information" refers to data such as the operating status of machines and equipment within a building and error messages.

[1477] A "web server" refers to a computer system that transmits and receives data over the Internet or an intranet.

[1478] "Analysis" refers to the process of processing received data to understand its contents and identify the cause of the problem.

[1479] "Simple repair" refers to repair work that can be done by the user themselves, without the need for a specialized technician.

[1480] A "repair manual" is a document that provides instructions and methods for repairing a specific problem.

[1481] "Manufacturer" refers to a company that designs, manufactures, and sells equipment.

[1482] "Repair Schedule" means the date and time for carrying out repair work.

[1483] "Trouble response history" refers to data that records the troubles that occurred, how they were resolved, and the results.

[1484] "Equipment images" refers to photographs and video data of facilities and equipment.

[1485] "Fault determination" refers to the operation of identifying the cause and location of a failure in equipment or devices based on analyzed data.

[1486] "Preventive maintenance" refers to maintenance activities such as inspections and repairs carried out before equipment or machinery breaks down.

[1487] "Failure prediction" refers to the operation of predicting the possibility of future failure of facilities or equipment based on accumulated data.

[1488] This invention provides a building maintenance system that allows for quick and efficient response to equipment failures. Specifically, users, terminals, and servers work together to collect and analyze equipment status information and take action as needed.

[1489] Users access the "Building Trouble SOS" platform using their smartphones or PCs. They enter information about the equipment's status and, if necessary, take and upload photos. This information includes specific equipment malfunctions, error messages, and images of the equipment. The user's device converts this information into JSON format and sends it to the server.

[1490] The server analyzes the received JSON data and separates the photo data from the text data. The photo data is processed as an image file (e.g., JPEG format), and the text data is extracted as string information. The server then sends the photo data to an image recognition AI model (e.g., using TensorFlow) and the text data to a natural language processing model (e.g., a model based on GPT-3 or BERT). This allows the server to analyze the cause of the failure and, based on that information, determine whether a simple repair is possible.

[1491] If a simple repair is determined to be possible, the server retrieves the corresponding repair manual from the database and sends it to the user's device. The repair manual contains detailed instructions for the repair, so the user can follow them to attempt the simple repair. For example, if the air conditioner's filter is diagnosed as clogged, a manual including instructions for cleaning the filter will be provided.

[1492] If the simple repair is unsuccessful, the user reports this on the application. The device again sends the information to the server, which then automatically contacts the manufacturer. The server then coordinates a repair date with the manufacturer, determines the earliest possible date and time for repair, and notifies the user's device of this information. The user can then confirm the notified repair date and wait for on-site repair.

[1493] The server also records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the proposed repair manual, the success or failure of simple repairs, communication with the manufacturer, and confirmation of repair dates. This data is used to predict future failures and for preventive maintenance.

[1494] As a specific example of how it works, let's say a user takes a photo of an air conditioning unit with their smartphone and uploads it to the "Building Trouble SOS" app. They also enter the contents of any error messages. The device converts this information into JSON format and sends it to the server. The server analyzes the photo and text based on an AI model and diagnoses that "the filter is likely clogged." Based on this diagnosis, it presents filter cleaning instructions and sends them to the user. The user attempts to clean the filter by following the instructions displayed on their smartphone, but if there is no improvement, they report it again. The server automatically contacts the manufacturer and arranges a repair date. All of this information is recorded in a database and will be used for later analysis.

[1495] An example of a prompt sentence might be:

[1496] "If the air conditioning in a building is not working, it is likely that the filter is clogged. Enter the filter cleaning procedure as a simple repair manual in this case."

[1497] In this way, the system provides a series of processes for quickly and efficiently resolving equipment problems in building maintenance.

[1498] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1499] Step 1:

[1500] Users access the "Building Trouble SOS" platform using their smartphones or PCs. They input information about the equipment's status (for example, a report that the air conditioning is not working or an error message), and take and upload photos of the equipment if necessary. This input process provides specific information about the equipment's malfunction.

[1501] Step 2:

[1502] The terminal converts the information entered by the user and the uploaded photo into a single JSON data, which contains text information and image data separately, and sends the converted JSON data to the server.

[1503] Step 3:

[1504] The server parses the received JSON data and separates the photo data from the text data. The server saves the photo data as an image file (e.g., JPEG format) and the text data as string information. This analysis prepares the data needed for the next step.

[1505] Step 4:

[1506] The server sends the photo data to an image recognition AI model (e.g., using TensorFlow) and the text data to a natural language processing model (e.g., a GPT-3 or BERT-based model). The server receives the diagnosis results returned by these models. For example, an image recognition model may return a diagnosis that "the filter is likely clogged." The input at this stage is image data and text data, and the output is the diagnosis result.

[1507] Step 5:

[1508] The server determines whether a simple repair is possible based on the diagnosis results obtained from the AI ​​model. If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual from the database and sends it to the user's device. For example, a repair procedure such as "filter cleaning procedure" is provided to the user. In this step, the diagnosis results of the AI ​​model are the input, and the simple repair manual is the output.

[1509] Step 6:

[1510] The user attempts to perform a simple repair according to the repair manual displayed on the terminal, for example, removing and cleaning the filter of the air conditioning unit, and checks the results of the simple repair (for example, whether the air conditioning works properly).

[1511] Step 7:

[1512] If the simple repair is not successful, the user reports this in the application. The device then sends the data containing the user's report back to the server. This report includes the repair procedure that was attempted and the results.

[1513] Step 8:

[1514] The server records the failure of the simple repair based on the data resent by the user and automatically contacts the manufacturer. The server then automatically coordinates a repair schedule with the manufacturer and determines the earliest possible date and time. This output is the repair schedule information, which is then notified to the user's terminal.

[1515] Step 9:

[1516] The server records each step of the troubleshooting process in a database, including information entered by the user, the diagnosis results from the AI ​​model, the repair manual provided, the success or failure of simple repairs, communication with the manufacturer, and confirmation of repair dates. This data is used to predict future failures and for preventive maintenance.

[1517] (Application example 1)

[1518] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1519] When a robot operating in a factory breaks down or malfunctions, a high level of specialized knowledge and experience is required to identify the cause and quickly repair it. This makes it difficult for on-site operators and managers to respond quickly, leading to production line shutdowns and unexpected increases in costs. Furthermore, if appropriate repair procedures and communication and coordination with equipment manufacturers are not carried out efficiently, repair delays may occur. There is a need for a system that can solve these problems and perform robot maintenance efficiently and effectively.

[1520] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1521] In this invention, the server includes a means for users to input equipment status information using a smart device, a means for using a generative AI model to analyze equipment photos and error messages, and a means for optimizing diagnostic and repair processes specific to the application category. This allows on-site operators and managers to easily input status information when a robot operating in a factory breaks down or malfunctions, enabling quick and accurate fault diagnosis based on that information. Furthermore, if a simple repair is possible, the system presents specific repair procedures and, if necessary, automatically contacts the equipment manufacturer and arranges a repair schedule, thereby minimizing repair delays and shortening production line downtime.

[1522] The "means for inputting equipment status information" refers to a means by which a user inputs information such as a photograph of the equipment or an error message into a terminal.

[1523] The "means for transmitting equipment status information to the server" is a communication means for transferring input equipment status information to the server.

[1524] The "means for analyzing the equipment status information received by the server" is an analysis function for processing the status information received by the server and identifying the cause of an abnormality or failure.

[1525] The "means for determining whether simple repair is possible based on the analyzed information" is a means for determining whether simple repair is possible based on the analyzed information.

[1526] The "means for presenting a repair manual when simple repair is possible" is a means for providing a user with a manual describing the repair method when simple repair is possible.

[1527] "Means for contacting the equipment manufacturer when simple repairs are not possible" refers to a means for automatically contacting the equipment manufacturer when repairs are difficult.

[1528] "Means for coordinating repair schedules with equipment manufacturers" refers to means for coordinating and deciding repair schedules with equipment manufacturers.

[1529] The "means for notifying the user of the repair schedule" is a means for notifying the user of the confirmed repair schedule.

[1530] "Means for recording the history of troubleshooting" refers to a means for saving the steps and results of each troubleshooting in a database.

[1531] "Means for a user to input equipment status information using a smart device" refers to means for a user to input equipment status information using a device such as a smartphone or smart glasses.

[1532] "Means using a generative AI model to analyze equipment photos and error messages" means means using a generative AI model to analyze equipment photos and error messages.

[1533] "Measures to optimize diagnostic and repair processes specific to the application category" are measures to streamline diagnostic and repair processes to suit the specific requirements and environment of the application.

[1534] This invention describes in detail a method for realizing a maintenance management system for robots operating in a factory. This system begins when a user inputs equipment status information using a smart device and sends that information to a server. Each step is described in detail below.

[1535] 1. User Input

[1536] Users (factory managers and operators) use their smartphones or smart glasses to input status information about robots that have malfunctioned or are experiencing an abnormality into the application. This status information includes a photo of the robot and an error message. The photographed and input information is converted into JSON format and sent to the server.

[1537] 2. Initial diagnosis by the server

[1538] The server receives JSON data sent from the user device. The received data is analyzed using a generative AI model (e.g., TensorFlow or PyTorch). The photo and text data are converted into an appropriate format and sent to the AI ​​model to predict the cause of the failure. During this analysis, anomalies are identified from the photo and the problem is extracted from the error message.

[1539] 3. Simple repair manual provided

[1540] If the server determines that simple repairs are possible based on the analysis results, it provides a repair manual to the user's terminal. The repair manual contains detailed instructions on specific procedures, and the user can attempt to repair the device by following the instructions displayed on the smart device.

[1541] 4. Contact the equipment manufacturer and arrange repair schedules

[1542] If the simple repair fails or if it is determined that a more complex repair is required, the user can report it again through the application. Based on the report, the server will automatically contact the equipment manufacturer and arrange a repair date. Once the arrangement is complete, the server will notify the user of the confirmed repair date.

[1543] 5. Recording and using troubleshooting history

[1544] The server records the history of each troubleshooting in a database. This data includes information entered by the user, the results of the AI ​​diagnosis, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. The accumulated data is used to predict future failures and for preventive maintenance.

[1545] Hardware and software used

[1546] Hardware: Smartphones, smart glasses, servers

[1547] Software: Generative AI models (TensorFlow, PyTorch), applications (for smart devices)

[1548] Specific examples

[1549] For example, if a robot arm in operation in a factory stops working properly, the operator takes a photo of the robot arm with the smart glasses and enters an error message. The information is then sent to the server via the app. The server analyzes the data using a generative AI model and diagnoses the malfunction as "malfunction due to motor overload." Based on the results, the user is presented with specific repair procedures. If the repair is unsuccessful, the operator contacts the equipment manufacturer again via the app to arrange a repair date.

[1550] Prompt Sentence Examples

[1551] "Please take a photo of any abnormalities in the robotic arms operating in the factory and enter the error message."

[1552] This system makes it possible to respond quickly and efficiently to malfunctions or abnormalities in robots operating in factories.

[1553] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1554] Step 1:

[1555] The user uses a smart device to input equipment status information. Specifically, they take a photo of the abnormal part of the robot and enter the error message into the application. The input information is converted into JSON format.

[1556] Input: Photo data, text data (error message)

[1557] Output: JSON format data

[1558] Step 2:

[1559] The user's device (smartphone or smart glasses) sends equipment status information converted into JSON format to the server.

[1560] Input: JSON format data

[1561] Output: Data transfer to the server

[1562] Step 3:

[1563] The server analyzes the received JSON data. Specifically, it converts the photo data and text data into an appropriate format and sends it to a generative AI model (TensorFlow or PyTorch). The AI ​​model performs image analysis, identifies abnormalities from the photo, and performs text analysis to extract the cause of the failure from the error message.

[1564] Input: JSON format data

[1565] Output: Image analysis results, text analysis results

[1566] Step 4:

[1567] The server determines whether a simple repair is possible based on the analysis results. Based on the diagnosis results returned by the AI ​​model, it determines whether a simple repair is possible. This process refers to past repair history and common failure patterns in the database.

[1568] Input: Image analysis results, text analysis results

[1569] Output: Simple repair possible / impossible result

[1570] Step 5:

[1571] If it is determined that simple repair is possible, the server sends the corresponding repair manual to the user's terminal. The repair manual contains detailed instructions on the specific repair procedures, and the user attempts to repair the device by following the instructions displayed on the smart device.

[1572] Input: Simple repair possibility determination result

[1573] Output: Repair manual

[1574] Step 6:

[1575] If the simple repair fails or if it is determined that a more complex repair is required, the user can report it again through the application. Based on the report, the server will automatically contact the equipment manufacturer and arrange a repair date. Once the arrangement is complete, the server will notify the user of the confirmed repair date.

[1576] Input: User report data (simple repair failure report)

[1577] Output: Contact with equipment manufacturer, results of repair schedule adjustment, notification to user

[1578] Step 7:

[1579] The server records the history of each troubleshooting in a database. The recorded data includes information entered by the user, the results of the AI ​​diagnosis, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmed repair schedules. This data is used to predict future failures and for preventive maintenance.

[1580] Input: Trouble response data (user input information, AI diagnosis results, repair results, communication logs, etc.)

[1581] Output: Records in the database

[1582] Prompt Sentence Examples

[1583] "Please take a photo of any abnormalities in the robotic arms operating in the factory and enter the error message."

[1584] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1585] This invention is a building maintenance system for responding quickly and efficiently to equipment breakdowns. The system supports responding to equipment breakdowns by having the user input equipment status information, analyzing it via a server, and providing a simple repair manual or contacting the equipment manufacturer as necessary. Furthermore, the system improves the user experience by combining it with an emotion engine that recognizes the user's emotions and flexibly adjusts its response depending on the situation.

[1586] User Input

[1587] Users (building managers and general users) use their smartphones or PCs to input equipment status information into the "Building Trouble SOS" platform. This status information includes photos of the equipment and text of error messages. The device converts the input information into JSON format and sends it to the server.

[1588] Initial diagnosis by the server

[1589] The server analyzes the received JSON data and converts the photo and text data into the appropriate format. It then sends the data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The generated AI model predicts the cause of the failure based on the received information. The server determines whether a simple repair is possible based on the diagnosis results returned by the AI ​​model.

[1590] Simple repair manual provided

[1591] If it is determined that a simple repair is possible, the server retrieves the corresponding repair manual and sends it to the user's device. The user then attempts the simple repair by following the repair manual displayed on their smartphone or PC. The manual contains detailed instructions on specific repair procedures, allowing the user to follow the steps to resolve the problem.

[1592] Contacting equipment manufacturers and arranging repair schedules

[1593] If the problem is not resolved by simple repairs, the user reports the situation on the application. The device then sends this report back to the server. The server records the information that the simple repairs failed and immediately and automatically contacts the equipment manufacturer. The server then automatically coordinates a repair schedule with the equipment manufacturer and determines the earliest possible date and time for repairs. The server then notifies the user's device of the confirmed schedule. The user then checks the notified repair schedule and waits for on-site repairs.

[1594] User support with emotion engine

[1595] Furthermore, this system has a function that allows the emotion engine to recognize the user's emotions based on the user's input information and behavior. For example, if the user frequently inputs error messages or shows behavior that suggests frustration with repair procedures, the emotion engine will detect the user's stress or dissatisfaction.

[1596] Flexible response based on emotions

[1597] Based on the user's emotions recognized by the emotion engine, the server can flexibly change the messages and repair manuals it presents. For example, if the user is feeling anxious, it can provide a message containing words of encouragement and support. If the user is confused by a complicated procedure, it can present a simpler explanation or a video tutorial.

[1598] Expert Notification and Additional Support

[1599] If the emotion engine detects a high level of stress or dissatisfaction in the user, the server will automatically notify the equipment manufacturer's experts and request additional support. This feature allows users to receive professional assistance quickly and without having to struggle alone, improving the speed and accuracy of problem resolution.

[1600] Recording and using troubleshooting history

[1601] The server records each troubleshooting step in a database. This data includes information entered by the user, the AI ​​diagnosis results, the presented repair manual, the success or failure of simple repairs, communication with the equipment manufacturer, confirmed repair schedules, and emotional information recognized by the emotion engine. This data can be used to predict future failures and for preventive maintenance.

[1602] Specific examples

[1603] For example, if a building's air conditioning stops working, the building manager takes a photo of the air conditioning unit with their smartphone and uploads it to the app. If an error message is displayed, they enter the details. The device then sends this information to the server. The server analyzes the photo and text based on a generative AI model and determines that the filter is likely clogged. Based on the diagnosis, the server presents filter cleaning instructions and sends them to the user. The user then attempts to clean the filter by following the instructions displayed on their smartphone. If cleaning does not improve the situation, the user reports it again through the application, and the server automatically contacts the equipment manufacturer and schedules a repair. All of this information is recorded in a database and will be used for future analysis.

[1604] Furthermore, if the user indicates a high level of stress in the emotion engine, the server notifies experts and prepares a means to provide additional support. For example, if a situation arises in which the user is "very troubled by frequent error messages," the emotion engine will recognize the stress and the server will immediately contact experts to request a prompt response. In this way, the system quickly and efficiently resolves equipment problems in building maintenance while simultaneously reducing the user's psychological burden.

[1605] The processing flow will be explained below.

[1606] Step 1:

[1607] When a user notices that a building facility has broken down, they access the "Building Trouble SOS" platform using their smartphone or PC, take a photo of the facility, and enter a description of the current situation (e.g., "The air conditioning is not working" or "An error message is displayed") in text.

[1608] Step 2:

[1609] The device receives the input photo and text data and converts it into JSON format, including the type of equipment (air conditioning, lighting, network equipment, etc.).

[1610] Step 3:

[1611] The device sends the generated JSON data to the server, which includes the user's input information and photo data.

[1612] Step 4:

[1613] The server parses the received JSON data and converts the photo and text data into the appropriate format, which prepares it for passing to the AI ​​model.

[1614] Step 5:

[1615] The server sends the converted data to an AI model for photo analysis and a natural language processing model for text analysis, requesting a diagnosis. The AI ​​model predicts the cause of the failure.

[1616] Step 6:

[1617] The AI ​​model analyzes the photo and text data to identify the cause of the malfunction, such as a clogged filter or a blown fuse, and returns the diagnosis results to the server.

[1618] Step 7:

[1619] The server receives the diagnosis results from the AI ​​model and determines whether a simple repair is possible. If a simple repair is possible, it searches for the corresponding repair manual.

[1620] Step 8:

[1621] If the server determines that simple repair is possible, it retrieves the corresponding repair manual and sends it to the user's device. The repair manual contains specific repair procedures.

[1622] Step 9:

[1623] The device displays the received repair manual on its screen. The user attempts simple repairs by following the instructions in the manual. For example, the manual contains instructions for cleaning the filter and turning the power on and off.

[1624] Step 10:

[1625] After the user performs the simple repair, they report the results in the application. If the repair is successful, the application simply ends.

[1626] Step 11:

[1627] If the problem is not resolved by the simple repair, the user reports it as "not resolved" on the application. The terminal then sends this report to the server.

[1628] Step 12:

[1629] The server records the information about the failure of the simple repair and immediately contacts the equipment manufacturer automatically, sending a repair request to the equipment manufacturer.

[1630] Step 13:

[1631] The server receives the response from the equipment manufacturer and arranges the earliest possible date for repair. It also acquires the schedule information received from the equipment manufacturer.

[1632] Step 14:

[1633] The server transmits the confirmed repair schedule to the user's terminal and notifies the user.

[1634] Step 15:

[1635] The terminal notifies the user of the received repair schedule, and the user confirms the notified repair schedule and waits for the equipment manufacturer to visit.

[1636] Step 16:

[1637] The server records each troubleshooting step in a database, including information entered by the user, the AI ​​diagnosis results, the repair manual provided, the success or failure of simple repairs, communication with the equipment manufacturer, and confirmation of repair dates.

[1638] Step 17:

[1639] The emotion engine analyzes the user's input information and behavior to recognize the user's emotional state. For example, it can detect stress or frustration based on frequent error messages or delayed responses to operations.

[1640] Step 18:

[1641] The server flexibly changes the messages and repair manuals it presents based on the user's emotions recognized by the emotion engine. For example, if the user feels anxious, it displays encouraging messages, and if the user is confused by a complicated procedure, it provides simpler explanations.

[1642] Step 19:

[1643] If the emotion engine detects high levels of stress or dissatisfaction, the server automatically notifies the equipment manufacturer's experts and requests additional support, allowing users to receive fast, professional assistance.

[1644] Step 20:

[1645] The server records the trouble response history, including emotion recognition information, in a database and updates the data model used for future failure prediction and preventive maintenance.

[1646] By taking the above specific steps, the system can quickly and efficiently resolve equipment problems while reducing the psychological burden on users and providing a better user experience.

[1647] Example 2

[1648] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1649] Conventional building maintenance systems tend to respond slowly when equipment malfunctions occur, resulting in the user having to spend time and effort to deal with the issue. Furthermore, the system only provides a uniform response without considering the user's emotions or stress level, which can lead to a decline in user satisfaction. Furthermore, the accumulation and analysis of trouble response data was insufficient, meaning that the data could not be used to predict future malfunctions or for preventive maintenance.

[1650] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1651] In this invention, the server includes means for a user to input equipment status information, means for a terminal to send the equipment status information to the server, means for analyzing the equipment status information received by the server, means for identifying the cause of the failure using a generative AI model based on the analyzed information and determining whether a simple repair is possible, means for presenting a repair manual to the user if a simple repair is possible, means for contacting the equipment manufacturer if a simple repair is not possible, means for coordinating a repair schedule with the equipment manufacturer, means for notifying the user of the adjusted repair schedule, means for analyzing the user's emotions using an emotion engine and coordinating a response based on the analysis, and means for recording the trouble response history. This not only enables equipment failures to be responded to quickly and efficiently, but also reduces the psychological burden on the user and enables the use of data for failure prediction and preventive maintenance.

[1652] "User" refers to a person who uses the system to input equipment status information and receive repairs and support.

[1653] "Terminal" refers to a device such as a computer or smartphone that a user uses to input equipment status information.

[1654] "Server" refers to a computer system that receives equipment status information sent from a terminal and performs functions such as analysis, providing repair manuals, and contacting the equipment manufacturer.

[1655] "Generative AI model" refers to an artificial intelligence model used to analyze photographs and text data and identify the cause of equipment failure.

[1656] "Cause of failure" refers to the analysis results used to identify the cause of equipment abnormalities or malfunctions.

[1657] "Simple repair" refers to a simple repair method that the user can perform themselves.

[1658] "Repair manual" refers to a written or digital file that details steps and methods for performing simple repairs.

[1659] "Equipment manufacturer" refers to the company or professional that manufactured or sold the faulty equipment.

[1660] "Repair Schedule" means the date and time set by the equipment manufacturer for on-site repair work.

[1661] An "emotion engine" refers to software that analyzes emotions from user input and behavior, and then takes appropriate action based on the results.

[1662] "Trouble response history" refers to a record of a series of responses made within the system, such as information on equipment failures, analysis results, repair manuals, repair status, and emotion analysis results from the emotion engine.

[1663] "Failure prediction" refers to predicting future equipment failures by analyzing accumulated trouble response history.

[1664] "Preventive maintenance" refers to maintenance activities to prevent equipment failures before they occur, based on the results of failure prediction.

[1665] This invention is a building maintenance system that supports responding to equipment failures by having users input equipment status information, analyzing it via a server, and providing simple repair manuals or contacting the equipment manufacturer as necessary. Furthermore, the user experience is improved by combining it with an emotion engine that recognizes the user's emotions and flexibly adjusts its response depending on the situation.

[1666] Hardware and software used

[1667] Users access the system using a smartphone or PC. The system uses the following hardware and software:

[1668] User device: Smartphone or PC

[1669] Server: Cloud server or on-premise server

[1670] Generative AI models: ResNet (image analysis), BERT (natural language processing)

[1671] Emotion engine: IBM Watson

[1672] Data processing and calculation

[1673] 1. User Input

[1674] Users log in to the Building Trouble SOS platform and enter equipment status information, including photos of the equipment and the text of any error messages.

[1675] Specific examples

[1676] For example, if the air conditioning in a building stops working, a user can take a photo of the air conditioning unit with their smartphone, upload it to the app, and enter the details of any error messages that may be displayed.

[1677] 2. Data Transmission

[1678] The terminal converts the input information into JSON format and sends it to the server.

[1679] Specific examples

[1680] The device encodes the photo data into Base64 format, retrieves the error message from the text field, generates a JSON containing these, and sends it to the server via an HTTP POST request.

[1681] 3. Initial diagnosis

[1682] The server parses the received JSON data, converts the photo data and text data into an appropriate format, and then sends the data to an AI model (e.g., ResNet) for photo analysis and a natural language processing model (e.g., BERT) for text analysis, requesting a diagnosis.

[1683] Specific examples

[1684] The server analyzes the photo and text based on a generative AI model and diagnoses whether the filter is likely clogged.

[1685] Repair manual provided

[1686] The server determines whether a simple repair is possible based on the diagnosis results, and if so, obtains the appropriate repair manual and sends it to the user's device.

[1687] Specific examples

[1688] The server retrieves the filter cleaning instructions from the database and sends the PDF file to the user's device.

[1689] 4. Use of Emotion Engine

[1690] The server sends the entered error message and the user's behavior log to an emotion engine (e.g., IBM Watson), which recognizes the user's emotional state (stress, frustration, anxiety, etc.) and provides messages of support and encouragement.

[1691] Specific examples

[1692] If the emotion engine detects a high level of stress in the user, the server generates a cheering message and sends it to the user's terminal.

[1693] Expert Notice

[1694] If simple repairs fail or if the user shows high stress, the server automatically contacts the equipment manufacturer and schedules a repair.

[1695] Specific examples

[1696] If the user reports again that the simple repair has failed, the server will contact the equipment manufacturer, set the earliest possible repair date, and notify the user.

[1697] Record of troubleshooting history

[1698] The server records all troubleshooting steps in a database, including information entered by the user, the diagnosis results of the AI ​​model, the proposed repair manual, the success or failure of the repair, communication with the equipment manufacturer, the confirmed repair schedule, and the emotional information recognized by the emotion engine.

[1699] Prompt Sentence Examples

[1700] Here is an example prompt:

[1701] Prompt statement

[1702] "Please identify the cause of the building equipment failure based on the error message and photo below. Error message: 'Air conditioner not working', Photo: [URL of photo of air conditioner]"

[1703] The above is a specific description of an embodiment of the present invention. The system aims to enable quick and efficient response to equipment failures and reduce the psychological burden on users.

[1704] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1705] Step 1:

[1706] Users input the equipment status information by logging into the Building Trouble SOS platform using a smartphone or PC, taking a photo of the equipment, and entering error messages and a detailed description of the situation.

[1707] Input: Equipment photo, error message, detailed description of the situation

[1708] Output: Equipment status information in JSON format

[1709] Specific actions

[1710] The user takes a photo of the air conditioning unit.

[1711] The user enters the error message "The air conditioner is not working."

[1712] This information is converted to JSON format for transmission to the server.

[1713] Step 2:

[1714] The device sends the status information entered by the user to the server. The data is converted to JSON format and sent to the server as an HTTP POST request.

[1715] Input: Equipment status information in JSON format

[1716] Output: Data sent to the server

[1717] Specific actions

[1718] The device encodes the photo data into Base64 format.

[1719] The terminal retrieves the error message from the text field.

[1720] The device generates JSON data and sends it to the server via an HTTP POST request.

[1721] Step 3:

[1722] The server parses the received JSON data, converts the photo data and text data into an appropriate format, and then sends the data to an AI model (e.g., ResNet) for photo analysis and a natural language processing model (e.g., BERT) for text analysis, requesting a diagnosis.

[1723] Input: Equipment status information in JSON format

[1724] Output: Diagnostic results (e.g., possible clogged filter)

[1725] Specific actions

[1726] The server extracts the photo data from the JSON data and inputs it into an image analysis model (e.g., ResNet).

[1727] The server inputs the error message into a natural language processing model (e.g., BERT).

[1728] The server receives the diagnostic results returned by the AI ​​model and determines whether a simple repair is possible.

[1729] Step 4:

[1730] If the server determines that a simple repair is possible based on the diagnostic results of the generated AI model, it retrieves the corresponding repair manual from the database and sends it to the user's device.

[1731] Input: Diagnostic results

[1732] Output: Repair manual

[1733] Specific actions

[1734] The server retrieves the filter cleaning instructions from the database.

[1735] The server sends a PDF file of the repair manual to the user's device.

[1736] Step 5:

[1737] The user attempts a simple repair by following the repair manual displayed on the terminal.

[1738] Input: Repair manual

[1739] Output: Simple repair success or failure

[1740] Specific actions

[1741] The user opens the PDF manual.

[1742] The user should follow the "filter removal method" and "cleaning procedure" written in the manual.

[1743] Step 6:

[1744] If the problem cannot be resolved by simple repairs, the user reports the situation. The device then sends this information back to the server, which then automatically contacts the equipment manufacturer. The server then automatically schedules a repair and notifies the user.

[1745] Input: Simple repair success / failure, status report

[1746] Output: Contact with equipment manufacturer and repair schedule

[1747] Specific actions

[1748] The user presses a button to report a "repair failed."

[1749] The terminal sends a request to the server to report the status.

[1750] The server automatically contacts the equipment manufacturer via their API and schedules the earliest possible repair date.

[1751] The server notifies the user of the confirmed repair schedule.

[1752] Step 7:

[1753] The server uses an emotion engine to analyze the user's emotions and detect their stress or frustration. Based on the results of the detection, it provides messages of encouragement and support.

[1754] Input: Error message, Action log

[1755] Output: Message of support

[1756] Specific actions

[1757] The server sends the entered error message and the user's action log to the emotion engine.

[1758] The emotion engine analyzes the user's emotions and determines that they are "feeling stressed."

[1759] The server generates a cheering message and sends it to the user's terminal.

[1760] Step 8:

[1761] If the emotion engine detects high levels of stress or dissatisfaction, the server automatically contacts the equipment manufacturer to request additional support.

[1762] Input: Sentiment analysis results

[1763] Output: Liaison with equipment manufacturers and additional support

[1764] Specific actions

[1765] The server sends a "high stress state" notification to the equipment manufacturer.

[1766] The server forwards any additional support instructions or messages from the experts to the user.

[1767] Step 9:

[1768] The server records each troubleshooting step in a database for future analysis and preventative maintenance.

[1769] Input: Troubleshooting data

[1770] Output: Accumulated troubleshooting history

[1771] Specific actions

[1772] The server stores all data in a database, including information entered by the user, the diagnosis results from the AI ​​model, the presented repair manual, and whether the repair was successful or not.

[1773] The stored data is used for analysis to predict failures and plan maintenance.

[1774] (Application example 2)

[1775] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1776] When equipment failure occurs, it is necessary to quickly and efficiently identify the cause of the failure and carry out simple repairs. However, conventional systems take time to identify the cause of the failure, which places a heavy psychological burden on users. In addition, they are unable to respond flexibly and take into account the user's emotions, which can lead to the accumulation of stress and dissatisfaction. To solve these issues, a system is needed that monitors the status of equipment in real time and uses an emotion engine to respond according to the user's emotions.

[1777] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1778] In this invention, the server includes a means for analyzing equipment status information, a means for determining whether simple repairs are possible, a means for presenting a repair manual, and a means for identifying the user's emotions and changing the response accordingly. This allows the cause of equipment failure to be identified quickly and efficiently, and enables flexible response while reducing user stress and frustration. Furthermore, the accumulated trouble response history can be analyzed and used for future failure prediction and preventive maintenance.

[1779] "Equipment status information" refers to data entered by users, such as the equipment's operating status, abnormal conditions, error messages, etc.

[1780] "Server" refers to a central management system that receives, analyzes, processes, and issues various decisions and notifications regarding the status of equipment.

[1781] "Means of analysis" refers to algorithms or programs for identifying the cause of a failure based on equipment status information.

[1782] "Means for determining whether simple repairs are possible" refers to algorithms or mechanisms that determine whether the target equipment can be easily repaired based on the analysis results.

[1783] "Means for presenting a repair manual" refers to a method or system that provides a user with specific repair procedures when simple repairs are possible.

[1784] "Equipment manufacturer" refers to the company or professional engineer who manufactured the equipment in question.

[1785] "Means for arranging repair schedules" refers to a system that communicates with equipment manufacturers and automatically determines the date and time for repairs.

[1786] "User" refers to anyone who operates or manages the facility, including building managers and general users.

[1787] "Means for recording trouble response history" refers to a system that stores the process of each trouble response in a database for later analysis and use.

[1788] "Means for identifying emotions and changing the response according to those emotions" refers to a function that recognizes the user's emotions such as stress or dissatisfaction, and appropriately changes the information provided and the response based on that emotional state.

[1789] "Means for acquiring and analyzing images of equipment" refers to the technology and programs for collecting photographic data of equipment taken with a camera, etc., and analyzing those images.

[1790] "Means for determining abnormality" refers to a function that determines whether the equipment is operating normally or whether an abnormality has occurred based on the acquired data.

[1791] "Failure prediction and preventive maintenance" refers to measures that analyze accumulated data to predict and prevent possible future failures.

[1792] The present invention is a system that analyzes equipment status information, presents repair manuals as needed, and automates communication and coordination with equipment manufacturers. It also has the ability to recognize a user's emotions and respond accordingly. A specific embodiment of the system is described below.

[1793] Hardware and Software Configuration

[1794] The system operates around three main players: the server, the terminal, and the user. The server performs analysis processing and database management, the terminal is responsible for inputting and receiving information, and the user provides equipment status information.

[1795] server:

[1796] Hardware: A server computer equipped with a high-performance CPU and GPU. A network interface card (NIC) that enables high-speed data transfer is used.

[1797] software:

[1798] Analysis module: Uses PhotoAIModel and TextAIModel to analyze equipment photo data and error messages.

[1799] Emotion Recognition Module: Uses the EmotionEngine to recognize emotions based on user input and actions.

[1800] Database: Use a reliable database system such as MySQL or PostgreSQL to store troubleshooting history and equipment status.

[1801] Device:

[1802] Hardware: Smartphone, tablet, smart glasses, head-mounted display (HMD), or PC.

[1803] software:

[1804] Input interface: Provides a dedicated application for users to input equipment status information.

[1805] Display Interface: GUI for displaying repair manuals and messages.

[1806] User:

[1807] Role: Inputs equipment status information (photos, error messages, etc.), refers to repair manuals, and performs simple repairs. Also, uses emotion recognition to reduce user stress and frustration.

[1808] Data processing and calculation

[1809] The terminal sends the equipment status information (photos and error messages) entered by the user to the server, which analyzes the data using the following procedure and takes the necessary action.

[1810] 1. Data reception and analysis:

[1811] The server receives the JSON format data sent from the terminal.

[1812] The received data is input into PhotoAIModel and TextAIModel to analyze the cause of equipment failure.

[1813] 2. Repair decision and manual presentation:

[1814] Based on the analysis results, it is determined whether simple repairs are possible.

[1815] If simple repair is possible, the corresponding repair manual is obtained and sent to the terminal.

[1816] 3. Liaison and coordination with equipment manufacturers:

[1817] If simple repair is not possible, the equipment manufacturer will be contacted automatically.

[1818] We will contact the equipment manufacturer, schedule a repair, and notify the user.

[1819] 4. Emotion recognition and response modification:

[1820] EmotionEngine recognizes the user's emotions based on the user's input information and actions.

[1821] To flexibly change messages to be displayed and contents of repair manuals according to emotional information of a user.

[1822] If it detects high levels of stress or dissatisfaction, it will notify experts and provide additional support.

[1823] Specific examples

[1824] For example, if a robot in a factory stops working, a worker can use smart glasses to take a picture of the robot's condition and send the photo and an error message to the system. The system then analyzes the failure and diagnoses that the sensor needs to be calibrated. The worker is then presented with a calibration procedure, which they follow to attempt calibration. If the worker becomes stressed, the system's emotion engine will recognize this early on and send a message of encouragement or request expert assistance.

[1825] Prompt Sentence Examples

[1826] Please diagnose the cause of the failure based on the following equipment photo and error message. Photo: [Image path], Message: [Error message]

[1827] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1828] Step 1:

[1829] The user uses smart glasses or a head-mounted display (HMD) to capture a photo of the equipment and input an error message, which is then stored on the device.

[1830] Input: Facility photo, error message

[1831] Output: JSON format data saved on the device

[1832] Step 2:

[1833] The device sends the saved state information in JSON format to the server.

[1834] Input: JSON data of facility photos and error messages

[1835] Output: JSON data sent to the server

[1836] Step 3:

[1837] The server analyzes the received JSON data. The photo data is sent to the PhotoAIModel, and the error message is sent to the TextAIModel. Each model analyzes the data and identifies the cause of the error.

[1838] Input: JSON data sent to the server

[1839] Output: Analysis results from PhotoAIModel and TextAIModel

[1840] Step 4:

[1841] The server determines whether simple repairs are possible based on the analysis results obtained from PhotoAIModel and TextAIModel.

[1842] Input: Analysis results of PhotoAIModel and TextAIModel

[1843] Output: Simple repair possible

[1844] Step 5:

[1845] If it is determined that simple repair is possible, the server obtains the corresponding repair manual and sends it to the terminal.

[1846] Input: Judgment result that simple repair is possible

[1847] Output: Repair manual

[1848] Step 6:

[1849] The user attempts to carry out simple repairs according to the repair manual displayed on the terminal, and when the repair is complete, inputs the results into the terminal.

[1850] Input: Repair manual

[1851] Output: Simple repair results

[1852] Step 7:

[1853] If simple repairs are impossible or have failed, the user sends the information back to the server from their terminal, which then automatically contacts the equipment manufacturer and arranges a repair date.

[1854] Input: Simple repair failure information

[1855] Output: Repair schedule adjustment results with equipment manufacturer

[1856] Step 8:

[1857] The server notifies the user of the adjusted repair schedule, and the user confirms the schedule and waits for the repair to be performed.

[1858] Input: Repair schedule adjustment results with equipment manufacturer

[1859] Output: Notification of repair schedule to user

[1860] Step 9:

[1861] The server uses the Emotion Engine to identify emotions based on user input and behavior, and flexibly changes the corresponding messages and repair manuals based on this emotional information.

[1862] Input: User input and behavioral information

[1863] Output: Emotion-based messages and repair manuals

[1864] Step 10:

[1865] If the emotion engine detects high levels of stress or dissatisfaction in the user, the server immediately notifies an expert and provides additional support. Subsequent responses are also recorded in the database.

[1866] Input: Emotional information about high levels of stress and frustration

[1867] Output: Notification to experts, database record of troubleshooting history

[1868] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1869] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1870] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1871] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1872] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1873] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1874] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1875] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition a...

Claims

1. a means for inputting equipment status information; means for transmitting equipment status information to a server; A means for analyzing the equipment status information received by the server; A means for determining whether simple repair is possible based on the analyzed information; a means for presenting a repair manual if simple repair is possible; A means of contacting the equipment manufacturer if simple repairs are not possible; A means of coordinating repair schedules with equipment manufacturers; means for notifying the user of the repair schedule; A means of recording the history of troubleshooting; A system including:

2. a means for capturing and analyzing photographs of the facility; A means for determining an abnormality based on the photographic data; The system of claim 1 , comprising:

3. A means of analyzing the accumulated trouble response history and using it for failure prediction and preventive maintenance; The system of claim 1 , comprising:

Images