Information processing method, program, and information processing apparatus

Abstract

To provide an information processing method and the like capable of sequentially generating reports based on images and log data acquired at a plurality of different timings by a language model.SOLUTION: An information processing method according to one aspect causes a computer to execute a process of acquiring an image of a target object captured in a plant and time-series log data in the plant, and sequentially generating a report on the plant for each timing by giving the image and the log data acquired at a plurality of different timings to a language model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing device. [Background technology]

[0002] In recent years, the development of automatic report generation technologies has been actively promoted in association with the automation of processes. For example, Patent Document 1 discloses a method for generating sensor data using a sensor system at an assembly site, identifying the context of the current stage of the assembly process, and generating a quality report of an assembly product based on the sensor data and the context of the current stage. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-064608 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the invention of Patent Document 1 has a problem in that it is not possible to use a language model to generate reports sequentially based on images and log data acquired at multiple different times.

[0005] One aspect of the present invention is to provide an information processing method and the like that can sequentially generate reports based on images and log data acquired at multiple different times using a language model. [Means for solving the problem]

[0006] In one aspect of the information processing method, a computer acquires images of objects photographed in a plant and time-series log data for the plant, and provides the images and log data acquired at multiple different times to a language model, thereby sequentially generating reports about the plant for each time period. [Effects of the Invention]

[0007] In one aspect, the language model allows for sequential report generation based on image and log data captured at multiple different times. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of a report generation system. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of a server. [Figure 3] 10A and 10B are explanatory diagrams showing examples of record layouts of an image DB and a report DB. [Figure 4] FIG. 2 is an explanatory diagram illustrating an example of the data configuration of a log file. [Figure 5] FIG. 2 is a block diagram showing an example configuration of an apparatus and a terminal. [Figure 6] FIG. 10 is an explanatory diagram illustrating a process of generating a report about a plant using a language model. [Figure 7] 10 is a flowchart showing a processing procedure for generating a report about a plant. [Figure 8] 10 is a flowchart showing a processing procedure for generating a report including statistical information. [Figure 9] 10 is a flowchart showing a processing procedure for generating a report including causal relationships. [Figure 10] 10 is a flowchart showing a processing procedure for generating a report including information on the progress of processing of an object. [Figure 11] FIG. 10 is a block diagram showing an example of the configuration of a server in the fourth embodiment. [Figure 12] FIG. 2 is an explanatory diagram illustrating an example of a record layout of a knowledge DB. [Figure 13] FIG. 10 is an explanatory diagram illustrating a process for generating a report including questions and answers. [Figure 14] FIG. 10 is an explanatory diagram showing examples of prompts and responses. [Figure 15] 10 is a flowchart showing a processing procedure for generating a report including questions and answers. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below with reference to the drawings showing embodiments thereof.

[0010] (Embodiment 1) The first embodiment relates to a form in which a report about a plant is generated using a language model. Fig. 1 is an explanatory diagram showing an overview of a report generation system. The system of this embodiment includes an information processing device 1, an imaging device 2, a device 3, and an information processing terminal 4, and each device transmits and receives information via a network N such as the Internet.

[0011] The information processing device 1 is an information processing device that processes, stores, and transmits / receives various types of information. The information processing device 1 is, for example, a server device or a personal computer. In the present embodiment, the information processing device 1 will be hereinafter referred to as a server 1 for the sake of simplicity.

[0012] The imaging device 2 is installed to capture images of objects in a plant, and is, for example, an imaging device such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera.

[0013] The imaging device 2 includes a wireless communication unit. The wireless communication unit is a wireless communication module for performing communication-related processing, and transmits the captured image of the object to the server 1 via the network N. The imaging device 2 may be connected to the server 1 via a wired connection. The imaging device 2 may be replaced by a personal computer capable of capturing images, a smartphone, a mobile surveillance robot capable of capturing images of the object, or the like.

[0014] The objects include food products or industrial products, or substances or materials used to manufacture food products or industrial products. Food products include, for example, confectionery (chocolate, ice cream, biscuits, rice crackers, candy, etc.), processed meat products (processed meat, minced meat, hamburger steak, sausage, etc.), kamaboko products (fish paste products made by shaping and heating minced fish paste), tamagoyaki (rolled eggs), rice balls, etc. Industrial products are molded products made of metal, resin, etc. (cast products, forged products, injection molded products, extrusion molded products, etc.).

[0015] The device 3 is a device within a plant. The plant may be, for example, an energy plant, an industrial plant (e.g., a food plant, a petrochemical plant, or a pharmaceutical plant), or an environmental plant. An energy plant is a device that produces energy from thermal or nuclear power. An industrial plant is a device that produces chemical products, food, medicines, metals, or the like used in daily life. An environmental plant is a device that treats sewage or waste and further recycles resources.

[0016] The equipment in the plant is a plurality of pieces of equipment such as manufacturing equipment, processing equipment, electrical equipment, or piping that are combined to produce a specific product or material. As an example, the equipment 3 in this embodiment is equipment that manufactures food products or industrial products. The equipment 3 is manufacturing equipment that includes a kettle, a shaft, or various sensors (e.g., a weight measurement sensor, a temperature sensor, a pressure data sensor, or a photoelectric sensor). The kettle is a container used to stir, extract, heat, bake, or dry a substance or material.

[0017] The information processing terminal 4 is a terminal device that receives and displays reports generated by a language model. The information processing terminal 4 is an information processing device such as a smartphone, a mobile phone, a wearable device such as a smart watch, a tablet, or a personal computer terminal. For simplicity, the information processing terminal 4 will be referred to as the terminal 4 below.

[0018] Although this report generation system is configured with the terminal 4 and the device 3 as separate entities, they may be integrated into one unit, or the device 3 may be directly connected to the terminal 4. The terminal 4 may also be a server that performs the functions of the server 1. Furthermore, the server 1 and the device 3 may be the same device.

[0019] Plant reports are a valuable source of information about plant operations or production processes to improve efficiency. Regular reports also provide information about plant safety and help prevent accidents or damage. Furthermore, reports can help improve plant performance by assessing indicators such as productivity, quality, cost efficiency, or yield.

[0020] However, creating reports manually takes time and effort, and there is a risk of human error or bias. Therefore, an automated report generation system is needed. Automation can streamline the process from data collection to analysis and report creation, providing fast and accurate information.

[0021] The server 1 according to this embodiment acquires images of objects photographed in a plant and time-series log data for the plant. The server 1 provides the images of objects and log data acquired at multiple different times to a language model, thereby generating reports on the plant sequentially for each time. The language model will be described later. The server 1 transmits the generated reports to a terminal 4.

[0022] 2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a storage unit 12, a communication unit 13, a reading unit 14, and a large-capacity storage unit 15. Each component is connected by a bus B.

[0023] The control unit 11 includes an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), an NPU (Neural Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), or a quantum processor. The control unit 11 reads and executes a control program 1P (program product) stored in the storage unit 12, thereby performing various information processing or control processing related to the server 1. The control program 1P described in this embodiment may be provided on a recording medium or may be distributed from an external computer.

[0024] It should be noted that the control program 1P can be deployed to run on a single computer, or on multiple computers located at one site, or distributed across multiple sites and interconnected by a communications network.

[0025] 2, the control unit 11 is described as a single processor, but it may be a multi-processor. The control unit 11 may execute various information processes or control processes by the same processor within the server 1, or may execute various processes by different processors within the server 1.

[0026] The storage unit 12 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores the control program 1P or data required for the control unit 11 to execute processing. The storage unit 12 also temporarily stores data required for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and transmits and receives information to and from the imaging device 2, device 3, terminal 4, etc. via the network N.

[0027] The reading unit 14 reads a portable storage medium 1a including a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 14 and store it in the mass storage unit 15. Alternatively, the control unit 11 may download the control program 1P from another computer via a network N or the like and store it in the mass storage unit 15. Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

[0028] The mass storage unit 15 includes a recording medium such as a hard disk drive (HDD) or a solid state drive (SSD), etc. The mass storage unit 15 includes a language model 151, an image database (DB) 152, a report DB 153, and a log file 154.

[0029] The language model 151 is a language generation model constructed by pre-training using large-scale text data (dataset). As the language model 151, for example, large language models (LLMs) such as Transformer, ALBERT (A Lite BERT), GPT (Generative Pre-trained Transformer)-2, GPT-3, GPT-4, LLaVA (Large Language and Vision Assistant), MiniGPT-4, or BERT (Bidirectional Encoder Representations from Transformers) can be used.

[0030] The image DB 152 stores images of objects photographed in the plant. The report DB 153 stores reports about the plant generated by the language model 151. The log file 154 is a file for recording log data.

[0031] In this embodiment, the storage unit 12 and the large-capacity storage unit 15 may be configured as an integrated storage device. Furthermore, the large-capacity storage unit 15 may be configured by a plurality of storage devices. Furthermore, the large-capacity storage unit 15 may be an external storage device connected to the server 1.

[0032] The server 1 may execute various information processing and control processing on a single computer, or may execute the processing in a distributed manner on multiple computers. The server 1 may also be realized by multiple virtual machines provided in a single server, or may be realized by using a cloud server.

[0033] FIG. 3 is an explanatory diagram showing an example of the record layout of the image DB 152 and the report DB 153. As shown in FIG. The image DB 152 includes an image ID column, an object ID column, an image data column, and an acquisition date and time column. The image ID column stores the ID of an image of a uniquely specified object in order to identify the image of each object. The object ID column stores the ID of an object in order to identify each object. The image data column stores image data of the object. The acquisition date and time column stores information on the date and time when the image of the object was acquired.

[0034] The report DB 153 includes a report ID column, an object ID column, an image ID column, a log file column, a report data column, and a generation date and time column. The report ID column stores a unique report ID to identify each report. The object ID column stores an object ID for identifying an object. The image ID column stores an image ID for identifying an image of an object.

[0035] The log file column stores the file names of the log files 154 containing log data recorded in chronological order. The log files 154 are, for example, files in CSV (Comma-Separated Values), SSV (Space-Separated Values), TSV (Tab-Separated Values), XML (Extensible Markup Language), JSON (JavaScript Object Notation), text (TXT), etc. The log file column may also store the chronological log data contained in the log files 154.

[0036] The report data column stores data of the report generated by the language model 151. The generated date and time column stores information about the date and time when the report was generated.

[0037] FIG. 4 is an explanatory diagram showing an example of the data structure of the log file 154. The log file 154 is a file that records time-series log data generated while the device 3 is operating. The log file 154 may record time-series log data for each operation, or may record time-series log data for a predetermined period. The predetermined period may include, for example, one hour, one day, or one week. For example, if the operation period is from 6:00 to 12:00, all time-series log data for that operation period may be stored in a single log file 154. Alternatively, all time-series log data for a specified period, such as one hour, one day, or one week, may be stored in a single log file 154. Note that the log data may be recorded as multiple log files 154.

[0038] As shown, the log file 154 includes log data such as number, operation mode (OpMode), image brightness (CamBright), image blur (CamBlur), imaging position (CamPos), classification class (UekamaClass), AI stage (AiStage), shaft low position (ShftLowLr), shaft rotation (ShftRot), shaft rotation low speed (ShftRotLow), shaft up / down load (ShftUDload), and recording date and time.

[0039] The number is an identifier indicating the execution order of the operation in the time series data within the period for which the log data is recorded. The operation mode is information indicating the operation mode of the device 3, and includes, for example, manual (MANUAL), artificial intelligence (AI-DRIVE), or AI finishing (AI-TUKIM).

[0040] Image brightness is a parameter indicating the level of brightness related to the imaging device 2 or image processing, and includes normal, bright, dark, too bright, or too dark. Image blur is a parameter indicating the degree of blur in the imaging device 2 or image processing, and includes clear, blur, or too blur. Imaging position is information indicating the position of the captured image, and includes in or out. In means an inside or close position, and out means an outside or far position.

[0041] The classification class is a class for identifying the classification of an object, obtained by a learning model installed on the device 3. The classification class includes, for example, a white class (WHT), a yellow class (YEL), a yellow-black class (YEL+BLK), a red class (RED), a red-black class (RED+BLK), and a black class (BLK).

[0042] The white class defines the following: "Background color: invisible, color inside the pot: white and yellow, overall brightness: bright." The yellow class defines the following: "Background color: visible, color inside the pot: white and yellow, overall brightness: bright." The yellow-black class defines the following: "Background color: visible, color inside the pot: yellow and black, overall brightness: bright."

[0043] The red class defines the content as "Background color: visible, color inside the pot: red or orange, overall brightness: normal." The red-black class defines the content as "Background color: visible, color inside the pot: red and black, overall brightness: dark." The black class defines the content as "Background color: visible or invisible, color inside the pot: black, overall brightness: dark."

[0044] It should be noted that the classification classes are not limited to those described above, and other classification classes may be set according to, for example, attribute information (color, state, etc.) of the object.

[0045] The AI ​​stage is stage information that indicates the inspection stage of the industrial process, obtained by a learning model installed on the device 3. The AI ​​stage includes, for example, IDLE, CHECK1, SHAFT-DOWN, SHAFT-UP, DOWN, KAKUHAN, ABEND, TUKIMI, or abnormal termination.

[0046] Idle indicates that the equipment or device is not active and is on standby. Check 1 indicates the first stage of inspection or checking. Shaft down indicates that the shaft is moving downwards. Shaft up indicates that the shaft is moving upwards. Down indicates that the equipment or line is stopped. Kakuhan indicates the stage of processing or treatment of the product. Abend indicates that some kind of abnormality or error has occurred. Tsukimi indicates that the product or part is waiting to be finished or inspected.

[0047] The classification class or stage information may be obtained from a learning model installed on the server 1 or an external information processing device.

[0048] The shaft low position is a parameter related to the low position of the shaft, and includes no limit (NO-LIMIT), low position (left) (LOWR-L), low position (right) (LOWR-R), etc. The shaft rotation indicates information about the rotation of the shaft (such as angle or rotation speed), and includes no rotation (NO-ROT), reverse rotation (RVS), forward rotation (FWD), etc.

[0049] The low shaft rotation speed indicates information (such as angle or rotation speed) about the rotation of the shaft at low speed, and includes standard (STD), high (HIGH), low (LOW), etc. The upper and lower shaft load indicates information (such as weight or pressure) about the upper and lower loads (loads) applied to the shaft, and includes standard (STD), high (HIGH), low (LOW), etc. The recording date and time is the date and time when each operation was performed.

[0050] The sensor data is not limited to the above-mentioned sensor data (image brightness, image blur, position of the imaging device 2, low position of the shaft, shaft rotation, low shaft rotation speed, or shaft vertical load, etc.). The sensor data may be time-series data obtained by various sensors mounted on the imaging device 2 and the device 3. For example, the sensor data may include shaft speed, temperature data, pressure data, etc.

[0051] The above-described log file 154 is merely an example, and is not limited to this. The log file 154 may include items such as the execution time of the operation, temperature, humidity, pressure, or vibration.

[0052] The timing for acquiring the log data includes "at the start of operation," "at the end of operation," "every predetermined unit time," or "when a request is received," etc.

[0053] "At the start of operation" is the timing at which log data for the most recent operation period is acquired to check the most recent operation. The most recent operation period includes, for example, the period from the start of the previous operation to the start of the most recent operation, or a period corresponding to a predetermined number of times in the past (for example, twice).

[0054] For example, if the previous operation was from midnight to 6:00, the log data for the period from midnight to 6:00 will be the target. For example, if the operation starts at 6:00, the log data for the previous operation period (for example, from midnight to 6:00) may be acquired as one log file 154 based on the recording date and time of each operation.

[0055] "End of operation" refers to the timing for acquiring log data for a predetermined period when an operation is ended. The predetermined period may be the period from the start to the end of the operation, or the period from a specified date and time (e.g., midnight on the current day) to the end. For example, if an operation ends between 12:00 and 18:00, log data from 12:00 to 18:00 may be acquired as a single log file 154, based on the recording date and time of each operation, or log data from midnight to 18:00 on the current day may be acquired as a single log file 154.

[0056] "Every predetermined unit time" refers to the timing at which log data is acquired periodically at every predetermined time. The predetermined unit time includes, for example, every 10 minutes, every hour, every day, every week, or every month. For example, when log data is acquired every predetermined unit time (for example, every 6 hours) within a day, the log data may be acquired as one log file 154 containing log data corresponding to each of the time periods "0:00 to 6:00," "6:00 to 12:00," "12:00 to 18:00," and "18:00 to 24:00" based on the recording date and time of each operation.

[0057] "When a request is received" refers to the timing when a log data acquisition request is received from a user and log data is acquired in response to the acquisition request. The acquisition request includes the period for which the log data is to be acquired (e.g., June 25th) or acquisition conditions. The acquisition conditions may be, for example, "including sensor data obtained by the imaging device 2." The acquisition of log data "when a request is received" is flexible because it is performed based on the period or conditions specified by the user. This allows the user to efficiently acquire only the data they need.

[0058] 4, the values ​​of various log data are recorded in the log file 154 in character string format, but this is not limiting. For example, the values ​​of the log data themselves may be directly recorded in the log file 154 in numeric format.

[0059] The storage formats of the DBs and files described above are merely examples, and other storage formats may be used as long as the relationships between the data are maintained.

[0060] 5 is a block diagram showing an example configuration of the device 3 and the terminal 4. The device 3 includes a control unit 31, a storage unit 32, and a communication unit 33. Note that the control unit 31, the storage unit 32, and the communication unit 33 are similar to the control unit 11, the storage unit 12, and the communication unit 13 of the server 1, and therefore a description thereof will be omitted.

[0061] The terminal 4 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, and a display unit 45. The control unit 41 includes an arithmetic processing device such as a CPU, an MPU, a GPU, or an NPU (Neural Network Processing Unit), and performs various information processing and control processing related to the terminal 4 by reading and executing a control program 4P (program product) stored in the storage unit 42. The control program 4P described in this embodiment may be provided on a recording medium or may be distributed from an external computer.

[0062] 5, the control unit 41 is described as a single processor, but it may be a multi-processor. The control unit 41 may execute various information processing or control processes by the same processor within the terminal 4, or may execute various information processing or control processes by different processors within the terminal 4.

[0063] The storage unit 42 includes a memory element such as a RAM or a ROM, and stores the control program 4P or data required for the control unit 41 to execute processing. The storage unit 42 also temporarily stores data required for the control unit 41 to execute arithmetic processing.

[0064] The communication unit 43 is a communication module for performing communication-related processing, and transmits and receives information to and from the server 1, etc. via the network N. The input unit 44 may be a keyboard, a mouse, or a touch panel integrated with the display unit 45. The display unit 45 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays various information according to instructions from the control unit 41.

[0065] 6 is an explanatory diagram illustrating a process for generating a report about a plant using a language model 151. The server 1 acquires images of an object taken in the plant at multiple different times within a predetermined period (for example, one day) from the imaging device 2, and acquires a log file 154 containing log data recorded in chronological order in the plant from the device 3. That is, the images acquired at each time point are associated with the log data.

[0066] For example, when acquiring log data for each predetermined unit time (e.g., 6 hours) within a day, the server 1 acquires the data as one log file 154 including log data corresponding to each time period of "0:00 to 6:00," "6:00 to 12:00," "12:00 to 18:00," and "18:00 to 24:00" based on the recording date and time of each operation. Note that the log data may be acquired as multiple log files 154.

[0067] The server 1 generates prompts to be given to the language model 151 based on images of the object acquired at a plurality of different times and time-series log data in the plant.

[0068] The language model 151 is a language model that uses images of an object and time-series log data, and is used as a program module that is part of artificial intelligence software. The language model 151 in this embodiment is a constructed language model (language generation model) that receives as input images of an object and time-series log data acquired at multiple different times, and a prompt including instructions (commands) for generating a report about a plant, and outputs a report about the plant.

[0069] Instead of storing the language model 151 in the mass storage unit 15, the server 1 may access an external language processing server or language processing platform and read it out.

[0070] A prompt is an instruction or an input sentence created in a format that can be understood by the language model 151 and given as an input to the language model 151. The language model 151 interprets the input object image, time-series log data, and prompt, and outputs an appropriate response (e.g., a report about a plant).

[0071] Specifically, when an image of an object, time-series log data, and a prompt are input, the language model 151 extracts features from the input image of the object using image processing technology (e.g., a deep learning model or an image recognition algorithm). The language model 151 identifies the object contained in the image by analyzing the extracted features. Alternatively, the language model 151 extracts text information from the image using, for example, diffusion models, and generates natural language text describing the objects in the image, a description of the scene, or the content of the image.

[0072] The language model 151 extracts features of the input time-series log data and extracts report information from the log data. The report information includes the frequency of changes in operation mode, the distribution of image brightness, the frequency of image blurring, the distribution of imaging positions, the distribution of classification classes, the distribution of each AI stage, the frequency of low shaft positions, statistical information on the shaft rotation speed (average, maximum, minimum, variance, etc.), or the fluctuation distribution of the shaft vertical load.

[0073] The content of the report information varies depending on the time-series log data. For example, if the log data includes the execution time of an operation, the report information may include the average time, maximum time, minimum time, etc. of the operation.

[0074] The language model 151 obtains additional information from the context of the prompt based on the text obtained by converting the identified object or image into text and the extracted report information, and generates a more accurate response. As an example, the language model 151 divides the prompt into tokens to convert it into a format that the language model 151 can process. The language model 151 performs context understanding processing by calculating the relationship between each token in the prompt and other tokens.

[0075] The language model 151 performs a response generation process for objects identified from images, report information extracted from time-series log data, and prompts based on linguistic knowledge obtained through advance learning and fine-tuning. For example, the language model 151 selects optimal tokens using a generation method such as greedy decoding, beam search, or sampling. The language model 151 performs a decoding process on the selected tokens to return them to text format and generate a report about the plant.

[0076] In this embodiment, the prompt includes instructions to extract report information from the log file 154, considerations, remedial actions, and instructions for generating the report.

[0077] As an example, the generated prompt is: Extract the following report information from the images and log files of the object taken every specified unit time (e.g., 6 hours) within a day: Frequency of use of operation modes, distribution of image brightness, type and frequency of blur applied to images, usage pattern of imaging positions, distribution of classification classes, distribution of AI stages, statistical information of shaft low position and rotation speed, and distribution of fluctuations in shaft up and down load. Also, please provide your thoughts and suggestions for improvement regarding operations based on this data. Additionally, generate a report on the plant every six hours that includes report information, observations, and remedial actions."

[0078] The server 1 inputs images of the object acquired every predetermined unit time (e.g., 6 hours) within a day, a log file 154 containing time-series log data, and the generated plant into the language model 151, and sequentially generates reports about the plant for each timing (e.g., 0:00 to 6:00, 6:00 to 12:00, 12:00 to 18:00, and 18:00 to 24:00).

[0079] As an example, the generated report is as shown: Report (midnight to 6am) <Report Information> Frequency of use of operation mode: MANUAL: 2 times AI-DRIVE: 2 times AI-TUKIM: 1 time Image luminance distribution: NORMAL: 2 times BRIGHT: 2 times DARK: 1 time TOO BRIGHT: 1 time The type and frequency of blur applied to the image: CLEARLY: 2 times BLUR: 2 times TOO-BLUR: 1 time Imaging location usage patterns: IN: 2 times OUT: 3 times Classification class distribution: WHT: 1 time YEL: 2 times YEL+BLK: 1 time RED: 2 times RED+BLK: 1 time AI Stage Distribution: IDLE: 1 time CHECK1: 1 time ABEND: 1 time Shaft low position and rotation speed statistics: LOWER-L: 4 times SHAFT-DOWN: 1 time SHAFT-UP: 1 time Shaft vertical load distribution: STD: 5 times HIGH: 2 times <Consideration> Image brightness is divided into four categories: NORMAL, BRIGHT, DARK, and TOO BRIGHT. For TOO BRIGHT images, excessive brightness can be a problem. In AI-DRIVE mode, the vertical load on the shaft is high, and the load on the device is large. <Improvement measures> TOO BRIGHTAdjust the brightness of the image to avoid excessive brightness. To reduce the load on the shaft in AI-DRIVE mode, optimize the rotation speed. (6:00 AM - 12:00 PM) <Report Information> xxx <Consideration> xxx <Improvement measures> xxx (12:00-18:00) <Report Information> xxx <Consideration> xxx <Improvement measures> xxx (6pm - midnight) <Report Information> xxx <Consideration> xxx <Improvement measures> It could also be "XXX".

[0080] The server 1 transmits the report generated by the language model 151 to the terminal 4. The terminal 4 receives the report transmitted from the server 1 and displays it on the screen.

[0081] In the above example, reports are generated sequentially at predetermined times (for example, from midnight to 6:00, 6:00 to 12:00, 12:00 to 18:00, and 18:00 to 24:00), but this is not limitative. For example, reports may be generated at times including "the start of operations" and "the end of operations."

[0082] Specifically, the server 1 acquires images and log data of the object from the previous time before the start of the most recent operation or a predetermined number of times in the past (for example, twice). When the operation ends, the server 1 acquires images and log data of the object from the start to the end of the most recent operation. The server 1 provides the images and log data of the object acquired at each of the timings of "start of operation" and "end of operation" to the language model 151, thereby sequentially generating reports for each timing.

[0083] When the server 1 receives a request to generate a report through the terminal 4, the server 1 may generate the report in response to the received request. The request includes, for example, an object ID, a target period for the report (e.g., June 27th), and a target timing (e.g., every six hours).

[0084] 7 is a flowchart showing a processing procedure for generating a report about a plant. The control unit 11 of the server 1 acquires images of an object photographed in the plant from the imaging device 2 via the communication unit 13 at multiple different times (such as "at the start of operation," "at the end of operation," "every predetermined unit time," or "when a request is received") within a predetermined period (for example, one day) (step S101).

[0085] The control unit 11 stores the images of the object acquired at different times in the image DB 152 of the mass storage unit 15 (step S102). Specifically, the control unit 11 assigns an image ID to each acquired image of the object. For each image of the object, the control unit 11 associates the object ID, image data, and acquisition date and time with the assigned image ID and stores the object ID, image data, and acquisition date and time as one record in the image DB 152.

[0086] The control unit 11 acquires, via the communication unit 13, one log file 154 including log data recorded in chronological order in the plant from the device 3 based on the recording date and time of each operation at different times within the same predetermined period (step S103). The control unit 11 stores the acquired log file 154 in the mass storage unit 15 (step S104).

[0087] The control unit 11 generates a prompt to be given to the language model 151 based on the images of the object acquired at multiple different times and the time-series log data (step S105). The generated prompt includes instructions to extract report information from the log file 154, considerations, improvements, instructions to generate the report, etc.

[0088] The control unit 11 inputs the images of the object and the log file 154 acquired at different times and the generated plant to the language model 151 (step S106), and sequentially generates (outputs) reports on the plant for each time (step S107). The control unit 11 transmits the reports sequentially generated for each time to the terminal 4 via the communication unit 13 (step S108).

[0089] The control unit 41 of the terminal 4 receives the report transmitted from the server 1 via the communication unit 43 (step S401). The control unit 41 displays the received report on the display unit 45 (step S402). The control unit 41 ends the process.

[0090] In the present embodiment, an example of an image of an object has been described, but the present invention is not limited to this, and a video of the object may also be used. For example, the server 1 acquires video of the object from the imaging device 2 at multiple different times, and acquires the log file 154 from the device 3. The server 1 extracts multiple frame images at predetermined intervals (e.g., 10 seconds) from the video acquired at different times. The server 1 may provide the extracted multiple frame images and log data to the language model 151, thereby sequentially generating reports related to the plant at each time.

[0091] According to this embodiment, by providing the language model 151 with images and log data of an object acquired at a plurality of different times, it becomes possible to generate reports on a plant sequentially for each time.

[0092] According to this embodiment, by generating reports about the plant sequentially at each timing, it becomes possible to grasp data or trends that differ at each timing and make appropriate decisions.

[0093] <Variation 1> A process of outputting statistical information by including it in a report will be described. First, the server 1 generates statistical information using the language model 151. The statistical information includes the availability and trend of the equipment 3 in the plant. The availability indicates the percentage of time that the equipment 3 is operating. The trend indicates whether the availability of the equipment 3 tends to increase or decrease over time. Note that the statistical information is not limited to the availability and trend, and may also include, for example, a failure rate, maintenance time, productivity (e.g., production volume per unit time), or a quality index (e.g., defective product rate).

[0094] Specifically, the server 1 acquires from the device 3 a log file 154 containing log data for a predetermined period (for example, May). If the log file 154 is stored in the mass storage unit 15, the server 1 may acquire the log file 154 from the mass storage unit 15. The log file 154 in this first modification further includes the operating time of the device 3, the stop time, and the start and end times of stoppages due to maintenance or errors. The operating time is the time during which the device 3 was operating. The stop time is the time during which the device 3 was stopped.

[0095] Server 1 generates a prompt containing instructions for generating statistical information. As an example, the generated prompt is: "Based on multiple log files for May, please graph the weekly utilization rate of equipment 3 for May. Also, please tell me the trends for device 3."

[0096] The server 1 inputs the acquired log file 154 and the generated prompt into the language model 151, and generates statistical information including the operating rate and trend of the device 3. As an example, the generated statistics are "<Operating rate> A graph was generated showing the utilization rate of equipment 3 by week in May. The utilization rate for each week is displayed as a bar graph. Week 1:――――――――――90% Week 2:――――――――――100% Week 3:――――――――80% Week 4:――――――――70% <Trends> An analysis of the operating rate trend for device 3 shows that it was relatively stable from the first week through to the middle of the month, but that the operating rate tended to decline towards the latter half of the month.

[0097] Next, the server 1 adds the generated statistical information to the report generated in embodiment 1. The server 1 transmits (outputs) the report including the statistical information to the terminal 4. The terminal 4 receives the report transmitted from the server 1 and displays the received report on the screen.

[0098] 8 is a flowchart showing the processing steps when generating a report including statistical information. The control unit 11 of the server 1 acquires a target report and a log file 154 corresponding to the report (step S111). Specifically, based on the report ID of the report generated by the language model 151, the control unit 11 acquires report data and the file names of one or more log files 154 corresponding to the report from the report DB 153 of the mass storage unit 15. Based on the file name of each acquired log file 154, the control unit 11 acquires each log file 154 from the mass storage unit 15.

[0099] The control unit 11 generates a prompt including an instruction to generate statistical information (step S112). The control unit 11 inputs the acquired log file 154 and the generated prompt into the language model 151 (step S113), and generates statistical information including the operating rate and trend of the device 3 (step S114). The control unit 11 adds the generated statistical information to the acquired report data (step S115). The control unit 11 transmits the report including the statistical information to the terminal 4 via the communication unit 13 (step S116).

[0100] The control unit 41 of the terminal 4 receives the report transmitted from the server 1 via the communication unit 43 (step S411). The control unit 41 displays the received report on the display unit 45 (step S412). The control unit 41 ends the process.

[0101] According to this modification, it is possible to use the language model 151 to generate a report that includes statistical information.

[0102] <Variation 2> The process of outputting a report containing the causal relationship between the image of the object and the log data will be described below. First, the server 1 generates the causal relationship between the image of the object and the log data using the language model 151. The log data includes, for example, the brightness of the image, the blur of the image, the position of the imaging device 2, the low position of the shaft, the rotation of the shaft, or the vertical load of the shaft.

[0103] The causal relationship is a relationship in which the image of the object and the log data influence each other, including, for example, the relationship between the rotation of the shaft and the blur of the image, the relationship between the vertical load of the shaft and the brightness of the image, or the relationship between the position of the imaging device and the low position of the shaft.

[0104] For example, when the shaft of device 3 is rotating, the faster the shaft's rotation speed, the more noticeable the blurring of the image tends to be. Alternatively, if the load causes minute irregularities on the shaft's surface, the light reflectance changes, resulting in non-uniformity in the image brightness. Furthermore, if imaging device 2 is located near a low position on the shaft (e.g., the bottom of the shaft), vibrations or noise caused by the rotation of the shaft may affect the image.

[0105] Specifically, the server 1 acquires images of the object within a predetermined period (for example, one day) from the imaging device 2, and acquires a log file 154 containing log data for the predetermined period from the device 3. Note that if the images of the object and the log file 154 are stored in the mass storage unit 15, the server 1 may acquire the images of the object and the log file 154 from the mass storage unit 15. The server 1 generates a prompt including an instruction for generating a causal relationship between the images of the object and the log data. As an example, the generated prompt is: "Based on multiple images of objects throughout the day and log files, please generate a causal relationship between the images of the objects and the log data."

[0106] The server 1 inputs the acquired images of multiple objects within a predetermined period, the log file 154, and the generated prompt into the language model 151. The server 1 uses the Advanced Data Analysis function of the language model 151 to generate a causal relationship between the images of the objects and the log data using the language model 151.

[0107] As an example, the generated causal relationships are: “Increased vertical shaft load can reduce the brightness of the image of the object. As the imaging device moves further away from the shaft, the brightness of the image of the object tends to decrease. When a shaft rotates at high speed, it is easy for motion blur to appear in the image of the object.

[0108] Next, the server 1 adds the generated causal relationship to the report generated in embodiment 1. The server 1 transmits (outputs) the report including the causal relationship to the terminal 4. The terminal 4 receives the report transmitted from the server 1 and displays the received report on the screen.

[0109] 9 is a flowchart showing the processing steps when generating a report including a causal relationship. The control unit 11 of the server 1 acquires the target report, an image of an object corresponding to the report, and a log file 154 (step S121). Specifically, based on the report ID of the report generated by the language model 151, the control unit 11 acquires report data, an image of one or more objects corresponding to the report, and the file name of the log file 154 from the report DB 153 of the mass storage unit 15. Based on the file name of each acquired log file 154, the control unit 11 acquires each log file 154 from the mass storage unit 15.

[0110] The control unit 11 generates a prompt including an instruction for generating a causal relationship between the image of the object and the log data (step S122). The control unit 11 inputs the acquired images of the object, the log file 154, and the generated prompt into the language model 151 (step S123), and generates a causal relationship between the image of the object and the log data (step S124). The control unit 11 adds the generated causal relationship to the acquired report data (step S125). The control unit 11 transmits the report including the causal relationship to the terminal 4 via the communication unit 13 (step S126).

[0111] The control unit 41 of the terminal 4 receives the report transmitted from the server 1 via the communication unit 43 (step S421). The control unit 41 displays the received report on the display unit 45 (step S422). The control unit 41 ends the process.

[0112] According to this modification, it is possible to use the language model 151 to generate a report including the causal relationship between the image of the object and the log data.

[0113] <Variation 3> The following describes a process for outputting a report containing information about the progress of processing of an object. The server 1 acquires information about the progress of processing of the object from an external information processing device, such as an inspection device or an analysis device. The information about the progress of processing of the object is information for evaluating whether a predetermined process or operation has been properly performed on the object and for evaluating the quality or effectiveness, and includes, for example, quality, accuracy (color, size, etc.), efficiency (time, resource usage efficiency, cost efficiency, etc.), progress (completed, in progress, etc.), safety, etc.

[0114] The server 1 adds the acquired information on the progress of the processing of the object to the report generated in embodiment 1. The server 1 transmits (outputs) the report including the information on the progress of the processing of the object to the terminal 4. The terminal 4 receives the report transmitted from the server 1 and displays the received report on the screen.

[0115] 10 is a flowchart showing the processing procedure for generating a report including information on the progress of processing of an object. The control unit 11 of the server 1 acquires report data from the report DB 153 of the mass storage unit 15 based on the report ID of the report generated by the language model 151 (step S131).

[0116] The control unit 11 acquires information on the progress of processing of the object from an inspection device, an analysis device, or the like via the communication unit 13 (step S132). The control unit 11 adds the acquired information on the progress of processing of the object to the acquired report data (step S133). The control unit 11 transmits the report including the information on the progress of processing of the object to the terminal 4 via the communication unit 13 (step S134).

[0117] The control unit 41 of the terminal 4 receives the report transmitted from the server 1 via the communication unit 43 (step S431). The control unit 41 displays the received report on the display unit 45 (step S432). The control unit 41 ends the process.

[0118] The report on the plant may include statistical information including the operating rate and trends of the equipment 3, the causal relationship between the image of the object and the log data, information on the progress of the processing of the object, or a combination of these.

[0119] According to this modification, it is possible to use the language model 151 to generate a report including information on the progress of processing of the object.

[0120] (Embodiment 2) The second embodiment relates to an embodiment in which a report including prediction data of occurrences in an object or a plant is generated using a language model 151. Note that a description of the contents overlapping with the first embodiment will be omitted. The prediction data includes replacement prediction data, failure prediction data, risk prediction data, performance prediction data, etc.

[0121] The replacement prediction data is data for predicting when parts (consumables) of the device 3 will wear out and optimizing the replacement timing. The replacement prediction data includes, for example, prediction of motor or shaft replacement based on the number of shaft overloads, or prediction of shaft or kettle replacement based on the amount of aluminum or ash leaking.

[0122] The failure prediction data is data for detecting and predicting signs of failure before a failure occurs in the device 3. The failure prediction data includes, for example, prediction of motor failure based on the magnitude of vibration in shaft rotation, prediction of heater failure based on temperature rise, prediction of switchboard failure based on abnormal current, or prediction of control device failure based on changes in sensor data values.

[0123] The risk prediction data is data for predicting the risk of an accident or danger occurring based on information obtained from the operating state of the device 3 or parameters of the surrounding environment. For example, the risk prediction data may cover situations such as an internal rupture of the device 3, or improper or delayed supply of medicines or materials.

[0124] The performance prediction data is data for predicting the quality of the object. The performance prediction data includes, for example, a prediction of whether the operation time is within a predetermined range, a prediction of whether the object at each stage belongs to a predetermined classification class, a prediction of whether ash is mixed in when the aluminum flows into the lower kettle, or a prediction of the amount or time of aluminum flowing out.

[0125] The server 1 acquires images of an object within a predetermined period (for example, one day or one week) from the imaging device 2. The server 1 acquires a log file 154 containing log data for the predetermined period from the device 3. The server 1 generates a prompt to be given to the language model 151.

[0126] The prompts include instructions to extract report information from the log file 154, considerations, remedial actions, instructions to generate the report, and instructions to generate predicted data for occurrences in the object or plant.

[0127] As an example, the generated prompt is: Extract the following report information from the images and log files of the object taken every specified unit time (e.g., 6 hours) within a day: Frequency of use of operation modes, distribution of image brightness, type and frequency of blur applied to images, usage pattern of imaging positions, distribution of classification classes, distribution of AI stages, statistical information of shaft low position and rotation speed, and distribution of fluctuations in shaft up and down load. Also, please provide your thoughts and suggestions for improvement regarding operations based on this data. Additionally, generate sequential reports on your plant by time, including report information, insights and remedial measures. Also, based on the images and log files of these multiple objects, please generate replacement prediction data, failure prediction data, risk prediction data, and performance prediction data that will occur in the object or plant."

[0128] The server 1 inputs the acquired images of multiple objects within a predetermined period, the log file 154, and the generated prompt into the language model 151, and sequentially generates reports about the plant including prediction data (replacement prediction data, failure prediction data, risk prediction data, performance prediction data, etc.) for each timing. The timings include, for example, 0:00 to 6:00, 6:00 to 12:00, 12:00 to 18:00, and 18:00 to 24:00. As an example, the generated report: Report (midnight to 6am) <Report Information> Frequency of use of operation mode: MANUAL: 2 times xxx <Consideration> xxx <Improvement measures> xxx (6:00 AM - 12:00 PM) xxx (12:00-18:00) xxx (6pm - midnight) xxx Based on images and log files of multiple objects collected over the course of a day, the following predictions were generated: Replacement prediction data: Based on the low shaft rotation speed and vertical shaft load, the next motor replacement may occur in approximately 300 hours. Failure Prediction Data: Motor failure is predicted based on operating mode and shaft low position. Risk prediction data: Based on the fluctuations in the vertical load on the shaft, there is a risk of rupture inside unit 3. Performance prediction data: Tends to belong to the red / black class.

[0129] The server 1 transmits the report generated by the language model 151 to the terminal 4. The terminal 4 receives the report transmitted from the server 1 and displays it on the screen.

[0130] The flowchart of the process for generating a report on a plant in this embodiment is the same as that shown in FIG. 7, and therefore will not be repeated.

[0131] Note that the generation process of prediction data is not limited to the above-described process using language model 151. Prediction data can be generated by utilizing a learning model. The learning model is a model that learns the relationship between the image and log data of the object in a first period (for example, 9:00 to 12:00) and prediction data (replacement prediction data, failure prediction data, risk prediction data, performance prediction data, etc.) in periods after the first period.

[0132] The learning model may be realized using, for example, Seq2Seq (Sequence to Sequence), which is a type of RNN (Recurrent Neural Network), a Transformer, a decision tree, a random forest, or an SVM (Support Vector Machine).

[0133] Specifically, the server 1 acquires images and log data of the object for a first period. The server 1 inputs the acquired images and log data of the object for the first period into a learning model and outputs prediction data for the first period and thereafter. The server 1 adds the prediction data output from the learning model to the data of the target report. The server 1 transmits the report including the prediction data to the terminal 4.

[0134] According to this embodiment, by providing image or log data to the language model 151, it is possible to generate a report including predicted data of what will occur in the object or plant.

[0135] According to this embodiment, replacement or failure of parts or the like is predicted in advance, and planned maintenance is performed, thereby minimizing downtime and improving productivity.

[0136] According to this embodiment, the risk of an accident or damage to equipment can be grasped in advance, and appropriate measures can be taken to improve work safety.

[0137] According to this embodiment, it is possible to predict the quality of an object based on the performance prediction data, thereby reducing the occurrence of defective products.

[0138] (Embodiment 3) The third embodiment relates to a form in which a report including a measure for improving yield is generated using a language model 151. Note that a description of the contents overlapping with the first and second embodiments will be omitted. The yield is an index indicating the quantity or quality of an output produced in a production process or production step. For example, the yield is used as a measure indicating the percentage of an output that can be efficiently obtained when raw materials or resources are processed in a production step to obtain a product or output.

[0139] The server 1 acquires images of an object within a predetermined period (for example, one day) from the imaging device 2. The server 1 acquires a log file 154 containing log data for the predetermined period from the device 3. The server 1 generates a prompt to be given to the language model 151 based on the acquired images of the object and the log data for each operation.

[0140] The prompts may include instructions to extract report information from the log file 154, considerations, remedies, instructions to generate the report, and instructions to generate yield remedies.

[0141] As an example, the generated prompt is: Extract the following report information from the images and log files of the object taken every specified unit time (e.g., 6 hours) within a day: Frequency of use of operation modes, distribution of image brightness, type and frequency of blur applied to images, usage pattern of imaging positions, distribution of classification classes, distribution of AI stages, statistical information of shaft low position and rotation speed, and distribution of fluctuations in shaft up and down load. Also, please provide your thoughts and suggestions for improvement regarding operations based on this data. Additionally, generate sequential reports on your plant by time, including report information, insights and remedial measures. Also, please come up with improvement plans to increase yield."

[0142] The server 1 inputs the acquired images of multiple objects within a predetermined period, the log file 154, and the generated prompt into the language model 151, and sequentially generates reports on the plant including measures to improve yield for each timing. The timings include, for example, 0:00 to 6:00, 6:00 to 12:00, 12:00 to 18:00, and 18:00 to 24:00.

[0143] As an example, the generated report: Report (midnight to 6am) <Report Information> Frequency of use of operation mode: MANUAL: 2 times xxx <Consideration> xxx <Improvement measures> xxx (6:00 AM - 12:00 PM) xxx (12:00-18:00) xxx (6pm - midnight) xxx Improvement ideas for increasing yield: 1. Consider introducing a new control algorithm to find the optimal balance between shaft speed and load in each mode. 2. Adjust the image processing parameters optimally for each operation mode. In manual operation mode: Image Brightness Adjustment: Set the brightness to medium and adjust to increase the contrast. Color Correction: Adjust the hue and enhance the clarity. In AI-DRIVE mode: xxx In AI-TUKIM mode: It could also be "XXX".

[0144] The server 1 transmits the report generated by the language model 151 to the terminal 4. The terminal 4 receives the report transmitted from the server 1 and displays it on the screen.

[0145] The flowchart of the process for generating a report on a plant in this embodiment is the same as that shown in FIG. 7, and therefore will not be repeated.

[0146] Furthermore, a report including measures to improve yield can be generated using the language model 151 based on log data relating to operations with good yields.

[0147] Specifically, the server 1 acquires a second log file 154 containing log data related to operations with good yields from the terminal 4 or an external information processing terminal. The server 1 generates a prompt to be given to the language model 151 based on the acquired second log file 154, images of a plurality of objects within a predetermined period, and the log file 154 (first log file 154).

[0148] As an example, the generated prompt is: Extract the following report information from the images and log files of the object taken every specified unit time (e.g., 6 hours) within a day: xxxxxx. Also, please provide your thoughts and suggestions for improvement regarding operations based on this data. Additionally, generate sequential reports on your plant by time, including report information, insights and remedial measures. The XXX log file records log data related to operations with good yields. Based on this log file, please create improvement plans to increase the yield during one day's operations."

[0149] The server 1 inputs the second log file 154, images of multiple objects within a predetermined period of time, the first log file 154, and the generated prompt into the language model 151, and sequentially generates reports on the plant including measures to improve yield at each timing.

[0150] According to this embodiment, by providing the log data for each operation to the language model 151, it becomes possible to generate a report including measures to improve yield.

[0151] According to this embodiment, by generating a report including measures to improve yield, the direction for solving specific problems in the production process or the equipment 3 becomes clear, enabling efficient improvement activities.

[0152] (Embodiment 4) The fourth embodiment relates to an embodiment in which a report including questions about an object or a plant and answers to the questions is generated using the language model 151. Note that a description of the contents overlapping with the first to third embodiments will be omitted.

[0153] Fig. 11 is a block diagram showing an example of the configuration of the server 1 in embodiment 4. Note that the same reference numerals are used to designate the same components as those in Fig. 2, and the description thereof will be omitted. The mass storage unit 15 includes a knowledge DB 155. The knowledge DB 155 stores information for reference when generating answers to questions.

[0154] FIG. 12 is an explanatory diagram showing an example of a record layout of the knowledge DB 155. The knowledge DB 155 includes a document number column, a type column, a document name column, a chapter column, and a content column. The document number column stores a unique document number to identify each document. The type column stores the type of document. The document type includes, for example, specifications, manuals, or user guides. Specifications include system design documents, hardware specifications, software specifications, or interface specifications. Manuals include device manuals (e.g., control panel instruction manuals), maintenance manuals, or troubleshooting manuals. The document name column stores the name of the document. The chapter column stores the name of each chapter of the document. The content column stores the content of each chapter.

[0155] 13 is an explanatory diagram illustrating the process of generating a report including a question and an answer. The terminal 4 accepts a question input by a user regarding an object or a plant. The terminal 4 transmits the accepted question to the server 1. The server 1 receives the question transmitted from the terminal 4. The server 1 generates an answer to the question by providing the received question to the language model 151.

[0156] Specifically, based on the received question, the server 1 generates a prompt to be given to the language model 151. The prompt includes the question, a reference DB, etc. The server 1 inputs the generated prompt into the language model 151 and generates an answer (response) to the question.

[0157] FIG. 14 is an explanatory diagram showing examples of prompts and responses. Three examples are shown in FIG. 14. The prompt in Example 1 is "Please refer to the knowledge DB and tell me the location of the execution start button on the control panel." When the prompt in Example 1 is input, the language model 151 acquires information about the control panel from the knowledge DB included in the prompt. The knowledge DB stores, for example, hardware specifications and control panel instruction manuals. Based on the acquired information about the control panel, the language model 151 extracts an image of the control panel from the knowledge DB or a related database.

[0158] The language model 151 identifies the position or shape of the blue button, which is the execution start button, from the extracted image of the control panel. The language model 151 generates an image of the control panel for highlighting the identified blue button. The language model 151 generates text (sentence), for example, "Please press this blue button for one second," along with the generated image of the blue button. The language model 151 outputs the generated image and text of the blue button as a response.

[0159] The prompt in Example 2 is "Please refer to the knowledge DB and report DB and tell me about limit control of mixing time." When the prompt in Example 2 is input, the language model 151 searches the knowledge DB included in the prompt for information about limit control of mixing time. The knowledge DB stores, for example, software specifications.

[0160] The language model 151 is generated based on the retrieved information on the limit control of the mixing time, for example, "We determine the class of the upper kettle and set the limit. White class 30 seconds Yellow class 1 minute Yellow and black class 1 minute 15 seconds Red class 1 minute 30 seconds Red and Black Class 1 minute 45 seconds The language model 151 generates the text "Black class 2 minutes." The language model 151 outputs the generated text as a response.

[0161] The prompt in Example 3 is "Please refer to the report DB and tell me the average operation time for the past week." When the prompt in Example 3 is input, the language model 151 acquires each operation time for the week from the report DB included in the prompt. The language model 151 calculates the average operation time for the week based on each acquired operation time.

[0162] Based on the calculated average weekly operation time, the language model 151 generates text such as "According to the report dated June 20, 2024, the average operation time is 5 minutes and 23 seconds." The language model 151 outputs the generated text as a response.

[0163] 13, the server 1 adds the question by the user and the answer (response) generated by the language model 151 to the report generated in the first embodiment. The server 1 transmits the report including the question and the answer to the terminal 4. The terminal 4 receives the report transmitted from the server 1 and displays the received report on the screen.

[0164] 15 is a flowchart showing the processing steps for generating a report including a question and an answer. The control unit 11 of the server 1 acquires report data from the report DB 153 in the mass storage unit 15 based on the report ID of the report generated by the language model 151 (step S141). The control unit 11 acquires the question posed by the user from the terminal 4 via the communication unit 13 (step S142).

[0165] Based on the acquired question, the control unit 11 generates a prompt to be given to the language model 151 (step S143). The prompt includes the question, a reference DB, etc. The control unit 11 inputs the generated prompt to the language model 151 (step S144) and generates an answer to the question (step S145). The control unit 11 adds the question and the generated answer to the acquired report data (step S146).

[0166] The control unit 11 transmits a report including the question and answer to the terminal 4 via the communication unit 13 (step S147). The control unit 41 of the terminal 4 receives the report transmitted from the server 1 via the communication unit 43 (step S441). The control unit 41 displays the received report on the display unit 45 (step S442). The control unit 41 ends the process.

[0167] According to this embodiment, the language model 151 can be used to generate reports that include questions and answers.

[0168] According to this embodiment, the report contains questions and answers, allowing the cause and solution of the problem to be traced, so that if a particular problem recurs, past answers can be referenced and a quick response can be taken.

[0169] The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0170] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the reference format. Furthermore, the claims use a format in which a claim references two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that references at least one other multiple claim may also be used. [Explanation of symbols]

[0171] 1. Information processing device (server) 11 Control section 12 Storage section 13 Communications Department 14 Reading unit 15 Mass storage 151 language models 152 Image DB 153 Report DB 154 Log files (first log file, second log file) 155 Knowledge DB 1a Portable storage media 1b semiconductor memory 1P control program 2. Imaging device 3 equipment 31 Control Unit 32 Storage section 33 Communications Department 4. Information processing terminal (terminal) 41 Control Unit 42 Storage section 43 Communications Department 44 Input section 45 Display section 4P control program B Bus N Network

Claims

1. Acquiring images of objects photographed in a plant and time-series log data in the plant; By providing the images and log data acquired at a plurality of different times to a language model, reports regarding the plant are generated sequentially for each time. An information processing method in which processing is performed by a computer.

2. The timing includes the start of operation, the end of operation, every predetermined unit time, and when a request is received. The information processing method according to claim 1 .

3. generating statistical information including utilization rates and trends of equipment in the plant using the language model; Generated statistical information is included in the above report and output.

3. The information processing method according to claim 1 or 2.

4. generating a causal relationship between the image and the log data using the language model based on the image and the log data; The generated causal relationships are included in the report and output.

3. The information processing method according to claim 1 or 2.

5. Acquire information on the processing status of the object; The acquired information on the progress is included in the report and output.

3. The information processing method according to claim 1 or 2.

6. The image or the log data is fed to the language model to generate the report including predicted data that will occur in the object or the plant.

3. The information processing method according to claim 1 or 2.

7. Acquire log data for each operation on the object; The acquired log data for each operation is fed to the language model to generate the report including measures to improve yield.

3. The information processing method according to claim 1 or 2.

8. generating an answer to a question about the object or the plant by providing the question to the language model; The question and the generated answer are included in the report and output.

3. The information processing method according to claim 1 or 2.

9. Acquiring images of objects photographed in a plant and time-series log data in the plant; By providing the images and log data acquired at a plurality of different times to a language model, reports regarding the plant are generated sequentially for each time. A program that causes a computer to perform a process.

10. An information processing device including a control unit, The control unit Acquiring images of objects photographed in a plant and time-series log data in the plant; By providing the images and log data acquired at a plurality of different times to a language model, reports regarding the plant are generated sequentially for each time. Information processing device.

Images