Learning apparatus, state reading system, learning method, and learning program

Abstract

To provide a learning apparatus which can read states of a plurality of instruments from a single captured image of a board to be created by an inspector, without using definition information associating the board with an image of a target object.SOLUTION: A learning apparatus 250 according to the present invention comprises a data acquiring unit 251 for acquiring learning data including captured images obtained by capturing a surface of an inspection target board mounted with a plurality of measuring instruments, respective images of the plurality of measuring instruments included in the captured image, and identification information as information for use in identifying and specifying each of the plurality of measuring instruments, and a model generating unit 252 for, by using the learning data, generating, from the captured image, a leaned model for use in inference of each of the images of the measuring instruments and the identification information.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present disclosure relates to a learning device, a status reading system, a learning method, and a learning program for reading the status of an instrument on a panel. [Background technology]

[0002] Conventionally, when identifying the type and model number of an equipment panel and automatically reading the numerical values ​​and status of the instruments installed on the panel, definition information linking the type and model number of the panel to be read with an image of the target object was used. When reading a panel for which no definition information was set, the worker had to create new definition information, which made management of the definition information cumbersome.

[0003] Patent Document 1 discloses a meter reading device that detects the area of ​​a meter and a character string of identification information from an image of a panel, and reads the state without using a definition. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-166587 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, while the meter reading device described in Patent Document 1 above can photograph the state of the panel and read the state without using definition information, it can only read one type of object with one photograph, and if multiple types of instruments are installed on the panel, it is necessary to photograph images of the panel multiple times, which is time-consuming.

[0006] The present disclosure has been made in consideration of the above, and aims to obtain a learning device that makes it possible to read the state of multiple instruments from a single captured image of a panel without using definition information that links the panel created by an inspector to an image of the object. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the objective, the learning device disclosed herein comprises a data acquisition unit that acquires learning data including a captured image of a panel surface on which multiple instruments are mounted on a panel to be inspected, images of each of the multiple instruments included in the captured image, and identification information that is information for identifying and specifying each of the multiple instruments, and a model generation unit that uses the learning data to generate a trained model for inferring the images and the identification information of each of the multiple instruments from the captured image. Effect of the Invention

[0008] The present disclosure has the effect of providing a learning device that makes it possible to read the status of multiple instruments from a single captured image of a panel without using definition information that links the panel created by an inspector to an image of the object. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a configuration of a status reading system according to a first embodiment; [Diagram 2] FIG. 1 is a diagram showing a configuration of a mobile terminal included in a status reading system according to a first embodiment; [Diagram 3] FIG. 1 is a diagram showing a configuration of a server included in a status reading system according to a first embodiment; [Figure 4] FIG. 1 is a diagram showing a flow of information between a mobile terminal and a server included in the status reading system according to the first embodiment. [Diagram 5] FIG. 1 is a diagram showing an example of a panel table stored in an equipment database of a server included in the status reading system according to the first embodiment; [Figure 6]FIG. 6 is a diagram showing an example of an image stored in the board table shown in FIG. 5. [Figure 7] FIG. 1 is a diagram showing an example of an object table stored in an inspection database of a server included in the condition reading system according to the first embodiment; [Figure 8] FIG. 1 is a diagram showing an example of an object whose status is read by a status reading unit of a server included in a status reading system according to a first embodiment; [Figure 9] FIG. 1 is a first diagram showing an example of an image captured by a mobile terminal included in a status reading system according to a first embodiment; [Figure 10] FIG. 2 is a second diagram showing an example of an image captured by a mobile terminal included in the status reading system according to the first embodiment; [Figure 11] FIG. 3 is a third diagram showing an example of an image captured by a mobile terminal included in the status reading system according to the first embodiment; [Figure 12] FIG. 4 is a fourth diagram showing an example of an image captured by the mobile terminal included in the status reading system according to the first embodiment; [Figure 13] FIG. 1 is a diagram showing a configuration of a learning device according to a first embodiment; [Figure 14] 1 is a flowchart showing a procedure of a learning process performed by the learning device according to the first embodiment; [Figure 15] FIG. 1 is a diagram showing a configuration of a neural network used by a learning device according to a first embodiment; [Figure 16] FIG. 1 is a diagram showing an example of learning data used by the learning device according to the first embodiment; [Figure 17] FIG. 1 is a diagram showing an example of an output of a trained model in the learning device according to the first embodiment. [Figure 18] FIG. 1 is a diagram showing an example of learning content about a board in a model generation unit of the learning device according to the first embodiment; [Figure 19] FIG. 1 is a diagram showing an example of learning content about a meter included in a captured image of a panel in a model generating unit of the learning device according to the first embodiment; [Figure 20]FIG. 1 is a diagram showing an example of learning content about a lamp included in a photographed image of a board in a model generating unit of the learning device according to the first embodiment; [Figure 21] FIG. 1 is a diagram showing a configuration of an inference device according to a first embodiment. [Figure 22] 1 is a flowchart showing a procedure of an inference process performed by the inference device according to the first embodiment; [Figure 23] FIG. 1 is a diagram showing an example of input data input to the inference device according to the first embodiment; [Figure 24] FIG. 1 is a diagram showing an example of output data output from the inference device according to the first embodiment; [Diagram 25] 1 is a flowchart showing a procedure of an operation of the status reading system according to the first embodiment; [Figure 26] 11 is a first flowchart showing an example of a procedure of a process for reading a status of an object in a status reading unit of a server included in the status reading system according to the first embodiment; [Figure 27] 2 is a second flowchart showing an example of a procedure of a process for reading a status of an object in a status reading unit of a server included in the status reading system according to the first embodiment; [Figure 28] FIG. 13 is a diagram showing an example of a read result display screen displayed on a display unit of a mobile terminal included in the status reading system according to the first embodiment; [Figure 29] FIG. 13 is a diagram showing an example of an output file of the reading result of the target object on a display unit of a mobile terminal included in the status reading system according to the first embodiment; [Diagram 30] FIG. 13 is a diagram showing a flow of information between a mobile terminal and a server included in the status reading system according to the second embodiment. [Diagram 31] FIG. 13 is a diagram showing an example of learning data used by a learning device in the status reading system according to the second embodiment; [Diagram 32] FIG. 13 is a diagram showing an example of learning content about a board in a learning unit of the learning device according to the second embodiment; [Diagram 33] 11 is a flowchart showing a procedure of an operation of the status reading system according to the second embodiment. [Diagram 34]11 is a flowchart showing a detailed procedure of a characteristic operation of the status reading system according to the second embodiment. [Diagram 35] FIG. 13 is a diagram showing a flow of information between a mobile terminal and a server included in the status reading system according to the third embodiment. [Diagram 36] 11 is a flowchart showing a procedure of an operation of the status reading system according to the third embodiment. [Figure 37] FIG. 13 is a diagram showing an example of a read result display screen displayed on a display unit of a mobile terminal included in the status reading system according to the third embodiment; [Figure 38] FIG. 11 is a first diagram showing an example of a read result editing screen displayed on a display unit of a mobile terminal included in a status reading system according to a third embodiment; [Figure 39] FIG. 2 is a second diagram showing an example of a read result editing screen displayed on a display unit of a mobile terminal included in a status reading system according to a third embodiment; [Diagram 40] FIG. 13 is a diagram showing a flow of information between a mobile terminal and a server included in the status reading system according to the fourth embodiment. [Diagram 41] 11 is a flowchart showing the procedure of the operation of the status reading system according to the fourth embodiment. [Diagram 42] FIG. 13 is a diagram showing an example of a panel table stored in an equipment database of a server included in the status reading system according to the fourth embodiment. [Diagram 43] FIG. 13 is a diagram showing an example of a report format table stored in an inspection database of a server included in the condition reading system according to the fourth embodiment. [Diagram 44] 11 is a flowchart showing a detailed procedure of a characteristic operation of the status reading system according to the fourth embodiment. [Diagram 45] FIG. 13 is a diagram showing an example of a read result display screen displayed on a display unit of a mobile terminal included in the status reading system according to the fourth embodiment; [Figure 46] FIG. 13 is a diagram showing an example of an output of a report file on a display unit of a mobile terminal included in the status reading system according to the fourth embodiment; [Figure 47] FIG. 13 is a diagram showing a configuration of a server included in a status reading system according to a fifth embodiment. [Figure 48] FIG. 13 is a diagram showing the flow of information between a mobile terminal and a server included in the status reading system according to the fifth embodiment. [Figure 49] FIG. 13 is a diagram showing an example of an inspection result table stored in an inspection database of a server included in the condition reading system according to the fifth embodiment. [Figure 50] A flowchart showing a procedure of a learning process performed by an operational state learning unit according to the fifth embodiment. [Figure 51] FIG. 23 is a diagram showing an example of learning data used by an operational state learning unit according to the fifth embodiment; [Figure 52] FIG. 23 is a diagram showing an example of an output of an operating state determination pattern in the learning device according to the fifth embodiment; [Figure 53] 13 is a flowchart showing the procedure of the operation of the status reading system according to the fifth embodiment. [Figure 54] 11 is a flowchart showing a detailed procedure of a characteristic operation of the status reading system according to the fifth embodiment. [Figure 55] FIG. 13 is a diagram showing an example of a read result display screen displayed on a display unit of a mobile terminal included in the status reading system according to the fifth embodiment; [Figure 56] FIG. 13 is a diagram showing an example of an output file of the reading result of the object on the display unit of the mobile terminal included in the status reading system according to the fifth embodiment. [Figure 57] FIG. 13 is a diagram showing the flow of information between a mobile terminal and a server included in the status reading system according to the sixth embodiment. [Figure 58] FIG. 23 is a diagram showing an example of learning data used by the learning device according to the sixth embodiment for learning about an image of an object. [Figure 59] FIG. 23 is a diagram showing an example of an output of a trained model in the learning device according to the sixth embodiment. [Figure 60] FIG. 23 is a diagram showing an example of learning data used by the learning device according to the sixth embodiment for learning about identification information; [Figure 61] FIG. 23 is a diagram showing an example of an output of a trained model in the learning device according to the sixth embodiment. [Figure 62]FIG. 23 is a diagram for explaining an example of information about a board, which is identification information output by a character detection model according to the sixth embodiment. [Figure 63] FIG. 23 is a diagram for explaining an example of information about a meter, which is identification information output by the character detection model according to the sixth embodiment. [Figure 64] FIG. 23 is a diagram showing an example of information about a lamp, which is identification information output by the character detection model according to the sixth embodiment; [Figure 65] 13 is a flowchart showing the procedure of the operation of the status reading system according to the sixth embodiment. [Figure 66] A flowchart showing an example of a procedure for a process of determining identification information related to an object in an object detection unit of a server included in a status reading system according to a sixth embodiment. [Figure 67] FIG. 23 is a diagram showing an example of a captured image of a board input to an object detection unit of a server included in a status reading system according to a sixth embodiment; [Figure 68] FIG. 22 is a diagram for explaining an example of a calculation process in an object detection unit of a server included in a status reading system according to a sixth embodiment. [Figure 69] FIG. 1 shows a configuration in which the functions of the control unit according to the first to sixth embodiments are realized by hardware. [Figure 70] FIG. 1 shows a configuration in which the functions of the control unit according to the first to sixth embodiments are realized by software. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A learning device, a state reading system, a learning method, and a learning program according to embodiments will be described in detail below with reference to the accompanying drawings.

[0011] Embodiment 1 FIG. 1 is a diagram showing a configuration of a status reading system according to a first embodiment. FIG. 2 is a diagram showing a configuration of a mobile terminal included in the status reading system according to the first embodiment. FIG. 3 is a diagram showing a configuration of a server included in the status reading system according to the first embodiment. FIG. 4 is a diagram showing a flow of information between the mobile terminal and the server included in the status reading system according to the first embodiment. FIG. 4 shows extracted components used to explain the operation of the status reading system 1.

[0012] The status reading system 1 according to the first embodiment includes a mobile terminal 100 and a server 200. The status reading system 1 is a system that automatically reads the status of a plurality of objects to be inspected that are mounted on equipment 300 to be inspected. The objects to be inspected that are mounted on the equipment 300 are instruments provided on the outer surface of the equipment 300. In the first embodiment, the equipment 300 is a panel. The mobile terminal 100 and the server 200 are connected to the Internet 400, which is a global information and communication network, and are capable of transmitting and receiving information to and from each other. In other words, the mobile terminal 100 and the server 200 are capable of transmitting and receiving information to and from each other via the Internet 400.

[0013] The mobile terminal 100 includes an operation unit 101, a board designation unit 102, a photographing unit 103, a display unit 104, an output unit 105, a terminal storage unit 106, a terminal communication unit 107, and a terminal control unit 108.

[0014] The operation unit 101 is an operation reception unit that receives user operations, i.e., setting operations from the user. The setting operations from the user include a setting operation for specifying a panel to be inspected, i.e., a setting operation for specifying a panel from which the state of an object is to be read. The operation unit 101 receives input of the user operation and transmits information corresponding to the user operation to the terminal control unit 108. The operation unit 101 is configured with input devices such as a keyboard, a mouse, and a touch panel display having a touch panel function, and operations on the mobile terminal 100 are performed by the user. In the first embodiment, the operation unit 101 is configured with a touch panel display together with the display unit 104.

[0015] The board designation unit 102 transmits information to be designated as the equipment 300 to be photographed from among the information on the equipment 300 transmitted from the board data acquisition unit 201 of the server 200, which will be described later. That is, the board designation unit 102 transmits board designation information, which is information on the board to be designated as the board to be photographed, from the information in a list of board data, which is information on the equipment 300 transmitted from the board data acquisition unit 201 of the server 200, which will be described later. The board designation unit 102 identifies the board to be designated as the board to be photographed, based on information input by a setting operation from the user. The board to be photographed is a board to be inspected.

[0016] When the board designation unit 102 receives information on the list of board data transmitted from the board data acquisition unit 201, it displays the list of board data on the display unit 104 based on the received information, and prompts the user to designate a board to be photographed. The user inputs information on the board to be designated as the board to be photographed to the board designation unit 102 by selecting the board to be designated as the board to be photographed from the list of board data displayed on the display unit 104.

[0017] The photographing unit 103 includes a camera (not shown), controls the operation of the camera to photograph an image of the board, and transmits the photographed image to the board data acquisition unit 201 of the server 200. Note that the photographing unit 103 may also control the operation of an external camera connected to the mobile terminal 100 to photograph an image of the board, and transmit the photographed image to the board data acquisition unit 201.

[0018] The display unit 104 has a display screen and displays various types of information inside the mobile terminal 100, including images captured by the imaging unit 103, information on the state of the object read by a state reading unit 203 (described later) of the server 200, and information such as an inspection result file acquired by the output unit 105. Note that the display on the display unit 104 may be controlled by the terminal control unit 108.

[0019] The output unit 105 acquires the inspection result file created by a read result storage unit 204 (described later) of the server 200 , and stores it in the terminal storage unit 106 of the mobile terminal 100 .

[0020] The terminal storage unit 106 stores information used to control the mobile terminal 100. The terminal storage unit 106 stores various information used for the functions of the condition reading system 1, including information such as the image captured by the image capturing unit 103 and the inspection result file acquired by the output unit 105.

[0021] The terminal communication unit 107 connects to the Internet 400 and communicates with devices external to the mobile terminal 100. The terminal communication unit 107 communicates with the server 200 via the Internet 400. The terminal communication unit 107 transmits information received from external devices such as the server 200 to internal components of the mobile terminal 100 such as the terminal control unit 108. The terminal communication unit 107 transmits various information such as board designation information and captured images to the server 200.

[0022] The terminal control unit 108 controls the overall operation of the mobile terminal 100. The terminal control unit 108 causes the terminal storage unit 106 to store various pieces of information transmitted from the terminal communication unit 107.

[0023] The server 200 has a computer on which a learning program is installed. The learning program is a program for realizing the steps of acquiring learning data including a captured image of a board surface on which multiple objects are mounted on a board to be inspected, an image of each of the multiple objects included in the captured image, and identification information that is information for identifying and specifying each of the multiple objects, and generating a trained model for inferring the image and the identification information of each of the multiple objects from the captured image using the learning data.

[0024] The server 200 comprises a board data acquisition unit 201, an object detection unit 202, a status reading unit 203, a reading result storage unit 204, an object learning unit 205, a server communication unit 206, a server control unit 207, and a server memory unit 208.

[0025] The board data acquisition unit 201 acquires a board table stored in an equipment database 2081 described below, and transmits information about the board stored in the board table to the mobile terminal 100. The board data acquisition unit 201 also receives board designation information, which is information about a board designated as the board to be photographed, transmitted from the board designation unit 102 of the mobile terminal 100. The board data acquisition unit 201 determines the board indicated in the board designation information as the board to be photographed. Note that the database may be referred to as DB.

[0026] The board table is stored in the equipment database 2081, and stores information necessary to uniquely identify a board, such as the name of the board, the location where the board is installed, and an image of the board. In other words, the board table can be said to include a list of information on boards registered in the status reading system 1. The board table stores and defines information about boards to be processed by the status reading system 1, such as the board's ID (Identification), the name of the board, where the board is installed, and the image of the board surface.

[0027] FIG. 5 is a diagram showing an example of a board table stored in an equipment database of a server included in the status reading system according to the first embodiment. FIG. 6 is a diagram showing an example of an image stored in the board table shown in FIG. 5. The board table shown in FIG. 5 has a plurality of items of information related to the board, such as a board ID, a board name, an installation location, and an image, and stores corresponding information. The board ID is identification information unique to the board. The installation location is the location where the board is installed. The image is a photographed image of the board transmitted from the mobile terminal 100. FIG. 6 shows an example of a photographed image of a board with a board ID of M001 in the board table shown in FIG. 5. As shown in FIG. 6, the photographed image of the board is an image of a board surface on which objects such as meters and lamps are arranged.

[0028] The object detection unit 202 is a detection unit having a photographed image acquisition function for acquiring information about a photographed image of the board transmitted from the mobile terminal 100, and an object detection function for detecting an object included in the photographed image of the board.

[0029] In the object detection function, the object detection unit 202 inputs information about a photographed image of the board into the object detection model 2083 to detect an object on the board that appears in the photographed image, and obtains an image of the object and identification information, which is character information for identifying the object. Examples of objects detected by the object detection unit 202 include meters, lamps, switches, and levers. The information about the object includes the position and type of the object, as well as character information contained inside the object, such as the model number and scale value, and the position information of the object.

[0030] Furthermore, the object detection unit 202 acquires information about the board by inputting information about a captured image of the board to the object detection model 2083. The board information includes the names of devices involved in the operation of the object, such as pump No. 1 and pump No. 2. The names of the devices are used by the status reading unit 203 when grouping the objects.

[0031] The identification information of the object includes information on the position of the object and the type of the object, as well as character information contained within the object. The character information contained within the object includes information on the model number, scale value, scale value, and character position.

[0032] The information on the position of the object is information on the position of the object on the surface of the board shown in the captured image, and is information indicating where the object is located on the surface of the board.

[0033] The object detection unit 202 includes an inference device 260, and inputs information about a captured image of the board to an object detection model 2083, which is a trained model 273 described later, to obtain an image of an object on the board shown in the captured image and identification information about the object. The object detection model 2083 and the inference device 260 will be described in detail later.

[0034] The object detection unit 202 transmits information on the generated image of the object and identification information on the object to the status reading unit 203 as object detection information.

[0035] The status reading unit 203 reads the object status from the image of the object using image processing based on the board information, the image information of the object, and the identification information acquired from the object detection unit 202, and acquires information on the object status. The status reading unit 203 transmits the acquired information on the object status to the mobile terminal 100.

[0036] That is, an image of an object is input to the status reading unit 203, and information on the state of the object is output. Examples of objects whose states are read by the status reading unit 203 include a meter, a lamp, a switch, and a lever. An example of the state of an object is information on the value indicated by the meter needle when the object is a meter, or information that the lamp is on or off when the object is a lamp.

[0037] The reading result storage unit 204 has the function of recording the information on the board, the captured image, and the information on the reading results of the board including the reading results of the target object together in a single inspection result file after receiving reading result confirmation information from the mobile terminal 100 indicating that the status reading results have been confirmed.

[0038] The read result storage unit 204 also has a function of transmitting a file that combines the board information, the captured image, and the information on the results of reading the board to the output unit 105 of the mobile terminal 100.

[0039] Furthermore, the reading result storage unit 204 has a function of storing information about the board, the captured image, and information about the reading results of the board including the reading results of the object in the inspection database 2082. This allows the object learning unit 205 to learn the object detection model 2083 based on the information stored in the inspection database 2082 by the reading result storage unit 204. Furthermore, the object learning unit 205 is capable of re-learning using information stored in the inspection database 2082 at any time.

[0040] The object learning unit 205 includes a learning device 250, and learns the image and identification information of the object using machine learning, generates an object detection model 2083 which is a learned model 273, and stores it in the server storage unit 208. Details of the learning device 250 will be described later.

[0041] The server storage unit 208 includes an equipment database 2081 and an inspection database 2082. The server storage unit 208 also stores a model 2083 for object detection.

[0042] The equipment database 2081 is a database that stores information about the equipment 300 to be inspected, that is, a board table that is information about the board.

[0043] The inspection database 2082 stores information about objects provided on the panel that is the equipment 300 to be inspected, i.e., an object table that is information about the objects. The inspection database 2082 also stores information about a report to be described later that is used when inspecting the objects.

[0044] The object table stores information necessary to uniquely identify an inspection object, such as the type of object, the output range of the object, etc. In other words, the object table stores information about objects whose status is to be read by the status reading system 1, and stores and defines information about the objects whose status is to be read, such as the object ID, the object name, the object type, the display format of the information on the object, information about the board on which the object is installed, and an image of the object.

[0045] Fig. 7 is a diagram showing an example of an object table stored in an inspection database of a server included in the condition reading system according to the embodiment 1. The object table shown in Fig. 7 has a plurality of items of information on the object, such as an object ID, a name, a type, an output range, a display, a board ID, and an object image, and stores corresponding information.

[0046] The object ID is identification information unique to the object. In addition, in the object table, when multiple objects are installed on one board, multiple object IDs are associated with one board ID. The type is the type of object, such as a meter, lamp, switch, lever, etc., and also includes more detailed classifications such as a circular meter. The output range is the range of the output of the object. The display is the display format of the output of the object, such as % display, running, and stopped. The board ID is identification information unique to the board that is also stored in the board table, and is information common to the board table. The board ID links the information stored in the board table and the information stored in the object table. The object image is an image of the object acquired by the object detection unit 202, and is an image cut out from the captured image or copied from the captured image.

[0047] Here, an example of an object whose state is read by the status reading unit 203 will be described. Fig. 8 is a diagram showing an example of an object whose state is read by the status reading unit of the server included in the status reading system according to the first embodiment. Fig. 8 shows a state in which various objects are placed on a sample board 410. As shown in Fig. 8, objects whose state is read by the status reading unit 203 of the server 200 include meters, lamps, switches, and levers.

[0048] Examples of meters include circular meters, vertical meters, square meters, and counter meters. In circular meters, vertical meters, and square meters, when the status is read by the status reading unit 203, the scale position and scale value are recognized, and then the needle value is read by comparing it with the needle position. In counter meters, when the status is read by the status reading unit 203, the characters of the part where the numerical value is counted from 0 to 9 are read.

[0049] Examples of lamps include circular lamps and square lamps. When the status of a lamp is read by the status reading unit 203, either the on state or the off state is read based on the brightness of the lamp part. Note that reading of the blinking state of the lamp is not included here.

[0050] Examples of the switch include a slide switch and a knob switch. In the slide switch shown in Fig. 8, the hatched knob moves up and down, and the character string "ON" or "OFF" which indicates the current switch state is visible. In the slide switch shown in Fig. 8, when the state is read by the state reading unit 203, the character string "ON" or "OFF" is read.

[0051] In the case of a knob-type switch, when the state is read by the state reading unit 203, the position of the knob part and the inclination of the knob part in the rotation direction are recognized, and then the correspondence between the knob part and the character strings around the knob part are calculated, that is, the distance between the knob part and the character strings around the knob part is calculated, and the state of the knob-type switch is read. In the knob-type switch shown in Fig. 8, when the state is read by the state reading unit 203, the character closest to the knob part out of the characters "automatic" or "manual" is read as the state of the knob-type switch.

[0052] An example of the lever is a rotary lever. In the lever shown in Fig. 8, when the state is read by the state reading unit 203, the position of the hatched handle is recognized, and then the correspondence between the handle and the character string around the handle is calculated, that is, the distance between the handle and the character string around the handle is calculated, and the state of the lever is read. In the lever shown in Fig. 8, when the state is read by the state reading unit 203, the characters "1", "2" or "3" are read as the state of the lever.

[0053] Here, the photographed image of the board will be described. In the photographed image of the board taken during inspection, the brightness changes depending on the photographing conditions such as the weather and the photographing location.

[0054] In an image of a meter captured outdoors on a sunny day, some of the characters may appear missing or the characters themselves may appear faint due to reflections. In an image of a lamp captured outdoors on a sunny day, the surrounding area of ​​the lamp appears bright, making the lamp appear dark even when it is lit.

[0055] Fig. 9 is a first diagram showing an example of an image captured by a mobile terminal included in the status reading system according to the first embodiment. Fig. 9 shows an image of a sample panel 410 on which a circular meter and a circular lamp are installed, captured in a sunny outdoor environment. Fig. 9 shows how the characters on the circular meter look faint because the surroundings of the panel are bright. Fig. 9 also shows how the lit circular lamp looks dark because the surroundings of the panel are bright.

[0056] In an image captured outdoors under rainy weather, the meter and lamps shown in the image have water droplets adhering to their surroundings.

[0057] Fig. 10 is a second diagram showing an example of an image captured by the mobile terminal included in the status reading system according to the first embodiment. Fig. 10 shows an image of the same sample board 410 as in Fig. 9 captured in an outdoor environment under rainy weather. Fig. 10 shows water droplets 411 adhering to the surfaces of both the circular meter and the circular lamp.

[0058] In images of meters and lamps captured in a photograph taken in a bright indoor environment, depending on the brightness of the photographing location, glare may occur, just as in images captured in a sunny outdoor environment.

[0059] Fig. 11 is a third diagram showing an example of an image captured by a mobile terminal included in the status reading system according to the first embodiment. Fig. 11 shows an image of the same sample board 410 as in Fig. 9 captured in a bright indoor environment. Fig. 11 shows that both the circular meter and the circular lamp look slightly dark due to the brightness of the shooting location.

[0060] In an image of a meter captured in a dark indoor environment, the outlines of the letters are difficult to see because of the dark surroundings. In an image of a lamp captured in a dark indoor environment, the brightness of the lamp appears to be the same as in a bright environment, even though the surroundings are dark.

[0061] Fig. 12 is a fourth diagram showing an example of an image captured by the mobile terminal included in the status reading system according to the first embodiment. Fig. 12 shows a state in which the outline of the characters on the circular meter is difficult to see because the periphery of the panel is dark.

[0062] The object detection model 2083 is a trained model that receives a photographed image of a board and outputs an image of an object contained in the photographed image of the board, information about the board, and identification information, which is character information contained in the photographed image of the board and is information for identifying the object. The object detection model 2083 acquires, by character recognition, the names of devices involved in the operation of the object, such as "Pump No. 1" and "Pump No. 2," as device information. The object detection model 2083 also combines character recognition and object recognition to acquire character information contained inside the object, as well as the position and type of the object, as information about the object.

[0063] The server communication unit 206 is connected to the Internet 400 and communicates with devices external to the server 200. In other words, the server communication unit 206 communicates with the mobile terminal 100 via the Internet 400.

[0064] The server control unit 207 controls the overall operation of the server 200 .

[0065] Next, the learning device 250 provided in the object learning unit 205 will be described. FIG. 13 is a diagram showing the configuration of the learning device according to the first embodiment. The learning device 250 includes a data acquisition unit 251, a model generation unit 252, and a trained model storage unit 253. The data acquisition unit 251 is a first data acquisition unit in the state reading system 1. The trained model storage unit 253 may be located outside the learning device 250, or may be provided in the server storage unit 208 of the server 200.

[0066] The learning device 250 learns the image and identification information of the object based on the learning data. That is, the learning device 250 learns the relationship between the image and identification information of the object and the photographed image of the board based on the photographed image of the board included in the board table stored in the equipment database 2081 and the image of the object and the identification information, which is the information of the object, included in the object table stored in the inspection database 2082, and generates a trained model 273, which is an object detection model 2083. The learning device 250 learns the image and identification information of the object based on the learning data for multiple boards input to the learning device 250. This allows the learning device 250 to learn the learning data for multiple boards. That is, the learning device 250 acquires the image of the board from the board table, acquires the information of the object from the object table, and learns the information of the position of the object and the information of the type of the object in the image of the entire board.

[0067] The data acquisition unit 251 acquires the input information 271 and the detection information 272 of a state corresponding to the input information 271 as learning data. The input information 271 includes a captured image of an object. The detection information 272 is an image of an object corresponding to a state indicated in the input information 271 and information for identifying the object. Here, the learning data is data in which the input information 271 and the detection information 272 are associated with each other. The data acquisition unit 251 acquires the learning data from the server storage unit 208 of the server 200. The learning data may also be input to the data acquisition unit 251 from a device external to the server 200. The data acquisition unit 251 generates the learning data by associating the input information 271 with the detection information 272.

[0068] The model generation unit 252 learns the image of the object, the board information, and the identification information corresponding to the input information 271, based on the learning data created based on a combination of the input information 271 and the detection information 272 sent from the data acquisition unit 251. The model generation unit 252 generates a trained model 273 for inferring the image of the object, the board information, and the identification information corresponding to the input information 271, from the input information 271 and the detection information 272. That is, the model generation unit 252 learns the information on the position of the object and the information on the type of the object in the image of the entire board, based on the learning data.

[0069] As described above, the brightness of images of a board taken during inspection varies depending on shooting conditions such as weather and shooting location. For this reason, the model generation unit 252 also performs learning using images of the board taken in different shooting environments such as those shown in Figures 9 to 12.

[0070] Next, a processing procedure of the learning process by the learning device 250 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing a processing procedure of the learning process by the learning device according to the first embodiment.

[0071] In step S110, the data acquiring unit 251 acquires learning data. Specifically, the data acquiring unit 251 acquires the input information 271 and the detection information 272 as the learning data. The data acquiring unit 251 may acquire the input information 271 and the detection information 272 at the same timing, or may acquire them at different timings. That is, the data acquiring unit 251 may acquire the input information 271 and the detection information 272 at any timing as long as the input information 271 and the detection information 272 can be associated with each other.

[0072] In step S120, the model generation unit 252 executes a learning process according to the learning data, which is a combination of the input information 271 and the detection information 272 acquired by the data acquisition unit 251. The model generation unit 252 generates a trained model 273 by so-called supervised learning according to the learning data, for example. The model generation unit 252 performs learning to recognize the shape of an object and characters contained inside the object, and generates the trained model 273 by performing learning by supervised learning.

[0073] In step S130, the trained model storage unit 253 stores the trained model 273. That is, the model generation unit 252 stores the generated trained model 273 in the trained model storage unit 253.

[0074] The learning algorithm used by the model generation unit 252 may be a known algorithm such as supervised learning, unsupervised learning, reinforcement learning, etc. As an example, a case will be described in which the model generation unit 252 executes supervised learning using a neural network.

[0075] The model generation unit 252 learns the detection information 272 by so-called supervised learning, for example, according to a neural network model. Here, supervised learning refers to a method of providing a learning device with a set of input and result (label) data, learning the features of the learning data, and inferring the result from the input.

[0076] A neural network is composed of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer, or two or more layers.

[0077] Fig. 15 is a diagram showing the configuration of a neural network used by the learning device according to the first embodiment. For example, in the case of a three-layered neural network as shown in Fig. 15, when a plurality of pieces of data are input to the input layers X1 to X3, the values ​​are multiplied by weights w11 to w16 and input to the intermediate layers Y1 to Y2, and the results are further multiplied by weights w21 to w26 and output from the output layers Z1 to Z3. This output result changes depending on the values ​​of the weights w11 to w16 and the weights w21 to w26.

[0078] In the first embodiment, the neural network learns the detection information 272 by so-called supervised learning according to learning data (data set) created based on a combination of the input information 271 acquired by the data acquisition unit 251 and the detection information 272.

[0079] That is, the neural network learns by inputting input information 271 to input layers X1 to X3 and adjusting weights w11 to w16 and w21 to w26 so that the results output from output layers Z1 to Z3 approach detected information 272.

[0080] The model generation unit 252 executes the above-mentioned learning to generate and output a trained model 273.

[0081] The trained model storage unit 253 stores the trained model 273 output from the model generation unit 252.

[0082] Next, a specific example of the learning data acquired in step S110 described above will be described. Fig. 16 is a diagram showing an example of the learning data used by the learning device according to the first embodiment. As shown in Fig. 16, input information 271 of the learning data used by learning device 250 includes information on an image of the board, i.e., information on a photographed image of the board. Detection information 272 of the learning data used by learning device 250 includes information on an image of an object included in the board, which is teacher data corresponding to input information 271 of the learning data, and information for identifying the object. The object included in the board is an object included in a photographed image of the board.

[0083] Fig. 17 is a diagram showing an example of output of the trained model in the learning device according to the embodiment 1. When the image of the board shown in Fig. 16 is input as learning data to the trained model 273, an image of an object included in the image of the board and identification information are output as output data from the trained model 273 as shown in Fig. 17.

[0084] 18 is a diagram showing an example of the learning content for the surface of a board in the model generation unit of the learning device according to the embodiment 1. The learning device 250 learns, as board information, information on the names of devices included in a photographed image of the board, such as "Pump No. 1" and "Pump No. 2," and information on the positions of each name in the photographed image of the board.

[0085] 19 is a diagram showing an example of the learning content for a meter included in a photographed image of a board in the model generation unit of the learning device according to the embodiment 1. The model generation unit 252 learns information such as "position on board", "range", "model number", "unit", "scale value", and "scale position" for the meter included in the photographed image of the board as identification information.

[0086] 20 is a diagram showing an example of the learning content about a lamp included in a photographed image of a board in the model generation unit of the learning device according to embodiment 1. The model generation unit 252 learns information such as "position on the board," "range," "shape," "color when lit," and "color when unlit" about the lamp included in the photographed image of the board as identification information.

[0087] Next, the inference device 260 included in the object detection unit 202 will be described. FIG. 21 is a diagram showing a configuration of the inference device according to the first embodiment. The inference device 260 includes a data acquisition unit 261 and an inference unit 262. The data acquisition unit 261 is a second data acquisition unit in the status reading system 1. The inference unit 262 is connected to the learned model storage unit 253.

[0088] Information on the captured image of the board, which is included in the input information 271, is input to the data acquisition unit 261. That is, information on the captured image of the board captured by the imaging unit 103 of the mobile terminal 100 is input to the data acquisition unit 261. The inference unit 262 uses a trained model 273 obtained by the learning device 250 to output information on the image of the object corresponding to the state indicated in the input information 271 and identification information as detection information 272.

[0089] The data acquisition unit 261 acquires information on the captured image of the board as inference data. The inference data is transmitted from the mobile terminal 100 and input to the data acquisition unit 261.

[0090] The inference unit 262 outputs the detection information 272 using the obtained trained model 273. The inference unit 262 reads out the trained model 273 from the trained model storage unit 253. The inference unit 262 inputs information on the captured image of the board to the trained model 273. As a result, the inference unit 262 infers information on the image of the object, information on the board, and identification information corresponding to the state indicated in the data for inference as the detection information 272. That is, the inference unit 262 inputs information on the captured image of the board, which is data for inferring the detection information 272 acquired by the data acquisition unit 261, to the trained model 273 for inferring the detection information 272, and can output information on the image of the object, information on the board, and identification information, which are the detection information 272 inferred from the data for inference.

[0091] Next, a procedure of the inference process by the inference device 260 will be described with reference to Fig. 22. Fig. 22 is a flowchart showing the procedure of the inference process by the inference device according to the first embodiment.

[0092] In step S210, the data acquisition unit 261 acquires data for inference. That is, the data acquisition unit 261 acquires information on a captured image of the board as data for inference.

[0093] In step S220, the inference unit 262 inputs information about the photographed image of the board, which is inference data acquired by the data acquisition unit 261, to the learned model 273 stored in the learned model memory unit 253, and obtains, as inference results obtained by the learned model 273, information about the image of the object corresponding to the input information, information about the board, and identification information.

[0094] In step S230, the inference result obtained by the trained model 273 is output to the object detection unit 202. Specifically, the inference unit 262 outputs the inference result obtained by the trained model 273 to the object detection unit 202. The object detection unit 202 transmits the inference result to the state reading unit 203 as object detection information.

[0095] The inference results obtained by the trained model 273 are information on the image of the object, information on the board, and identification information corresponding to the inference data acquired by the data acquisition unit 261.

[0096] In step S240, the status reading unit 203 reads the status of the object using the object image information, which is the object detection information, and the identification information.

[0097] FIG. 23 is a diagram showing an example of input data input to the inference device according to the first embodiment. FIG. 23 shows an example of a photographed image of a board input to the data acquisition unit 261. FIG. 24 is a diagram showing an example of output data output from the inference device according to the first embodiment. FIG. 24 shows identification information from the inference result output from the inference device 260 based on the photographed image of the board shown in FIG. 23. In the example shown in FIG. 24, "device information" and "object information" are shown as the identification information. "Device information" includes information such as "device number," "device name," and "position." "Object information" includes information such as "object number," "position on board," "range," "type," "model number," "scale value," and "scale position."

[0098] Next, a description will be given of the operation of the status reading system 1 according to the first embodiment. Fig. 25 is a flowchart showing the procedure of the operation of the status reading system according to the first embodiment.

[0099] First, in step S1001, the status reading system 1 is started up in the mobile terminal 100. The terminal control unit 108 of the mobile terminal 100 transmits start-up information to the board data acquisition unit 201 of the server 200.

[0100] Next, in step S2001, the board data acquisition unit 201 of the server 200 searches for information on boards that can be used by the user. That is, when the board data acquisition unit 201 receives the startup information, it accesses the equipment database 2081 of the server storage unit 208 of the server 200 based on the startup information, and searches for information on boards that can be used by the user.

[0101] Next, in step S2002, the board data acquisition unit 201 of the server 200 acquires data on boards available to the user. That is, the board data acquisition unit 201 acquires a board table stored in the equipment database 2081, and transmits data on multiple boards stored in the board table to the board designation unit 102 of the mobile terminal 100. The board data is, for example, a board ID. The data on multiple boards stored in the board table is information on a list of board data, which is information on the equipment 300, for example, information on a list of board IDs.

[0102] Next, in step S1002, the display unit 104 of the mobile terminal 100 displays the board data. That is, when the display unit 104 receives the board data transmitted from the server 200, it displays the board data.

[0103] Next, in step S1003, the board designation unit 102 of the mobile terminal 100 designates the board to be photographed. That is, based on instructions from the user, the board designation unit 102 selects the board to be photographed from the board data acquired from the board data acquisition unit 201, and transmits board designation information, which is information on the board designated as the board to be photographed, to the board data acquisition unit 201 of the server 200. The board designation information includes a board ID or a board name as information for uniquely identifying the board. The board to be photographed is a board to be inspected, which is to be inspected using the status reading system 1.

[0104] Next, in step S2003, the board data acquisition unit 201 of the server 200 determines the board to be photographed. That is, the board data acquisition unit 201 receives board designation information transmitted from the board designation unit 102, and determines the board indicated in the board designation information as the board to be photographed.

[0105] Next, in step S1004, the photographing unit 103 of the mobile terminal 100 photographs an image including the board, and transmits information about the photographed image of the board and information about the board to the object detection unit 202 of the server 200.

[0106] Next, in step S2004, the object detection unit 202 of the server 200 receives information on the captured image of the board transmitted from the imaging unit 103 of the mobile terminal 100. The object detection unit 202 of the server 200 also receives information on the board.

[0107] Next, in step S2005, the object detection unit 202 of the server 200 detects an object appearing in the captured image of the board. That is, the object detection unit 202 inputs information on the captured image of the board to the object detection model 2083 stored in the server storage unit 208 of the server 200. The object detection model 2083 detects an object included in the board appearing in the captured image, generates information on the image of the object and information on the object, which is identification information, and outputs the information to the object detection unit 202. The object detection unit 202 also acquires information on the specified board from the object table. The object detection unit 202 transmits the information on the specified board, the image of the object, and the object information on the object to the state reading unit 203 as object detection information. Note that, in a typical board, the number of objects appearing in the captured image is more than one.

[0108] Next, in step S2006, the status reading unit 203 of the server 200 reads the object status. That is, the status reading unit 203 reads the object status, which is the state of the object, from the information of the image of the object using image processing based on the information of the image of the object and the identification information acquired from the object detection unit 202, and acquires information on the state of the object.

[0109] Status reading unit 203 acquires information on the status of each object detected by object detection unit 202, i.e., for each piece of image information of the object acquired from object detection unit 202. That is, status reading unit 203 repeats the process of reading the object status, i.e., the process of acquiring information on the object status, a number of times corresponding to the number of objects detected by object detection unit 202.

[0110] Fig. 26 is a first flowchart showing an example of a procedure of a process for reading the status of an object in the status reading unit of the server included in the status reading system according to the embodiment 1. Fig. 26 shows a case where the status reading unit 203 reads the status of a circular meter, which is the object.

[0111] First, in step S310, the status reading unit 203 acquires an image of a circular meter, which is information on an image of an object, and identification information of the circular meter.

[0112] Next, in step S320, the status reading unit 203 associates the panel, which is a device, with the circular meter based on the panel information and the information on the position of the circular meter on the panel included in the identification information. That is, the status reading unit 203 groups the objects based on the object position information and the device position information. For example, the status reading unit 203 classifies the object into either "Pump No. 1" or "Pump No. 2".

[0113] Next, in step S330, the status reading unit 203 extracts an image of the area within which the needle of the circular meter moves from the image of the circular meter, which is information about the image of the object.

[0114] Next, in step S340, the status reading unit 203 detects the value of the needle of the circular meter, i.e., the value indicated by the needle of the circular meter, from the image information of the circular meter, which is the image information of the object. In this way, the status of the circular meter is read.

[0115] For example, in the case of a meter, the status reading unit 203 obtains the range in which the needle moves by using the position of the scale included in the identification information obtained in step S310. Furthermore, the status reading unit 203 extracts an image included in the above range. Then, in the case of a meter, for example, the status reading unit 203 associates the image obtained as described above with the value of the scale, detects the position of the needle from the change in pixel values ​​in the range in which the needle moves using a histogram or the like, and detects the value taken by the needle.

[0116] For example, in the case of a lamp, the status reading unit 203 obtains the luminance, i.e., pixel value, of the central part of the image of the lamp included in the identification information obtained in step S310. The status reading unit 203 then compares the luminance value obtained as described above with a luminance value for when the lamp is on or a luminance value for when the lamp is off that is defined in advance in the status reading unit 203, to detect the status of the lamp.

[0117] Fig. 27 is a second flowchart showing an example of the procedure of the reading process of the status of the object in the status reading unit of the server included in the status reading system according to the embodiment 1. Fig. 27 shows a case where the status reading unit 203 reads the status of a lamp, which is the object.

[0118] First, in step S410, the status reading unit 203 acquires an image of a lamp, which is information on an image of an object, and identification information of the lamp.

[0119] Next, in step S420, the status reading unit 203 associates the board, which is the device, with the lamp based on the information on the position of the lamp included in the identification information.

[0120] Next, in step S430, the status reading unit 203 obtains information on the luminance of the center of the lamp from the image of the lamp, which is information on the image of the object.

[0121] Next, in step S440, the status reading unit 203 compares the luminance information of the center of the lamp with a predetermined reference value of luminance to determine the status of the lamp. In this way, the status reading unit 203 obtains information on the status of the lamp, that is, whether the lamp is in a lit state or an unlit state. In this way, the status of the lamp is read.

[0122] Next, in step S2007, the status reading unit 203 of the server 200 tally up the information on the reading results of the object. That is, the status reading unit 203 ties up the image information of the object, the board information, and the object information for the object, and tally up the information. As a result, for each object, the object type information is linked to the object state information read in step S2006. The status reading unit 203 transmits the board information, the image information of the object, the object information for the object, and the status reading result information, which is the information on the reading result of the object obtained by tallying up the acquired object state information, to the display unit 104 of the mobile terminal 100. Note that, hereinafter, the status reading result may be referred to as the reading result.

[0123] Next, in step S1005, the display unit 104 of the mobile terminal 100 displays the reading result of the entire board. That is, when the display unit 104 receives the information transmitted from the status reading unit 203 of the server 200, it displays information on the image of the object, object information about the object, and information on the reading result of the object for the entire board.

[0124] Fig. 28 is a diagram showing an example of a reading result display screen displayed on a display unit of a mobile terminal included in the status reading system according to the first embodiment. Fig. 28 shows a reading result display screen on which information on the reading result of an object, Pump No. 1, which is an object, is displayed. The reading result display screen shown in Fig. 28 displays a photographed image of the panel, information on Pump No. 1 which is information on the panel, and information on the reading result of the circular meter and circular lamp which is information on the reading result of the object.

[0125] On the reading result display screen shown in Fig. 28, the position of the object detected by the object detection unit 202 is displayed on top of the captured image with a number and a dashed line. Also, on the reading result display screen shown in Fig. 28, the name and installation location of the panel to be photographed specified by the panel specification unit 102 are displayed. Also, on the reading result display screen shown in Fig. 28, the inspection results for each group associated by the status reading unit 203, that is, the inspection results for each of Pump No. 1 and Pump No. 2, are displayed as inspection results.

[0126] Next, in step S1006, the terminal control unit 108 of the mobile terminal 100 determines whether the reading result of the object has been confirmed. That is, the terminal control unit 108 determines that the reading result of the object has been confirmed when the user inputs confirmation information to the terminal control unit 108 to confirm the reading result of the object. The terminal control unit 108 determines that the reading result of the object has not been confirmed when the user inputs discard information to the terminal control unit 108 to discard the reading result of the object.

[0127] In the reading result display screen shown in FIG. 28, when the "Register" button is selected, confirmation information indicating that the reading result of the object is confirmed is input to the terminal control unit 108. In addition, in the reading result display screen shown in FIG. 28, when the "Retake" button is selected, discard information indicating that the reading result of the object is discarded is input to the terminal control unit 108. The user selects the "Retake" button, for example, when the state of the object is not properly read, or when the information of the reading result of the state of the object displayed on the reading result display screen does not match the state of the object that the user is actually viewing. In the reading result display screen shown in FIG. 28, the group can be switched by the "Next" button and the "Previous" button. In the reading result display screen shown in FIG. 28, when the "Next" button is selected, a reading result display screen is displayed in which information of the reading result of the object for Pump No. 2 is displayed. In the reading result display screen shown in FIG. 28, when the "Cancel" button is selected, the status reading system is terminated. When the “Confirm” button is selected on the read result display screen shown in FIG. 28, the photographed image, board information, and inspection result information are transmitted to the server 200.

[0128] If it is determined that the reading result of the object has been confirmed, the result is Yes in step S1006, and the terminal control unit 108 transmits the information displayed on the reading result display screen to the reading result storage unit 204 of the server 200, and the process proceeds to step S2008. That is, the terminal control unit 108 transmits to the reading result storage unit 204 of the server 200 the board information, the object image information, the object information about the object, and the status reading result information which is the information on the reading result of the object obtained by aggregating the obtained information on the object status.

[0129] If it is determined that the reading result of the object has not been confirmed, the result is No in step S1006, the terminal control unit 108 discards the information displayed on the reading result display screen, and the process returns to step S1004.

[0130] In step S2008, the read result storage unit 204 of the server 200 compiles the data received from the mobile terminal 100 into one file of the reading results of the object, and registers it in the inspection database 2082 of the server storage unit 208 of the server 200. That is, the read result storage unit 204 compiles the board information, the object image information, the object information about the object, and the condition reading result information, which is the information of the reading results of the object obtained by aggregating the acquired information on the object's condition, into one file of the reading results of the object, and stores it in the inspection database 2082 as an inspection result table.

[0131] Furthermore, the read result storage unit 204 transmits a file of the compiled read results of the object to the output unit 105 of the mobile terminal 100.

[0132] Next, in step S1007, the output unit 105 of the mobile terminal 100 stores the file of the reading result of the object in the terminal storage unit 106 of the mobile terminal 100.

[0133] Fig. 29 is a diagram showing an example of an output of a file of the reading result of an object on the display unit of a mobile terminal included in the condition reading system according to the first embodiment. Fig. 29 shows an inspection result display screen on which information on the reading result of the board is displayed using the file of the reading result of the object. The inspection result display screen shown in Fig. 29 displays the photographed image of the board, the board information, and information on the condition reading result which is information on the reading result of the object.

[0134] In the above, the status reading system 1 reads the status of an instrument mounted on a panel, but the inspection target is not limited to this. The status reading system 1 can also be applied to reading the status of instruments such as a speedometer meter and a lamp indicating the remaining amount of gasoline, etc., installed in equipment such as an automobile.

[0135] In the condition reading system 1 according to the above-mentioned first embodiment, a learning device is realized which includes a data acquisition unit which acquires learning data including a captured image of a panel surface on which multiple instruments are mounted on a panel to be inspected, images of each of the multiple instruments included in the captured image, and identification information which is information for identifying and specifying each of the multiple instruments, and a model generation unit which uses the learning data to generate a trained model for inferring the image and the identification information of each of the multiple instruments from the captured image.

[0136] The state reading system 1 inputs a photographed image of the board, and stores in the server storage unit 208 an object detection model 2083, which is a trained model that outputs an image of an object contained in the photographed image of the board and identification information, which is character information contained in the photographed image of the board and information for identifying the object. The object detection unit 202 of the server 200 inputs information of the photographed image of the board to the object detection model 2083, thereby detecting an object contained in the board shown in the photographed image, and generates information on the image of the object and identification information, which is information on the object about the object. The state reading unit 203 of the server 200 uses image processing to read the object state, which is the state of the object, from the information on the image of the object, based on the information on the image of the object and the identification information acquired from the object detection unit 202, and acquires information on the state of the object.

[0137] As a result, the status reading system 1 can read the status of all objects of multiple types and numbers mounted on the board from a single photographed image of the board and output the reading results of all objects simultaneously, making inspection work easier and more efficient.

[0138] As a result, the condition reading system 1 eliminates the need for the inspector to define and create in advance identification information that links the type of board to be read and the model number of the board with an image of the object, and the need to specify the position of the object, making inspection work easier and more efficient.

[0139] Therefore, according to the status reading system 1 of embodiment 1, an effect is achieved in which a learning device is obtained that makes it possible to read the status of multiple instruments from a single photographed image of the panel without using definition information created by an inspector.

[0140] Embodiment 2 FIG. 30 is a diagram showing the flow of information between a mobile terminal and a server included in a status reading system according to the second embodiment. FIG. 30 shows extracted components used to explain the operation of a status reading system 2. The status reading system 2 according to the second embodiment basically has the same configuration and functions as the status reading system 1 according to the first embodiment. The status reading system 2 according to the second embodiment includes a mobile terminal 100 and a server 200a. The server 200a is obtained by adding a detection result determination unit 209 to the server 200 according to the first embodiment.

[0141] The detection result determination unit 209 determines whether or not all objects included in the image of the object have been detected by the object detection unit 202 by comparing the defined number of objects, which is the number of objects per board defined in the object table, with information on the number of objects detected by the object detection unit 202.

[0142] In the state reading system 2, the trained model 273 also outputs the number of objects, i.e., information on how many objects are included in the image of the object input to the trained model 273. That is, the object detection model 2083 also outputs the number of objects, i.e., information on how many objects are included in the image of the object input to the trained model 273. Therefore, in the second embodiment, the learning device 250 learns the number of objects for each type of object.

[0143] The learning device 250 in the status reading system 2 basically performs learning in the same manner as the learning device 250 in the status reading system 1 according to the first embodiment. FIG. 31 is a diagram showing an example of learning data used by the learning device in the status reading system according to the second embodiment. As shown in FIG. 31, the input information 271 of the learning data used by the learning device 250 in the status reading system 2 includes information on an image of the board, that is, information on a photographed image of the board. Furthermore, the detection information 272 of the learning data used by the learning device 250 includes information on an image of an object included in the board, information on the number of objects included in the board, and information for identifying the object, which are teacher data corresponding to the input information 271 of the learning data. The number of objects included in the board is distinguished according to the type of object.

[0144] 32 is a diagram showing an example of the learning content for the surface of a board in the learning unit of the learning device according to the second embodiment. The model generation unit 252 learns, as identification information, information on the names of devices included in the photographed image of the board, such as "Pump No. 1" and "Pump No. 2," and information on the positions of each name in the photographed image of the board. The learning device 250 learns the number of objects for each type of object.

[0145] The learning content of model generation unit 252 of learning device 250 in embodiment 2 is the same as that in embodiment 1, but model generation unit 252 learns so that the number of pieces of object position information and the number of pieces of object type information detected from the photographed image of the board match the number of objects. That is, although the number of objects is described as the learning content in Fig. 32, this description does not mean that model generation unit 252 learns to output the number of objects itself, but rather that model generation unit 252 learns so that the number of pieces of object position information and the number of pieces of object type information detected from the photographed image of the board match the number of objects.

[0146] Moreover, the output from the trained model 273 is the same as in the case of embodiment 1. That is, when the information on the image of the board shown in FIG. 31 is input as learning data to the trained model 273, the output data from the trained model 273 is the same as in the case of embodiment 1.

[0147] Next, the operation of the status reading system 2 according to the second embodiment will be described. Fig. 33 is a flowchart showing the procedure of the operation of the status reading system according to the second embodiment. Below, the differences from the operation of the status reading system 1 according to the first embodiment shown in Fig. 25 will be described.

[0148] In the second embodiment, in step S2002, the board data acquisition unit 201 of the server 200 acquires data on boards that can be used by the user, as described above. The board data acquisition unit 201 also acquires information on the number of objects linked to the board IDs of boards that can be used by the user, and transmits the information on the number of objects to the detection result determination unit 209. The board data acquisition unit 201 acquires an object table stored in the inspection database 2082, and acquires information on the number of objects linked to the board IDs of boards that can be used by the user from the object table. The board data acquisition unit 201 acquires information on the number of objects for each board for all boards that can be used by the user. The board data acquisition unit 201 transmits information on the number of objects for all boards that can be used by the user to the detection result determination unit 209.

[0149] 7, the object table stores information about objects provided on each board, linked by the board ID. That is, the object table defines information about the number of objects on each board, and includes information about the defined number of objects, which is the number of objects on each board.

[0150] In the second embodiment, in step S2005, object detection unit 202 transmits information about the board, information about the captured image of the object, object information about the object, and information about the number of objects included in the captured image of the object as object detection information to detection result determination unit 209. The information about the number of objects acquired by object detection unit 202 from object detection model 2083 is, for example, information such as 2 lamps or 4 meters.

[0151] In the second embodiment, in step S2011, the detection result determination unit 209 determines whether or not all objects have been detected. The detection result determination unit 209 acquires information on the number of objects detected by the object detection unit 202 from the object detection unit 202. The detection result determination unit 209 then compares the information on the defined number, i.e., the information on the number of objects acquired from the board data acquisition unit 201, with the information on the number of objects detected by the object detection unit 202. The information on the defined number, i.e., the information on the number of objects acquired from the board data acquisition unit 201, and the information on the number of objects detected by the object detection unit 202 can be linked by a board ID.

[0152] The detection result determination unit 209 determines that all objects have been detected when the number of objects detected by the object detection unit 202 matches the defined number. The detection result determination unit 209 determines that all objects have not been detected when the number of objects detected by the object detection unit 202 does not match the defined number.

[0153] If it is determined that all objects have been detected, the result in step S2011 is Yes, and the detection result determination unit 209 transmits object image information, which is object detection information, object information about the object, and board information to the status reading unit 203, and the process proceeds to step S2006. If it is determined that all objects have not been detected, the result in step S2011 is No, and the process returns to step S1004.

[0154] Fig. 34 is a flowchart showing a detailed procedure of a characteristic operation of the status reading system 2 according to the embodiment 2. Fig. 34 shows a characteristic process in the status reading system 2 according to the embodiment 2.

[0155] Step S510 corresponds to step S2002 described above, in which the panel data acquisition unit 201 of the server 200 acquires data on a plurality of panels stored in a panel table stored in the equipment database 2081.

[0156] Step S520 corresponds to step S2002 described above, in which the board data acquisition unit 201 of the server 200 acquires information on the number of objects linked to the board ID of the board that the user can use, and transmits the information on the number of objects to the detection result determination unit 209.

[0157] Step S530 corresponds to step S2004 described above, and the object detection unit 202 of the server 200 receives information about the captured image of the board transmitted from the image capturing unit 103 of the mobile terminal 100.

[0158] Step S540 corresponds to step S2005 described above, and the object detection unit 202 of the server 200 acquires information on the image of the object, object information about the object, and information on the number of objects from the object detection model 2083.

[0159] Step S550 corresponds to step S2011 described above, where the detection result determination unit 209 determines whether or not the defined number, which is the defined number, matches the number of objects detected by the object detection unit 202. If the answer is Yes in step S550, the characteristic process of the status reading system 2 according to the second embodiment ends. If the answer is No in step S550, the process returns to step S530, and the captured image is acquired again.

[0160] The above-mentioned condition reading system 2 according to the second embodiment includes a detection result determination unit 209 that determines whether or not all objects included in the image of the object have been detected by the object detection unit 202 by comparing the defined number of objects, which is the number of objects per board defined in the object table, with information on the number of objects detected by the object detection unit 202. This allows the condition reading system 2 to reliably read the conditions of all objects included in the image of the object.

[0161] Embodiment 3 FIG. 35 is a diagram showing the flow of information between a mobile terminal and a server included in the status reading system according to the third embodiment. FIG. 35 shows extracted components used to explain the operation of the status reading system 3 described later. The status reading system 3 according to the third embodiment basically has the same configuration and functions as the status reading system 2 according to the second embodiment. The status reading system 3 according to the third embodiment includes a mobile terminal 100a and a server 200a. The mobile terminal 100a is obtained by adding a reading result editing unit 109 to the mobile terminal 100 according to the second embodiment.

[0162] The read result editing unit 109 has a function of editing information on the expression of the read result transmitted from the server 200a into an arbitrary character string based on an instruction input by the user. The expression on the read result includes an expression on the state of the object, such as ON or OFF, and a unit. That is, the read result editing unit 109 edits the expression of the read result of the entire board displayed on the display unit 104 in the above-mentioned step S1005 into an arbitrary character string based on an instruction input by the user. Therefore, the status reading system 3 is obtained by adding a function for an inspector to correct the read result to the status reading system 2 according to the second embodiment.

[0163] Next, the operation of the status reading system 3 according to the third embodiment will be described. Fig. 36 is a flowchart showing the procedure of the operation of the status reading system according to the third embodiment. Below, the differences from the operation of the status reading system 2 according to the second embodiment shown in Fig. 33 will be described.

[0164] If it is determined in step S1006 that the reading result of the object has been confirmed, the result is Yes in step S1006, and the process proceeds to step S1011.

[0165] Next, in step S1011, the read result editing unit 109 of the mobile terminal 100a edits the expression related to the read result into an arbitrary character string based on an instruction input by the user.

[0166] The reading result editing unit 109 acquires information on the output value of the reading result held by the terminal control unit 108. That is, the reading result editing unit 109 acquires information on the board, information on the image of the object, information on the object about the object, and information on the state reading result, which is information on the reading result of the object obtained by aggregating the acquired information on the state of the object, from the terminal control unit 108. The reading result editing unit 109 displays the reading result display screen and the reading result editing screen on the display unit 104, and modifies the information on the expression related to the state reading result displayed on the reading result display screen according to a user operation. Then, the reading result editing unit 109 updates the display content of the reading result editing screen to correspond to the modified content.

[0167] Fig. 37 is a diagram showing an example of a reading result display screen displayed on a display unit of a mobile terminal included in the status reading system according to embodiment 3. Fig. 38 is a first diagram showing an example of a reading result editing screen displayed on a display unit of a mobile terminal included in the status reading system according to embodiment 3. Fig. 39 is a second diagram showing an example of a reading result editing screen displayed on a display unit of a mobile terminal included in the status reading system according to embodiment 3.

[0168] In the reading result display screen shown in Fig. 37, the "Register" button in the reading result display screen shown in Fig. 28 has been changed to a "Confirm" button. When the "Confirm" button is selected in the reading result display screen shown in Fig. 37, confirmation information to confirm the reading result of the object is input to reading result editing unit 109. Then, reading result editing unit 109 causes display unit 104 to display the reading result editing screen shown in Fig. 38.

[0169] When the "Edit" button is selected on the read result editing screen shown in FIG. 38, the read result editing screen is updated, and the read result editing screen shown in FIG. 39 is displayed on the display unit 104. On the read result editing screen, a popup for editing is displayed, and an editing form for inputting an input value of editing data for editing the read result is displayed. When an inspector inputs an input value into the editing form and selects the "Update" button, the read result editing unit 109 corrects and updates the information of the status read result held in accordance with the contents of the input value. In addition, the read result editing unit 109 closes the popup of the read result editing screen, and updates the display contents of the read result shown on the read result editing screen. When the display contents of the read result shown on the read result editing screen are updated, the updated part is highlighted, for example, by being displayed in red. In addition, the read result editing screen is provided with a "Clear" button for clearing the edited data.

[0170] Thereafter, the read result editing unit 109 transmits the information displayed on the updated read result editing screen to the read result storage unit 204 of the server 200a, and proceeds to step S2008. That is, the read result editing unit 109 transmits the board information, the object image information, the object information about the object, and the edited state read result information, which is the edited state read result information, to the read result storage unit 204 of the server 200a. The edited state read result information reflects the editing of the state read result by the inspector. Note that if the state read result is not edited in step S1011, the state read result information is transmitted to the read result storage unit 204, instead of the edited state read result information.

[0171] The above-described functions of the status reading system 3 are also applicable to the status reading system 1 and the status reading system 2 described above.

[0172] The condition reading system 3 according to the third embodiment described above has a reading result editing unit 109 that edits the information on the expression of the reading result transmitted from the server 200a into an arbitrary character string based on an instruction input by the user. This allows the condition reading system 3 to correct the information on the expression of the reading result into an appropriate expression by the user, thereby obtaining a more appropriate inspection result.

[0173] Embodiment 4 Fig. 40 is a diagram showing the flow of information between a mobile terminal and a server included in the status reading system according to the fourth embodiment. Fig. 40 shows extracted components used to explain the operation of the status reading system 4 described later. The status reading system 4 according to the fourth embodiment basically has the same configuration and function as the status reading system 3 according to the third embodiment. In the fourth embodiment, the information on the board and the correspondence between the reading result of the target object and the inspection items written in the report are used to obtain the entry location for the status reading result in the report, and the reading result is entered in the report and output.

[0174] The status reading system 4 according to the fourth embodiment includes a mobile terminal 100a and a server 200b. The server 200b includes a report format acquisition unit 210, a report entry position acquisition unit 211, and a report storage unit 212 in addition to the server 200a according to the third embodiment.

[0175] After the board to be photographed is determined, the report format acquisition unit 210 acquires a report format ID corresponding to the board ID of the board to be photographed, acquires a report format table stored in the inspection database 2082 based on the acquired report format ID, and acquires information on inspection items that match the report format ID. The information on inspection items acquired by the report format acquisition unit 210 includes, for example, information such as the inspection items and the object ID.

[0176] The report format table is information that defines the inspection items for each format of the inspection result report. The inspection items defined in the report format table include, for example, the equipment to be inspected, the name of the inspection item, and the method of inputting the inspection results. In the report format table, the inspection item ID, which is individual identification information, is linked to report format information items such as the report format ID, inspection target, item name, input method, and object ID.

[0177] The report entry position acquisition unit 211 acquires information on the status reading result from the reading result editing unit 109, and acquires the inspection item for which the reading result is to be entered in the report from the correspondence between the object information included in the status reading result and the inspection item information acquired by the report format acquisition unit 210. The object information here includes information such as the object name and object ID.

[0178] The report storage unit 212 writes in the report the inspection item names and the status reading results defined in the report format table according to the report format. The report here is an electronic document. The report storage unit 212 also creates a file of the information on the completed report and transmits the file to the mobile terminal 100a. The report storage unit 212 also stores the current inspection results in the inspection database 2082. That is, the report storage unit 212 writes in the report the information on the reading results corresponding to the inspection items for the board to be inspected that are written in the report, which is a predetermined electronic document.

[0179] Next, the operation of the status reading system 4 according to the fourth embodiment will be described. Fig. 41 is a flowchart showing the procedure of the operation of the status reading system according to the fourth embodiment. Below, the differences from the operation of the status reading system 3 according to the third embodiment shown in Fig. 36 will be described.

[0180] In the fourth embodiment, in step S1011, the read result editing unit 109 transmits information to the report entry position acquisition unit 211, and the process proceeds to step S2021.

[0181] Fig. 42 is a diagram showing an example of a panel table stored in an equipment database of a server included in the status reading system according to the embodiment 4. Fig. 43 is a diagram showing an example of a report format table stored in an inspection database of a server included in the status reading system according to the embodiment 4. Since there are cases where multiple panels are inspected using the same report format, multiple panel IDs correspond to one report format ID.

[0182] The panel table shown in FIG. 42 is linked to the object table shown in FIG. 7 by a panel ID. The object table shown in FIG. 7 is linked to the report format table shown in FIG. 43 by an object ID. Therefore, the panel table shown in FIG. 42 is linked to the report format table shown in FIG. 43 by the panel ID and the object ID. For example, "Object ID: T001" in the object table shown in FIG. 7 corresponds to "Type: Circular Meter", which is an object indicating "Name: Pump 1 Output". "Object ID: T001" in the object table shown in FIG. 7 is linked to "Item Name: Output" in the report format table shown in FIG. 43.

[0183] "Pump equipment 01" and "Motor 01," which have the same report format ID "F001" in the panel table shown in FIG. 42, have the same "items" for inspection listed in the report format table shown in FIG. 43. This "item" corresponds to the "item name" in FIG. 43. The same report format ID means that the image of the panel is the same, and the "items" for inspection are the same. In other words, the same report format ID means that the panel is the same, and different report format IDs mean that the panel is different. Report format IDs are prepared individually for each panel. In other words, during inspection, the "items" for inspection are set individually for each panel.

[0184] In step S2021, the report format acquisition unit 210 acquires the report format of the board. The report format acquisition unit 210 acquires the board ID of the board to be photographed from the board data acquisition unit 201 of the server 200b. The report format acquisition unit 210 acquires the board table shown in FIG. 42 stored in the equipment database 2081, and acquires multiple report format IDs corresponding to the acquired board ID from the board table. The report format acquisition unit 210 acquires the report format table shown in FIG. 43 stored in the inspection database 2082, and acquires, as the report format, information on the inspection item whose report format ID matches from the report format table based on the acquired report format ID. The report format acquisition unit 210 acquires information such as the item name and the object ID as the information on the inspection item.

[0185] Next, in step S2022, the report book entry position acquisition unit 211 acquires information on the location where the reading result is to be entered on the report book. That is, the report book entry position acquisition unit 211 associates the information on the inspection item included in the report book format acquired from the report book format table with the information on the target object included in the information on the post-editing state reading result acquired from the reading result editing unit 109, and associates the inspection item with the reading result. The report book entry position acquisition unit 211 acquires the entry location on the report book for the item where the inspection item and the reading result match, as information on the location where the reading result is to be entered on the report book.

[0186] Next, in step S2023, the report storage unit 212 generates a report file that summarizes the information of the edited state reading result in the form of a report, and registers the report file in the inspection database 2082 of the server storage unit 208 of the server 200b. The report storage unit 212 acquires the information of the edited state reading result and information of the position where the reading result is to be entered on the report from the report entry position acquisition unit 211.

[0187] Furthermore, the transcript storage unit 212 transmits the transcript file to the output unit 105 of the mobile terminal 100a.

[0188] Next, in step S1021, the output unit 105 of the mobile terminal 100a stores the transcript file in the terminal storage unit 106 of the mobile terminal 100a.

[0189] Fig. 44 is a flowchart showing a detailed procedure of a characteristic operation of the status reading system 4 according to the embodiment 4. Fig. 44 shows a characteristic process in the status reading system 4 according to the embodiment 4.

[0190] Step S610 corresponds to step S2021 described above, where the report format acquisition unit 210 of the server 200b acquires information on inspection items from the report format table as the report format of the report linked to the board ID to be read.

[0191] Steps S620 to S660 correspond to steps S2022 and S2023 described above, and the processing is repeated for each object included in the information of the post-editing state reading result, that is, a number of times corresponding to the number of objects detected by object detection unit 202.

[0192] In step S630, the report entry position acquisition unit 211 of the server 200b acquires information on the object acquired by the condition reading unit 203. That is, the report format acquisition unit 210 acquires information on the edited condition reading result from the reading result editing unit 109.

[0193] In step S640, the report card entry position acquisition unit 211 determines whether or not an entry field for the read result exists on the report card.

[0194] If the answer is Yes in step S640, the report entry position acquisition unit 211 associates the inspection item with the reading result, that is, associates the entry field of the reading result with the reading result included in the information of the post-editing state reading result, and proceeds to step S650. If the answer is No in step S640, proceeds to step S660, and the process of step S2022 ends.

[0195] In step S650, the report storage unit 212 of the server 200b writes the information of the post-editing state read result into the report format and generates a report file compiled in the report format.

[0196] Fig. 45 is a diagram showing an example of a reading result display screen displayed on a display unit of a mobile terminal included in a status reading system according to embodiment 4. Fig. 45 shows a reading result display screen on which information on the reading result of an object, Pump No. 1, which is an object, is displayed. The reading result display screen shown in Fig. 45 displays the same contents as the reading result display screen in embodiment 3 shown in Fig. 38.

[0197] Fig. 46 is a diagram showing an example of output of a report file on the display unit of a mobile terminal included in the status reading system according to the fourth embodiment. Fig. 46 shows an inspection result display screen on which information on the results of reading the board is displayed using the report file. The following describes the points where the inspection result display screen according to the fourth embodiment shown in Fig. 46 differs from the inspection result display screen according to the first embodiment shown in Fig. 29.

[0198] In the inspection result display screen in the first embodiment shown in Fig. 29, the reading results are output for each type of object, such as for each circular meter, for each circular lamp, etc. On the other hand, in the inspection result display screen in the fourth embodiment shown in Fig. 46, the reading results are output for each group of object, such as for pump No. 1, pump No. 2, etc.

[0199] In the inspection result display screen in the first embodiment shown in FIG. <1> On the other hand, in the inspection result display screen in the fourth embodiment shown in FIG. 46, the inspection item and the reading result are output in that order, such as (output: 50%).

[0200] In the inspection result display screen in the first embodiment shown in Fig. 29, a dashed line indicating the position of the object and an identification number are output on the captured image. On the other hand, in the inspection result display screen in the fourth embodiment shown in Fig. 46, the dashed line indicating the position of the object and the identification number are not output.

[0201] The above-mentioned functions of the status reading system 4 are also applicable to the above-mentioned status reading systems 1 to 3.

[0202] The condition reading system 4 according to the above-mentioned fourth embodiment includes a report storage unit 202 that automatically writes in a report, which is a predetermined electronic document, information on the reading results corresponding to the inspection items for the board. This allows the condition reading system 4 to efficiently and automatically write in a report the information on the reading results of the conditions of all of the multiple types and multiple objects mounted on the board, which is read from a photographed image of one board, and to realize the efficiency of inspection work on site.

[0203] Embodiment 5. FIG. 47 is a diagram showing the configuration of a server provided in the status reading system according to the fifth embodiment. FIG. 48 is a diagram showing the flow of information between a mobile terminal and a server provided in the status reading system according to the fifth embodiment. FIG. 48 shows extracted components used in explaining the operation of the status reading system 5 described later. The status reading system 5 according to the fifth embodiment basically has the same configuration and functions as the status reading system 4 according to the fourth embodiment. In the fifth embodiment, the reading result of the object is input to a pattern that outputs the operating status of the board from the state of the object, and the state of the board, such as operating or abnormal, is determined.

[0204] The status reading system 5 according to the fifth embodiment includes a mobile terminal 100a and a server 200c. The server 200c includes an operation status determining unit 213, an operation status learning unit 214, and an operation status determining pattern 2084 in addition to the components of the server 200b according to the fourth embodiment.

[0205] The operational state determination unit 213 inputs the information on the result of reading the object received from the mobile terminal 100a to the operational state determination pattern 2084, and acquires the operational state of the board that is the subject of the image capture.

[0206] The operation state determination pattern 2084 is a pattern that outputs information on the operation state of the board on which an object is installed by inputting board information including the board ID and the reading result of the object, and is stored in the server storage unit 208. The information on the operation state of the board includes information on operation, failure, etc. In other words, the operation state determination pattern 2084 defines a combination of the reading result of the object and the operation state of the board for the reading result. The operation state determination pattern 2084 is a pattern created based on the inspection result, rather than a learned model by machine learning.

[0207] The operating state learning unit 214 learns the operating state determination pattern 2084 by supervised learning. That is, the operating state learning unit 214 learns the relationship between the state of the object and the operating state of the panel. The operating state learning unit 214 acquires information on the state of the object and information on items related to the operating state of the panel from information on past inspection results of the panel stored in a report table stored in an inspection result table stored in the inspection database 2082. The operating state learning unit 214 acquires information on a combination of information on the state of the object, such as the value of a meter and the state of a lamp, and information on the operating state of the panel corresponding to the information on the state of the object, from the acquired information on the inspection results, to create a pattern, and stores the created pattern in the operating state determination pattern 2084.

[0208] Fig. 49 is a diagram showing an example of an inspection result table stored in an inspection database of a server included in a status reading system according to the fifth embodiment. As shown in Fig. 49, the inspection result table has a plurality of items of information relating to the inspection result, such as an inspection result ID, an inspection item ID, an inspection date, and an inspection result, and stores corresponding information. The inspection result ID is identification information unique to the inspection result. The inspection item ID is identification information unique to the inspection item.

[0209] A description will be given of a procedure for learning the operation state determining pattern 2084 by the operation state learning unit 214. Fig. 50 is a flowchart showing a procedure for learning the operation state determining pattern 2084 by the operation state learning unit according to the fifth embodiment.

[0210] In step S710, the operating state learning unit 214 acquires learning data. Specifically, the operating state learning unit 214 acquires information on the object to be inspected from the object table as learning data. In addition, the operating state learning unit 214 acquires information on the state of the object based on the information on the object from information on the past inspection results of the panel stored in the report table stored in the inspection result table. That is, the operating state learning unit 214 acquires information on the past inspection results of the panel, which is information on the state of the object, from the report table as learning data. The information on the object to be inspected includes information such as the meter type and the output range of the object. The report table is a table in which the inspection date and the inspection result of each inspection item are added to the report format table, and is stored in the inspection result table of the inspection database 2082.

[0211] Fig. 51 is a diagram showing an example of learning data used by the operating status learning unit according to the fifth embodiment. As shown in Fig. 51, the learning data used by the operating status learning unit 214 includes, as information on the object, the name of the object, such as Pump No. 1, information on the state of the object, such as the meter value and the lamp state, and information on the inspection result of the panel, such as state: operating, state: stopped, and state: operating (lamp failure).

[0212] In step S720, the operating status learning unit 214 learns the pattern. The operating status learning unit 214 creates a combination of the value taken by the object in the information on the state of the object acquired in step S710 and the inspection result information, which is the state of the panel corresponding to the information on the state of the object. For example, the operating status learning unit 214 creates a set of learning data including information such as "Input: circular meter No. 1 is 50%, lamp No. 1 is on" and "Output: operating status". Then, the operating status learning unit 214 provides the set of learning data to the pattern and learns the pattern.

[0213] In step S730, the operating state learning unit 214 inputs the learned pattern to the operating state determination pattern 2084, and stores the learned pattern in the operating state determination pattern 2084.

[0214] Fig. 52 is a diagram showing an example of an output of an operating state determination pattern in the learning device according to embodiment 5. When information on the state of an object among the learning data shown in Fig. 51 is input as learning data to operating state determination pattern 2084, information on the operating state of the panel is output as output data from operating state determination pattern 2084 as shown in Fig. 52.

[0215] Through the above processing, the operating state learning unit 214 learns the operating state determining pattern 2084.

[0216] Next, the operation of the status reading system 5 according to the fifth embodiment will be described. Fig. 53 is a flowchart showing the procedure of the operation of the status reading system according to the fifth embodiment. The following describes the differences from the operation of the status reading system 4 according to the fourth embodiment shown in Fig. 41.

[0217] In the fifth embodiment, in step S1011, the reading result editing unit 109 of the mobile terminal 100a transmits information to the operating state determination unit 213, and the process proceeds to step S2031. The reading result editing unit 109 transmits board information including the board ID, object image information, object information about the object, and edited status reading result information, which is edited status reading result information, to the operating state determination unit 213 of the server 200c.

[0218] In step S2031, the operating state determination unit 213 of the server 200c determines the operating state of the board to be photographed. The operating state determination unit 213 inputs the board information including the board ID and the information of the reading result of the object included in the information of the edited state reading result received from the reading result editing unit 109 to the pattern for operating state determination 2084 to acquire information on the operating state of the board to be photographed. The pattern for operating state determination 2084 acquires multiple patterns related to the operating state of the object based on the board ID of the board information during the above-mentioned learning. The pattern is information of a combination of information on the state of the object and information on the operating state of the board corresponding to the information on the state of the object. Then, in step S2031, the pattern for operating state determination 2084 compares the information of the reading result of the object included in the information of the edited state reading result with the multiple patterns related to the operating state of the object held, and outputs a result having a high similarity to the information of the reading result of the object as information on the operating state of the board to be photographed. In other words, the pattern 2084 for determining an operating state outputs information on the operating state of the panel that is included in a pattern that is highly similar to the information on the reading result of the object included in the information on the post-edited state reading result, from among a plurality of patterns that are combinations of information on the state of the object and information on the operating state of the panel corresponding to the information on the state of the object stored in the pattern 2084 for determining an operating state.

[0219] Furthermore, the operating state determination unit 213 transmits the board information, the image information of the object, the object information about the object, the edited state read result information which is the edited state read result information, and the operating state information of the board to be photographed to the report card entry position acquisition unit 211 of the server 200c. Here, the operating state determination unit 213 adds an item of "board state" which is an item about the operating state of the board to be photographed to the edited state read result information, enters the determined operating state information of the board in the "board state" item, and transmits it to the report card entry position acquisition unit 211. Then, proceed to step S2021.

[0220] When the report card entry position acquisition unit 211 acquires the inspection item for which the reading result is to be entered in the report card in step S2022, it corresponds to the item string "board condition" being "condition" in the report card format table.

[0221] Fig. 54 is a flowchart showing a detailed procedure of a characteristic operation of the status reading system according to the embodiment 5. Fig. 54 shows detailed processing in the above-mentioned step S2031.

[0222] In step S810, the operational state determination unit 213 acquires information about the object installed on the panel to be inspected from the object table.

[0223] In step S820, the operating status determination unit 213 obtains information about the board, information about the image of the object, information about the object about the object, and information about the edited status reading result, which is information about the edited status reading result, from the reading result editing unit 109 of the mobile terminal 100a.

[0224] In step S830, the operating state determination unit 213 inputs the information on the reading result of the object contained in the information on the edited state reading result acquired in step S820 into the operating state determination pattern 2084, and acquires the determination result of the operating state of the board.

[0225] Fig. 55 is a diagram showing an example of a reading result display screen displayed on the display unit of a mobile terminal included in the status reading system according to the fifth embodiment. The reading result display screen shown in Fig. 55, which is displayed on the display unit 104 of the mobile terminal 100a in the status reading system 5 according to the fifth embodiment, displays the same content as the reading result display screen shown in Fig. 45 in the fourth embodiment. That is, the function of the operating state determination unit 213 is executed based on the edited status reading result information transmitted from the mobile terminal 100a or the status reading result information confirmed without editing, so that the reading result display screen displayed on the display unit 104 of the mobile terminal 100a is the same as that in the fourth embodiment.

[0226] Fig. 56 is a diagram showing an example of an output file of the reading result of an object on the display unit of a mobile terminal included in the status reading system according to the fifth embodiment. Fig. 56 shows an inspection result display screen on which information on the reading result of the board is displayed using a report file. The difference between the inspection result display screen according to the fifth embodiment shown in Fig. 56 and the inspection result display screen according to the fourth embodiment shown in Fig. 46 will be described.

[0227] On the inspection result display screen in the fourth embodiment shown in Fig. 46, only information related to the reading results from the photographed image of the board is output. On the other hand, on the inspection result display screen in the fifth embodiment shown in Fig. 56, the pattern of the board's operating state is determined from the reading results, and the operating state of the board is output as "Status: Operating".

[0228] The status reading system 5 according to the fifth embodiment described above includes an operating status determination unit 213 that acquires the operating status of the board based on information on the reading result of the object. As a result, the status reading system 5 can automatically acquire information on the status of the board when the status of the object is read, in addition to the reading result of the object read by the status reading system 5, and can present the reading result of the object, including the information on the status of the board, to an inspector.

[0229] Embodiment 6 Fig. 57 is a diagram showing the flow of information between a mobile terminal and a server included in a status reading system according to the sixth embodiment. Fig. 57 shows extracted components used to explain the operation of a status reading system 6, which will be described later. The status reading system 6 according to the sixth embodiment basically has the same configuration and functions as the status reading system 5 according to the fifth embodiment.

[0230] The state reading system 6 according to the sixth embodiment includes a mobile terminal 100a and a server 200d. The server storage unit 208 of the server 200d stores an image detection model 2085 and a character detection model 2086 as an object detection model 2083. In the sixth embodiment, the object detection model 2083 does not "output an image of an object and character information for identification simultaneously" using one object detection model, but "detects an image of an object in a photographed image of a board by the image detection model 2085" and "detects character information in a photographed image of a board by the character detection model 2086". In the sixth embodiment, the object detection unit 202 detects an object by inputting a photographed image of a board to the image detection model 2085 and the character detection model 2086. In addition, the state reading unit 203 identifies identification information by associating the position of the object with the position of the character.

[0231] The image detection model 2085 is a trained model that receives a photographed image of the board and outputs an image of an object contained in the photographed image of the board and information on the display position of the object.

[0232] The character detection model 2086 is a trained model that receives a photographed image of a board and outputs character information contained in the photographed image of the board and information on the display position of the character information. That is, the character detection model 2086 outputs information such as the model number of the object and the scale value, and information on the display position of the character information.

[0233] In embodiment 6, the status reading unit 203 obtains a correspondence between the object and the character information possessed by the object based on information on the display position of the object in the photographed image of the board obtained by the image detection model 2085 and information on the position of the characters contained in the photographed image of the board obtained by the character detection model 2086, and uses this correspondence when reading the status.

[0234] In embodiment 6, the learning device 250 learns the relationship between the photographed image of the board and the image of the object based on the photographed image of the board included in the board table stored in the equipment database 2081 and the image of the object included in the object table stored in the inspection database 2082, and generates a learned model 273 corresponding to the image detection model 2085.

[0235] In addition, in embodiment 6, the learning device 250 learns the relationship between the photographed image of the board included in the board table stored in the equipment database 2081 and the identification information, which is information on the object included in the object table stored in the inspection database 2082, based on the photographed image of the board and the identification information, and generates a learned model 273 corresponding to the character detection model 2086.

[0236] Fig. 58 is a diagram showing an example of learning data used by the learning device according to the sixth embodiment for learning about an image of an object. Fig. 58 shows learning data of image detection model 2085, i.e., data used by learning device 250 for learning to generate image detection model 2085. As shown in Fig. 58, input information 271 of the learning data used by learning device 250 for learning about an image of an object includes information on an image of the board, i.e., information on a photographed image of the board. Detection information 272 of the learning data used by learning device 250 includes information on an image of an object included in the board, which is teacher data corresponding to input information 271 of the learning data.

[0237] Fig. 59 is a diagram showing an example of output of a trained model in the learning device according to embodiment 6. When the image of the board shown in Fig. 58 is input as learning data to trained model 273 of image detection model 2085, an image of an object included in the image of the board is output as output data from trained model 273 as shown in Fig. 59.

[0238] FIG. 60 is a diagram showing an example of learning data used by the learning device according to the sixth embodiment for learning about identification information. FIG. 60 shows learning data of character detection model 2086, i.e., data used by learning device 250 for learning to generate character detection model 2086. As shown in FIG. 60, input information 271 of the learning data used by learning device 250 for learning about identification information, which is character information contained in a photographed image of a board and information for identifying an object contained in the photographed image of the board, includes information on the image of the board, i.e., information on the photographed image of the board. Furthermore, detection information 272 of the learning data used by learning device 250 includes identification information of an object contained in the photographed image of the board, which is teacher data corresponding to input information 271 of the learning data.

[0239] Fig. 61 is a diagram showing an example of output of a trained model in the learning device according to embodiment 6. When the image of the board shown in Fig. 60 is input as learning data to trained model 273 of character detection model 2086, object identification information, which is character information included in the photographed image of the board, is output as output data from trained model 273 as shown in Fig. 61.

[0240] Fig. 62 is a diagram for explaining an example of information about the board, which is identification information output by the character detection model according to the sixth embodiment. The character detection model 2086 outputs the name of the board, which is the name of the device, as information about the board, based on the captured image of the board shown in Fig. 62.

[0241] Fig. 63 is a diagram for explaining an example of information about a meter, which is identification information output by the character detection model according to the sixth embodiment. The character detection model 2086 outputs information about the model number, unit, and scale value as information about the meter, based on an image of the meter in the photographed image of the panel shown in Fig. 63.

[0242] Fig. 64 is a diagram showing an example of information about a lamp, which is identification information output by the character detection model according to the sixth embodiment. The character detection model 2086 does not output information about the image of the lamp in the photographed image of the board shown in Fig. 64, since no character information exists. However, if character information is displayed on the lamp, the character information is output.

[0243] Fig. 65 is a flowchart showing the procedure of the operation of the status reading system according to the embodiment 6. The following describes the differences from the operation of the status reading system 5 according to the embodiment 5 shown in Fig. 53.

[0244] In the sixth embodiment, in step S2041, the object detection unit 202 separates the captured image of the board into image information of the object and text information, and detects the object reflected in the captured image. That is, the object detection unit 202 uses the image detection model 2085 to acquire the image of the object included in the captured image of the board and information on the display position of the object. The object detection unit 202 also uses the text detection model 2086 to acquire the text information included in the captured image of the board and information on the display position of the text information.

[0245] In step S2042, the distance between the display position of the object acquired by the image detection model 2085 and the display position of the character acquired by the character detection model 2086 is less than a predetermined threshold, and this is determined as "identification information related to the object."

[0246] In step S2006, the identification information related to the object determined in step S2042 is used to perform a state reading process in the same manner as in the first embodiment, to read the state of the object.

[0247] Fig. 66 is a flowchart showing an example of a procedure of a process for determining identification information related to an object in an object detection unit of a server included in a status reading system according to embodiment 6. Fig. 66 shows a case where the status reading unit 203 reads the status of a lamp, which is an object.

[0248] First, in step S910, the object detection unit 202 acquires information on the coordinate position of the object detected by the image detection model 2085 in the photographed image of the board, i.e., the display position of the object in the photographed image of the board, from the information output by the image detection model 2085.

[0249] Next, in step S920, the object detection unit 202 obtains information on the coordinate positions in the photographed image of the board of characters such as the model number and scale value detected by the character detection model 2086 from the information output by the character detection model 2086, i.e., information on the display position in the photographed image of the board of the object.

[0250] Next, in step S930, the object detection unit 202 associates information on the display position of the object in the photographed image of the board with information on the display position of the characters in the photographed image of the board to identify the type of object and the identification information of the object, and determines the identified identification information as "identification information related to the object." In other words, the object detection unit 202 calculates the distance between the coordinate position of the object in the photographed image of the board and the coordinate position of the characters in the photographed image of the board. Then, the object detection unit 202 associates and determines the character information for which the calculated distance is within a predetermined threshold range as "identification information of the object to be read this time."

[0251] Next, in step S940, the status reading unit 203 performs a status reading process in the same manner as in the first embodiment, using the identification information related to the object determined in step S930, and reads the status of the object.

[0252] Fig. 67 is a diagram showing an example of a captured image of a board input to the object detection unit of the server provided in the status reading system according to the sixth embodiment. Fig. 68 is a diagram explaining an example of a calculation process in the object detection unit of the server provided in the status reading system according to the sixth embodiment. In the captured image of the board shown in Fig. 67, for example, if the predetermined threshold is 50px, the object detection unit 202 determines that the characters "20", "0", "%", ... and "100" are below the threshold, and therefore are "identification information of the object to be read this time".

[0253] The condition reading system 6 according to the above-mentioned embodiment 6, like the condition reading system 1 according to the embodiment 1, can read the status of all objects of multiple types and numbers mounted on the board from a single photographed image of the board and output the reading results of all objects simultaneously, thereby facilitating inspection work and making the inspection work more efficient.

[0254] Next, the hardware configuration of each of the control units 80 according to the first to sixth embodiments will be described. The control unit 80 according to the first to sixth embodiments corresponds to the board designation unit 102, the photographing unit 103, the output unit 105, and the terminal control unit 108 in the mobile terminals 100 and 100a, the reading result editing unit 109 in the mobile terminal 100a, the board data acquisition unit 201, the object detection unit 202, the state reading unit 203, the reading result storage unit 204, the object learning unit 205, and the server control unit 207 in the servers 200, 200a, 200b, 200c, and 200d, the detection result determination unit 209 in the server 200a, the report entry position acquisition unit 211, the report format acquisition unit 210, and the report storage unit 212 in the server 200b, and the operating state determination unit 213, the operating state learning unit 214, and the operating state determination pattern 2084 in the server 200c, respectively. Each function of the control unit 80 according to the first to sixth embodiments is realized by a processing circuit. The processing circuit may be a dedicated hardware, or may be a processing device that executes a program stored in a storage device.

[0255] When the processing circuit is a dedicated hardware, the processing circuit may be a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application specific integrated circuit, a field programmable gate array, or a combination of these. Figure 69 is a diagram showing a configuration in which the functions of the control unit according to the first to sixth embodiments are realized by hardware. A logic circuit 81a that realizes the function of the control unit 80 is incorporated in the processing circuit 81.

[0256] When the processing circuit 81 is a processing device, the functions of the control unit 80 are realized by software, firmware, or a combination of software and firmware.

[0257] FIG. 70 is a diagram showing a configuration in which the functions of the control unit according to the first to sixth embodiments are realized by software. The processing circuit 81 has a processor 811 that executes a program 81b, a random access memory 812 that the processor 811 uses as a work area, and a storage device 813 that stores the program 81b. The processor 811 deploys the program 81b stored in the storage device 813 on the random access memory 812 and executes it, thereby realizing the function of the control unit 80. The software or firmware is written in a program language and stored in the storage device 813. The processor 811 can be exemplified by a central processing unit, but is not limited to this. The storage device 813 can be a semiconductor memory such as a random access memory (RAM), a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM), or an electrically erasable programmable read only memory (EEPROM (registered trademark)). The semiconductor memory may be a non-volatile memory or a volatile memory. Further, in addition to a semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc) can be applied to the storage device 813. The processor 811 may output data such as a calculation result to the storage device 813 for storage, or may store the data in an auxiliary storage device (not shown) via the random access memory 812. By integrating the processor 811, the random access memory 812, and the storage device 813 on one chip, the functions of the control unit 80 can be realized by a microcomputer.

[0258] The processing circuit 81 realizes the functions of the control unit 80 by reading and executing the program 81b stored in the storage device 813. It can also be said that the program 81b causes a computer to execute procedures and methods for realizing the functions of the control unit 80.

[0259] In addition, the processing circuit 81 may be configured so that some of the functions of the control unit 80 are realized by dedicated hardware, and some of the functions of the control unit 80 are realized by software or firmware.

[0260] Thus, the processing circuitry 81 can realize each of the above-described functions by hardware, software, firmware, or a combination of these.

[0261] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or the embodiments may be combined with each other. Also, parts of the configurations may be omitted or modified without departing from the spirit of the invention.

[0262] Various aspects of the present disclosure are summarized below as appendices.

[0263] (Appendix 1) a data acquisition unit that acquires learning data including a captured image of a panel surface on which a plurality of instruments are mounted in an inspection target panel, images of each of the plurality of instruments included in the captured image, and identification information that is information for identifying and specifying each of the plurality of instruments; a model generation unit that generates a trained model for inferring images of each of a plurality of instruments and the identification information from the captured image using the learning data; A learning device comprising: (Appendix 2) The trained model is An image detection model that is a trained model that outputs images of each of the instruments included in the captured image by inputting the captured image; A character detection model that is a trained model that outputs character information contained in a photographed image of a board by inputting the photographed image; 2. The learning device according to claim 1, comprising: (Appendix 3) A status reading system comprising a mobile terminal and a server capable of communicating with the mobile terminal, the status reading system reading the status of a plurality of instruments mounted on a panel to be inspected, The mobile terminal includes an image capturing unit configured to capture an image of a panel surface on which the plurality of instruments are mounted, The server, a detection unit that detects, from the captured image, images of each of the plurality of instruments included in the captured image and identification information that is information for identifying and specifying each of the plurality of instruments; a status reading unit that obtains status information of each of the plurality of instruments by using an image of each of the plurality of instruments and the identification information; Equipped with the detection unit detects an image of each of a plurality of instruments and the identification information from the captured image by using a trained model for inferring the image of each of the plurality of instruments and the identification information; A status reading system comprising: (Appendix 4) The detection unit detects the number of the instruments included in the captured image, the server includes a detection result determination unit that determines whether or not all of the instruments included in the captured image have been detected by the detection unit by comparing information on the number of the instruments mounted on the board with information on the number of the instruments detected by the detection unit; 4. A status reading system as described in claim 3. (Appendix 5) The mobile terminal includes: a display unit that displays information on the read result, which is information on the status of each of the multiple instruments acquired by the status reading unit; a reading result editing unit that edits information of an expression related to the reading result; 5. A status reading system according to claim 3 or 4, comprising: (Appendix 6) a report storage unit that records information on the reading results corresponding to inspection items for the board that are described in a report, which is a predetermined electronic document, in the report; 6. A status reading system according to any one of claims 3 to 5, (Appendix 7) An operating state determination unit that acquires an operating state of the board based on information of the read result; 6. A status reading system according to any one of claims 3 to 5, (Appendix 8) A step of acquiring learning data including a captured image of a panel surface on which a plurality of instruments are mounted in an inspection target panel, images of each of the plurality of instruments included in the captured image, and identification information which is information for identifying and specifying each of the plurality of instruments; generating a trained model for inferring images of each of a plurality of instruments and the identification information from the captured image using the training data; A learning method comprising: (Appendix 9) A step of acquiring learning data including a captured image of a panel surface on which a plurality of instruments are mounted in an inspection target panel, images of each of the plurality of instruments included in the captured image, and identification information which is information for identifying and specifying each of the plurality of instruments; generating a trained model for inferring images of each of a plurality of instruments and the identification information from the captured image using the training data; A learning program characterized by causing a computer to execute the above. [Explanation of symbols]

[0264] 1,2,3,4,5,6 Status reading system, 80 control unit, 81 processing circuit, 81a logic circuit, 81b program, 100,100a mobile terminal, 101 operation unit, 102 board designation unit, 103 photography unit, 104 display unit, 105 output unit, 106 terminal memory unit, 107 terminal communication unit, 108 terminal control unit, 109 read result editing unit, 200,200a,200b,200c,200d server, 201 board data acquisition unit, 202 object detection unit, 203 status reading unit, 204 read result storage unit, 205 object learning unit, 206 server communication unit, 207 server control unit, 208 server memory unit, 209 detection result judgment unit, 210 report format acquisition unit, 211 report entry position acquisition unit, 212 Report storage unit, 213 operation state determination unit, 214 operation state learning unit, 250 learning device, 251 data acquisition unit, 252 model generation unit, 253 learned model storage unit, 260 inference device, 261 data acquisition unit, 262 inference unit, 271 input information, 272 detection information, 273 learned model, 300 equipment, 400 Internet, 410 sample board, 411 water droplet, 811 processor, 812 random access memory, 813 storage device, 2081 equipment database, 2082 inspection database, 2083 object detection model, 2084 operation state determination pattern, 2085 image detection model, 2086 character detection model.

Claims

1. A data acquisition unit acquires learning data including a photograph of the panel surface on which multiple instruments are mounted in the panel to be inspected, an image of each of the multiple instruments included in the photograph, and identification information which is information for identifying and specifying each of the multiple instruments. A model generation unit generates a trained model for inferring the images of each of the multiple instruments and the identification information from the captured images using the aforementioned training data. A learning device characterized by being equipped with the following features.

2. The aforementioned trained model, An image detection model, which is a trained model that takes the aforementioned captured image as input and outputs images of each of the multiple instruments contained in the captured image, A character detection model is a trained model that outputs character information contained in a captured image of a board when the captured image of the board is input. The learning device according to claim 1, characterized by having the following features.

3. A status reading system comprising a mobile terminal and a server capable of communicating with the mobile terminal, which reads the status of multiple instruments mounted on a panel to be inspected, The mobile terminal includes a shooting unit that captures a captured image which is an image of the panel surface on which the multiple instruments are mounted. The aforementioned server, A detection unit that detects from the captured image each of the multiple instruments included in the captured image and identification information which is information for identifying and specifying each of the multiple instruments, A state reading unit that acquires information on the state of each of the multiple instruments using the image of each of the multiple instruments and the identification information, Equipped with, The detection unit detects each image of a plurality of instruments and the identification information from the captured image using a trained model for inferring each image of a plurality of instruments and the identification information. A status reading system characterized by the following.

4. The detection unit detects the number of instruments included in the captured image, The server includes a detection result determination unit that determines whether all of the instruments included in the captured image have been detected by the detection unit by comparing information on the number of instruments mounted on the panel with information on the number of instruments detected by the detection unit. The state reading system according to claim 3, characterized by the following:

5. The aforementioned mobile device is A display unit that displays reading result information, which is information about the state of each of the multiple instruments acquired by the state reading unit, A reading results editing department edits information regarding the expression of the aforementioned reading results, The status reading system according to claim 3 or 4, characterized by comprising the above.

6. The system includes a performance report storage unit that records in the performance report information the reading results, which are information about the status of each of the multiple instruments acquired by the status reading unit corresponding to the inspection items for the panel described in the performance report, which is a predetermined electronic document. The state reading system according to claim 3, characterized by the following:

7. The operating state determination unit is provided with information that reads the state of each of the plurality of instruments obtained by the state reading unit, and the operating state of the panel is determined based on the reading result information. The state reading system according to claim 3, characterized by the following:

8. Steps to acquire training data including a photograph of the panel surface on which multiple instruments are mounted on the panel to be inspected, an image of each of the multiple instruments included in the photograph, and identification information which is information for identifying and specifying each of the multiple instruments; The steps include generating a trained model using the aforementioned training data to infer the images of each of the multiple instruments and the identification information from the captured images, A learning method characterized by including the following.

9. Steps to acquire training data including a photograph of the panel surface on which multiple instruments are mounted on the panel to be inspected, an image of each of the multiple instruments included in the photograph, and identification information which is information for identifying and specifying each of the multiple instruments; The steps include generating a trained model using the aforementioned training data to infer the images of each of the multiple instruments and the identification information from the captured images, A learning program characterized by having a computer execute it.

Images