Monitoring control device, monitoring control system, monitoring control method, and monitoring control program

Abstract

To obtain a monitoring control device capable of performing control while effectively preventing an erroneous operation by an operator.SOLUTION: A monitoring control device 1 comprises a line-of-sight information acquisition unit 22, an operation history storage unit 30, an erroneous operation determination unit 25, and a warning display unit 14. The line-of-sight information acquisition unit 22 acquires line-of-sight information 5, which is a movement of a line of sight when an operator performs a monitoring control operation. The operation history storage unit 30 stores operation history 6 including the line-of-sight information 5, which is a history of operations. The erroneous operation determination unit 25 determines whether an operation is the erroneous operation based on the line-of-sight information 5 acquired by the line-of-sight information acquisition unit 22 and the operation history 6 stored in the operation history storage unit 30. The warning display unit 14 displays a warning screen when the erroneous operation determination unit 25 determines that there is the erroneous operation.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a monitoring control device, a monitoring control system, a monitoring control method, and a monitoring control program that acquires the line of sight movement of an operator and issues a warning if an erroneous operation is suspected, thereby preventing the erroneous operation. [Background technology]

[0002] In a monitoring and control device, for example, an operator controls opening and closing a pump by performing operations on a screen. To prevent erroneous operations, a pop-up display or the like may be used to alert the operator to each operation.

[0003] However, even if a warning message such as a pop-up message is displayed, an operator who is accustomed to the operation may skip over the message and perform the operation, and thus may not be able to effectively prevent erroneous operations.

[0004] As a means for preventing such erroneous operations, a technique is known in which the operator's line of sight movement is acquired and an erroneous operation is recognized based on the acquired line of sight movement. For example, Patent Document 1 discloses a technique in which the operator's line of sight movement is acquired and an input on the screen is permitted only when the line of sight movement satisfies a predetermined pattern registered in advance. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2015-153195 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 is configured such that screen input is permitted only when a predetermined pattern registered in advance is met. Therefore, even if an operator is gazing at the screen, for example, operation is not possible if the predetermined pattern is not met, making it difficult to effectively prevent erroneous operation.

[0007] The present disclosure has been made to solve the above-mentioned problems, and has an object to provide a monitoring control device that can perform control while effectively preventing erroneous operation by an operator. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the monitoring control device according to the present disclosure includes a gaze information acquisition unit, an operation history storage unit, an erroneous operation determination unit, and a warning display unit. The gaze information acquisition unit acquires gaze information, which is the movement of the operator's gaze during the operation of the monitoring control. The operation history storage unit stores an operation history, which is a history of operations and includes the gaze information. The erroneous operation determination unit performs a process of determining the gaze information acquired by the gaze information acquisition unit and the operation history stored in the operation history storage unit. The operation is determined to be an erroneous operation when it is determined to be similar to the historical gaze information recorded in association with the operation that was an erroneous operation among the historical gaze information that is a history of the gaze information. The warning display unit displays a warning screen when the operation error determination unit determines that an operation error has occurred. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to obtain a monitoring control device that can perform control while effectively preventing erroneous operations by an operator. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing a configuration of a monitoring and control system according to a first embodiment. [Figure 2] 1 is a diagram illustrating a configuration of a monitoring and control device according to a first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of an operation screen of an operation unit according to the first embodiment. [Figure 4] 4 is a diagram showing an example of a warning screen displayed by a warning display unit according to the first embodiment; FIG. [Figure 5]FIG. 4 is a diagram showing an example of an operation history according to the first embodiment. [Figure 6] FIG. 3 is a diagram showing an example of historical line-of-sight information according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing another example of historical line-of-sight information according to the first embodiment. [Figure 8] 10 is a diagram showing an example of an operation history screen displayed by the operation error setting unit according to the first embodiment. FIG. [Figure 9] 4 is a diagram showing an example of line-of-sight information indicating a line-of-sight trajectory according to the first embodiment. FIG. [Figure 10] FIG. 2 is a diagram showing an example of dividing the trajectory of the line of sight into a plurality of points according to the first embodiment. [Figure 11] 4 is a flowchart showing the processing operation of the monitoring control device according to the first embodiment. [Figure 12] 5 is a flowchart showing the processing operation of the error manipulation setting unit according to the first embodiment. [Figure 13] 2 is a diagram illustrating an example of a hardware configuration of a monitoring and control device according to the first embodiment. FIG. [Figure 14] FIG. 10 is a diagram illustrating a configuration of a monitoring and control device according to a second embodiment. [Figure 15] FIG. 10 is a diagram illustrating a configuration of an operation history processing unit according to the second embodiment. [Figure 16] 10 is a flowchart showing the processing operation of the monitoring control device according to the second embodiment. [Figure 17] 10 is a flowchart showing the processing operation of a learning unit according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiment 1 A description will be given of a monitoring control device according to embodiment 1. Fig. 1 shows an example of a monitoring control system 100 including a monitoring control device 1 according to embodiment 1.

[0012] The monitoring and control system 100 comprises a monitoring and control device 1, a line-of-sight acquisition device 2, and a display unit 3.

[0013] The gaze acquisition device 2 acquires gaze movements of the operator during monitoring and control operations as gaze information 5. The gaze information 5 is not particularly limited as long as it is information related to the gaze. For example, it may be image data visualizing the gaze time of the operator at a location where the operator is gazing, coordinate changes at the location where the gaze is being performed, the direction and speed of the gaze movement, or the trajectory formed by the gaze change over time, or a combination of these. The gaze acquisition device 2 may be, for example, a camera-type device attached to the display unit 3, or may be glasses-type device worn by the operator. The display unit 3 displays information for the operator to perform control operations in the monitoring and control device 1. The display unit 3 is, for example, a liquid crystal display or an organic EL (organic electroluminescence) display, and may be a stationary monitor or a tablet terminal.

[0014] The configuration of the monitoring control device 1 will be described below. In this disclosure, only operations related to monitoring control are described. FIG. 2 is a diagram showing the configuration of the monitoring control device 1 according to the first embodiment. The monitoring control device 1 is composed of a monitoring control unit 10, an operation analysis unit 20, and an operation history storage unit 30. The monitoring control unit 10, the operation analysis unit 20, and the operation history storage unit 30 are capable of communicating with each other, and are capable of exchanging data such as gaze information 5 described below.

[0015] The monitoring control unit 10 is composed of an operation unit 11, a line-of-sight acquisition unit 12, an information output unit 13, a warning display unit 14, and an operation error setting unit 15. The operation analysis unit 20 is composed of an operation information acquisition unit 21, a line-of-sight information acquisition unit 22, an operation history acquisition unit 23, a similarity calculation unit 24, and an operation error determination unit 25.

[0016] The operation unit 11 performs processing to allow the operator to perform operations via the display unit 3 to control the opening and closing of the pump, etc. Fig. 3 shows an example of an operation screen related to the operation unit 11. For example, the operator clicks on an operation target in the drawing and performs the operation according to the instructions in a pop-up displayed on the screen.

[0017] The gaze acquisition unit 12 acquires gaze information 5 acquired by the gaze acquisition device 2 when the operator performs an operation on the operation unit 11. The gaze information 5 when performing an operation may be, for example, gaze information from when the operator finishes a previous operation to when the operator finishes a next operation, or may be gaze information from when a pop-up such as that shown in Fig. 3 appears to when the operator actually performs control when performing an operation.

[0018] The information output unit 13 associates operation information 4, which is the content of an operation performed by an operator, such as "operating the pump," with gaze information 5 acquired when the operation was performed, and outputs the associated information to the operation history storage unit 30. The information output unit 13 also outputs the operation information 4 to the operation information acquisition unit 21 and the gaze information 5 acquired when the operation was performed to the gaze information acquisition unit 22. The operation information 4 is not particularly limited as long as it is information that represents the content of the operation. For example, it may be a single piece of information that represents an overview of the operation, such as "operation of the pump," or it may be information having multiple items, such as "operation target," "operation target type," and "operation type," and the multiple items may be in one-to-one correspondence with the overview of the operation.

[0019] The warning display unit 14 displays a warning to the operator when an operation performed by the operator is determined to be an erroneous operation by the erroneous operation determination unit 25 described below. FIG. 4 shows an example of a display when a warning is displayed by the warning display unit 14. For example, when the warning display unit 14 is displayed, the operation unit 11 does not immediately perform control processing of the operation object, and a warning screen is displayed. If the operation is performed according to the instructions displayed on the warning screen, the operation object is controlled again. In other words, when the warning is displayed, the operator can check it again and review the operation.

[0020] The operation error setting unit 15 displays an operation history 6a, which is a simplified version of the operation history 6 stored in the operation history storage unit 30, and accepts an operation by the operator to register whether or not an operation was an operation error. The registration of an operation error is reflected in the operation history storage unit 30. FIG. 5 shows an example of the operation history 6 stored in the operation history storage unit 30. The operation history 6 is history information of each operation performed by the operator and includes operation information 4 and gaze information 5. Hereinafter, the operation information 4 and gaze information 5 in the operation history 6 stored in the operation history storage unit 30 will be referred to as historical operation information 4a and historical gaze information 5a, respectively. For example, in the columns of the table in FIG. 5, the operation target, operation target type, and operation type correspond to the historical operation information 4a, and the gaze information No. corresponds to the historical gaze information 5a. Hereinafter, an example will be described in which the gaze information 5 is one or more pieces of information that associate a gaze point on which the operator gazed with a gaze time, which is the time for which the operator gazed at the gaze point. Furthermore, the operation history 6 is history information of each operation, but hereinafter, when no distinction is made between operations, the history information stored in the operation history storage unit 30 will be collectively referred to as the operation history 6.

[0021] A gaze point refers to a location on the screen where an operator gazes at the same location for a certain period of time or more. Whether or not the operator gazes at the same location is determined, for example, by determining whether the change in the gaze coordinates is equal to or less than a predetermined value for a certain period of time or more. The gaze time is, for example, the time during which the change in the gaze coordinates is equal to or less than a predetermined value. The gaze point may be a coordinate on the screen, but if a location where operation-related content such as "Pump B" is displayed is close to the coordinate of the gaze point, the gaze point may be "Pump B," for example. In this case, it is necessary to store in a storage device (not shown) what is displayed at which coordinate on the screen for each operation screen. Also, a location gazed at for a certain period of time or more may be treated as a gaze point only if the location where operation-related content is displayed. In the following description, an operation object related to an operation or the like is considered to be a gaze point.

[0022] In FIG. 5, a gaze information No. is assigned as a serial number for the historical gaze information 5a associated with each operation. FIG. 6 shows an example of historical gaze information 5a in FIG. 5 where the gaze information No. is "1." The gaze information 5 and historical gaze information 5a each contain at least one piece of information that associates a gaze point, which is a point on the screen where the operator gazed, with the time spent gazing at the gaze point. In this case, in the operation to select valve C in FIG. 5, the operator gazed at valve B for 6 seconds, pump B for 3 seconds, and valve C for 4 seconds, in that order. Note that if there is no gaze point, i.e., if the operator has not gazed at a certain point for a certain period of time or longer, the gaze information 5 may be configured with the gaze point set to "none" and the gaze time set to 0 seconds.

[0023] Furthermore, the historical gaze information 5a may include information other than the gaze point and gaze time for each operation. Fig. 7 shows another example of the historical gaze information 5a. In the table of Fig. 7, in addition to the gaze point and gaze time, the device status, system status, and whether or not an alarm is present are stored in association with the gaze point as status information 9 representing the status of the gaze point. As a result, the gaze information 5, which includes conditions such as the device status and whether or not an alarm is present that represent the status of the gaze point when the operator performs an operation, can be used to calculate the similarity 8 described below.

[0024] In the above explanation, the historical gaze information 5a is stored as separate tables associated with each other as shown in Figures 5 and 6, but each operation may be stored in a single table with the number of rows corresponding to the number of gaze records in the historical gaze information 5a. For example, in the case of the historical gaze information 5a shown in Figure 6, the operation of selecting valve C in the operation history 6 is shown in three rows in the table.

[0025] The operation history 6 also includes an operation error flag 7 that indicates whether or not each operation was an operation error. In Fig. 6, "◯" indicates that it was an operation error. The operation error flag 7 is applied by operating the operation error setting unit 15 as described below. Note that the operation history 6 is not limited to the table format as shown in Fig. 5, and it is sufficient that the historical operation information 4a, the historical line-of-sight information 5a, and the operation error flag 7 are associated with each operation.

[0026] FIG. 8 is a diagram showing an example of an operation history 6a that is displayed on the display unit 3 by the operation error setting unit 15 and that the operator can operate. Among the columns in FIG. 8, the operation summary corresponds to the historical operation information 4a, and the operation error registration corresponds to the operation error flag 7. If an operation is an operation error, the operator changes the operation error flag in the operation history 6 from "unregistered" to "registered" by clicking the operation error registration column. That is, the operation error flag in the operation history 6 is changed to "operation error." In this case, the change in the operation error flag 7 is reflected in the corresponding operation in the operation history 6 stored in the operation history storage unit 30. For example, "◯" is recorded in the corresponding operation error flag in FIG. 5.

[0027] Furthermore, in the above example, the operator registers an operation error by setting the operation error flag 7 in the operation error setting unit 15, but, for example, when a warning is displayed by the warning display unit 14, the corresponding operation may be stored as an operation error in the operation history storage unit 30. If the operation for which a warning is displayed by the warning display unit 14 was not actually an operation error, the operator can change the corresponding operation from "registered" to "not registered" by operating the operation error setting unit 15.

[0028] Furthermore, the operation history 6a displayed by the operation error setting unit 15 has been described above as a simple display of the operation history 6, but it may also be a display of the operation history 6 itself.

[0029] The operation analysis unit 20 analyzes whether the operation performed by the operator was an error based on the operation information 4 and the gaze information 5. The operation information acquisition unit 21 acquires the operation information 4 for each operation. The gaze information acquisition unit 22 acquires the gaze information 5 at the time of each operation. In the present disclosure, the gaze acquisition unit 12 in the monitoring control unit 10 constantly receives the gaze information 5, and when an operation is performed by the operation unit 11, the corresponding gaze information 5 is transmitted to the gaze information acquisition unit 22 by the information output unit 13. However, the gaze information acquisition unit 22 may be configured to acquire the gaze information 5 directly from the gaze acquisition device 2.

[0030] The operation history acquisition unit 23 acquires an operation history 6 from the operation history storage unit 30 to determine whether an operation performed by the operator was an operation error. The operation history acquisition unit 23 acquires, from the operation history 6 stored in the operation history storage unit 30, those operation histories that are registered as an operation error by the operation error flag 7. That is, the operation history 6 acquired by the operation history acquisition unit 23 and indicating an operation error is used to determine whether an operation performed by the operator was an operation error. Alternatively, the operation history acquisition unit 23 may acquire only the operation history 6 including historical operation information 4a similar to the operation information 4 of an operation performed by the operator. This makes it possible to determine whether an operation performed by the operator was an operation error based on the operation history 6 of similar operations performed in the past. In determining the similarity of the operation information 4, operation information 4 having the same specific item, such as the operation target, may be deemed similar, or operation information 4 having the same items among a predetermined number of items may be deemed similar. Alternatively, operation information 4 may be deemed similar based on the number of identical items among the items of the plurality of operation information 4.

[0031] The similarity calculation unit 24 calculates a similarity 8 between the gaze information 5 acquired by the gaze information acquisition unit 22 and the historical gaze information 5a corresponding to the operation history 6 that was an erroneous operation acquired by the operation history acquisition unit 23. The similarity 8 is not particularly limited as long as it represents the degree of similarity between the gaze information 5 and the historical gaze information 5a. The similarity 8 may be a numerical value, or may be one of two types of character strings representing "similar" and "dissimilar." Furthermore, when the similarity 8 is a numerical value, the larger the numerical value, the higher the similarity, or the lower the numerical value, the higher the similarity.

[0032] For example, if the gaze information 5 is data represented by gaze points and gaze times, the similarity 8 is calculated from the number of common gaze points and the difference in gaze times when the gaze points are common. Furthermore, if the gaze information 5 includes other information such as a "device status" in addition to the gaze points and gaze times as shown in FIG. 7, the similarity 8 may be calculated using the other information, or the gaze points may be determined to match when the gaze points and the other information match. Furthermore, the total gaze time may be calculated, and if the total gaze time of the gaze information 5 is less than the total gaze time of the historical gaze information 5a, the similarity 8 may be output as "similar." Furthermore, if the gaze points represent coordinates on the screen, the gaze points may be determined to be common when the difference in coordinate positions is less than a predetermined value.

[0033] In addition, an example of a method for calculating the similarity 8 when the gaze information 5 is a trajectory as indicated by the arrow in FIG. 9 is shown below. FIG. 10 is a diagram in which the gaze information 5 is a trajectory, and the trajectory is divided into N points based on the start point and end point, and each point is represented as X1, X2, . . . , XN. N is a natural number greater than or equal to 2, with X1 being the start point and XN being the end point. The division may be based on the time from the start point to the end point or the length of the trajectory. When comparing the gaze information 5 with the historical gaze information 5a, the distance in the coordinate space between X1 of the gaze information 5 and X1 of the historical gaze information 5a is calculated as D1, the distance in the coordinate space between X2 of the gaze information 5 and X2 of the historical gaze information 5a is calculated as D2, and the distance in the coordinate space between XN of the gaze information 5 and XN of the historical gaze information 5a is calculated as DN. Then, the square root of the sum of the squares of the distances D1, D2, ..., DN between the respective points is calculated as the similarity 8. In this case, the smaller the value of the similarity 8, the more similar the gaze information 5 and the historical gaze information 5a are. Alternatively, a threshold may be set for the square root of the sum of the squares of the distances, and the similarity 8 may be output as "similar" if the value is below the threshold, or "dissimilar" if the value is equal to or greater than the threshold.

[0034] The above-described method for calculating the similarity 8 is merely an example, and the similarity 8 may be calculated using another publicly known technique related to gaze similarity, or by combining other publicly known techniques. Furthermore, for example, if the gaze information 5 includes both the gaze location and gaze duration, and the trajectory, the similarity 8 may be calculated by combining the above-described examples.

[0035] The operation error determination unit 25 determines whether an operation performed by the operator was an operation error based on the similarity 8 calculated by the similarity calculation unit 24 by comparing the gaze information 5 acquired by the gaze information acquisition unit 22 with the historical gaze information 5a corresponding to the operation history 6 that was an operation error. For example, if the similarity 8 is a numerical value indicating greater similarity as the similarity 8 is smaller, a threshold may be set, and the operation may be determined to be similar if there is historical gaze information 5a for which the similarity 8 does not meet the threshold. Alternatively, if the similarity 8 represents either "similar" or "dissimilar," the operation may be determined to be similar if there is historical gaze information 5a that is "similar." In other words, if the gaze information 5 is similar to at least one of the historical gaze information 5a corresponding to operations for which the operator previously registered an operation error in the operation error setting unit 15, the operation performed by the operator is determined to be an operation error.

[0036] Next, the processing operations in the monitoring control device 1 according to embodiment 1, from when the operator operates the operation unit 11 to when the warning display unit 14 displays a warning based on the judgment of the error operation judgment unit 25, will be explained using the flowchart of FIG.

[0037] When the operator performs an operation using the operation unit 11 (S10 YES), the operation information acquisition unit 21 acquires the operation information 4, and the line-of-sight information acquisition unit 22 acquires the line-of-sight information 5 (S11). Next, the operation history acquisition unit 23 acquires the operation history 6 of the erroneous operation from the operation history storage unit 30 (S12). The similarity calculation unit 24 compares the line-of-sight information 5 with the history line-of-sight information 5a of the operation history 6 to calculate the similarity (S13). Next, the operation error determination unit 25 determines whether the operation performed by the operator was an operation error based on the similarity 8 calculated by the similarity calculation unit 24 (S14). If the operation error determination unit 25 determines that the operation was an operation error (S14 YES), the warning display unit 14 displays a warning on the display unit 3 (S15). If the operation error determination unit 25 determines that the operation was not an operation error, no warning is displayed (S14 NO). The operator determines whether to execute or abort the control command based on the displayed warning.

[0038] Next, in the monitoring control device 1 of embodiment 1, the processing operation when the error setting unit 15 displays the operation history 6a and the error flag 7 of the operation history 6a is changed by the operator will be explained using the flowchart of Figure 12.

[0039] When the operator requests display of the operation history 6a, the operation error setting unit 15 acquires the operation history 6 from the operation history storage unit 30 (S20). Next, the operation history 6a, which is a simple display of the operation history 6, is displayed on the display unit 3 (S21). When the operator performs an operation to change the operation error flag 7 in the operation history 6a (YES in S22), the change to the operation error flag 7 in the corresponding operation history 6 is reflected in the operation history storage unit 30 (S23). Thereafter, the screen is updated by displaying the operation history 6a that reflects the change to the operation error flag 7 on the display unit 3 (S24).

[0040] In the above description of the processing of the error operation setting unit 15, the operator first requests the display of the operation history 6a, but the operation history 6a may be always displayed on the display unit 3. Also, the operation history 6a may be displayed on a screen separate from the monitoring control screen, and the display may be switched by the operator's operation.

[0041] Next, an example of the hardware configuration of the monitoring control device 1 according to embodiment 1 will be described. Fig. 13 is a diagram showing an example of the hardware configuration of the monitoring control device 1. The monitoring control device 1 includes a computer including a processor 101, a memory 102, and an input device 103.

[0042] The processor 101, memory 102, and input device 103 can transmit and receive information to and from each other via, for example, a bus 104. Operations by the operator on the operation unit 11 and operation error setting unit 15 are realized by the input device 103. The processor 101 executes functions of the operation unit 11, warning display unit 14, operation error setting unit 15, similarity calculation unit 24, etc. by reading and executing programs stored in the memory 102. The processor 101 is, for example, an example of a processing circuit, and includes one or more of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a system LSI (Large Scale Integration).

[0043] The memory 102 includes one or more of a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), and an EEPROM (Electrically Erasable Programmable Read Only Memory). The memory 102 also includes a recording medium on which a computer-readable program is recorded. Such recording medium includes one or more of a non-volatile or volatile semiconductor memory, a magnetic disk, a flexible memory, an optical disk, a compact disk, and a DVD (Digital Versatile Disc). The monitoring and control device 1 may also include an integrated circuit such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array).

[0044] As described above, the monitoring control device 1 according to the first embodiment includes the line-of-sight information acquisition unit 22, the operation history storage unit 30, the operation error determination unit 25, and the warning display unit 14. The line-of-sight information acquisition unit 22 acquires line-of-sight information 5, which is the movement of the operator's line of sight when performing monitoring control operations. The operation history storage unit 30 stores an operation history 6, which is a history of operations and includes the line-of-sight information 5. The operation error determination unit 25 determines whether an operation is an operation error based on the line-of-sight information 5 acquired by the line-of-sight information acquisition unit 22 and the operation history 6 stored in the operation history storage unit 30. The warning display unit 14 displays a warning screen when the operation error determination unit 25 determines that an operation is an operation error. This makes it possible to obtain a monitoring control device that can perform control while effectively preventing an operation error by the operator by determining an operation error based on the operation history and prompting the operator to confirm it.

[0045] In the description of the first embodiment, the operation history 6 does not particularly limit who performed the operation, but the operation history storage unit 30 may store the operation history 6 for each operator. In this case, for example, the operation history acquisition unit 23 acquires erroneous operations from the operation history 6 of past operations performed by the operator who performed the operation using the operation unit 11. That is, based on the historical gaze information 5a corresponding to the operation history 6 of erroneous operations performed by the operator himself in the past, the erroneous operation determination unit 25 determines whether or not the operation performed by the operator was an erroneous operation. This provides the effect of preventing erroneous operations according to the characteristics and habits of each individual operator.

[0046] Embodiment 2 The monitoring control device according to the second embodiment differs from the monitoring control device 1 according to the first embodiment in that it determines whether an operation error has occurred by using machine learning based on the operation history 6. In the following, components having the same functions as those in the first embodiment are denoted by the same symbols and explanations thereof are omitted, and the explanation will focus on the differences from the monitoring control device 1 of the first embodiment.

[0047] The following describes the configuration of a monitoring control device 1a according to embodiment 2. Fig. 14 is a diagram showing an example of the configuration of the monitoring control device 1a according to embodiment 2. The monitoring control device 1a includes a monitoring control unit 10, an operation analysis unit 20a, and an operation history processing unit 40.

[0048] The operation analysis unit 20a is composed of an operation information acquisition unit 21, a gaze information acquisition unit 22, a learning model acquisition unit 26, and an operation error determination unit 25a. The learning model acquisition unit 26 acquires a learning model 42a from a learning model storage unit 42 of the operation history processing unit 40 described below. The operation error determination unit 25a determines whether or not an operation performed by the operator is an operation error, based on the operation information 4 acquired by the operation information acquisition unit 21, the gaze information 5 acquired by the gaze information acquisition unit 22, and the learning model 42a acquired by the learning model acquisition unit 26. In other words, the operation error determination unit 25a corresponds to an inference unit in machine learning.

[0049] 15 is a diagram showing the configuration of the operation history processing unit 40. The operation history processing unit 40 is made up of an operation history storage unit 30a, a learning unit 41, and a learning model storage unit .

[0050] The operation history storage unit 30a stores the operation history 6, similar to the operation history storage unit 30 according to the first embodiment.

[0051] The learning unit 41 uses the operation history 6 stored in the operation history storage unit 30a as learning data to generate a learning model 42a for determining whether or not an operation performed by an operator is an operation error. The learning data has, for example, historical operation information 4a and historical gaze information 5a as input data, and the operation error flag 7 as a correct answer label. In other words, the learning model 42a is a learning model for the operation analysis unit 20 to use the operation information 4 and gaze information 5 as input to determine whether or not the operation error flag 7 is set, i.e., whether or not the operation is an operation error. The learning model storage unit 42 stores the learning model 42a generated by the learning unit 41.

[0052] Furthermore, when an operation history 6 is added to the operation history storage unit 30a, the learning unit 41 further trains the learning model 42a. Furthermore, when the operation error setting unit 15 changes the operation error flag 7 of the operation history 6, i.e., the correct label, the learning unit 41 reflects the change in the learning model 42a.

[0053] Next, in the monitoring control device 1a according to embodiment 2, the processing operations from when the operator operates the operation unit 11 to when the warning display unit 14 displays a warning based on the judgment of the error operation judgment unit 25 will be explained using the flowchart of FIG.

[0054] When the operator performs an operation using the operation unit 11 (S30 YES), the operation information acquisition unit 21 acquires operation information 4, and the gaze information acquisition unit 22 acquires gaze information 5 (S31). Next, the learning model acquisition unit 26 acquires a learning model 42a from the learning model storage unit 42 of the operation history processing unit 40 (S32). Next, the operation error determination unit 25 receives the operation information 4 and the gaze information 5 as input and uses the learning model 42a to determine whether the operation performed by the operator was an operation error (S33). If the operation error determination unit 25 determines that the operation was an operation error (S33 YES), the warning display unit 14 displays a warning on the display unit 3 (S34). If the operation error determination unit 25 determines that the operation was not an operation error, no warning is displayed (S33 NO).

[0055] Next, the processing operations from acquiring learning data to training the learning model 42a in the learning unit 41 according to the second embodiment will be described with reference to the flowchart of FIG.

[0056] When a new operation history 6 is added to the operation history storage unit 30a or when the data of the operation history 6 is updated by the operation error setting unit 15 changing the operation error flag 7 (YES in S40), the learning unit 41 acquires the corresponding operation history 6 as learning data (S41). Next, the learning unit 41 uses the acquired learning data to train a learning model 42a stored in the learning model storage unit 42 (S42).

[0057] An example of the hardware configuration of the monitoring control device 1a according to the second embodiment is the same as the example of the hardware configuration of the monitoring control device 1 shown in Fig. 13. The processor 101 can execute the functions of the learning unit 41 and the operation error determination unit 25a by reading and executing a program stored in the memory 102.

[0058] As described above, the monitoring control device 1a according to the second embodiment includes a learning model acquisition unit 26, an operation error determination unit 25a, a learning unit 41, and the like, instead of the configuration of the monitoring control device 1 according to the first embodiment. This makes it possible to determine an operation error through machine learning using information on the operation history 6 performed by the operator, thereby preventing an operation error based on the operator's past operations. Furthermore, when a large amount of operation history 6 is stored, the operation analysis unit 20a does not need to communicate data on the large amount of operation history 6, which also has the effect of reducing the processing load during operation analysis.

[0059] In the description of the monitoring control device 1a according to embodiment 2, the learning unit 41 is configured to learn the learning model 42a each time the operator's operation history 6 is updated, but in order to prevent overlearning, the learning unit 41 may be configured to not perform new learning even when the operation history 6 is updated.

[0060] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention. [Explanation of symbols]

[0061] 1,1a monitoring control device, 2 gaze acquisition device, 3 display unit, 4 operation information, 4a historical operation information, 5 gaze information, 5a historical gaze information, 6,6a operation history, 7 error flag, 8 similarity, 9 status information, 10 monitoring control unit, 11 operation unit, 12 gaze acquisition unit, 13 information output unit, 14 warning display unit, 15 error operation setting unit, 20,20a operation analysis unit, 21 operation information acquisition unit, 22 gaze information acquisition unit, 23 operation history acquisition unit, 24 similarity calculation unit, 25,25a error operation determination unit, 26 learning model acquisition unit, 30,30a operation history memory unit, 40 operation history processing unit, 41 learning unit, 42 learning model memory unit, 42a learning model, 100 monitoring control system, 101 processor, 102 memory, 103 input device, 104 bus.

Claims

1. a gaze information acquisition unit that acquires gaze information, which is the movement of the operator's gaze during the operation of the monitoring control; an operation history storage unit that records the operation history including the line-of-sight information for each operation; an operation error determination unit that determines whether the operation is an operation error based on the line-of-sight information acquired by the line-of-sight information acquisition unit and the operation history stored in the operation history storage unit; a warning display unit that displays a warning screen when the operation error determination unit determines that an operation error has occurred; The operation error determination unit compares the gaze information acquired by the gaze information acquisition unit with historical gaze information that is a history of the gaze information stored in the operation history storage unit and that is recorded in association with the operation that was an erroneous operation, and determines that the operation is an erroneous operation when it determines that the gaze information is similar. A monitoring and control device characterized by:

2. A gaze information acquisition unit that acquires gaze information, which is the movement of an operator's gaze during a monitoring control operation; an operation history storage unit that records the operation history including the line-of-sight information for each operation; an operation error determination unit that determines whether the operation is an operation error based on the line-of-sight information acquired by the line-of-sight information acquisition unit and the operation history stored in the operation history storage unit; a warning display unit that displays a warning screen when the operation error determination unit determines that an operation error has occurred; an operation information acquisition unit that acquires operation information that is the content of the operation, The operation history is a history in which historical operation information, which is a history of the operation information, and historical gaze information, which is a history of the gaze information, are recorded in association with each other, When historical operation information similar to the operation information acquired by the operation information acquisition unit exists in the operation history storage unit, the operation error determination unit compares the gaze information acquired by the gaze information acquisition unit with the historical gaze information recorded in association with the similar historical operation information and recorded in association with the operation that was an error, and when it determines that they are similar, determines that the operation is an error. A monitoring and control device characterized by:

3. The operation history is a history in which the historical line of sight information and an operation error flag indicating whether the operation was an operation error are associated with each other and recorded.

3. The monitoring and control device according to claim 1 or 2.

4. the operation information has one or more items that represent characteristics of the operation, The operation error determination unit compares the operation information acquired by the operation information acquisition unit with the historical operation information, and determines that the operation information acquired by the operation information acquisition unit and the historical operation information are similar based on the number of identical items among the one or more items.

3. The monitoring and control device according to claim 2.

5. the operation information includes an operation target that is a target of the operation, The operation error determination unit compares the operation information acquired by the operation information acquisition unit with the historical operation information, and if the operation target is the same, determines that the operation information acquired by the operation information acquisition unit and the historical operation information are similar.

3. The monitoring and control device according to claim 2.

6. the line-of-sight information includes one or more pieces of information associating a gaze point, which is a point on which the operator gazes for a certain period of time or more during the operation, with a gaze time, which is a time for which the operator gazes at the gaze point; The operation error determination unit compares the gaze information acquired by the gaze information acquisition unit with the historical gaze information recorded in association with the operation that was an erroneous operation, and determines whether or not they are similar based on the number of common gaze points and the gaze time of the common gaze points.

6. The monitoring and control device according to claim 1.

7. When the gaze point is a point displaying an operation object that is a target of the operation, the gaze information includes, for each gaze point, one or more pieces of state information that represent a state of the operation object; The operation error determination unit determines whether or not there is similarity based on the state information.

7. The monitoring and control device according to claim 6.

8. The operation error determination unit determines that the operation is an operation error when the total gaze time of the historical gaze information recorded in association with the operation that was an operation error is less than the total gaze time of the gaze information acquired by the gaze information acquisition unit.

8. The monitoring and control device according to claim 6 or 7.

9. The line of sight information is trajectory data representing a trajectory of a line of sight, The operation error determination unit compares the line of sight information acquired by the line of sight information acquisition unit with the historical line of sight information recorded in association with the operation that was an operation error, and determines that the operation is an operation error when it determines that the trajectory data are similar.

6. The monitoring and control device according to claim 1.

10. The operation error determination unit divides the line of sight trajectory into a plurality of points passing through the trajectory, compares the plurality of points in the line of sight information acquired by the line of sight information acquisition unit with the plurality of points in the history line of sight information recorded in association with the operation that was an erroneous operation, and determines whether the trajectory data are similar based on the distance between each of the points. The monitoring and control device according to claim 9 .

11. A gaze information acquisition unit that acquires gaze information, which is the movement of the operator's gaze during the operation of the monitoring control; an operation history storage unit that records the operation history including the line-of-sight information for each operation; an operation error determination unit that determines whether the operation is an operation error based on the line-of-sight information acquired by the line-of-sight information acquisition unit and the operation history stored in the operation history storage unit; a warning display unit that displays a warning screen when the operation error determination unit determines that an operation error has occurred; a trained model storage unit that stores a trained model for determining whether the operation is an erroneous operation; The trained model is a trained model generated by training using historical gaze information, which is a history of the gaze information included in the operation history, as input data, and historical gaze information corresponding to the operation that was an erroneous operation among the historical gaze information as a correct answer label of the input data, The operation error determination unit determines whether the operation is an operation error by using the trained model based on the line-of-sight information acquired by the line-of-sight information acquisition unit. A monitoring and control device characterized by:

12. A gaze information acquisition unit that acquires gaze information, which is the movement of the operator's gaze during the operation of the monitoring control; an operation history storage unit that records the operation history including the line-of-sight information for each operation; an operation error determination unit that determines whether the operation is an operation error based on the line-of-sight information acquired by the line-of-sight information acquisition unit and the operation history stored in the operation history storage unit; a warning display unit that displays a warning screen when the operation error determination unit determines that an operation error has occurred; an operation information acquisition unit that acquires operation information that is the content of the operation; a trained model storage unit that stores a trained model for determining whether the operation is an erroneous operation; The operation history is a history in which historical operation information, which is a history of the operation information, and historical gaze information, which is a history of the gaze information, are recorded in association with each other, The trained model is a trained model generated by training using the historical operation information and the historical gaze information as input data, and the historical operation information and historical gaze information corresponding to the operation that was an erroneous operation in the operation history as a correct answer label of the input data, The operation error determination unit determines whether the operation is an operation error based on the operation information acquired by the operation information acquisition unit and the gaze information acquired by the gaze information acquisition unit, using the trained model. A monitoring and control device characterized by:

13. The monitoring control device described in claim 11 or 12, further comprising a learning unit that, when the operation history of the operation history memory unit is updated, further trains the trained model using the updated operation history.

14. The operation history is a history in which the historical gaze information and an error flag indicating whether the operation was an error are associated and recorded, The trained model is a trained model generated by training using the operation error flag as the correct label.

13. The monitoring and control device according to claim 11 or 12.

15. further comprising an operation error setting unit that accepts an operation to switch the operation error flag recorded in association with each operation in the operation history storage unit, When the operator performs an operation to switch the operation error flag, the operation error setting unit switches the corresponding operation error flag stored in the operation history storage unit.

15. The monitoring and control device according to claim 3 or 14.

16. the operation history storage unit stores the operation history for each operator; The operation error determination unit determines whether the operation is an operation error based on the operation history corresponding to the operator who performed the operation.

16. The monitoring and control device according to claim 1.

17. a monitoring control device that performs monitoring control; a line-of-sight acquisition device that acquires line-of-sight information, which is a line-of-sight movement of the operator, when the operator performs the monitoring control operation; a display unit that displays the operation of the monitoring control; A monitoring and control system comprising: The monitoring and control device includes: a line-of-sight information acquisition unit that acquires the line-of-sight information acquired by the monitoring control device when an operator performs the operation; an operation history storage unit that records the operation history including the line-of-sight information for each operation; an error operation determination unit that compares the gaze information acquired by the gaze information acquisition unit with historical gaze information that is a history of the gaze information stored in the operation history storage unit and that is recorded in association with the operation that was an error operation, and determines that the operation is an error operation when it is determined that the gaze information is similar; a warning display unit that displays a warning screen on the display unit when the operation error determination unit determines that an operation error has occurred. A monitoring and control system characterized by:

18. A monitoring control device that performs monitoring control; a line-of-sight acquisition device that acquires line-of-sight information, which is a line-of-sight movement of the operator, when the operator performs the monitoring control operation; a display unit that displays the operation of the monitoring control; A monitoring and control system comprising: The monitoring and control device includes: a line-of-sight information acquisition unit that acquires the line-of-sight information acquired by the monitoring control device when an operator performs the operation; an operation history storage unit that records the operation history including the line-of-sight information for each operation; an operation information acquisition unit that acquires operation information that is the content of the operation; the operation history is a history in which historical operation information, which is a history of the operation information, and historical gaze information, which is a history of the gaze information, are recorded in association with each other, and when historical operation information similar to the operation information acquired by the operation information acquisition unit exists in the operation history storage unit, an error operation determination unit compares the gaze information acquired by the gaze information acquisition unit with the historical gaze information recorded in association with the operation that was an error operation among the historical gaze information recorded in association with the similar historical operation information, and determines that the operation is an error operation when it is determined that they are similar; a warning display unit that displays a warning screen on the display unit when the operation error determination unit determines that an operation error has occurred. A monitoring and control system characterized by:

19. When an operator performs a monitoring control operation, acquiring line-of-sight information which is a line-of-sight movement of the operator during the operation; a step of comparing the acquired gaze information with historical gaze information that is a history of the gaze information and that is recorded in association with the operation that was an erroneous operation, and determining that the operation is an erroneous operation when it is determined that the acquired gaze information is similar to the historical gaze information that is a history of the gaze information; displaying a warning screen when it is determined that the operation is an erroneous operation; A monitoring and control method comprising:

20. When an operator performs a monitoring control operation, a step of acquiring gaze information which is the movement of the operator's gaze during the operation; acquiring operation information that is the content of the operation; a step of recording an operation history for each of the operations, the operation history being a history of the operation information and a history of gaze information associated with each other; a step of comparing the acquired gaze information with the historical gaze information recorded in association with the operation that was an erroneous operation among the historical gaze information when the historical operation information similar to the operation information exists, and determining that the operation is an erroneous operation when it is determined that the acquired gaze information is similar to the historical operation information; displaying a warning screen when it is determined that the operation is an erroneous operation; A monitoring and control method comprising:

21. When an operator performs a monitoring control operation, acquiring line-of-sight information which is a line-of-sight movement of the operator during the operation; a step of comparing the acquired gaze information with historical gaze information that is a history of the gaze information and that is recorded in association with the operation that was an erroneous operation, and determining that the operation is an erroneous operation when it is determined that the acquired gaze information is similar to the historical gaze information that is a history of the gaze information; displaying a warning screen when it is determined that the operation is an erroneous operation; A monitoring and control program that causes a computer to execute the above.

22. A step of acquiring line-of-sight information, which is the movement of the operator's line of sight during a monitoring control operation performed by an operator; acquiring operation information that is the content of the operation; a step of recording an operation history for each of the operations, the operation history being a history of the operation information and a history of gaze information associated with each other; a step of comparing the acquired gaze information with the historical gaze information recorded in association with the operation that was an erroneous operation among the historical gaze information when the historical operation information similar to the operation information exists, and determining that the operation is an erroneous operation when it is determined that the acquired gaze information is similar to the historical operation information; displaying a warning screen when it is determined that the operation is an erroneous operation; A monitoring and control program that causes a computer to execute the above.

Images