Method for analyzing the cognitive process of a remote operator, system for analyzing the cognitive process of a remote operator, and method for designing a remote operation system.

Abstract

To provide an analysis method for a cognitive process for a remote ship operator, an analysis system for a cognitive process for a remote ship operator, and a design method for a remote ship operation system for improving safety of remote ship operation necessary in designing a remote ship operation system.SOLUTION: An analysis method for a cognitive process for a remote ship operator executes by using a computer 30: a scenario reading step S1 of reading a predetermined scenario for conducting an experiment of remote ship operation; a ship operation and wake data acquisition step S2 of acquiring and storing ship operation and wake data of simulated remote ship operation performed by a remote ship operator α by using ship operation means 10 on the basis of the scenario; line-of-sight data acquisition step S3 of acquiring and storing line-of-sight data of the remote ship operator α; and a scenario execution determination step S6 of determining if the predetermined scenario is executed.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a method for analyzing the cognitive process of a remote operator who remotely operates a ship, an analysis system for the cognitive process of a remote operator for executing the analysis method, and a design method for a remote control ship system using the analysis method.

Background Art

[0002] In recent years, the development of automatic ship operation technology aimed at improving safety and reducing the burden on crew has been carried out. One of the automatic operation technologies is remote control of a ship, in which the ship is operated from a location physically separated from the ship via a communication line such as the Internet.

[0003] Here, Patent Document 1 treats the fact that the driver's line of sight detected by the line-of-sight detection unit continuously exists within a specific range for a threshold time as fixation on the specific range, and does not treat the fact that the line of sight detected by the line-of-sight detection unit continuously exists within the specific range for a time less than the threshold time and the fact that the line of sight detected by the line-of-sight detection unit does not exist within the specific range as fixation on the specific range. When fixation on the range of the mirror surface of the rearview mirror is not performed by the driver during a specific period immediately before the simulation target vehicle changes its traveling direction, a driving simulator (driving information processing device) that executes a notification in the simulation to alert that safety confirmation using the rearview mirror has not been performed is disclosed. Paragraph 0176 describes that the driving information processing device can also be applied to the operation of a ship. Furthermore, Patent Document 2 discloses a remote control device comprising: an operation detection unit that detects the gaze of a user facing a display device that displays an image captured by an imaging device attached to a moving object such as a multicopter, and outputs the user's gaze point on the image, and acquires gaze point information from the gaze detection device and determines whether the gaze point indicated by the acquired information is included in a predetermined area; and a signal output control unit that, if the gaze point indicated by the information is included in the predetermined area, transmits a signal for controlling the moving object associated with that area. Furthermore, Patent Document 3 discloses an over-attention state determination device comprising: object detection means for detecting objects that may be the object of gaze of a person to be determined, such as a driver of a car, around the person to be determined, and calculating the direction in which the detected object exists; gaze direction detection means for detecting the direction of the person's gaze; eye movement speed detection means for detecting the speed of the person's eye movements; and over-attention determination means for determining that the person to be determined is in an over-attention state where their gaze is focused on a specific object if, for a predetermined time, the direction of the person's gaze is directed towards a specific object detected by the object detection means, and the speed of eye movements detected by the eye movement speed detection means is below a first threshold. Paragraph 0063 states that the device is also applicable to ships. Furthermore, Patent Document 4 discloses a driving assistance device comprising: a viewing distance estimation means for detecting the state of the driver's eyes and estimating the viewing distance when the driver is visually observing an object; an actual distance detection means for detecting the actual distance from the driver to the object; and an assistance means for providing driving assistance based on a focus evaluation index corresponding to the difference between the viewing distance estimated by the viewing distance estimation means and the actual distance detected by the actual distance detection means. Paragraph 0068, etc., states that it is also applicable to ship operators. Furthermore, Patent Document 5 discloses a data creation device comprising: a collection unit for collecting time-series data from equipment in a production site such as a factory; a storage unit for storing storage rules that define the items of related data to be saved for each event; and a conversion unit that extracts related data from the time-series data for each event according to the storage rules and converts it into event data. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2019-124892 [Patent Document 2] Japanese Patent Publication No. 2020-5114 [Patent Document 3] Japanese Patent Publication No. 2016-62330 [Patent Document 4] Japanese Patent Publication No. 2018-147479 [Patent Document 5] Japanese Patent Publication No. 2020-71570 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] While remote ship operation environments consist of observation and communication equipment, the amount of information transferred and the range of observation are limited by the capabilities of the equipment and the communication environment. Therefore, leaks or delays in necessary information could affect the remote operator's ship handling, potentially raising safety concerns. On the other hand, the inventors of this invention focused on the fact that the information required by a remote operator for ship handling operations is not uniformly detailed in all directions, but rather the necessary information is limited depending on the ship handling operation. Therefore, by observing the information necessary for the operation that the remote operator is about to perform on the target vessel and transmitting it to the remote operator, it is possible to improve the amount of information transmitted and the safety even within the limited range of observation. However, no method has been proposed to date for accurately determining what information is necessary for each ship handling operation. Therefore, the present invention aims to provide a method for analyzing the cognitive process of a remote operator, a system for analyzing the cognitive process of a remote operator, and a method for designing a remote ship handling system, which can grasp the information necessary for each ship handling operation performed by the remote operator when designing a remote ship handling system. It should be noted that while Patent Document 1 attempts to improve safety by detecting the driver's gaze, it assumes a state of being seated in the driver's seat and does not relate to driving operations from a physically distant location, such as remote operation of a ship. Nor does it attempt to analyze the cognitive process of a remote operator. Patent Document 2 aims to improve the operability of remote control of a moving object using line of sight, and does not attempt to analyze the cognitive process of the remote operator. Patent Document 3 attempts to determine a state of excessive attention with higher accuracy by detecting the gaze and eye movement speed of the person being assessed, but it does not relate to driving operations from a physically distant location, such as remote ship operation. Nor does it attempt to analyze the cognitive process of a remote ship operator. Patent Document 4 aims to provide appropriate driving assistance in accordance with the driver's visual perception of the object being seen, but it does not relate to driving operations from a physically distant location, such as remote operation of a ship. Nor does it attempt to analyze the cognitive process of a remote operator. Patent Document 5 aims to improve the overall production efficiency of a factory by accumulating information that organizes the operating status when a specific event occurs in the equipment, and does not attempt to analyze the cognitive process of a remote operator. [Means for solving the problem]

[0006] A method for analyzing the cognitive process of a remote operator corresponding to claim 1, wherein a computer is used to analyze the cognitive process of a remote operator who remotely operates a vessel, Computers A scenario loading step for loading a predetermined scenario for conducting a remote ship operation experiment; a ship operation and track data acquisition step for acquiring and saving ship operation and track data of a simulated remote ship operation performed by a remote ship operator based on the scenario using a ship operation means; and a gaze data acquisition step for acquiring and saving the gaze data of the remote ship operator. A video data acquisition step involves acquiring and saving video data of a remote operator performing simulated remote operation in a remote operation experiment, and an interview data acquisition step involves acquiring and saving interview data of the remote operator during the simulated remote operation. Scenario execution determination step to determine whether a predetermined scenario has been executed. and Execute If the scenario execution decision step determines that the scenario has been executed, the system proceeds to an analysis step to analyze the cognitive process of the remote operator. The computer reads the correspondence definition result, which defines the correspondence between the gaze target, including the maneuvering means in the simulated remote operation of the remote operator, and the acquired information, based on the video data and interview data. Based on the correspondence definition result, it performs a time-series gaze target data conversion step to convert the gaze data into time-series gaze target data, a steering / speed adjustment action data conversion step to convert the maneuvering / track data into steering / speed adjustment action data, and a time-series action data conversion step to convert the time-series gaze target data and steering / speed adjustment action data into time-series action data. In the analysis step, the system analyzes the cognitive process that associates the gaze target and actions of the remote operator based on the time-series action data. It is characterized by doing so. According to the present invention as described in claim 1, it becomes possible to analyze the cognitive process of the remote operator based on data related to the operation content and gaze in simulated remote operation obtained through observation in experiments, thereby grasping the information necessary for each operation performed by the remote operator. The analysis results can then be used to develop a remote operation system that improves the safety of remote operation.

[0007] Claim 2 The present invention is characterized in that multiple remote operators perform simulated remote operation in a remote operation experiment for the same scenario. Claim 2 According to the present invention described above, by acquiring experimental data from multiple remote operators, it becomes possible to analyze the cognitive processes of remote operators in a more multifaceted and objective manner. Furthermore, it becomes possible to analyze differences based on the experience of remote operators, such as between veterans and novice operators.

[0008] Claim 3 The present invention described herein is characterized in that the scenario includes a handover action that transitions from simulated automated navigation of a vessel to simulated remote operation. Claim 3 According to the present invention described above, it becomes possible to analyze the cognitive process of the remote operator based on data obtained from experiments during the handover process, allowing for remote monitoring of the autonomous vessel and remote fallback in emergencies, thereby improving the safety of remote operation.

[0009] Claim 4 The present invention is characterized in that the acquisition of the remote operator's gaze data is performed by an eye tracker worn by the remote operator. Claim 4 According to the present invention described herein, gaze data can be acquired efficiently.

[0010] Claim 5 The present invention is characterized in that the ship handling means is capable of providing scenery information, nautical chart information, control panel information, and rudder information. Claim 5According to the present invention described in [reference], it is possible to use, for example, the timing when a remote operator was gazing at landscape information, nautical chart information, console information, and rudder information necessary for remote operation during an experiment, etc. for analysis of the remote operator's cognitive process.

[0011] Claim 6 The present invention described in [reference] Computers includes a time-series vessel position data conversion step of converting a predetermined scenario read in a scenario loading step and the read operation / trajectory data into time-series vessel position data, and an event definition step of defining an event including start and end times based on the time-series vessel position data and executing In the analysis step, the cognitive process of the remote operator in events associated with the ship's position that occurred during simulated remote operation is analyzed. and is characterized by this. Claim 6 According to the present invention described in [reference], it is possible to analyze the cognitive process of a remote operator in an event associated with a vessel position that occurred in a simulated remote operation.

[0012] Claim 7 The present invention described in [reference] Computers includes an event-by-event time-series behavior data conversion step of converting the time-series behavior data converted in the time-series behavior data conversion step and the events defined in the event definition step into time-series behavior data for each event, and a time-series behavior log recording step of recording a time-series behavior log based on the time-series behavior data for each event and executing In the analysis step, the cognitive process of the remote operator is analyzed using a time-series behavior log based on the target of attention and actions taken when an event occurred. and is characterized by this. Claim 7 According to the present invention described in [reference], it is possible to analyze the cognitive process of a remote operator in detail and accurately based on a time-series behavior log based on the gazing target and actions at the time of event occurrence.

[0013] Claim 8The remote operator cognitive process analysis system corresponding to the description is a system for performing a method for analyzing the cognitive process of a remote operator, and is characterized by comprising a computer, a scenario providing means for providing scenarios and progress, a ship handling simulator as a ship handling means, a ship handling and track data recording device for storing acquired ship handling and track data, and a gaze data recording device for storing acquired gaze data. Claim 8 According to the present invention described above, the remote operation of the remote operator required for analysis can be easily performed using a ship handling simulator, and the necessary data can be recorded, thereby facilitating the analysis of the remote operator's cognitive process and improving the efficiency of the analysis.

[0014] Claim 9 The present invention described herein is characterized by recording acquired gaze data in a gaze data recording device via a gaze analysis device. Claim 9 According to the present invention described herein, by analyzing and recording the observed gaze data, the use of gaze data can be made easier.

[0015] Claim 10 The present invention is characterized by comprising a video data recording device that stores video data of simulated remote operation by a remote operator acquired with a video camera for filming ship operation, and interview video data of an interview with the remote operator acquired with a video camera for recording interviews. Claim 10 According to the present invention described above, when analyzing the cognitive process of a remote operator, it becomes possible to perform the analysis using video data of simulated remote operation and interview data, thereby enabling a more detailed and accurate analysis of the remote operator's cognitive process.

[0016] Claim 11 The present invention is characterized by comprising an interview data recording device that stores acquired interview data with a remote operator. Claim 11According to the present invention described herein, interview data stored in an interview data recording device can be used to analyze the cognitive processes of a remote operator.

[0017] Claim 12 The present invention is characterized in that the ship handling simulator has a landscape screen, a nautical chart screen, a control panel, and a rudder confirmation screen including the rudder angle. Claim 12 According to the present invention described above, in the experiment, the simulator has a landscape screen, a nautical chart screen, a control panel, and a rudder confirmation screen necessary for remote operation, so that the timing at which the remote operator was looking at or operating these can be used to analyze the remote operator's cognitive process.

[0018] Claim 13 In the design method for a remote ship handling system corresponding to the description, based on the time-series behavior log recorded in the time-series behavior log recording step of the method for analyzing the cognitive process of the remote ship operator, Based on the cognitive processes of the remote operator, the computer can provide the remote operator with information about what to focus on and what actions to take when an event occurs, at the necessary time. This invention is characterized by its design of remote control systems for ships. Claim 13 According to the present invention described herein, a highly safe remote ship handling system can be provided by using a time-series action log based on the target of attention and actions taken when an event occurs. [Effects of the Invention]

[0019] According to the method for analyzing the cognitive process of a remote operator of a vessel, based on data on the operation content and gaze direction in simulated remote operation obtained through observations in experiments, it becomes possible to analyze the cognitive process of the remote operator and grasp the information necessary for each operation performed by the remote operator. The analysis results can then be used to develop a remote operation system that improves the safety of remote operation.

[0020] Furthermore, when multiple remote operators perform simulated remote operation in remote operation experiments under the same scenario, obtaining experimental data from multiple remote operators makes it possible to analyze the cognitive processes of the remote operators from a more multifaceted and objective perspective. It also allows for the analysis of differences based on the experience of the remote operators, such as between veterans and novice operators.

[0021] Furthermore, if the scenario includes a handover action from simulated automated ship operation to simulated remote operation, it becomes possible to analyze the cognitive process of the remote operator based on the data obtained during the handover action in the experiment. This allows for remote monitoring of the automated ship and remote fallback in emergencies, thereby improving the safety of remote operation.

[0022] Furthermore, if the gaze data of the remote operator is acquired using an eye tracker worn by the remote operator, the gaze data can be acquired efficiently.

[0023] Furthermore, if the ship handling system can provide scenery information, nautical chart information, control panel information, and rudder information, the timing at which the remote operator focused on the scenery information, nautical chart information, control panel information, and rudder information necessary for remote operation during the experiment can be used to analyze the remote operator's cognitive process.

[0024] Also, Computers A time-series ship position data conversion step that converts a predetermined scenario loaded in the scenario loading step and the loaded ship handling and track data into time-series ship position data, and an event definition step that defines an event including the start and end times based on the time-series ship position data. and Execute In the analysis step, the cognitive process of the remote operator in events associated with the ship's position that occurred during simulated remote operation is analyzed. In this case, it becomes possible to analyze the cognitive process of the remote operator in events associated with the ship's position that occurred during simulated remote operation.

[0025] Also, ComputersThe time-series behavior data converted in the time-series behavior data conversion step and the event defined in the event definition step are converted into event-specific time-series behavior data in the event-specific time-series behavior data conversion step, and the time-series behavior log recording step is recorded based on the event-specific time-series behavior data. and Execute In the analysis step, the cognitive process of the remote operator is analyzed using a time-series behavior log based on the target of attention and actions taken when an event occurred. In such cases, a time-series behavior log based on the focus and actions taken at the time of the event allows for a detailed and accurate analysis of the remote operator's cognitive process.

[0026] Furthermore, according to the remote operator cognitive process analysis system of the present invention, the remote operation of the remote operator necessary for analysis can be easily performed using a ship operation simulator, and the necessary data can be recorded, thereby facilitating the analysis of the remote operator's cognitive process and improving the efficiency of the analysis.

[0027] Furthermore, when recording acquired gaze data to a gaze data recording device via a gaze analysis device, analyzing the observed gaze data before recording it makes it easier to utilize the gaze data.

[0028] Furthermore, if the system is equipped with a video data recording device that stores video data of simulated remote operation by the remote operator acquired with a video camera for filming ship handling operations, and interview video data of interviews with the remote operator acquired with a video camera for recording interviews, then when analyzing the cognitive process of the remote operator, it becomes possible to use the video data of the simulated remote operation and the interview data for analysis, thereby enabling a more detailed and accurate analysis of the cognitive process of the remote operator.

[0029] Furthermore, if the system is equipped with an interview data recording device that stores the interview data acquired from the remote operator, the interview data stored in the interview data recording device can be used to analyze the remote operator's cognitive processes.

[0030] Furthermore, if the ship handling simulator includes a landscape screen, a nautical chart screen, a control panel, and a rudder confirmation screen including the rudder angle, then in the experiment, the simulator having the landscape screen, nautical chart screen, control panel, and rudder confirmation screen necessary for remote operation allows for the analysis of the remote operator's cognitive process, such as the timing at which the remote operator was looking at or operating these screens.

[0031] Furthermore, according to the design method for the remote ship operation system of the present invention, a highly safe remote ship operation system can be provided by using a time-series action log based on the target of attention and actions taken when an event occurs. [Brief explanation of the drawing]

[0032] [Figure 1] Data acquisition flow in the method for analyzing the cognitive process of a remote operator according to an embodiment of the present invention. [Figure 2] This block diagram focuses on the equipment used for data acquisition within the analysis system for the cognitive processes of the remote operator. [Figure 3] Conceptual image of the ship handling simulator. [Figure 4] Data analysis flow in the method for analyzing the cognitive process of the remote operator. [Figure 5] This block diagram focuses on the equipment used for data analysis within the system for analyzing the cognitive processes of the remote operator. [Figure 6] This figure shows an example of analysis based on a list of the remote operator's actions for each event. [Modes for carrying out the invention]

[0033] This document describes an embodiment of the present invention that explains a method for analyzing the cognitive process of a remote operator, a system for analyzing the cognitive process of a remote operator, and a method for designing a remote operation system. The method for analyzing the cognitive processes of remote ship operators involves observing the person (remote ship operator) performing the task in remote ship operation experiments using a simulator or an actual ship, collecting data, and then analyzing the remote ship operator's ship operation work from the perspective of cognitive processes based on the acquired data. Figure 1 shows the data acquisition flow in the method for analyzing the cognitive process of a remote operator according to this embodiment, Figure 2 is a block diagram focusing on the equipment used for data acquisition in the analysis system for the cognitive process of a remote operator, and Figure 3 is an image diagram of a ship handling simulator. The analysis system shown in Figure 2 includes a ship handling simulator 10 having an operation panel 11 and a display 12 and a function to simulate the marine environment in which a ship operates; a ship handling and track data recording device 20 that acquires and stores ship handling and track data from the ship handling simulator 10 during the experiment (simulation); a computer 30 that processes various data obtained from the experiment; a scenario providing means 40 that provides the ship handling simulator 10, etc., with the scenario and progress to be used in the experiment; an eye-tracking analysis device 50 that analyzes the gaze of the remote operator α and generates eye-tracking data; an eye-tracking data recording device 60 that acquires and stores the eye-tracking data from the eye-tracking analysis device 50; two video cameras 70 for filming ship handling operations that film the remote operator α performing simulated remote ship handling; and a device for filming the interview with the remote operator α conducted by interviewer β. 80 video cameras for recording interviews The system includes a video data recording device 90 for recording video data from a video camera 70 for filming ship handling operations and a video camera 80 for recording interviews, an interview data recording device 100 for recording data summarizing the results of interviews with the remote ship operator α, and a scenario display and operation device 110 such as a touch panel monitor for checking the progress of the scenario and inputting operations related to the scenario. The scenario display and operation device 110 can also be configured as part of the scenario providing means 40.

[0034] As shown in Figure 1, when a remote ship operation experiment is started, the ship operation simulator 10 loads a predetermined scenario for conducting the remote ship operation experiment provided by the scenario provisioning means 40 (S1: Scenario Loading Step). The scenario provisioning means 40 includes a storage means capable of storing multiple scenarios and a CPU, etc., that manages the progress of the scenarios provided to the ship operation simulator 10. The scenario display and operation device 110 is positioned near the seat of experimenter γ, who is conducting the remote ship operation experiment. The scenario display and operation device 110 displays a scenario display screen 110A that shows the progress of the scenario to experimenter γ, etc., and a scenario operation screen 110B that accepts operations related to the scenario from experimenter γ, etc. The scenario is predefined by a person (experimenter γ, etc.), and the defined scenario is stored in the scenario provisioning means 40, etc., and is provided from the scenario provisioning means 40 to the ship handling simulator 10 automatically or by selection by experimenter γ, etc., at the start of the experiment. Examples of scenarios include a scenario that reproduces encounters with other ships and weather and sea conditions based on past operational records, a scenario that involves navigating in weather and sea conditions that are expected to occur in the future, or a scenario in which approaches to other ships and changes in weather and sea conditions occur randomly. Furthermore, if the scenario includes a handover action from simulated automated ship operation to simulated remote operation, it becomes possible to analyze the cognitive process of the remote operator α based on the data obtained during the handover action in the experiment. This allows for remote monitoring of the automated ship and remote fallback in emergencies, thereby improving the safety of remote operation.

[0035] When the ship handling simulator 10 starts a simulation based on a predetermined scenario, information to be provided to the remote operator α is displayed on the display 12. The display 12 provided in this embodiment is a scenery screen 12A that displays the scenery in front of the ship, a nautical chart screen 12B that displays nautical charts, and a rudder confirmation screen 12C that displays information related to the rudder, such as the rudder angle. The remote operator α is provided with scenery information from the scenery screen 12A, nautical chart information from the nautical chart screen 12B, and rudder information from the rudder confirmation screen 12C. In addition, control panel information is provided from the control panel 11. Control panel information is information that shows the status of the equipment operated by the remote operator α. For example, if it is a touch panel type control panel 11, it would be images of the operation of the rudder or engine, and if it is a control panel 11 with devices such as a steering wheel or engine control levers installed, it would be the current status of those devices. Remote operator α is a subject (seafarer) who performs simulated remote operation of a ship based on a scenario, and operates the control panel 11 while looking at the display 12. The ship handling simulator 10 calculates state variables based on the inputs of remote operator α to the control panel 11, and proceeds with the simulation while reflecting the calculated state variables in changes such as the scenery and rudder angle displayed on the display 12. The maneuvering and track data acquired during the simulated remote ship operation is transmitted from the ship operation simulator 10 to the ship operation and track data recording device 20 for storage (S2: Ship operation and track data acquisition step). The ship operation and track data includes, for example, the time and content of operations performed by the remote operator α on the control panel 11, as well as the course change information and speed information of the target vessel.

[0036] The remote operator α is fitted with an eye tracker to observe their gaze and eye movements during simulated remote operation. The observation results are input to the gaze analysis device 50, and the gaze data obtained from the analysis is output to the gaze data recording device 60 and saved along with the recording time (S3: Gaze data acquisition step). By using an eye tracker worn by the remote operator α to acquire the operator's gaze data, the gaze data can be acquired efficiently. Furthermore, by analyzing the acquired gaze data with the gaze analysis device 50 and recording it in the gaze data recording device 60, the use of the gaze data can be made easier. Furthermore, as described above, the ship handling simulator 10 provides scenery information, nautical chart information, control panel information, and rudder information to the remote operator α. This makes it possible to understand when the remote operator α was looking at or operating the scenery information, nautical chart information, control panel information, and rudder information necessary for remote operation during the experiment, and to use this information to analyze the remote operator α's cognitive process.

[0037] During the experiment, the simulated remote operation (operation work) by remote operator α is captured by a video camera 70 for recording operation work, and the acquired video data is saved in the video data recording device 90 along with the recording time (S4: video data acquisition step). The video camera 70 for recording ship handling operations consists of a first video camera 71 pointed towards the display 12 and a second video camera 72 pointed towards the remote operator α. As a result, the video data of the ship handling operations includes video data recording the information displayed on the display 12 and video data recording the head and body movements of the remote operator α performing the ship handling operations.

[0038] Interviewer β conducts an interview with remote operator α once a scenario is completed. Note that interviewer β may also be experimenter γ. During the interview, for example, questions may be asked about the amount of information displayed on display 12 during the simulated remote operation, such as whether the information was sufficient and whether the timing of its presentation was appropriate. The information obtained from the interview is entered into computer 30 in a predetermined format and saved in interview data recording device 100 (S5: Interview data acquisition step). By acquiring and saving post-experiment interview data with remote operator α, in addition to video data recording the operation during the experiment, it becomes possible to analyze the cognitive process of remote operator α in more detail and accurately. Furthermore, the interview is recorded using an interview recording video camera 80, and the acquired interview video data is saved in a video data recording device 90. By acquiring and saving the interview video data, it becomes possible to analyze the cognitive process of the remote operator α using both the video data of the simulated remote operation and the interview data, thereby enabling a more detailed and accurate analysis of the cognitive process of the remote operator α.

[0039] After the interview data acquisition step S5, experimenter γ determines whether the predetermined scenario has been completed based on the progress of the scenario provided by the scenario provision means 40 (S6: Scenario execution determination step). In the scenario execution determination step S6, for example, in the case of an experiment in which the same scenario is executed only once or repeatedly, it is determined whether the number of scenarios has been executed the prescribed number of times. In the case of an experiment in which different scenarios are executed consecutively, such as Experiment 1, Experiment 2, ..., it is determined whether all scenarios have been executed. If experimenter γ determines in scenario execution decision step S6 that the predetermined scenario has been completed (Yes), the experiment is terminated. On the other hand, if experimenter γ determines in scenario execution decision step S6 that the predetermined scenario has not been completed (No), the experimenter instructs the ship handling simulator 10 to execute the next scenario. In this case, steps S2 to S6 are repeated until it is determined that the predetermined scenario has been completed. Furthermore, it is also possible to separately place the scenario display and operation device 110 near the ship handling means 10, or to integrate it with the scenario provision means 40, so that the remote operator α can check the progress of the scenario on the scenario display screen 110A, make a decision in the scenario execution decision step S6, and then operate the ship handling simulator 10 using the scenario operation screen 110B based on the result of that decision, thereby enabling the remote operator α to conduct the experiment themselves. Furthermore, the decision in the scenario execution decision step S6 and the subsequent actions based on that decision can be performed by the ship handling simulator 10, rather than by a person such as experimenter γ or remote operator α.

[0040] In this way, by using the computer 30 to perform a scenario loading step S1 in which a predetermined scenario for conducting a remote operation experiment is loaded, a ship operation and track data acquisition step S2 in which ship operation means 10 is used to acquire and save ship operation and track data of the simulated remote operation performed by the remote operator α based on the scenario, a gaze data acquisition step S3 in which the gaze data of the remote operator α is acquired and saved, and a scenario execution determination step S6 in which it is determined whether the predetermined scenario has been executed, it becomes possible to analyze the cognitive process of the remote operator α based on the ship operation content and gaze data obtained from observations in the experiment, and to grasp the information necessary for each ship operation operation performed by the remote operator α, and to develop a remote operation system that improves the safety of remote operation by utilizing the analysis results. Furthermore, by including a computer 30, a scenario providing means 40 that provides scenarios and progress, a ship handling simulator as a ship handling means 10, a ship handling / track data recording device 20 that stores acquired ship handling / track data, and a gaze data recording device 60 that stores acquired gaze data, the remote operation of the remote operator α necessary for analysis can be easily performed using the ship handling simulator 10, and the necessary data can be recorded, thereby facilitating the analysis of the cognitive process of the remote operator α and improving the efficiency of the analysis.

[0041] Furthermore, it is preferable for multiple remote operators α to perform simulated remote operation in the remote operation experiment for the same scenario. This allows for the acquisition of experimental data from multiple remote operators α for a single scenario, making it possible to analyze the cognitive processes of remote operators α from a more multifaceted and objective perspective. In addition, it becomes possible to analyze differences based on the experience of remote operators α, such as between veterans and novice operators.

[0042] Figure 4 shows the data analysis flow in the method for analyzing the cognitive process of a remote operator, and Figure 5 is a block diagram focusing on the equipment used for data analysis in the system for analyzing the cognitive process of a remote operator. In addition to the equipment shown in Figure 2, the analysis system includes an interview data display device 120 such as a monitor and a time-series behavior log recording device 130 such as a storage device. The interview data display device 120 is used to display data stored in the interview data recording device 100 and to play back interview video data recorded in the video data recording device 90. The interview video data is recorded supplementarily to the data stored in the interview data recording device 100 and is used as appropriate for verification during analysis.

[0043] In analyzing the cognitive process of the remote operator, the first step is to load the correspondence definition results into the computer 30 (S11: Correspondence definition loading step). The correspondence definition result defines the correspondence between the gaze target, including the ship handling means (ship handling simulator) 10 in the simulated remote operation of the remote ship operator α, and the acquired information, based on the video data acquired in the video data acquisition step S4 and stored in the video data recording device 90, and the interview data acquired in the interview data acquisition step S5 and stored in the interview data recording device 100. The gaze target is, for example, the landscape screen 12A or the nautical chart screen 12B, and the acquired information is, for example, the position of other ships or the coastal topography. By loading the correspondence definition result into the computer 30, it becomes possible to analyze the cognitive process of the remote ship operator α in detail and accurately.

[0044] After the correspondence definition reading step S11, the computer 30 reads the gaze data stored in the gaze data recording device 60 and the ship handling / track data stored in the ship handling / track data recording device 20. Then, based on the acquired correspondence definition results, it processes the gaze data into time-series gaze target data (S12: time-series gaze target data conversion step) and the ship handling / track data into steering / ship speed adjustment action data (S13: steering / ship speed adjustment action data conversion step). The time-series gaze target data is data in which the gaze targets (gaze) of the remote operator α are arranged in chronological order, and the steering / ship speed adjustment action data is data of the time when the rudder was operated and the time when the ship speed was changed. The computer 30 then performs a conversion process from time-series gaze target data and steering / ship speed adjustment action data to time-series action data (S14: Time-series action data conversion step). The time-series action data is data arranged in chronological order, with the gaze target and whether or not an action was performed associated with the remote operator α. This makes it possible to analyze in detail the cognitive process by which the gaze target and actions of the remote operator α are associated, based on the time-series action data in which the gaze target and steering / ship speed adjustment actions are associated.

[0045] Furthermore, the computer 30 converts the predetermined scenario read by the ship handling simulator 10 in the scenario reading step S1 and the ship handling and track data read after the correspondence definition reading step S11 into time-series ship position data (S15: time-series ship position data conversion step). The time-series ship position data is a sequence of the positions of the ships being handled, arranged in chronological order. The computer 30 then defines an event, including its start and end times, based on the time-series ship position data (S16: event definition step). An event could be, for example, an emergency that occurred during navigation, avoidance of another ship, or keeping watch for other ships or obstacles. This makes it possible to analyze the cognitive process of the remote operator α in events associated with the ship's position that occurred during the simulated remote operation.

[0046] Furthermore, the computer 30 converts the time-series behavior data converted in the time-series behavior data conversion step S14 and the events defined in the event definition step S16 into time-series behavior data for each event (S17: event-specific time-series behavior data conversion step), and records a time-series behavior log based on the time-series behavior data for each event (S18: time-series behavior log recording step). The time-series behavior data for each event is a sequence of the remote operator α's gaze targets and actions from the start to the end of each event, arranged in chronological order. This makes it possible to analyze the cognitive process of the remote operator α in detail and accurately using the time-series behavior log based on the gaze targets and actions at the time of event occurrence.

[0047] Here, FIG. 6 is a diagram showing an analysis example based on the list of actions for each event of a remote operator, and is based on the results of Experiment 1 of Subject 1 when a plurality of experiments (scenarios) are also conducted for a plurality of subjects (remote operators). The upper side of FIG. 6 is a list of actions for each event, which shows the line of sight destination and the presence or absence of operations of Subject 1 for each task in each event in chronological order. The row of "line of sight" indicates the point of fixation (gaze point) of Subject 1 regarding the display 12 and the operation panel 11. "Scene" refers to the landscape screen 12A, "sea" refers to the nautical chart screen 12B, "rudder" refers to the rudder confirmation screen 12C, and "panel" refers to the operation panel 11. Also, in the row of "operation", the operation of the operation panel 11 by Subject 1 is indicated by the character "operation". In the design method of the remote ship control system, based on the data shown on the upper side of FIG. 6, a remote ship control system for a ship that allows remote operator α to sail safely is designed. The lower side of FIG. 6 shows an example of analyzing the fixation time for each fixation destination in Task 1-1 and Task 1-2 for the first event based on the upper data.

[0048] As described above, although there are limitations in the information transfer amount and the observation range in the remote ship control environment, for example, based on the analysis results shown in FIG. 6, considering the information with different high necessities and the provision timing depending on the ship operation, in constructing the remote ship control environment, such as setting to preferentially observe and transfer the information necessary for the operation that remote operator α is about to perform on the ship to be controlled rather than other information, by appropriately allocating the limited capabilities of the observation equipment and communication equipment to support the required ship operation, it is possible to prevent insufficient provision or delay of highly necessary information and improve the safety of the remote ship control system. In addition, as other analysis examples based on the list of actions for each event shown on the upper side of FIG. 6, there are analysis of the required time for each element operation from when remote operator α recognizes certain information until steering is performed, analysis of the required time for operations in a situation where a communication failure or communication delay occurs, etc.

Industrial Applicability

[0049] This invention can be used in the development of remote ship operation systems to help design appropriate remote ship operation systems that enable remote operators to safely control ships remotely. It can also optimize the ship operation environment, observation equipment, and communication equipment placement. [Explanation of Symbols]

[0050] 10. Maneuvering methods (Ship handling simulator) 11 Control panel 12A Landscape Screen 12B Chart screen 12C Rudder Confirmation Screen 20. Ship handling and track data recording device 30 Computers 40 Scenario Provisioning Methods 50 Line of sight analysis device 60 Eye-tracking data recording device 70 Video camera for recording ship handling operations 80 Video cameras for recording interviews 90 Video data recording device 100 Interview data recording device 110 Scenario display and operation means (scenario provision means) 120 Interview Data Display Device 130 Time-series activity log recording device α Remote operator S1 Scenario Loading Step S2 Ship handling and track data acquisition step S3 Eye-tracking data acquisition step S4 Video data acquisition step S5 Interview Data Acquisition Step S6 Scenario Execution Decision Step S11 Correspondence definition loading step S12 Time-series data conversion step S13 Steering and ship speed adjustment action data conversion step S14 Time-series behavioral data transformation step S15 Time-series ship position data conversion step S16 Event Definition Step S17 Event-by-Event Time-Series Behavioral Data Transformation Step S18 Time-series action logging step

Claims

1. A method for analyzing the cognitive process of a remote operator who remotely controls a ship using a computer, The aforementioned computer, A scenario loading step involves loading a predetermined scenario for conducting the aforementioned remote ship operation experiment, A ship handling and track data acquisition step involves acquiring and saving ship handling and track data of a simulated remote ship handling operation performed by the remote ship operator based on the scenario using a ship handling means, A gaze data acquisition step which involves acquiring and saving gaze data of the remote operator, A video data acquisition step of acquiring and saving video data of the simulated remote operation performed by the remote operator in the experiment of remote operation, An interview data acquisition step involves acquiring and saving interview data from the remote operator when they perform the simulated remote operation of the vessel, Scenario execution determination step to determine whether the predetermined scenario has been executed. And execute, If the scenario execution decision step determines that the scenario has been executed, the process proceeds to an analysis step that analyzes the cognitive process of the remote operator. The aforementioned computer, Based on the aforementioned video data and the aforementioned interview data, the correspondence definition result is read, which defines the correspondence between the gaze target and acquired information, including the maneuvering means, in the simulated remote operation of the remote operator. A time-series gaze target data conversion step is performed to convert the gaze data into time-series gaze target data based on the correspondence definition result, Based on the correspondence definition results, the steering and track data is converted into steering and speed adjustment action data in a steering and speed adjustment action data conversion step, Time-series behavioral data conversion step: Performs a conversion process from the aforementioned time-series target data and the steering and ship speed adjustment behavioral data into time-series behavioral data. And execute, In the aforementioned analysis step, the cognitive process that associates the target of the remote operator's gaze with their actions is analyzed based on the time-series behavioral data. A method for analyzing the cognitive process of a remote operator, characterized by the features described above.

2. The method for analyzing the cognitive process of a remote operator according to claim 1, characterized in that multiple remote operators perform the simulated remote operation in the remote operation experiment for the same scenario.

3. The method for analyzing the cognitive process of a remote operator according to claim 1, characterized in that the scenario includes a handover action that transitions from simulated automated operation of the vessel to simulated remote operation.

4. The method for analyzing the cognitive process of a remote operator according to claim 1, characterized in that the acquisition of the gaze data of the remote operator is performed by an eye tracker worn by the remote operator.

5. The method for analyzing the cognitive process of a remote operator according to claim 1, characterized in that the ship handling means can provide scenery information, nautical chart information, control panel information, and rudder information.

6. The computer A time-series ship position data conversion step that converts the predetermined scenario read in the scenario reading step and the read ship handling and track data into time-series ship position data, An event definition step that defines an event including a start and end time based on the aforementioned time-series ship position data. And execute, The analysis step involves analyzing the cognitive process of the remote operator in the event associated with the ship's position that occurred during the simulated remote operation. The method for analyzing the cognitive process of a remote operator as described in feature 1.

7. The computer The time-series behavior data converted in the time-series behavior data conversion step and the event-specific time-series behavior data conversion step which converts the time-series behavior data converted in the time-series behavior data conversion step into event-specific time-series behavior data based on the events defined in the event definition step, A time-series behavior log recording step, which records a time-series behavior log based on the time-series behavior data for each event. And execute, In the analysis step, the cognitive process of the remote operator is analyzed based on the time-series action log, which is derived from the object of focus and the action taken at the time the event occurred. The method for analyzing the cognitive process of a remote operator according to feature 6.

8. A system for performing the method for analyzing the cognitive process of a remote operator as described in Claim 1, comprising: a computer; a scenario providing means for providing the scenario and progress; a ship handling simulator as the ship handling means; a ship handling and track data recording device for storing the acquired ship handling and track data; and a gaze data recording device for storing the acquired gaze data.

9. The remote operator's cognitive process analysis system according to claim 8, characterized in that the acquired gaze data is recorded in the gaze data recording device via a gaze analysis device.

10. The remote operator cognitive process analysis system according to claim 8, further comprising a video data recording device that stores video data of the simulated remote operation performed by the remote operator, acquired with a video camera for filming the operation, and interview video data of the interview with the remote operator, acquired with a video camera for recording the interview.

11. The remote operator cognitive process analysis system according to claim 8, further comprising an interview data recording device for storing the acquired interview data with the remote operator.

12. The remote operator's cognitive process analysis system according to claim 8, characterized in that the ship handling simulator has a scenery screen, a nautical chart screen, a control panel, and a rudder confirmation screen including rudder angle.

13. Based on the time-series behavior log recorded in the time-series behavior log recording step in the method for analyzing the cognitive process of a remote operator according to Claim 7, The aforementioned computer, Based on the analyzed cognitive processes of the remote operator, a remote operation system for the vessel is designed that can provide the remote operator with information about the object of their gaze and the actions they will be taking when the event occurs, at the necessary time. A method for designing a remote ship handling system, characterized by the following features.

Images