Information processing device, information processing method, and computer program

Abstract

We provide information processing devices, information processing methods, and computer programs that contribute to the safe navigation of ships. [Solution] The information processing device is installed on the bridge of a ship and includes a camera that captures crew members inside the bridge, an acquisition unit that acquires navigation data relating to the ship's navigation, a behavior determination unit that determines the behavior of crew members in the images based on the images acquired from the camera, a work determination unit that determines the work content of the crew members based on the behavior determined by the behavior determination unit, and an output unit that outputs an evaluation of the crew members' behavior based on the navigation data acquired by the acquisition unit and the work content determined by the work determination unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a computer program that contribute to the safe navigation of ships.

Background Art

[0002] In order to maintain and improve the safety of ship navigation, large cargo ships or passenger ships that continue to navigate at night are obliged to install a preventive alarm device called BNWAS (Bridge Navigational Watch Alarm System) to prevent drowsiness or inattention. For safer navigation, not only drowsiness prevention by BNWAS, but also a system has been proposed that photographs crew members sitting at the helm with a camera and determines whether they are in a drowsy state or their physical condition has deteriorated from the images taken by the camera (Patent Document 1, etc.).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In addition to preventing drowsiness or inattention by BNWAS, it is possible to monitor whether the crew members sitting at the helm are looking ahead by capturing their faces with a camera, as in the technology proposed in Patent Document 1 or the technology for monitoring vehicle drivers. However, there may be no space to place seats on the bridge. In many cases, multiple crew members work on the bridge respectively. In addition to this, the work content of the crew members in the bridge is diverse in addition to monitoring the surrounding sea area. Because various devices are arranged in the bridge, crew members often move around in the bridge. A technology that can support crew members according to the situation of the surrounding sea area and the state of the ship during ship navigation contributes to safe navigation.

[0005] This disclosure is made in light of the circumstances described above and aims to provide an information processing device, an information processing method, and a computer program that contribute to the safe navigation of ships. [Means for solving the problem]

[0006] An information processing device in one aspect of this disclosure includes a camera installed on the bridge of a ship that captures crew members inside the bridge, an acquisition unit that acquires navigation data relating to the ship's navigation, a behavior determination unit that determines the behavior of crew members captured in the images based on the images acquired from the camera, a work determination unit that determines the work content of the crew members based on the behavior determined by the behavior determination unit, and an output unit that outputs an evaluation of the crew members' behavior based on the navigation data acquired by the acquisition unit and the work content determined by the work determination unit.

[0007] One aspect of this disclosure involves an information processing device that uses cameras to capture images of crew members on the bridge, analyzes the images to determine their behavior on the bridge, and further determines what tasks they were performing. This disclosure outputs an evaluation that includes determining whether the crew member's behavior, as determined from the images obtained from the cameras, was appropriate for the task at that time and appropriate in light of the conditions of the surrounding sea area at that time. This prevents the misjudgment that a crew member who was performing appropriate work-related behavior, such as quietly observing equipment rather than sleeping, was sleeping simply because they were motionless, and enables positive evaluation of appropriate work-related behavior. Image data of what constitutes appropriate behavior according to the task can be accumulated and used for crew training and other purposes.

[0008] In one aspect of the information processing device of this disclosure, the behavior determination unit may use a learning model that has been trained to output data on the position of the crew member's head and the orientation of the crew member's face, respectively, when given an image, and determine the direction of the crew member's line of sight or the position of the crew member within the bridge at the time corresponding to the image, based on the face position and face orientation data obtained by providing the learning model with an image acquired from the camera.

[0009] One aspect of this disclosure involves the use of a learning model for image recognition. The learning model may be divided into two or more models, such as a model for detecting the position of a person's head in an image and a model for detecting the orientation of the person's face in an image. By using the learning model, the position of the head in the image makes it possible to determine the location of the crew member in the bridge based on the camera layout. Depending on the face orientation detection result and the camera's layout in the bridge, the information processing device can determine the direction of the crew member's gaze from the face orientation detection result.

[0010] In one aspect of the information processing device of this disclosure, the acquisition unit may acquire the navigation position of the vessel, the weather, wind speed and direction of the sea area in which the vessel is navigating, the degree of congestion in the sea area, and the navigation status of whether or not the vessel is navigating.

[0011] One aspect of this disclosure involves acquiring navigation data such as the vessel's position, weather conditions, wind speed and direction in the area of ​​navigation, the degree of congestion in the area, and whether or not the vessel is underway. First, the necessity of crew monitoring changes depending on whether or not the vessel is underway. If the vessel is underway, the frequency of crew monitoring and the types of equipment to be operated may change depending on whether the vessel is in port or in the open sea. The frequency of crew monitoring may also change depending on the weather conditions, wind speed and direction in the area of ​​navigation. The frequency of crew monitoring may also change depending on the degree of congestion in the area of ​​navigation. These diverse conditions and the determined work content allow for a proper evaluation of crew behavior.

[0012] In one aspect of the information processing device of this disclosure, the behavior determination unit may determine the equipment in front of the crew member based on the image, or based on the design drawing data in the bridge and the crew member's position, and determine the crew member's behavior in relation to the equipment.

[0013] One aspect of this disclosure is that the work performed by crew members on the bridge can be automatically determined based on the location of the crew members in the image and the location of the equipment located on the bridge. This makes it possible to properly evaluate the crew members' behavior based on what constitutes appropriate behavior according to their work.

[0014] In one aspect of the information processing device of this disclosure, the output unit stores a reference for the range of movement of the crew member within the bridge or for their line of sight, associated with identification data that identifies each of a plurality of work contents, and outputs an evaluation value depending on whether the behavior determined by the behavior determination unit satisfies the reference corresponding to the determined work contents.

[0015] One aspect of this disclosure is that it becomes easier to evaluate the behavior of crew members on the bridge based on either the criterion that the position of the crew member in the image falls within a specific range of movement, or the criterion that the crew member's gaze is directed in a specific direction.

[0016] In one aspect of the information processing device of this disclosure, the output unit may store the criteria according to the navigation position of the vessel, the weather, wind speed and direction of the sea area during navigation, the time of day, and the navigation status, as indicated by the navigation data acquired by the acquisition unit, and output an evaluation value depending on whether the criteria corresponding to the navigation position, weather, wind speed, wind direction, time of day, or situation are met.

[0017] One aspect of this disclosure is that the criteria for what constitutes appropriate behavior differ depending on the vessel's position, weather, wind speed and direction, time of day, and whether or not it is underway. This makes it possible to properly evaluate the behavior of the crew members based on what constitutes appropriate behavior according to the nature of their work, rather than simply whether or not they are moving, in accordance with these diverse conditions.

[0018] In one aspect of the information processing device of this disclosure, the output unit may output an evaluation value of the crew's behavior regarding the safety aspects of the navigation of the vessel.

[0019] One aspect of this disclosure is that evaluations of seafarers may be output based on whether or not there is a high risk of an accident occurring in terms of safety. The evaluation may be based on the level of risk of a maritime accident occurring due to drowsiness or distraction, and if the risk is higher than a predetermined level, it may be recorded as a log. Even if the risk does not reach a predetermined level, if it is judged to be a near miss and the risk is deemed high, the information processing device may record the output evaluation in a referable format and use it for subsequent education, warnings, etc.

[0020] In one aspect of the information processing device of this disclosure, the output unit may change the criteria for evaluating the behavior based on the position of the vessel at the time corresponding to the image, the weather in the sea area where the vessel is navigating, or the navigation status of the vessel.

[0021] One aspect of this disclosure is that, in terms of safety, the criteria for determining whether or not the risk of an accident is high differ depending on the navigation conditions, such as whether or not the vessel is underway. This makes it possible to make the evaluation more appropriate.

[0022] In one aspect of the information processing device of this disclosure, the output unit may change the criteria for evaluating the behavior corresponding to the determined business content based on the business content of the vessel or the presence or severity of past accident cases in the vessel's navigation position and surrounding area.

[0023] One aspect of this disclosure is that information processing equipment may be evaluated more rigorously to keep the crew alert, especially if there are past accident cases identical or similar to the operations of the vessel at sea. Information processing equipment may also be evaluated more rigorously if there are past accident cases in or around the position of the vessel at sea, as this indicates an area that should be monitored frequently.

[0024] In an information processing apparatus according to an aspect of the present disclosure, the output unit may output an evaluation value regarding the health of the crew member with respect to the behavior of the crew member.

[0025] In an aspect of the present disclosure, an evaluation of a crew member may be output based on whether there is a high risk of being unable to continue work in terms of health. If a crew member is sleepy or looks around frequently, it may be evaluated as a state of lax attention, and if the risk of the health state is higher than a predetermined level, it may be recorded as a log. The information processing apparatus may record the output evaluation in a retrievable manner and make it available for use as a criterion for subsequent health observation, crew member assignment change, etc.

[0026] An information processing method according to an aspect of the present disclosure is provided on the bridge of a ship. A computer that acquires an image from a camera that captures crew members inside the bridge acquires navigation data regarding the navigation of the ship from equipment mounted on the ship, discriminates the behavior of the crew members shown in the image based on the image acquired from the camera, discriminates the work content of the crew members based on the discriminated behavior, and outputs an evaluation of the behavior of the crew members based on the acquired navigation data and the discriminated work content.

[0027] A computer program according to an aspect of the present disclosure causes a computer that acquires an image from a camera that captures crew members inside the bridge provided on the bridge of a ship to acquire navigation data regarding the navigation of the ship from equipment mounted on the ship, discriminates the behavior of the crew members shown in the image based on the image acquired from the camera, discriminates the work content of the crew members based on the discriminated behavior, and executes a process of outputting an evaluation of the behavior of the crew members based on the acquired navigation data and the discriminated work content.

Brief Description of Drawings

[0028] [Figure 1] It is a schematic diagram of the bridge duty management system of the first embodiment. [Figure 2] It is a block diagram showing the configuration of the information processing apparatus. [Figure 3] It is an explanatory diagram of the functions of the information processing apparatus based on the information processing program. [Figure 4] This is an explanatory diagram showing an example of an image obtained from a camera. [Figure 5] This is an explanatory diagram showing an example of an image obtained from a camera. [Figure 6] This is a schematic diagram of a learning model used in information processing equipment. [Figure 7] This diagram shows the correspondence between areas within Funabashi and the positions of heads in the images. [Figure 8] This flowchart shows an example of a processing procedure performed by an information processing device. [Figure 9] This is a flowchart showing an example of the evaluation output procedure. [Figure 10] This flowchart shows an example of the output processing procedure for evaluations in a data server. [Figure 11] This figure shows an example of the evaluation output that can be viewed by the client. [Figure 12] This flowchart shows an example of the evaluation output procedure in the second embodiment. [Figure 13] This is a schematic diagram of the bridge watch management system in the third embodiment. [Modes for carrying out the invention]

[0029] This disclosure will be described in detail with reference to drawings illustrating its embodiments. The following embodiments describe a bridge watch management system including the information processing device of this disclosure.

[0030] [First Embodiment] Figure 1 is a schematic diagram of the first embodiment of the bridge watch management system 100. The schematic diagram in Figure 1 shows the arrangement of each device on a schematic design drawing of the interior of the bridge of the ship S. The bridge watch management system 100 includes a camera 2 installed inside the bridge of the ship S, an information processing device 1 that performs processing on images captured by the camera 2, a group of devices 3 installed on the ship S, an alarm device 4, a data server 5 installed on land, and a client 6 for the data server 5.

[0031] Camera 2 uses a visible light image sensor to output images. Camera 2 may also be equipped with a far-infrared image sensor for nighttime use. Camera 2 is installed on the ceiling or side wall of the bridge to capture the crew members inside the bridge with the widest possible field of view. Camera 2 continuously outputs images in a time series at a predetermined rate (e.g., 10 frames per second).

[0032] Information processing device 1 is an edge computer installed on the ship S. Information processing device 1 acquires image data from camera 2 via the ship's communication network SN or signal lines (not shown). Based on the acquired image data, information processing device 1 determines the behavior of the crew members captured in the images, derives and stores an evaluation of the crew members according to their behavior, or outputs the evaluation of the crew members to alarm device 4 or data server 5. The detailed configuration and processing of information processing device 1 will be described later.

[0033] Equipment group 3 is a group of instruments that measure navigation data of the vessel S. Equipment group 3 includes a navigation control system 31, a GPS (Global Positioning System) receiver 32, an anemometer 33, an AIS (Automatic Identification System) communicator 34, and an Electronic Chart Display and Information System (ECDIS) 35. Equipment group 3 may also include radar, sonar, a speedometer, etc. (none of which are shown). Equipment group 3 is connected to the information processing device 1 and the navigation control system of the vessel S via an onboard communication network SN or signal lines (not shown).

[0034] Alarm device 4 is a BNWAS (Bridge Navigational Alarm System). Alarm device 4 is a device that issues an alarm if a reset signal is not received from equipment installed in the bridge or if the device is not reset due to the detection of crew movement during a waiting period, for example, 3 minutes. Alarm device 4 can communicate with information processing device 1 via the ship's communication network SN or a signal line (not shown). Alarm device 4 can determine whether or not to issue an alarm using the reset signal output by information processing device 1.

[0035] The shipboard communication network SN is a communication medium installed within the ship S. The shipboard communication network SN may be wired or wireless. The shipboard communication network SN enables communication with external communication equipment via connected satellite communication, carrier network, or AIS communication equipment 34.

[0036] The data server 5 is a server located on land, not on the ship S. The data server 5 consists of one server computer, or multiple server computers connected via a dedicated line or public communication network, and may be implemented as a cloud server that can be communicated to from the information processing device 1 and client 6 via a network. The data server 5 stores data related to the navigation of the ship S. The data server 5 stores data of the ship S's crew so that it can be accessed from others. The data server 5 can store personnel data such as working hours associated with the crew's identification data. The data server 5 stores data transmitted from the information processing device 1. The data server 5 can acquire data from external services such as weather forecasts, sea condition forecasts, or sea condition information provision via a public communication network. The information processing device 1 can send and receive data with the data server 5 wirelessly via satellite communication, carrier network, or WiFi communication. The information processing device 1 may send and receive data with the data server 5 via its own communication module or AIS communicator 34 that supports AIS dedicated frequencies. The data server 5 has a web server function that can output the data transmitted from the information processing device 1 and stored on a web page.

[0037] Client 6 is a personal computer having a user interface such as a keyboard or pointing device that accepts input from a user or operator. Client 6 can display web pages provided by data server 5 on its display. Client 6 may perform processing based on a client program for the Funabashi watch management system 100 of this disclosure and display data obtained from data server 5 via communication on its display.

[0038] In the bridge watch management system 100, the information processing device 1 monitors the behavior of crew members on the bridge, as shown in Figure 1, to determine whether they are dozing off or distracted during their watch, and to evaluate their behavior according to the type of work they are doing. As shown in Figure 1, the bridge is equipped with monitors 30 of the equipment group 3, and crew members on watch perform a variety of tasks, such as visually checking the route from the bridge and checking the conditions outside the ship by visually checking the monitors of the equipment group 3, while ensuring the continuation of safe navigation. The following describes in detail how the information processing device 1 uses images from camera 2 to determine the behavior of crew members on the bridge and how it evaluates their behavior.

[0039] Figure 2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 uses a small, high-performance computer used as a so-called edge computer. The information processing device 1 may also be a server computer or a personal computer. The information processing device 1 comprises a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0040] The processing unit 10 includes one or more arithmetic processing units such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). The processing unit 10 also includes a temporary storage medium such as SRAM (Static Random Access Memory) and DRAM (Dynamic Random Access Memory). The processing unit 10 reads the information processing program P1 stored in the storage unit 11 into the temporary storage medium and executes it, thereby causing a general-purpose computer to perform various processes described later, and to function as the information processing device 1 of this disclosure.

[0041] The storage unit 11 is a relatively large-capacity non-volatile storage medium such as an SSD (Solid State Drive) or a hard disk. The storage unit 11 stores the program (program product) necessary for the processing unit 10 to execute processing, and reference configuration data. The configuration data may include the type of ship S on which the information processing device 1 is installed, and identification information. The configuration data also includes data used for image processing decisions, which will be described later. The program product includes an information processing program P1 and a learning model M1.

[0042] The information processing program (program product) P1 and learning model M1 stored in the memory unit 11 may be obtained by the processing unit 10 reading the information processing program P9 and learning model M9 stored in the non-temporary storage medium 9 readable from the computer and storing them in the memory unit 11. Alternatively, the information processing program P1 and learning model M1 may be obtained by the processing unit 10 downloading them from the download server via the second communication unit 13 and storing them in the memory unit 11.

[0043] The memory unit 11 stores image data acquired from the camera 2 in a time-series format, linked to time information, so that it can be referenced in that order. The memory unit 11 also stores evaluations of the crew's behavior obtained as a result of processing the image data as described later.

[0044] The memory unit 11 contains an accident database 111 that stores data on past maritime accidents. The accident database 111 stores the date and time of the accident, location data of the accident's location, type of accident, and severity, associated with the accident identification number. The location data may be data that distinguishes sea areas, or it may be latitude and longitude information. The type of accident is data that identifies collision, capsizing, grounding, loss of navigation, engine failure, flooding, fire, etc.

[0045] The first communication unit 12 is a communication device that enables communication with the camera 2. The first communication unit 12 enables communication via the ship's communication network SN to which the camera 2 is connected. The first communication unit 12 may be a wireless communication device for WiFi® or a wired communication device for Ethernet®.

[0046] The second communication unit 13 is a communication device that enables communication with the data server 5. The second communication unit 13 may be a communication device that enables carrier communication via a carrier network, a communication device that supports wireless networks such as WiFi or Bluetooth (registered trademark), a communication device that communicates via satellite communication, or a communication device that communicates using an AIS-dedicated frequency. The second communication unit 13 may be replaced by equipment for communication with external devices connected to the shipboard communication network SN.

[0047] Figure 3 is an explanatory diagram of the functions of the information processing device 1 based on the information processing program P1. The processing unit 10 of the information processing device 1 performs the various functions shown in Figure 3 based on the information processing program P1. The processing unit 10 functions as an image acquisition unit 101, a behavior determination unit 102, a navigation data acquisition unit 103, a business determination unit 104, and an output unit 105.

[0048] The processing unit 10 of the information processing device 1 functions as an image acquisition unit 101 that acquires multiple images in chronological order of crew members inside the bridge of the ship S. As the image acquisition unit 101, the processing unit 10 acquires images (see Figure 5) from the camera 2.

[0049] The processing unit 10 of the information processing device 1 functions as a behavior determination unit 102 that determines the behavior of the crew members in the image based on the image acquired by the image acquisition unit 101 from the camera 2. The behavior determination unit 102 determines the direction of the gaze of the crew members in the image and their position within the bridge based on the position of their heads and the orientation of their faces. The behavior determination unit 102 may also determine the posture of the crew members in the image.

[0050] The behavior determination unit 102 uses the learning model M1 to detect the position of the crew member's head within the image. Based on the depth map of the image pre-set for camera 2 using the bridge design data, the behavior determination unit 102 identifies coordinate data indicating the crew member's position within the bridge from the position of the head within the image. The behavior determination unit 102 may also use data that pre-divides the bridge into multiple areas and the depth map of the image pre-set for camera 2 to determine which area within the bridge the crew member in the image belongs to, using data that identifies each area.

[0051] The behavior determination unit 102 uses the learning model M1 to determine the orientation of the crew member's face from the area where the head is visible. The behavior determination unit 102 identifies data that represents the orientation of the face in terms of yaw (rotation on the horizontal plane, head movement), roll (rotation on the vertical plane, tilt of the face), and pitch axis (degree of downward tilt of the neck and head). Based on the settings for the camera 2 layout, the behavior determination unit 102 determines from the identified face orientation whether the direction of the gaze is forward, starboard, port, rear, downward, or other (undeterminable), and outputs the determination result as gaze identification data.

[0052] The behavior discrimination unit 102 may determine the posture, such as whether the person is sitting, standing, or leaning forward, based on the detected head position and the identified face orientation, and output data that identifies each posture. The discrimination method used by the behavior discrimination unit 102 is not limited to discrimination using the learning model M1, but may also use pattern recognition.

[0053] The processing unit 10 functions as a navigation data acquisition unit 103 that acquires navigation data related to the navigation of the vessel S. The navigation data acquisition unit 103 acquires data output from the navigation control system 31, GPS receiver 32, wind direction and speed meter 33, AIS communication device 34, and electronic chart display system 35 at any time. The navigation data acquisition unit 103 may also acquire data from other devices 3.

[0054] The processing unit 10 acquires the speed, direction, and inclination of the vessel S from the navigation control system 31 using the navigation data acquisition unit 103. The processing unit 10 may also acquire whether or not the anchor has been lowered using the navigation data acquisition unit 103. Based on the vessel speed, the processing unit 10 can acquire the navigation status of the vessel S, indicating whether it is sailing or at anchor. The processing unit 10 may also acquire the navigation status indicating whether it is sailing or at anchor from the navigation control system 31.

[0055] Depending on whether the ship is underway or at anchor, the processing unit 10 can determine whether or not it should perform external monitoring as part of its duties. If the ship is underway and external monitoring should be performed, the processing unit 10 can determine whether or not the crew members on the bridge are performing external monitoring or operating specific equipment.

[0056] The navigation data acquisition unit 103 acquires weather forecasts, sea condition forecasts, or sea condition information from the data server 5 via the second communication unit 13 or the AIS communication device 34.

[0057] The processing unit 10 acquires the time and position during navigation using the navigation data acquisition unit 103. The processing unit 10 may also acquire weather, wind speed, and wind direction using the navigation data acquisition unit 103. Based on the navigation position, time, weather, wind direction, and wind speed, it is possible to determine whether the area under navigation has good visibility, etc.

[0058] The processing unit 10 functions as a task discrimination unit 104 that determines the work of a crew member in an image based on the behavior (position and direction of gaze) determined by the behavior discrimination unit 102. The task discrimination unit 104 determines the work of a crew member as "monitoring," "equipment operation," "other," or "unknown" from a combination of the behavior output by the behavior discrimination unit 102, i.e., the coordinate data of the crew member's position or area identification data (see Figure 7), and the identification data indicating the result of gaze discrimination. For example, if the area containing the crew member's position is an equipment area, the task discrimination unit 104 will determine it as "equipment operation" even if the gaze is directed downwards.

[0059] If the identified crew member's task is "surveillance," the task discrimination unit 104 further distinguishes between the front, starboard, port, and rear based on the gaze discrimination result by the behavior discrimination unit 102. If the identified crew member's task is "equipment operation," the task discrimination unit 104 further distinguishes the equipment to be operated based on the area containing the crew member's position or by recognition of the image. The task discrimination unit 104 may distinguish the task by distinguishing, for example, "radar (confirmation) operation" or "nautical chart (confirmation) operation."

[0060] The discrimination method used by the business discrimination unit 104 may also be based on another learning model that has been trained to output business content identification data when behavior data including location coordinate data or area identification data and gaze identification data is input.

[0061] The processing unit 10 functions as an output unit 105 that outputs the work content determined by the work determination unit 104 and an evaluation of the crew member's behavior in that work content. The output unit 105 outputs an evaluation of whether the behavior is appropriate or not, based on the behavior determined by the behavior determination unit 102, the navigation data acquired by the navigation data acquisition unit 103, and the work content determined by the work determination unit 104. The output unit 105 may output the evaluation as a numerical value indicating the degree to which an alarm should be issued, or it may output the high or low evaluation as a binary value, or as a discrete value or symbol such as A / B / C. The output unit 105 outputs the calculated evaluation, the work content, and a part of the image that served as the judgment criterion to the storage unit 11 or the data server 5. Based on the evaluation, the output unit 105 may output a reset signal that is referenced by the alarm device 4.

[0062] For example, if the work content of a crew member determined by the work determination unit 104 is "monitoring" and the crew member's gaze is determined to be straight ahead, the output unit 105 outputs a low risk for both safety and health aspects as an evaluation of the monitoring work. The output unit 105 may also output a reset signal. For example, during "monitoring," the output unit 105 outputs a low risk of accident occurrence in terms of safety as long as the direction of the gaze is facing straight ahead, starboard, or port, but calculates and outputs a high risk if the gaze is facing any other direction.

[0063] For example, if the task determination unit 104 determines that the crew member's task is "equipment (confirmation) operation" and the crew member's gaze is determined to be forward or downward towards the electronic chart display system 35 located forward in the bridge, the output unit 105 outputs "low risk" as an evaluation of the chart confirmation task. In this case, the output unit 105 may determine that the task is appropriate and output a reset signal to the alarm device 4. For example, even during "equipment (confirmation) operation," the output unit 105 outputs "high risk" if the direction of the gaze is directed downward for a long period of time. In addition, various methods can be used for evaluation by the output unit 105.

[0064] Some of the functions of the information processing device 1 described above may be executed on a data server 5 that can communicate with the information processing device 1. For example, any one of the functions of the behavior determination unit 102, the business determination unit 104, and the output unit 105 may be executed on the data server 5.

[0065] Figures 4 and 5 are explanatory diagrams showing examples of images obtained from camera 2. Figures 4 and 5 schematically represent images acquired from camera 2 by the image acquisition unit 101 when the processing unit 10 is used. Figures 4 and 5 show images taken while the vessel is in motion. Both images in Figures 4 and 5 show a crew member. In the image shown in Figure 4, the crew member is looking forward in the direction of travel of the vessel S, i.e., straight ahead, and is operating the steering wheel. In the image shown in Figure 5, the crew member is leaning back on a stool, looking downwards, and operating a smartphone.

[0066] Figure 6 is a schematic diagram of the learning model M1 used in the information processing device 1. The learning model M1 includes a head detection model M11 and a head direction discrimination model M12. When an image like the one shown in Figure 4 or Figure 5, acquired from camera 2, is input to the head detection model M11, it detects the region containing the head in that image. The head direction discrimination model M12 is a model that discriminates the direction of the head for the region detected by the processing of head detection model M11 and outputs the yaw, pitch, and roll of the head as vectors. Each model is a learning model that has been trained by deep learning using a general-purpose training dataset with a neural network. Each model may also be tuned to improve the recognition accuracy of images of the inside of a ship using images of crew members inside the ship that can be captured by camera 2.

[0067] The head detection model M11 is trained to output, upon receiving an image, the coordinate data of the region containing the head within that image, and a score indicating the likelihood that it is a head. The head detection model M11 is, for example, an SSD (Single Shot MultiBox Detector) and outputs coordinate data of a predetermined area, such as a rectangle, that contains the head.

[0068] The head orientation discrimination model M12 is trained to output vector data as features that indicate the direction the head is facing when it receives an image of the head range extracted based on coordinate data output from the head detection model M11. The vector data is the yaw, pitch, and roll of the head in a space whose coordinate axes are the left-right, up-down, and depth directions of the original image.

[0069] The head detection model M11 and the head orientation discrimination model M12 may employ models without convolutional layers, such as Transformers, or other algorithms. The head detection model M11 and the head orientation discrimination model M12 may each employ models without convolutional layers, such as Vision Transformers, or other architectures. The learning model M1 may be a model using a support vector machine or the like.

[0070] The processing unit 10, using the behavior determination unit 102, determines the position of the captured sailor's head within the image from the coordinate data of the area including the head output from the head detection model M11. For example, the processing unit 10 determines the center point of the rectangular area output from the head detection model M11 as the position within the image. The processing unit 10, using the behavior determination unit 102, calculates the coordinates of the position within the bridge from the position of the sailor's head in the image previously captured by camera 2, based on the bridge design drawing data.

[0071] The processing unit 10, using the behavior determination unit 102, may store information that identifies the area where a person with a head is standing if that head is captured in the image, based on the design drawing data of the bridge, and determine the area. Figure 7 is a diagram showing the correspondence between areas within the bridge and the positions of heads in the image. The upper part of Figure 7 shows divided areas on a map of the bridge based on the design drawing data. The area identification data is divided into, for example, "1: In front of the steering wheel", "2: In front of the communication device", "3: In front of the monitor", "4: Other", "5: In front of the workbench", and "6: Other". The lower part of Figure 7 shows the relationship between the position of a head in the image and the area identification data of the standing position of a crew member whose head is in that position. As shown in Figure 7, for each pre-divided area, the range of the position of a crew member's head standing in that area is stored in the image.

[0072] This allows the behavior determination unit 102 to determine the position and area of ​​the crew member from the position of the head in the image. The behavior determination unit 102 may determine only one of the crew member's coordinates within the bridge or area identification data.

[0073] The processing unit 10, using the behavior determination unit 102, determines identification data indicating the line of sight of the detected sailor based on the vector data output from the head direction determination model M12, with the left-right direction, up-down direction, and depth direction as coordinate axes, from "1: Front", "2: Starboard", "3: Port", "4: Up", "5: Down", and "6: Unknown" in the space of the bridge. The processing unit 10 may also determine this after converting the data into vectors with the left-right direction, vertical direction, and the fore-aft direction of the ship S as axes in the space of the bridge using the behavior determination unit 102.

[0074] The processing unit 10 may output posture identification data using the behavior determination unit 102. For example, if the detected head position is "3: In front of the monitor" as shown in Figure 7, and the gaze is directed downwards, the person may be "crouching" not in front of the monitor, but in the area of ​​"1: In front of the steering wheel", the area of ​​"4: Other", or the area of ​​"6: Other". The processing unit 10 determines this "crouching" posture based on the angle of the gaze. If the processing unit 10 determines that the gaze is directed downwards at a predetermined angle or more, it may correct the area identification data identified based on the position of the head in the image. The processing unit 10 may also determine the sailor's posture in the behavior determination unit 102 based on the recognition of parts of the image other than the head. In this case, the processing unit 10 may use, for example, a posture determination model that has been trained to output posture identification data when an image of a person is given.

[0075] The processing in the Funabashi watch management system 100, which is configured in this way, will now be explained. Figure 8 is a flowchart showing an example of the processing procedure by the information processing device 1. The processing unit 10 of the information processing device 1 continuously executes the following processes during startup.

[0076] The processing unit 10 acquires image data from the camera 2 using the image acquisition unit 101 (step S101). The processing unit 10 provides the acquired image data to the learning model M1 (step S102). Based on the position and face orientation data output from the learning model M1, the processing unit 10 determines the behavior using the function of the behavior determination unit 102 (step S103).

[0077] The processing unit 10 acquires data on the behavior of the crew members in the image from the behavior determination unit 102 (step S104). In step S104, the processing unit 10 acquires position coordinate data or area identification data indicating an area, and identification data indicating the crew member's line of sight from the behavior determination unit 102. The processing unit 10 may also acquire identification data on the crew member's posture. In step S104, if there are multiple crew members in the image, the processing unit 10 distinguishes each crew member in the image and acquires data on their behavior.

[0078] The processing unit 10, using the navigation data acquisition unit 103, acquires navigation data from the group of equipment 3, including the navigation status of the vessel S, the navigation position of the vessel S, data identifying the sea area in which the vessel S is navigating, and the weather, wind speed, and wind direction of the sea area in which the vessel is navigating (step S105).

[0079] The processing unit 10, using the functions of the task discrimination unit 104, determines the task being performed by the crew member in the image based on the acquired behavioral data (step S106).

[0080] In step S107, the processing unit 10 determines, based on the area identification data of the crew member's position and the line of sight identification data, that if the crew member is at "1: in front of the steering wheel" and looking "straight ahead", it determines that the task is "monitoring". Similarly, if the processing unit 10 determines that the crew member is at "in front of the steering wheel" and looking "downward", it determines that the task is "unknown". Similarly, if the processing unit 10 determines that the crew member is at "3: in front of the monitor" and looking "downward", it determines that the task is "equipment (confirmation) operation". These methods for determining tasks based on area identification data and line of sight identification data may also be a method in which a table of correspondences between combinations of position and line of sight and "task content" is stored in the storage unit 11 in advance, and the task is identified from that correspondence.

[0081] The processing unit 10 stores the crew behavior data acquired in step S104, the navigation data acquired in step S105, and the work content determined in step S106 in the database of the storage unit 11, along with the time information (step S107).

[0082] The processing unit 10 outputs an evaluation value for the crew member's behavior in the task determined in step S106 (step S108) using the function of the output unit 105. In step S108, the processing unit 10 may evaluate the level of risk, such as accident occurrence, regarding the safety of the vessel S. The processing unit 10 may also evaluate the difference between the behavior of the model crew member and the behavior of the crew member when performing the task in question. The procedure for outputting the evaluation value in step S108 will be described in detail later.

[0083] The processing unit 10 stores the output evaluation value in the database of the storage unit 11 (step S109) and terminates the process. If multiple crew members are in the image, the processing unit 10 distinguishes between the first crew member and the second crew member and stores the behavior data, navigation data, work content data, and evaluation. If the learning model M1 includes a model capable of face recognition, the processing unit 10 stores the data performed on the image containing multiple crew members in association with face recognition data. Even if faces can be identified across images at multiple points in time and across images separated by time, it is preferable to perform the processing in association with face recognition data and store the data.

[0084] Figure 9 is a flowchart showing an example of the evaluation output procedure. The processing procedure shown in the flowchart of Figure 9 corresponds to the function of the output unit 105 and corresponds to the details of the processing procedure in step S108 of Figure 8.

[0085] The processing unit 10 reads out behavioral data, navigation data, and operational data for a predetermined time period immediately preceding the processing time (step S801). The predetermined time period may be longer than the waiting time referenced when determining whether the alarm device 4 is active, for example, 3 minutes.

[0086] The processing unit 10 determines whether the ship is underway based on the navigation data (step S802). In step S802, the processing unit 10 obtains the ship speed from the navigation control system 31 and may determine that the ship is underway if the ship speed is above a predetermined value, or it may determine whether the ship is at anchor based on whether the anchor is deployed from the navigation control system 31. If it is determined that the ship is underway (S802: YES), the processing unit 10 adds an additional value corresponding to the time period included in the navigation data to the evaluation value indicating the level of risk of accident occurrence in terms of safety (step S803). If the time period is daytime, the additional value is zero or a small value, and if the time period is early morning or evening, the additional value is slightly larger than that for daytime. If the time period is nighttime, the additional value is larger than that for early morning or evening. The processing unit 10 adds an additional value corresponding to the ship speed obtained from the ship speedometer to the evaluation value (step S804). The faster the ship speed, the larger the additional value.

[0087] The processing unit 10 adds an additional value to the evaluation value according to the wind speed and wind direction included in the navigation data (step S805). In step S805, the processing unit 10 calculates an additional value corresponding to the level of accident risk according to predetermined criteria, such as the higher the wind speed, the larger the additional value, and the closer the wind direction is to the direction of navigation, and adds it to the evaluation value. The processing unit 10 adds an additional value to the evaluation value according to the weather (step S806). If the weather is clear, the additional value is small, and if it is raining or snowing, the additional value is large. The additional value may be larger if there is a lot of cloud cover, and the weather may include good visibility. The processing in steps S805 and S806 may be combined into an additional value according to the weather.

[0088] The processing unit 10 adds an additional value to the evaluation value corresponding to the degree of congestion in the area under navigation, as indicated by the navigation data (step S807). The processing unit 10 may obtain the degree of congestion from the second communication unit 13 and the data server 5, or it may obtain it by communicating with other vessels using the AIS communication device 34. The processing unit 10 may detect other vessels using radar or the like, and increase the additional value the closer the distance to other vessels is. The processing unit 10 increases the additional value if there are facilities related to the fishing industry in the sea area under navigation.

[0089] The processing unit 10 references the accident database 111 for records of accidents in the area under navigation, as indicated by the navigation data (step S808). The processing unit 10 adds a value to the evaluation value according to the frequency and severity of accidents in the area under navigation (step S809). The more accidents there are, the larger the added value, and the higher the severity, the larger the added value.

[0090] The processing unit 10 adds or subtracts an evaluation value (step S810) according to the number of times or duration in which the work content was determined to be "monitoring" during the most recent predetermined time while sailing. In step S810, the processing unit 10 increases or decreases the evaluation value in such a way that the more times or longer the time the work was determined to be monitoring, the lower the risk of an accident occurs (the more safe it is judged to be). The processing unit 10 may also calculate the evaluation value in such a way that the more times "forward monitoring" occurs, the lower the risk of an accident occurs.

[0091] During navigation, the processing unit 10 adds or subtracts an evaluation value according to the distance traveled by the crew member in the most recent predetermined time (step S811). In step S811, the processing unit 10 increases or decreases the evaluation value in terms of safety, such that the greater the distance traveled and the more diverse the locations within the bridge the crew member has moved to, that is, the closer the possibility of falling asleep is to zero, the lower the evaluation of the risk of an accident occurring.

[0092] The processing unit 10 adds or subtracts an evaluation value (step S812) according to the number of times and duration during the most recent predetermined time when a crew member's gaze is determined to be "downward" and the nature of their work is determined to be "other," and then terminates the process. In step S812, the processing unit 10 increases or decreases the evaluation value to increase the risk of an accident, the more often or for the longer the crew member is determined to be looking down and doing something other than work.

[0093] During navigation, the processing unit 10 increases or decreases the evaluation value via the second communication unit 13 and the data server 5, depending on the degree of congestion in the area being navigated and the presence or absence of aquaculture rafts for marine products being produced in the area.

[0094] If the processing unit 10 determines in step S802 that the ship is not underway (S802: NO), it can determine that the ship is at anchor (step S813). When the ship is at anchor, there is no particular need to evaluate the behavior of the crew members on the bridge regarding the operation of the ship S, so the processing is terminated. Depending on the type of ship S and the type of equipment located on the bridge, monitoring may be required while the ship is at anchor. In such cases, instead of adding evaluation values ​​based on ship speed, weather, past accidents, etc., an evaluation may be calculated and stored based on whether the crew members are facing forward, starboard, or port in the content of the "monitoring" task.

[0095] In the processing procedure shown in the flowchart of Figure 9, the processing unit 10 output an evaluation by comparing it with a standard behavior according to the content of the work. However, the processing unit 10 is not limited to this, and may also use a learning model that has been trained to output an evaluation value when it receives data on the navigation state and the behavior during a predetermined time period.

[0096] The processing procedure shown in Figure 9 is not limited to this. Some processes may be omitted, and the added values ​​may be adjusted according to the design.

[0097] If the evaluation value output by the output unit 105 is greater than or equal to a predetermined value, the processing unit 10 evaluates the crew member's behavior at the time of judgment as having a high risk of accident occurrence and outputs this. The processing unit 10 may choose not to output a reset signal to the alarm device 4 if the risk of accident occurrence is high. If the risk of accident occurrence is high, the information processing device 1 may also notify an external ship monitoring system via the data server 5. This ensures that when the ship S is underway, the alarm device 4 will emit an alarm if the crew member is dozing off or distracted, while at anchor, it will be automatically controlled not to emit an alarm unnecessarily. Furthermore, when underway, under poor visibility conditions, the risk of accident occurrence is evaluated as higher and an alarm is more likely to be emitted if the crew member is dozing off or distracted, compared to when visibility is good, thus increasing the sense of urgency. In this way, the information processing device 1 of this disclosure, based on the analysis of the crew member's behavior and evaluation criteria that are modified by referring to navigation data or work content, reduces the stress on the crew member from alarms emitted by the alarm device 4 and enables appropriate and effective alarms according to the situation.

[0098] The database in the storage unit 11 of the information processing device 1 stores data on behavior determined at each point in time, navigation data, and business content data. Image data captured at each point in time may also be stored in the database. When the processing unit 10 becomes able to communicate with the data server 5 via the second communication unit 13, it sends the data stored in the database to the data server 5, increasing the capacity of the usable storage area of ​​the storage unit 11.

[0099] The data server 5 receives and stores data on behavior determined at each point in time, navigation data, and work content data stored in the database from the information processing device 1 installed on each vessel S. The data server 5 can perform analysis processing on the stored data and output the analysis results. The analysis processing is similar to the evaluation output processing shown in Figure 9, for example. When the data server 5 calculates an evaluation value, it performs the calculation for a predetermined time, such as the watch time. This makes it possible to refer to the results of the evaluation output for the behavior of the crew members during their watch time.

[0100] The data server 5 uses behavioral data, navigation data, and work content data stored in the database of the information processing device 1 to derive evaluations for each crew member and create screen data that can be viewed by the client 6. Figure 10 is a flowchart showing an example of the evaluation output processing procedure in the data server 5.

[0101] Data server 5 identifies the target date (step S501) and selects each crew member who was on watch on that date (step S502). Data server 5 retrieves the watch hours of the selected crew members during the identified date from the personnel data (step S503). Data server 5 extracts data on behavior, navigation data, and work content data corresponding to the retrieved watch hours from the stored data (step S504).

[0102] The data server 5 creates a web page that graphs the time distribution of the identified behavior and the time distribution of the identified business content, making the navigation data accessible at each time point (step S505). The data server 5 calculates the evaluation at each point in time (step S506), graphs the time distribution of the evaluation, adds it to the web page (step S507), stores the data of the created web page (step S508), and terminates the process.

[0103] Data server 5 executes the processing procedure shown in Figure 10 for each day and for each crew member. This makes the results of the determination of behavior and work content for each day and each crew member, as well as the derived evaluation results, accessible from client 6.

[0104] Figure 11 shows an example of the evaluation output viewed by client 6. Figure 11 shows an example of a web page displayed based on the web browser program of client 6. In client 6, the crew selection or date selection screen is displayed first. The web page shown in Figure 11 includes the evaluation of "Crew Member C" and the date "October 1, 2024" on which that crew member was on watch. The web page shown in Figure 11 displays a bar graph showing the time distribution of work content during "Crew Member C's" watch time. When the cursor is placed on this bar graph, it would be desirable for text indicating the work content, ship speed, wind direction, etc., corresponding to the cursor's position to be superimposed. In the example in Figure 11, the time corresponding to the cursor's position on the bar graph is shown to be, for example, while sailing in "Osaka Bay".

[0105] The webpage shown in Figure 11 further displays a scatter plot of the behavior of "Seafarer C" at various points in time during his watch. The behavior includes changes in line of sight and changes in position. Changes in line of sight are shown using yaw, roll, and pitch data at each point in time. Position data is shown using coordinate data within the bridge.

[0106] The webpage shown in Figure 11 includes a pie chart showing the percentage of tasks identified during "Seafarer C's" watch time. The webpage also includes text indicating which of several levels "Seafarer C's" evaluation falls to, such as "B". This "Evaluation B" is given when the evaluation value calculated using the processing procedure in Figure 9 is lower than a predetermined value, meaning that the period during which the risk of accident occurrence is judged to be high falls within a predetermined percentage range (e.g., 75% to 90%) during the watch time. If the period during which the calculated evaluation value is lower than the predetermined value is a predetermined percentage or higher (e.g., 90%), it may result in an "Evaluation A" or "Evaluation S".

[0107] As shown in the example in Figure 11, the time distribution of crew members' behavior on the bridge and the time distribution of their work are visualized on a web page. This allows users to see what tasks each crew member performed and what movements they were making. By comparing the movements of each crew member, it is also possible to visualize analysis results such as what movements should be made, which can be used for crew member training. Since the data server 5 stores behavior data of multiple crew members from multiple ships S, it becomes possible to calculate the difference between the behavior of high-ranking crew members and the behavior of other crew members and use this for evaluation, etc.

[0108] [Second Embodiment] In the second embodiment, the processing unit 10 of the information processing device 1 outputs an evaluation of the crew's behavior as an evaluation value related to the crew's health, as an output unit 105. The configuration of the bridge watch management system 100 in the second embodiment is the same as that of the bridge watch management system 100 in the first embodiment, except for the evaluation value output by the output unit 105, so the same reference numerals are used for common components and detailed explanations are omitted.

[0109] In the second embodiment, the processing unit 10, as an output unit 105, outputs an evaluation value indicating the possibility of health problems, based on the work content determined by the work discrimination unit 104 and the crew member's behavior in that work content, based on whether the behavior is appropriate or not. The output unit 105 may output the evaluation level as a binary value, or as a discrete value or symbol such as A / B / C.

[0110] In the second embodiment, the processing unit 10 executes the processing procedure shown in Figure 8, similar to the first embodiment. In the second embodiment, in step S108 of the processing procedure shown in Figure 8, the processing unit 10 calculates an evaluation value regarding the health of the crew member at the time of processing.

[0111] Figure 12 is a flowchart showing an example of the evaluation output procedure in the second embodiment. The processing procedure shown in the flowchart of Figure 12 corresponds to the function of the output unit 105 and corresponds to the details of the processing procedure in step S108 of Figure 8.

[0112] The processing unit 10 reads out behavioral data, navigation data, and operational data for a predetermined time period immediately preceding the processing time (step S821). The predetermined time period may be longer than the waiting time referenced when determining whether the alarm device 4 is active, for example, 3 minutes.

[0113] The processing unit 10 calculates the percentage, number, or frequency of time within a predetermined period in which the work content was determined to be "monitoring" (step S822). The processing unit 10 calculates an evaluation value according to the calculated percentage, number, or frequency of time determined to be "monitoring" (step S823). The evaluation value here represents the level of health risk to the seafarer. In step S823, the processing unit 10 calculates a higher risk as an evaluation value for the seafarer's health status the smaller the percentage of time determined to be monitoring, or the smaller the frequency of time determined to be monitoring.

[0114] The processing unit 10 adjusts the evaluation value according to the number, duration, or frequency of times in the most recent predetermined time when the crew member's gaze was determined to be "downward" and the work content was determined to be "other" (step S824), and then terminates the process. In step S824, the processing unit 10 increases the risk as an evaluation value for health status if the number of times or duration of times when the crew member is determined to be looking down and doing something other than appropriate work is high.

[0115] When the processing unit 10 determines data indicating posture, it may calculate the percentage, number, or frequency of time within a predetermined period in which the crew member's posture was in a specific posture. A specific posture is, for example, a posture judged as "sleeping," a crouching posture, or a sitting posture. In this case, the processing unit 10 adjusts the evaluation value according to the calculated percentage, number, or frequency of time the crew member was in the specific posture. In this case, the higher the percentage, number, or frequency of time the crew member was in the specific posture, the higher the risk value for health status.

[0116] If the output evaluation value of the output unit 105 is above a predetermined value and it is determined that there is a high risk to the health of the crew members, the processing unit 10 outputs "high risk". In this case, the output unit 105 may output a reset signal to the alarm device 4 if it is determined that the situation is appropriate for business. In this case, the output unit 105 may not output a reset signal to the alarm device 4 if it is determined that the situation is dangerous. The alarm device 4 does not receive a reset signal and therefore sounds an alarm. This makes it possible to increase vigilance by sounding an alarm if a crew member is dozing off or if they are dozing off or distracted while underway. If there is a high risk of an accident, the information processing device 1 may notify an external ship monitoring system via the data server 5.

[0117] As shown in Figure 12, the bridge watch management system 100 can retrospectively evaluate a crew member's behavior as indicating a state of impaired attention if they frequently look away (downward). The bridge watch management system 100 may issue an alarm if the crew member's health is at high risk. The alarm device 4 may be directed to the captain of the vessel S. In this case, for example, the bridge watch management system 100 may alert the captain if a crew member spends a long time crouching, allowing for crew member reassignment or other measures before the crew member's condition becomes serious. The evaluation may be recorded in the information processing device 1 or data server 5 for reference, enabling retrospective analysis such as health monitoring and crew member reassignment.

[0118] [Third Embodiment] In the third embodiment, the information processing device 1 is installed on the ground. Figure 13 is a schematic diagram of the bridge watch management system 200 in the third embodiment. The configuration of the bridge watch management system 200 in the third embodiment is the same as that of the bridge watch management system 100 in the first embodiment, except for the changes due to the information processing device 1 being installed on the ground. Therefore, the same reference numerals are used for common components and detailed explanations are omitted.

[0119] In the third embodiment, the information processing device 1 is installed on land instead of the data server 5. The bridge watch management system 200 of the third embodiment includes a camera 2 installed inside the bridge of the ship S, a communication device 21 that acquires images taken by the camera 2 and transmits them to the information processing device 1, a group of devices 3 installed on the ship S, an alarm device 4, an information processing device 1 installed on land, and a client 6.

[0120] The communication device 21 communicates with the ship's internal communication network SN to acquire images from camera 2 and navigation data from the equipment group 3. The communication device 21 is a device that enables communication with the outside world via network N, which includes communication media such as satellite communication and carrier networks. The communication device 21 may include a communication device that communicates with land-based equipment or other ships using an AIS-dedicated frequency, or it may be a wireless communication device that connects to a carrier network, or it may be a wireless communication device for WiFi.

[0121] When communication with the information processing device 1 becomes possible, the communication device 21 transmits image data and navigation data acquired from the camera 2 and the group of devices 3 to the information processing device 1, along with time information indicating the time of acquisition.

[0122] In the third embodiment, the information processing device 1 can acquire images and navigation data transmitted wirelessly from the communication device 21 via satellite communication, carrier network, or WiFi communication. The information processing device 1 may also acquire images and navigation data via the AIS communication device 34 installed on the ship S using its own communication module that corresponds to the AIS dedicated frequency.

[0123] In the third embodiment, the first communication unit 12 of the information processing device 1 does not realize communication with the shipboard communication network SN, but is a device that realizes communication via a network including a public communication network and a carrier network. In the third embodiment, the information processing device 1 enables communication connections from external services such as weather forecasts, sea condition forecasts, or sea condition information provision outside the system via the first communication unit 12.

[0124] In the third embodiment, the processing unit 10 of the information processing device 1 performs the functions shown in Figure 3 of the first embodiment. The processing unit 10, as an image acquisition unit 101, receives image data from the camera 2 via the communication device 21, associated with time information, via the second communication unit 13, and acquires the necessary data. The processing unit 10, as a navigation data acquisition unit 103, acquires navigation data received by the second communication unit 13 via the communication device 21, along with the image data, associated with time information. The processing unit 10 of the third embodiment functions as a behavior determination unit 102 and a business determination unit 104. The processing unit 10 of the information processing device 1 in the third embodiment, as an output unit 105, transmits an evaluation of whether or not to issue an alarm by the alarm device 4 to the communication device 21 of the ship S via the second communication unit 13.

[0125] The processing unit 10 uses the image data acquired by the image acquisition unit 101 and the navigation data acquired by the navigation data acquisition unit 103 to execute the processing procedure shown in Figures 8-10, similar to the first embodiment. Thus, the entity that performs the behavior discrimination processing and evaluation output processing may be an information processing device 1 installed on the ship S, or an information processing device 1 installed on land. Part of each process may be executed by the information processing device 1 installed on the ship S, and other parts may be executed by the information processing device 1 installed on land.

[0126] The embodiments disclosed above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, and all modifications within the meaning and scope equivalent to the claims are included.

[0127] Furthermore, independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, while the claims use a multi-claim format in which claims refer to two or more other claims (multi-claim format), this is not the only format. A multi-claim format in which at least one multi-claim is referenced (multi-multi-claim format) may also be used.

[0128] The following additional information is disclosed regarding the embodiments described above.

[0129] (Note 1) A camera installed on the bridge of a ship to capture the crew inside the bridge, An acquisition unit that acquires navigation data related to the navigation of the aforementioned vessel, A behavior determination unit that determines the behavior of the crew members in the image based on the image acquired from the camera, A task determination unit determines the content of the crew member's duties based on the behavior determined by the behavior determination unit, An output unit outputs an evaluation of the crew member's behavior based on the navigation data acquired by the acquisition unit and the work content determined by the work determination unit. An information processing device equipped with the following features.

[0130] (Note 2) The behavior determination unit is, Given the aforementioned image, a trained model is used that outputs data on the position of the crew member's head and the orientation of the crew member's face, respectively. By providing the image acquired from the camera to the learning model, the data obtained regarding the position and orientation of the face is used to determine the direction of the crew member's gaze or the crew member's position within the bridge at the time corresponding to the image. The information processing device described in Appendix 1.

[0131] (Note 3) The acquisition unit acquires the ship's navigation position, the weather, wind speed and direction of the sea area in which the ship is navigating, the degree of congestion in the sea area, and the navigation status, whether or not the ship is navigating. The information processing device described in Appendix 1 or 2.

[0132] (Note 4) The behavior determination unit is, Based on the aforementioned image, or based on the design drawing data inside the bridge and the position of the crew member, the equipment in front of the crew member is identified. The behavior of the crew member in relation to the equipment is determined. An information processing device as described in any one of the appendices 1 to 3.

[0133] (Note 5) The output unit is, The system stores the range of movement of the crew member within the bridge, or the reference point for their line of sight, in association with identification data that identifies each of the multiple tasks. The behavior determination unit outputs an evaluation value depending on whether the behavior it determines meets the criteria corresponding to the determined business content. An information processing device as described in any one of the appendices 1 to 4.

[0134] (Note 6) The output unit is, The navigation data acquired by the aforementioned acquisition unit indicates the ship's position, the weather in the area of ​​navigation, wind speed and direction, time of day, and the criteria according to the navigation conditions, and the system stores these criteria accordingly. An evaluation value is output based on whether the aforementioned criteria corresponding to the navigation position, weather, wind speed, wind direction, time of day, or conditions are met. The information processing device described in Appendix 5.

[0135] (Note 7) The output unit outputs an evaluation value of the crew's behavior regarding the safety aspects of the ship's navigation. An information processing device as described in any one of the appendices 1 to 6.

[0136] (Note 8) The output unit changes the criteria for evaluating the behavior based on the position of the vessel at the time corresponding to the image, the weather in the sea area where the vessel is navigating, or the navigation status of the vessel. The information processing device described in Appendix 7.

[0137] (Note 9) The output unit changes the criteria for evaluating the behavior corresponding to the determined work content, based on the work content of the vessel or the presence or severity of past accident cases in the vessel's navigation position and surrounding area. The information processing device described in Appendix 7 or 8.

[0138] (Note 10) The output unit outputs an evaluation value relating to the crew member's health in relation to the crew member's behavior. An information processing device as described in any one of the appendices 1 to 9.

[0139] (Note 11) A computer installed on the bridge of a ship, which acquires images from a camera that captures the crew inside the bridge, Navigation data relating to the navigation of the said vessel is acquired from equipment installed on the said vessel. Based on the image acquired from the aforementioned camera, the behavior of the crew members shown in the image is determined. Based on the identified behavior, the duties of the crew member are determined. Based on the acquired navigation data and the identified work content, an evaluation of the crew member's behavior is output. Information processing methods.

[0140] (Note 12) A computer that acquires images from a camera installed on the bridge of a ship, which captures the crew inside the bridge, Navigation data relating to the navigation of the said vessel is acquired from equipment installed on the said vessel. Based on the image acquired from the aforementioned camera, the behavior of the crew members shown in the image is determined. Based on the identified behavior, the duties of the crew member are determined. Based on the acquired navigation data and the identified work content, an evaluation of the crew member's behavior is output. A computer program that executes a process. [Explanation of Symbols]

[0141] 100 Funabashi Duty Management System 1. Information Processing Device 10 Processing Unit 101 Image acquisition unit 102 Behavior determination unit 103 Navigation Data Acquisition Unit 104 Business Discrimination Department 105 Evaluation Output Unit M1 Learning Model M11 Head Detection Model M12 Head Direction Discrimination Model P1 Information Processing Program (Computer Program) 2 cameras 31 Navigation control system 32 GPS receivers 33 Wind direction anemometer 34 AIS communication device 35 Chart Display Systems 4 Alarm device 5. Data Server

Claims

1. A camera installed on the bridge of a ship to capture the crew inside the bridge, An acquisition unit that acquires navigation data related to the navigation of the aforementioned vessel, A behavior determination unit that determines the behavior of the crew members in the image based on the image acquired from the camera, A task determination unit determines the content of the crew member's duties based on the behavior determined by the behavior determination unit, An output unit outputs an evaluation of the crew member's behavior based on the navigation data acquired by the acquisition unit and the work content determined by the work determination unit. An information processing device equipped with the following features.

2. The behavior determination unit is, Given the aforementioned image, a trained model is used that is trained to output data on the position of the crew member's head and the orientation of the crew member's face, respectively. By providing the image acquired from the camera to the learning model, the data obtained regarding the position and orientation of the face is used to determine the direction of the crew member's gaze or the crew member's position within the bridge at the time corresponding to the image. The information processing apparatus according to claim 1.

3. The acquisition unit acquires the ship's navigation position, the weather, wind speed and direction of the sea area in which the ship is navigating, the degree of congestion in the sea area, and the navigation status, whether or not the ship is navigating. The information processing apparatus according to claim 1.

4. The behavior determination unit is, Based on the aforementioned image, or based on the design drawing data inside the bridge and the position of the crew member, the equipment in front of the crew member is identified. The behavior of the crew member in relation to the equipment is determined. The information processing apparatus according to any one of claims 1 to 3.

5. The output unit is, The system stores the range of movement of the crew member within the bridge, or the reference point for their line of sight, in association with identification data that identifies each of the multiple tasks. The behavior determination unit outputs an evaluation value depending on whether the behavior it determines meets the criteria corresponding to the determined business content. The information processing apparatus according to any one of claims 1 to 3.

6. The output unit is, The navigation data acquired by the aforementioned acquisition unit indicates the ship's position, the weather in the area of ​​navigation, wind speed and direction, time of day, and the criteria according to the navigation conditions, and the system stores these criteria accordingly. An evaluation value is output based on whether the aforementioned criteria corresponding to the navigation position, weather, wind speed, wind direction, time of day, or conditions are met. The information processing apparatus according to claim 5.

7. The output unit outputs an evaluation value of the crew's behavior regarding the safety aspects of the ship's navigation. The information processing apparatus according to any one of claims 1 to 3.

8. The output unit changes the criteria for evaluating the behavior based on the position of the vessel at the time corresponding to the image, the weather in the sea area where the vessel is navigating, or the navigation status of the vessel. The information processing apparatus according to claim 7.

9. The output unit changes the criteria for evaluating the behavior corresponding to the determined work content, based on the work content of the vessel or the presence or severity of past accident cases in the vessel's navigation position and surrounding area. The information processing apparatus according to claim 8.

10. The output unit outputs an evaluation value relating to the crew member's health in relation to the crew member's behavior. The information processing apparatus according to any one of claims 1 to 3.

11. A computer installed on the bridge of a ship, which acquires images from a camera that captures the crew inside the bridge, Navigation data relating to the navigation of the said vessel is acquired from equipment installed on the said vessel. Based on the image acquired from the aforementioned camera, the behavior of the crew members shown in the image is determined. Based on the identified behavior, the duties of the crew member are determined. Based on the acquired navigation data and the identified work content, an evaluation of the crew member's behavior is output. Information processing methods.

12. A computer that acquires images from a camera installed on the bridge of a ship, which captures the crew inside the bridge, Navigation data relating to the navigation of the said vessel is acquired from equipment installed on the said vessel. Based on the image acquired from the aforementioned camera, the behavior of the crew members shown in the image is determined. Based on the identified behavior, the duties of the crew member are determined. Based on the acquired navigation data and the identified work content, an evaluation of the crew member's behavior is output. A computer program that executes a process.

Images