Notification device, notification method, and computer program

Abstract

We provide notification devices, notification methods, and computer programs that contribute to the safe navigation of ships. [Solution] The notification device is installed on the bridge of a ship and includes a camera that captures crew members inside the bridge, a behavior determination unit that determines the behavior of crew members in the image based on the image acquired from the camera, an acquisition unit that acquires navigation data relating to the ship's navigation at the time corresponding to the image, a necessity determination unit that determines whether notification is necessary for the behavior of the crew members based on the behavior determined by the behavior determination unit and the navigation data acquired by the acquisition unit, and a notification unit that notifies the crew members if it is determined that notification is necessary.

Description

Technical Field

[0001] The present invention relates to a notification device, a notification method, and a computer program that contribute to the safe navigation of ships.

Background Art

[0002] In order to maintain and improve the safety of ship navigation, large cargo ships or passenger ships that continue to navigate at night are obliged to install a preventive alarm device called BNWAS (Bridge Navigational Watch Alarm System) to prevent drowsiness or inattention. For safer navigation, not only drowsiness prevention by BNWAS, but also a system has been proposed that photographs crew members sitting at the helm and operating the ship with a camera, and determines whether they are in a drowsy state or their physical condition has deteriorated from the images taken by the camera (Patent Document 1, etc.).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] BNWAS issues an alarm when it is not reset by an operation on the connected device or by detection of the movement of the crew by a motion sensor during a predetermined reset waiting time. Although there is also an operation in which BNWAS does not issue an alarm while the ship is at anchor, since it issues an alarm according to a uniform standard regardless of the situation during navigation, there is a concern that the crew will become accustomed to the notification. In addition, there is also a concern that in a situation where visual monitoring operation is insufficient due to long-term device operation, BNWAS alone cannot issue an alarm appropriately.

[0005] This disclosure is made in light of the circumstances described above and aims to provide a notification device, notification method, and computer program that contribute to the safe navigation of ships. [Means for solving the problem]

[0006] A notification device according to one aspect of this disclosure is installed on the bridge of a ship and comprises a camera that captures crew members inside the bridge, a behavior determination unit that determines the behavior of crew members captured in the image based on the image acquired from the camera, an acquisition unit that acquires navigation data relating to the ship's navigation at a point in time corresponding to the image, a necessity determination unit that determines whether or not to notify the crew member of the behavior based on the behavior determined by the behavior determination unit and the navigation data acquired by the acquisition unit, and a notification unit that notifies the crew member if it is determined that notification is necessary.

[0007] In one aspect of this disclosure, the notification system determines the behavior of crew members in the bridge by analyzing images captured by cameras, and then determines whether notification is necessary based on the determined behavior. Unlike BNWAS, which immediately notifies if no operation or movement is detected during a uniform waiting period, the notification system determines whether notification is necessary based on the crew member's behavior, and only notifies the crew member if it is determined that notification is necessary. This allows for appropriate timing of notifications.

[0008] In a notification device according to one aspect of the present disclosure, the behavior determination unit may use a learning model that has been trained to output data on the orientation of the crew member's face when an image is provided, and determine the direction of the crew member's gaze at the time corresponding to the image based on the face orientation data obtained by providing the learning model with an image acquired from the camera.

[0009] One aspect of this disclosure involves the use of a learning model for image recognition. The learning model detects the heads of people in an image and further detects the orientation of their faces. By using the learning model, the notification device can determine the direction of the crew member's gaze based on the face orientation detection results.

[0010] In a notification device relating to one aspect of this disclosure, the behavior determination unit may use a head detection model that has been trained to output the position of the crew member's head when given an image, and use the position of the face obtained by providing the image to the head detection model, in addition to the data on the orientation of the face, to determine the direction of the crew member's line of sight and the position of the crew member within the bridge at the time corresponding to the image.

[0011] In one aspect of this disclosure, a model may be used as a learning model for image recognition that detects the heads of people in an image and outputs the position of the heads. The position of the heads in the image makes it possible to determine the location of the crew member in the bridge. Depending on the detection result of the face orientation and the layout of the camera in the bridge, the notification device can determine the crew member's behavior in detail.

[0012] In a notification device relating to one aspect of this disclosure, the acquisition unit may acquire the navigation position of the vessel, the weather, wind speed and direction of the sea area in which the vessel is navigating, the degree of congestion in the sea area, and the navigation status of whether or not the vessel is navigating.

[0013] One aspect of this disclosure is that navigation data includes the ship's position, weather conditions in the area of ​​navigation, wind speed and direction, and the degree of congestion in the area. Depending on whether the ship is in port or in the open sea, the crew's operational movements may vary. When sailing near a port, the crew is likely to be busy with monitoring and maneuvering. On the other hand, when sailing away from port, with good visibility and low wave height, the crew is likely to be less active, such as quietly monitoring in one place. Thus, appropriate crew behavior can vary depending on the situation, and based on the acquired navigation data, it becomes possible to appropriately determine whether or not notification is necessary.

[0014] In a notification device relating to one aspect of this disclosure, the necessity determination unit may raise the criteria for determining whether the notification is necessary if the navigation data acquired by the acquisition unit indicates that the vessel is at anchor.

[0015] One aspect of this disclosure is that when a vessel is at anchor, the risk of an accident is low, so the criteria for determining whether or not to issue a notification may be set higher, either by not issuing a notification or by making it less likely that a notification will be deemed necessary. Compared to when the vessel is underway, when it is at anchor, the crew is not on the bridge, so it is possible to automatically control the system to prevent unnecessary notifications from being issued.

[0016] In a notification device relating to one aspect of this disclosure, the necessity determination unit may lower the criteria for determining whether notification is necessary if the navigation data acquired by the acquisition unit indicates that the vessel is underway and visibility is poor.

[0017] One aspect of this disclosure is that when a vessel is at sea and visibility is poor, the risk of an accident is high, so the criteria for determining that notification is necessary may be set lower to make it easier to determine that notification is necessary. Even under the same conditions of navigation, when visibility is poor, compared to when the vessel is sailing away from port, visibility is good, and the wave height is not high, the crew will likely be busy monitoring and maneuvering the vessel. In such circumstances, it is expected that the effectiveness of the warning notification will be maintained because it will be easier to determine that notification is necessary if the crew is looking behind rather than forward, or if they are operating equipment for a long period of time.

[0018] In a notification device according to one aspect of this disclosure, the necessity determination unit may lower the criteria for determining whether notification is necessary if the navigation data acquired by the acquisition unit indicates a high degree of congestion in the sea area where the vessel is navigating.

[0019] One aspect of this disclosure is that when a vessel is at sea and the area it is navigating is congested, the risk of an accident is high, so the criteria for determining that notification is necessary may be set lower to make it easier to determine that notification is required. Even under the same navigational conditions, if a vessel is navigating a congested area, an area where aquaculture is prevalent, or an area with a history of many accidents, the crew should monitor carefully. In such cases, the criteria will be adjusted to make it easier to determine that notification is necessary, and it is expected that the effectiveness of the warning notification will be maintained.

[0020] In the notification device according to one aspect of the present disclosure, the necessity determination unit may change the determination criterion for the necessity of the notification according to the navigation position of the ship indicated by the navigation data acquired by the acquisition unit and the presence or absence or severity of past accident cases in the vicinity.

[0021] In one aspect of the present disclosure, when there is a past accident case that is the same as or similar to the business content of a ship during navigation, especially when there is a more serious accident case, it is preferably determined that notification is necessary in order to give the crew a sense of tension. Conversely, even when there has been a past accident case at the navigation position and its surroundings, since it is a sea area that should be monitored frequently, it is preferably determined that notification is necessary.

[0022] In the notification device according to one aspect of the present disclosure, when the determination criterion is raised, the necessity determination unit may increase the reference time for determining that notification is necessary with respect to the duration of a specific behavior of the crew that has been determined.

[0023] In one aspect of the present disclosure, when raising the determination criterion, the length until it is determined that notification is necessary is set longer with respect to the duration of a specific behavior of the crew, and the notification frequency is reduced. The specific behavior is, for example, a stationary state suspected of drowsiness, or a downward line of sight corresponding to during equipment operation. Thereby, it can be expected that the timing of the notification becomes appropriate.

[0024] In the notification device according to one aspect of the present disclosure, when the determination criterion is raised, the necessity determination unit may increase the reference frequency for determining that notification is necessary with respect to the frequency of a specific behavior of the crew that has been determined.

[0025] In one aspect of the present disclosure, when raising the determination criterion, the reference frequency until it is determined that notification is necessary is set higher with respect to the frequency of a specific behavior of the crew, and the notification frequency is reduced. It can be expected that the timing of the notification becomes appropriate.

[0026] The notification method according to one aspect of the present disclosure is provided in the bridge of a ship. A computer that acquires an image from a camera that captures crew members inside the bridge determines the behavior of the crew members shown in the image based on the image acquired from the camera, acquires navigation data related to the navigation of the ship at the time corresponding to the image, determines whether notification of the behavior of the crew members is necessary based on the determined behavior and the acquired navigation data, and notifies the crew members when it is determined that notification is necessary.

[0027] A computer program according to one aspect of the present disclosure causes a computer that is provided in the bridge of a ship and acquires an image from a camera that captures crew members inside the bridge to determine the behavior of the crew members shown in the image based on the image acquired from the camera, acquire navigation data related to the navigation of the ship at the time corresponding to the image, determine whether notification of the behavior of the crew members is necessary based on the determined behavior and the acquired navigation data, and execute a process of notifying the crew members when it is determined that notification is necessary.

Brief Description of Drawings

[0028] [Figure 1] It is a schematic diagram of the bridge duty management system of the first embodiment. [Figure 2] It is a block diagram showing the configuration of the notification device. [Figure 3] It is an explanatory diagram of the function of the notification device based on an information processing program. [Figure 4] It is an explanatory diagram showing an example of an image obtained from a camera. [Figure 5] It is an explanatory diagram showing an example of an image obtained from a camera. [Figure 6] It is a schematic diagram of the learning model used in the notification device. [Figure 7] It is a diagram showing the correspondence between the area inside the bridge and the position of the head in the image. [Figure 8] It is a flowchart showing an example of a processing procedure by the notification device. [Figure 9] It is a flowchart showing an example of a determination process of necessity. [Figure 10] This flowchart shows an example of a process for determining necessity. [Figure 11] This flowchart shows an example of the processing procedure by the notification device 1 in a modified example. [Figure 12] This flowchart shows an example of the process for determining necessity in a modified example. [Figure 13] This flowchart shows an example of the process for determining necessity in a modified example. [Figure 14] This is a schematic diagram of the second embodiment of the bridge watch management system. [Figure 15] This is an explanatory diagram of the function of the notification device according to the second embodiment. [Figure 16] This flowchart shows an example of the processing procedure by the notification device of the second embodiment. [Modes for carrying out the invention]

[0029] This disclosure will be described in detail with reference to drawings illustrating its embodiments. The following embodiments describe a bridge watch management system including the notification device of this disclosure.

[0030] [First Embodiment] Figure 1 is a schematic diagram of the bridge watch management system 100 according to the first embodiment. The schematic diagram in Figure 1 shows the arrangement of each device on a schematic design drawing of the interior of the bridge of the ship S. The bridge watch management system 100 includes a camera 2 installed inside the bridge of the ship S, a notification device 1 that determines whether notification is necessary based on the image captured by the camera 2, a group of devices 3 installed on the ship S, and a data server 4 installed on land.

[0031] Camera 2 uses a visible light image sensor to output images. Camera 2 may also be equipped with a far-infrared image sensor for nighttime use. Camera 2 is installed on the ceiling or side wall of the bridge to capture the crew members inside the bridge with the widest possible field of view. Camera 2 continuously outputs images in a time series at a predetermined rate (e.g., 10 frames per second).

[0032] The notification device 1 is an edge computer installed on the ship S. The notification device 1 acquires image data from camera 2 via the ship's communication network SN or a signal line (not shown). Based on the acquired image data, the notification device 1 determines the behavior of the crew members captured in the image, decides whether notification is necessary based on the crew members' behavior, and issues an alert if it determines that notification is necessary. The detailed configuration and processing of the notification device 1 will be described later.

[0033] Equipment group 3 is a group of instruments that measure navigation data of the vessel S. Equipment group 3 includes a navigation control system 31, a GPS (Global Positioning System) receiver 32, an anemometer 33, an AIS (Automatic Identification System) communicator 34, and an Electronic Chart Display and Information System (ECDIS) 35. Equipment group 3 may also include radar, sonar, a speedometer, etc. (none of which are shown). Equipment group 3 is connected to the notification device 1 and the navigation control system of the vessel S via an onboard communication network SN or signal lines (not shown).

[0034] The shipboard communication network SN is a communication medium installed within the ship S. The shipboard communication network SN may be wired or wireless. The shipboard communication network SN enables communication with external communication equipment via connected satellite communication, carrier network, or AIS communication equipment 34.

[0035] The data server 4 is a server installed on land. The data server 4 may consist of one server computer or multiple server computers connected via a dedicated line or public communication network, and may be implemented as a cloud server that can be communicated to from the notification device 1 via a network. The data server 4 can acquire data from external services such as weather forecasts, ocean condition forecasts, or ocean condition information provision via the public communication network. The notification device 1 can send and receive data with the data server 4 wirelessly via satellite communication, carrier network, or WiFi communication. The notification device 1 may also send and receive data with the data server 4 via its own communication module or AIS communicator 34 that supports AIS dedicated frequencies. The notification device 1 can acquire weather forecasts, ocean condition forecasts, or ocean condition information acquired by the data server 4 and use it to determine whether or not notification is necessary.

[0036] In the bridge watch management system 100, the notification device 1 is a system that identifies the behavior of crew members on the bridge as shown in Figure 1, determines what kind of work they are doing, and provides appropriate notifications according to the crew members' behavior. In the bridge watch management system 100, the notification device 1 may also store the behavior identification results in association with time information. As shown in Figure 1, the bridge is equipped with monitors 30 of the equipment group 3, and crew members on watch perform various tasks such as visually checking the sea route from the bridge and visually checking the underwater conditions by looking at the monitors of the equipment group 3, thereby ensuring the continuation of safe navigation. The following describes in detail the process by which the notification device 1 identifies the behavior of crew members on the bridge using images from camera 2 and determines whether or not to notify the crew members.

[0037] Figure 2 is a block diagram showing the configuration of the notification device 1. The notification device 1 uses a small, high-performance computer used as a so-called edge computer. The notification device 1 may also be a server computer or a personal computer. The notification device 1 comprises a processing unit 10, a storage unit 11, a first communication unit 12, a second communication unit 13, and a notification unit 14.

[0038] The processing unit 10 includes one or more arithmetic processing units such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). The processing unit 10 also includes a temporary storage medium such as SRAM (Static Random Access Memory) and DRAM (Dynamic Random Access Memory). The processing unit 10 reads the information processing program P1 stored in the storage unit 11 into the temporary storage medium and executes it, thereby causing a general-purpose computer to perform various processes described later, and to function as the notification device 1 of this disclosure.

[0039] The storage unit 11 is a relatively large-capacity non-volatile storage medium such as an SSD (Solid State Drive) or a hard disk. The storage unit 11 stores the program (program product) necessary for the processing unit 10 to execute processing, and reference setting data. The setting data may include the type of ship S on which the notification device 1 is installed, and identification information. The setting data also includes data used for making decisions in the image processing described later. The program product includes an information processing program P1 and a learning model M1.

[0040] The information processing program (program product) P1 and learning model M1 stored in the memory unit 11 may be obtained by the processing unit 10 reading the information processing program P9 and learning model M9 stored in the non-temporary storage medium 9 readable from the computer and storing them in the memory unit 11. Alternatively, the information processing program P1 and learning model M1 may be obtained by the processing unit 10 downloading them from the download server via the second communication unit 13 and storing them in the memory unit 11.

[0041] The memory unit 11 stores image data acquired from the camera 2 in a time-series format, linked to time information, so that it can be referenced in that order. The memory unit 11 stores data of behavior determined from the image data through processing described later, linked to time information. The memory unit 11 may also store evaluation values ​​for the crew's behavior obtained as a result of the processing of the image data described later.

[0042] The memory unit 11 contains an accident database 111 that stores data on past maritime accidents. The accident database 111 stores the date and time of the accident, location data of the accident's location, type of accident, and severity, associated with the accident identification number. The location data may be data that distinguishes sea areas, or it may be latitude and longitude information. The type of accident is data that identifies collision, capsizing, grounding, loss of navigation, engine failure, flooding, fire, etc.

[0043] The first communication unit 12 is a communication device that enables communication with the camera 2. The first communication unit 12 enables communication via the ship's communication network SN to which the camera 2 is connected. The first communication unit 12 may be a wireless communication device for WiFi® or a wired communication device for Ethernet®.

[0044] The second communication unit 13 is a communication device that enables communication with the data server 4. The second communication unit 13 may be a communication device that enables carrier communication via a carrier network, a communication device that supports wireless networks such as WiFi or Bluetooth (registered trademark), a communication device that communicates via satellite communication, or a communication device that communicates using an AIS-dedicated frequency. The second communication unit 13 may be replaced by equipment for communication with external devices connected to the shipboard communication network SN.

[0045] The notification unit 14 includes a speaker. The notification unit 14 notifies an alert by outputting a beep sound or voice based on a control signal from the processing unit 10.

[0046] Figure 3 is an explanatory diagram of the functions of the notification device 1 based on the information processing program P1. The processing unit 10 of the notification device 1 performs the various functions shown in Figure 3 based on the information processing program P1. The processing unit 10 functions as an image acquisition unit 101, a behavior determination unit 102, a navigation data acquisition unit 103, a business determination unit 104, and a necessity determination unit 105.

[0047] The processing unit 10 of the notification device 1 functions as an image acquisition unit 101 that acquires multiple images in chronological order of crew members inside the bridge of the ship S. As the image acquisition unit 101, the processing unit 10 acquires images (see Figure 5) from the camera 2.

[0048] The processing unit 10 of the notification device 1 functions as a behavior determination unit 102 that determines the behavior of the crew members in the image based on the image acquired by the image acquisition unit 101 from the camera 2. The behavior determination unit 102 determines the direction of the crew member's gaze, their position on the bridge, and their posture based on the position of their head and the orientation of their face.

[0049] The behavior determination unit 102 uses the learning model M1 to detect the position of the crew member's head within the image. Based on the bridge design data and the depth map of the image pre-set for camera 2, the behavior determination unit 102 identifies coordinate data indicating the crew member's position within the bridge from the position of the head within the image. The behavior determination unit 102 may also use data that pre-divides the bridge into multiple areas and the depth map of the image pre-set for camera 2 to determine which area within the bridge the crew member in the image is located in, using data that identifies each area.

[0050] The behavior determination unit 102 uses the learning model M1 to determine the orientation of the crew member's face from the area where the head is visible. The behavior determination unit 102 identifies data that represents the orientation of the face in terms of yaw (rotation on the horizontal plane, head movement), roll (rotation on the vertical plane, tilt of the face), and pitch axis (degree of downward tilt of the neck and head). Using the bridge design drawing data, the behavior determination unit 102 determines from the identified orientation of the face whether the direction of the gaze is forward, starboard, port, rear, downward, other, or undeterminable, and outputs the determination result using gaze identification data that identifies each direction.

[0051] The behavior discrimination unit 102 may determine the posture, such as whether the person is sitting, standing, or leaning forward, based on the detected head position and the identified face orientation, and output data that identifies each posture. The discrimination method used by the behavior discrimination unit 102 is not limited to discrimination using the learning model M1, but may also use pattern recognition.

[0052] The behavior determination unit 102 may determine that the crew member is maintaining a specific posture if the detected head position has not changed from the position detected in the previous image. Alternatively, the behavior determination unit 102 may determine whether the crew member is maintaining a specific posture based on the difference between images, rather than the head position. The specific posture may be, for example, the head, i.e., the crew member, being still, or the gaze being directed downwards.

[0053] The processing unit 10 functions as a navigation data acquisition unit 103 that acquires navigation data related to the navigation of the vessel S. The navigation data acquisition unit 103 acquires data output from the navigation control system 31, GPS receiver 32, wind direction and speed meter 33, AIS communication device 34, and electronic chart display system 35 at any time. The navigation data acquisition unit 103 may also acquire data from other devices 3.

[0054] The processing unit 10 acquires the speed, direction, and inclination of the vessel S from the navigation control system 31 using the navigation data acquisition unit 103. The processing unit 10 may also acquire whether or not the anchor has been lowered using the navigation data acquisition unit 103. Based on the vessel speed, the processing unit 10 can acquire the navigation status of the vessel S, indicating whether it is sailing or at anchor. The processing unit 10 may also acquire the navigation status indicating whether it is sailing or at anchor from the navigation control system 31.

[0055] Depending on whether the ship is underway or at anchor, the processing unit 10 can determine whether or not it should perform external monitoring as part of its duties. If the ship is underway and external monitoring should be performed, the processing unit 10 can determine whether or not the crew members on the bridge are performing external monitoring or operating specific equipment.

[0056] The navigation data acquisition unit 103 acquires weather forecasts, sea condition forecasts, or sea condition information from the data server 4 via the second communication unit 13 or the AIS communication device 34.

[0057] The processing unit 10 acquires the time and position during navigation using the navigation data acquisition unit 103. The processing unit 10 may also acquire weather, wind speed, and wind direction using the navigation data acquisition unit 103. Based on the navigation position, time, weather, wind direction, and wind speed, it is possible to determine whether the area under navigation has good visibility, etc.

[0058] The processing unit 10 functions as a task determination unit 104 that determines the work of a crew member in an image based on the behavior (position, direction of gaze, and whether a specific posture is being maintained) determined by the behavior determination unit 102. The function of the task determination unit 104 is not essential in this disclosure. The task determination unit 104 determines the work of a crew member as "monitoring," "equipment operation," "other," or "unknown" from a combination of the behavior output by the behavior determination unit 102, i.e., the coordinate data of the crew member's position or area identification data (see Figure 7), and the identification data indicating the result of gaze determination. For example, if the area containing the crew member's position is an equipment area, the task determination unit 104 may determine it as "equipment operation" even if the gaze is directed downwards, distinguishing it from "unknown" outside of the equipment area.

[0059] If the identified crew member's task is "surveillance," the task discrimination unit 104 further distinguishes the front, starboard, port, and rear based on the gaze discrimination result by the behavior discrimination unit 102. If the identified crew member's task is "equipment operation," the task discrimination unit 104 further distinguishes the equipment to be operated based on the area containing the crew member's position or by recognition of the image. The task discrimination unit 104 may distinguish the task by distinguishing it as, for example, "radar (confirmation) operation" or "nautical chart (confirmation) operation."

[0060] The discrimination method used by the business discrimination unit 104 may also be based on another learning model that has been trained to output business content identification data when behavior data including location coordinate data or area identification data and gaze identification data is input.

[0061] The processing unit 10 functions as a necessity determination unit 105 that determines whether or not to send a notification from the notification unit 14, based on the behavior of the crew determined by the behavior determination unit 102 and the navigation data acquired by the navigation data acquisition unit 103. As a necessity determination unit 105, the processing unit 10 determines the necessity using a determination criterion calculated based on the behavior and navigation data, specifically a reference time. The necessity determination unit 105 may also take into account the content of the work determined by the work determination unit 104 when determining the necessity.

[0062] The necessity determination unit 105, for example, if it determines that a crew member's gaze is directed forward during navigation, raises the criteria for determining whether notification is necessary, i.e., lengthens the reference time, making it easier to determine that notification is unnecessary. This is because, when a crew member's gaze is directed forward, the necessity determination unit 105 determines that appropriate behavior is being performed according to the conditions during navigation and the risk of an accident is low, thus reducing the need to issue an alert. If the necessity determination unit 105 can determine that the crew member's work content, as determined by the work determination unit 104, is "monitoring," and the crew member's gaze is directed forward, it may also raise the criteria for determining whether notification is necessary, i.e., lengthens the reference time, making it easier to determine that notification is unnecessary, since appropriate behavior is being performed and the risk of an accident is low.

[0063] Conversely, if the necessity determination unit 105 determines that a crew member's gaze is directed downwards during navigation, it lowers the criteria for determining whether notification is necessary, i.e., shortens the standard time, making it easier to determine that notification is necessary. This is because if a crew member's gaze is directed downwards, there is a possibility that they are looking away, and the necessity determination unit 105 determines that the risk of an accident increases, and therefore it is necessary to issue an alert.

[0064] The determination method used by the necessity determination unit 105 can employ various other methods. Furthermore, some of the functions of the notification device 1 described above may be executed by a data server 4 that can communicate with the notification device 1. For example, any one of the functions of the behavior determination unit 102, the business determination unit 104, and the necessity determination unit 105 may be executed by the data server 4.

[0065] Figures 4 and 5 are explanatory diagrams showing examples of images obtained from camera 2. Figures 4 and 5 schematically represent images acquired from camera 2 by the image acquisition unit 101 when the processing unit 10 is used. Figures 4 and 5 show images taken while the vessel is in motion. Both images in Figures 4 and 5 show a crew member. In the image shown in Figure 4, the crew member is looking forward in the direction of travel of the vessel S, i.e., straight ahead, and is operating the steering wheel. In the image shown in Figure 5, the crew member is leaning back on a stool, looking downwards, and operating a smartphone.

[0066] Figure 6 is a schematic diagram of the learning model M1 used in the notification device 1. The learning model M1 includes a head detection model M11 and a head direction discrimination model M12. When an image like the one shown in Figure 4 or Figure 5, acquired from camera 2, is input to the head detection model M11, it detects the region containing the head in that image. The head direction discrimination model M12 is a model that discriminates the direction of the head for the region detected by the processing of head detection model M11 and outputs the yaw, pitch, and roll of the head as vectors. Each model is a learning model that has been trained by deep learning using a general-purpose training dataset with a neural network. Each model may also be tuned to improve the recognition accuracy for images of the inside of the bridge using images of crew members inside the bridge that can be captured by camera 2.

[0067] The head detection model M11 is trained to output, upon receiving an image, the coordinate data of the region containing the head within that image, and a score indicating the likelihood that it is a head. The head detection model M11 is, for example, an SSD (Single Shot MultiBox Detector) and outputs coordinate data of a predetermined area, such as a rectangle, that contains the head.

[0068] The head orientation discrimination model M12 is trained to output vector data as features that indicate the direction the head is facing when it receives an image of the head range extracted based on coordinate data output from the head detection model M11. The vector data is the yaw, pitch, and roll of the head in a space whose coordinate axes are the left-right, up-down, and depth directions of the original image.

[0069] The head detection model M11 and the head orientation discrimination model M12 may employ models without convolutional layers, such as Transformers, or other algorithms. The head detection model M11 and the head orientation discrimination model M12 may each employ models without convolutional layers, such as Vision Transformers, or other architectures. The learning model M1 may be a model using a support vector machine or the like.

[0070] The processing unit 10, using the behavior determination unit 102, determines the position of the captured sailor's head within the image from the coordinate data of the area including the head output from the head detection model M11. For example, the processing unit 10 determines the center point of the rectangular area output from the head detection model M11 as the position within the image. The processing unit 10, using the behavior determination unit 102, calculates the coordinates of the position within the bridge from the position of the sailor's head in the image previously captured by camera 2, based on the bridge design drawing data.

[0071] The processing unit 10, using the behavior determination unit 102, may store information that identifies the area where a person with a head is standing if that head is captured in the image, based on the design drawing data of the bridge, and determine the area. Figure 7 is a diagram showing the correspondence between areas within the bridge and the positions of heads in the image. The upper part of Figure 7 shows divided areas on a map of the bridge based on the design drawing data. The area identification data is divided into, for example, "1: In front of the steering wheel", "2: In front of the communication device", "3: In front of the monitor", "4: Other", "5: In front of the workbench", and "6: Other". The lower part of Figure 7 shows the relationship between the position of a head in the image and the area identification data of the standing position of a crew member whose head is in that position. As shown in Figure 7, for each pre-divided area, the range of the position of a crew member's head standing in that area is stored in the image.

[0072] This allows the behavior determination unit 102 to determine the position and area of ​​the crew member from the position of the head in the image. The behavior determination unit 102 may determine only one of the crew member's coordinates within the bridge or area identification data.

[0073] The processing unit 10, using the behavior determination unit 102, determines identification data indicating the line of sight of the detected sailor based on the vector data output from the head direction determination model M12, with the left-right direction, up-down direction, and depth direction as coordinate axes, from "1: Front", "2: Starboard", "3: Port", "4: Up", "5: Down", and "6: Unknown" in the space of the bridge. The processing unit 10 may also determine this after converting the data into vectors with the left-right direction, vertical direction, and the fore-aft direction of the ship S as axes in the space of the bridge using the behavior determination unit 102.

[0074] The processing unit 10 may output posture identification data using the behavior determination unit 102. For example, if the detected head position is "3: In front of the monitor" as shown in Figure 7, and the gaze is directed downwards, the person may be "crouching" not in front of the monitor, but in the area of ​​"1: In front of the steering wheel", the area of ​​"4: Other", or the area of ​​"6: Other". The processing unit 10 determines this "crouching" posture based on the angle of the gaze. If the processing unit 10 determines that the gaze is directed downwards at a predetermined angle or more, it may correct the area identification data identified based on the position of the head in the image. The processing unit 10 may also determine the sailor's posture in the behavior determination unit 102 based on the recognition of parts of the image other than the head. In this case, the processing unit 10 may use, for example, a posture determination model that has been trained to output posture identification data when an image of a person is given.

[0075] The processing in the bridge watch management system 100 configured in this way will now be explained. Figure 8 is a flowchart showing an example of the processing procedure by the notification device 1. The processing unit 10 of the notification device 1 continuously executes the following processes while it is running.

[0076] The processing unit 10 acquires image data from the camera 2 using the image acquisition unit 101 (step S101). The processing unit 10 provides the acquired image data to the learning model M1 (step S102). Based on the position and face orientation data output from the learning model M1, the processing unit 10 determines the behavior using the function of the behavior determination unit 102 (step S103).

[0077] The processing unit 10 acquires data on the behavior of the crew members in the image from the behavior determination unit 102 (step S104). In step S104, the processing unit 10 acquires data from the behavior determination unit 102 indicating whether or not the crew member is stationary, coordinate data of the position or area identification data indicating an area, and identification data indicating the crew member's line of sight. The processing unit 10 may also acquire identification data of the crew member's posture. In step S104, if there are multiple crew members in the image, the processing unit 10 distinguishes each crew member in the image and acquires data on their behavior.

[0078] The processing unit 10, using the navigation data acquisition unit 103, acquires navigation data from the group of equipment 3, including the navigation status of the vessel S, the navigation position of the vessel S, data identifying the sea area in which the vessel S is navigating, and the weather, wind speed, and wind direction of the sea area in which the vessel is navigating (step S105).

[0079] The processing unit 10, using the functions of the task discrimination unit 104, determines the task being performed by the crew member in the image based on the acquired behavioral data (step S106). As mentioned above, task discrimination in step S106 is not mandatory.

[0080] In step S106, the processing unit 10 determines, for example, that the crew member is at "1: in front of the steering wheel" and looking "straight ahead" based on the area identification data of the crew member's position and the line of sight identification data, and identifies the task as "monitoring". Similarly, the processing unit 10 determines that the task is "unknown" if the crew member is at "1: in front of the steering wheel" and looking "downward". Similarly, the processing unit 10 determines that the task is "equipment (confirmation) operation" if the crew member is at "3: in front of the monitor" and looking "downward". These methods for determining tasks based on area identification data and line of sight identification data may also be a method in which a table of correspondences between combinations of position and line of sight and "task content" is stored in the storage unit 11 in advance, and the task is identified from that correspondence.

[0081] The processing unit 10 stores the crew behavior data acquired in step S104, the navigation data acquired in step S105, and the work content determined in step S106 in the database of the storage unit 11, along with the time information (step S107).

[0082] The processing unit 10 determines whether the crew member is continuing a specific behavior based on the function of the necessity determination unit 105 (step S108). The specific behavior is, for example, a state of being still or looking downwards. If it is determined that the crew member is continuing a specific behavior (S108: YES), the duration of the specific behavior from the time the crew member's movement was most recently detected is calculated (step S109).

[0083] The processing unit 10, using the necessity determination unit 105, determines whether notification from the notification unit 14 is necessary based on the duration of the specific behavior calculated in step S109, the behavior acquired in step S104, and the navigation data acquired in step S105 (step S110). Details of the determination process in step S110 will be described later.

[0084] If the processing unit 10 determines that notification is required (S110: YES), it sends an alert to the crew from the notification unit 14 (step S111). The processing unit 10 stores time or count data indicating the time the alert was sent in the storage unit 11 (step S112). The processing unit 10 stores the evaluation value derived in step S110 in the database of the storage unit 11 (step S113) and terminates the process.

[0085] If the processing unit 10 determines in step S110 that notification is unnecessary (S110: NO), it proceeds to step S113 and terminates the process.

[0086] If the processing unit 10 determines in step S108 that the crew member is not continuing a specific behavior (S108: NO), it determines that notification is unnecessary (step S114) and stores the time of the determination as the time when the crew member's movement was detected (step S115). The processing unit 10 proceeds to step S113 and terminates the process.

[0087] If multiple crew members are visible in the image, the processing unit 10 should adopt the judgment result with the lower assessment value of the accident risk based on their behavior. Also, if multiple crew members are visible in the image, the processing unit 10 should distinguish between the first crew member and the second crew member and store the behavior data, navigation data, work content data, and assessment value separately. If the learning model M1 includes a model capable of face recognition, the processing unit 10 may store the data processed on images with multiple crew members in association with face recognition data. Even when faces can be identified across images at multiple points in time and images separated by time, it is good practice to process and store the data in association with face recognition data.

[0088] Figures 9 and 10 are flowcharts illustrating an example of the necessity determination process. The processing steps shown in the flowcharts of Figures 9 and 10 correspond to the functions of the necessity determination unit 105 and to the detailed processing steps of step S110 in Figure 8.

[0089] The processing unit 10 reads out behavioral data and navigation data for a predetermined time period immediately preceding the processing time (step S501). In step S501, the processing unit 10 may also read out data on the business content. The predetermined time period may be longer than the initial setting for the duration of stationary time, for example, 3 minutes.

[0090] The processing unit 10 determines whether the navigation status included in the read navigation data is underway (step S502). If it is determined that the navigation status is underway (S502: YES), the processing unit 10 adds an additional value corresponding to the time period included in the navigation data to the evaluation value indicating the level of safety risk (step S503). If the time period is daytime, the additional value is zero or a small value, and if the time period is early morning or evening, the additional value is larger than that for daytime. The larger the additional value, the shorter the reference time described later tends to be, and the smaller the additional value, the longer the reference time tends to be. If the time period is nighttime, the additional value is larger than that for early morning or evening. The processing unit 10 adds an additional value corresponding to the ship speed obtained from the ship speedometer to the evaluation value (step S504). The faster the ship speed, the larger the additional value.

[0091] The processing unit 10 adds an additional value to the evaluation value according to the wind speed and wind direction included in the navigation data (step S505). The faster the wind speed, the larger the additional value, and the more perpendicular the wind direction is to the direction of navigation, the larger the additional value. The processing unit 10 adds an additional value to the evaluation value according to the weather (step S506). If the weather is clear, the additional value is small, and if it is raining or snowing, the additional value is large. The more cloud cover there is, the larger the additional value may be, and good visibility may be included in the weather. It is desirable that the additional value be large when visibility is poor and small when visibility is good. The processing in steps S505 and S506 may be combined into an additional value according to the weather.

[0092] The processing unit 10 adds a value to the evaluation value corresponding to the degree of congestion in the area under navigation, as indicated by the navigation data (step S507). The processing unit 10 may obtain the degree of congestion from the second communication unit 13 and the data server 4, or it may obtain it by communicating with other vessels using the AIS communication device 34. The processing unit 10 may detect other vessels using radar or the like, and increase the added value the closer the distance to other vessels is. The processing unit 10 increases the added value if there are facilities related to the fishing industry in the area under navigation.

[0093] The processing unit 10 references the accident database 111 for records of accidents in the area under navigation, as indicated by the navigation data (step S508). The processing unit 10 adds a value to the evaluation value according to the frequency and severity of accidents in the area under navigation (step S509). The more accidents there are, the larger the added value, and the higher the severity, the larger the added value.

[0094] The processing unit 10 adds or subtracts an evaluation value (step S510) according to the number of times and duration during navigation when the work content is determined to be "monitoring" in the most recent predetermined time. In step S510, the processing unit 10 increases or decreases the evaluation value in such a way that the risk of accident occurrence decreases (it is judged to be safer) as the number of times monitoring is determined to be monitoring increases or the duration of monitoring is determined to be longer. The processing unit 10 may also calculate in such a way that the evaluation of the level of risk decreases as the number of times "forward monitoring" is determined to be higher.

[0095] During navigation, the processing unit 10 adds or subtracts an evaluation value according to the distance traveled by the crew member in the most recent predetermined time (step S511). In step S511, the processing unit 10 increases or decreases the evaluation value in such a way that the greater the distance traveled and the more diverse the locations within the bridge the crew member has moved to, that is, the closer the possibility of falling asleep is to zero, the lower the evaluation of the risk of an accident occurring.

[0096] The processing unit 10 adds or subtracts an evaluation value (step S512) according to the number and duration of times during the most recent predetermined period when a crew member's gaze was determined to be "downward" and the nature of their work was determined to be "other".

[0097] The processing unit 10 calculates a reference time for the duration of a specific crew member's behavior, which serves as the criterion for determining whether notification is necessary, according to the evaluation value calculated up to that point (step S513). For example, the evaluation value and the reference time (length relative to the duration) are inversely proportional; if the evaluation value is high and the risk of an accident is high, the reference time is calculated to be short, and if the evaluation value is low and the risk of an accident is low, the reference time is calculated to be long.

[0098] The processing unit 10 determines whether the calculated duration of a specific behavior is equal to or greater than the reference time (step S514). If it determines that the duration of the specific behavior is equal to or greater than the reference time (S514: YES), the processing unit 10 determines that notification is necessary (step S515) and terminates the process.

[0099] In step S514, if it is determined that the duration of a particular behavior is less than the reference time (S514: NO), the processing unit 10 determines that notification is unnecessary (step S516) and terminates the process.

[0100] If the processing unit 10 determines in step S502 that the ship is not underway (S502: NO), it can determine that the ship is at anchor (step S517). While at anchor, the processing unit 10 does not need to notify the crew on the bridge regarding the operation of the ship S, so it calculates the risk assessment value for accident occurrence to the lowest value (step S518), proceeds to step S513, and calculates the reference time to the maximum value (for example, the time until the scheduled departure time). While at anchor, notifications are automatically stopped until the ship gains speed and is determined to be underway.

[0101] In the processing procedure shown in the flowcharts of Figures 9 and 10, the processing unit 10 calculated an evaluation value by comparing it with a standard behavior according to the content of the work. However, the processing unit 10 is not limited to this, and may also use a learning model that has been trained to output a standard time for a given duration when it receives data on the navigation mode and the behavior during a predetermined time period as input.

[0102] The processing procedures shown in Figures 9 and 10 are not limited to these. Some processing steps may be omitted, and the addition values ​​may be adjusted according to the design. Some of the processing procedures shown in Figures 9 and 10 may be executed by the data server 4.

[0103] If the determination result by the necessity determination unit 105 shown in Figures 9 and 10 indicates that notification is required, the processing unit 10 will send an alert from the notification unit 14. For example, if it is determined that the vessel S is underway (S502: YES) and the evaluation value calculated in steps S503-S512 indicates a high safety risk, then in step S513, the determination criterion for duration is low, meaning the standard time is calculated to be shorter than the initial setting time. For example, if it is determined that the vessel is underway at night and the line of sight is directed towards the rear or downwards rather than straight ahead, starboard, or port, the determination criterion is low, meaning the standard time is calculated to be short. This makes it possible to increase the sense of urgency. Conversely, if it is determined that the vessel is underway during the day, visibility is good, and the line of sight is directed straight ahead, starboard, or port, the determination criterion is high, meaning the standard time is calculated to be long. During daytime navigation, if the gaze is directed downwards, the standard time is calculated to be shorter, and if the specific behavior of directing the gaze downwards continues for longer than the standard time, a notification is sent from the notification unit 14.

[0104] When the vessel S is at anchor, the reference time is calculated to be as long as the maximum value, as described above. Even if the maximum value is not set, it is preferable to set the system to determine that notification is unnecessary until the vessel gains speed and is judged to be underway. In this way, the notification device 1 of this disclosure processes data based on an analysis of the crew's behavior and an evaluation of the navigation data, making it possible to determine whether notification is necessary or not appropriately. This allows for appropriate and effective notification, while also providing a suitable level of tension for the crew.

[0105] The database in the storage unit 11 of the notification device 1 stores data on behavior, navigation data, and work content data identified at each point in time. Image data captured at each point in time may also be stored in the database. When the processing unit 10 becomes able to communicate with the data server 4 via the second communication unit 13, it sends the data stored in the database to the data server 4, increasing the capacity of the usable storage area of ​​the storage unit 11. The behavior data, navigation data, and work content data stored in the data server 4 are analyzed and processed, and together with the watch duty data of each crew member, it becomes possible to calculate an evaluation of their behavior during watch time.

[0106] In the first embodiment, as shown in Figures 9 and 10, an evaluation value was added or subtracted based on the behavior and navigation data identified for the crew member, according to the level of risk of an accident occurring, i.e., the likelihood of a situation requiring an alert, and a reference time was calculated according to the evaluation value. The calculation of the evaluation value is not limited to adding or subtracting from an initial value as described above; a learning model using a neural network that has been trained to output an evaluation value when behavior data (three-axis vectors of position and gaze direction) is input may also be used. Furthermore, in order to determine whether a crew member is stationary as one of their specific behaviors, detection may be performed using a motion sensor instead of determination by the behavior determination unit 102.

[0107] Compared to conventional configurations that issue alerts based on a fixed time during navigation when there is no operation or crew activity, the bridge watch management system 100 of the first embodiment can provide appropriate notifications according to the crew's behavior and the navigation conditions. By timing notifications appropriately, it is possible to maintain a moderate level of alertness among the crew.

[0108] [Differentiation] In a modified example, the necessity determination unit 105 determines that notification is necessary when the frequency at which the behavior determination unit 102 determines that the crew is stationary exceeds a standard frequency. The necessity determination unit 105 further changes the standard frequency (the criteria for determining necessity) based on the behavior determined by the behavior determination unit 102, the navigation data acquired by the navigation data acquisition unit 103, and the content of the work determined by the work determination unit 104, and appropriately implements notification according to the situation. The necessity determination unit 105 may also store the evaluation used as the criterion for determining the necessity of notification in the storage unit 11 for reference.

[0109] In the modified example, the necessity determination unit 105 raises the criteria for determining whether notification is necessary, i.e., increases the standard frequency, when it is determined that the crew member's line of sight is forward, making it easier to determine that notification is unnecessary. Regardless of the nature of the work, if the crew member's line of sight is determined to be forward, the necessity determination unit 105 may raise the criteria for determining whether notification is necessary, i.e., increases the standard frequency, making it easier to determine that notification is unnecessary, since appropriate behavior is being performed and the safety risk is low.

[0110] If the necessity determination unit 105 determines that the work is "monitoring," and if it determines that the crew member's gaze is directed downwards for an extended period, it may lower the criteria for determining whether notification is necessary, i.e., lower the frequency of notification, making it easier to determine that notification is necessary.

[0111] Furthermore, the necessity determination unit 105 lowers the criteria for determining whether notification is necessary, i.e., lowers the frequency of notification, even if the crew member's work content determined by the work determination unit 104 is "equipment (confirmation) operation," if the crew member's gaze is determined to be directed downwards, making it easier for the unit to determine that notification is necessary.

[0112] Figure 11 is a flowchart showing an example of the processing procedure by the notification device 1 in a modified example. The processing unit 10 of the notification device 1 continuously executes the following processes during startup. Of the processing procedures shown in Figure 11, those that are common to the processing procedures shown in Figure 8 are given the same step numbers and detailed explanations are omitted.

[0113] In the modified example, if the processing unit 10 of the notification device 1 determines in step S108 that a specific behavior is continuing (S108: YES), it calculates the frequency of that specific behavior in the most recent predetermined time (step S121). In the modified example, the processing unit 10 determines whether or not to issue a notification from the notification unit 14 based on the frequency of the specific behavior calculated in step S121 and the evaluation value for the crew member's behavior in the work determined in step S106 (step S122). Other processes are the same as the processing procedure shown in Figure 8.

[0114] Figures 12 and 13 are flowcharts illustrating an example of the process for determining the necessity of a modified example. The processing steps shown in the flowcharts of Figures 12 and 13 correspond to the functions of the modification necessity determination unit 105 and to the detailed processing steps of step S122 in Figure 11. For the processing steps shown in Figures 12 and 13 that are common with the processing steps shown in Figures 9 and 10, the same step numbers are used and detailed explanations are omitted.

[0115] In the modified example, in any mode, the processing unit 10 calculates an evaluation value and then calculates a reference frequency for the frequency of a specific crew member's behavior, which serves as the criterion for determining whether notification is necessary, according to the evaluation value calculated up to that point (step S541). For example, the evaluation value and the reference frequency are inversely proportional; if the evaluation value is high and the risk of an accident is high, the reference frequency is calculated to be low, and if the evaluation value is low and the risk of an accident is low, the reference frequency is calculated to be high.

[0116] For example, if a ship is sailing at night and its line of sight is determined to be facing backward or downward rather than straight ahead, starboard, or port, the threshold frequency used for detection will be calculated to be low. This increases the likelihood of an alert being issued even if the frequency is not very high, thus raising awareness. Conversely, if a ship is sailing during the day with good visibility and its line of sight is determined to be facing straight ahead, starboard, or port, the threshold frequency used for detection will be calculated to be high. Even when sailing during the same daytime, the likelihood of an alert being issued is lower when sailing in a less congested area compared to sailing in a more congested area around a port.

[0117] When ship S is at anchor, the risk assessment value for accident occurrence is set to the lowest value, so conversely, the baseline frequency is high, and for example, it may be calculated up to the maximum value. Even if the maximum value is not set, it would be good to set it so that notification is deemed unnecessary until ship speed is generated and it is determined that the ship is underway.

[0118] The processing unit 10 determines whether the calculated frequency of a specific behavior is equal to or greater than the reference frequency (step S542). If it determines that the frequency of the specific behavior is equal to or greater than the reference frequency (S542: YES), the processing unit 10 determines that notification is necessary (S515) and terminates the process.

[0119] In step S542, if it is determined that the frequency of a particular behavior is below the standard frequency (S542: NO), the processing unit 10 determines that notification is unnecessary (S516) and terminates the process.

[0120] Even in modified cases, the determination of whether or not notification is necessary is made appropriately based on frequency, analysis of crew behavior, and evaluation referencing navigation data. This allows for a balanced level of alertness among crew members, enabling appropriate and effective notification.

[0121] [Second Embodiment] The second embodiment of the bridge watch management system 100 is linked with an alarm system called BNWAS (Bridge Navigational Alarm System). Figure 14 is a schematic diagram of the second embodiment of the bridge watch management system 100. The configuration of the second embodiment of the bridge watch management system 200 is the same as that of the first embodiment of the bridge watch management system 100, except for the changes due to the linkage with the alarm system, so the same reference numerals are used for common components and detailed explanations are omitted.

[0122] In the second embodiment, the bridge watch management system 100 includes a camera 2 installed inside the bridge of the ship S, a notification device 1 that determines whether notification is necessary based on the image captured by the camera 2, a group of devices 3 installed on the ship S, an alarm device 5, and a data server 4 installed on land.

[0123] Alarm device 5 is a BNWAS (Bridge Navigational Alarm System). Alarm device 5 is a device that issues an alarm if a reset signal is not received from equipment installed in the bridge or if the device is not reset due to detection of crew movement during a waiting period, for example, 3 minutes. Alarm device 5 can communicate with notification device 1 via the ship's communication network SN or a signal line (not shown). Alarm device 5 can determine whether or not to issue an alarm by utilizing the reset signal output by notification device 1 of the second embodiment. Alarm device 5 may also refer to the waiting time or notification status output by notification device 1 and issue an alarm according to the information referred to.

[0124] Figure 15 is an explanatory diagram of the functions of the notification device 1 in the second embodiment. The processing unit 10 of the notification device 1 performs various functions shown in Figure 15 based on the information processing program P1. In the second embodiment, the processing unit 10, as a necessity determination unit 105, controls the output and stopping of the reset signal to the alarm device 5 according to the determination of whether notification is necessary or not. The processing unit 10, as a necessity determination unit 105, detects a specific behavior of a crew member using the behavior determination unit 102, and determines that notification is necessary if the duration of the specific behavior exceeds a standard time. The processing unit 10, as a necessity determination unit 105, stops the reset signal and causes the alarm device 5 to issue an alarm when it determines that notification is necessary, and outputs a reset signal when it determines that notification is unnecessary. The necessity determination unit 105 may output a standard time calculated from the behavior determined by the behavior determination unit 102 and the navigation data acquired by the navigation data acquisition unit 103 as a waiting time to the alarm device 5 for use by the alarm device 5.

[0125] Other functions of the processing unit 10 are the same as in the first embodiment or its modified form, so a detailed explanation will be omitted.

[0126] Figure 16 is a flowchart showing an example of the processing procedure by the notification device 1 of the second embodiment. The processing unit 10 of the notification device 1 continuously executes the following processes during startup. Of the processing procedures shown in Figure 16, those that are common with the processing procedures shown in Figure 8 of the first embodiment are given the same step numbers and detailed explanations are omitted.

[0127] In the second embodiment, if the processing unit 10 determines in step S110 that notification is required (S110: YES), it stops the reset signal from the notification unit 14 to the alarm device 5 (step S131) ​​and terminates the process. As a result, the alarm device 5 issues an alert.

[0128] In the second embodiment, if the processing unit 10 determines in step S108 that the crew member is not continuing a specific behavior (S108: NO), it determines that notification is unnecessary (S114), stores the time of the determination as the time when the crew member's movement was detected (S115), outputs a reset signal (step S132), and terminates the process. If the processing unit 10 determines in step S110 that notification is unnecessary (S110: NO), it outputs a reset signal (S132) and terminates the process.

[0129] The processing unit 10 may output the reference time calculated in the determination process of step 110 as the waiting time, along with whether or not notification is necessary, to the alarm device 5 for reference, using the necessity determination unit 105.

[0130] Alarm device 5 does not issue an alarm (alert) while a reset signal is output from notification device 1, and issues an alarm each time the waiting period elapses once the reset signal stops. The waiting period at this time may be the standard time output from notification device 1. Alarm device 5 does not issue an alarm while notification device 1 has determined that notification is unnecessary. By setting the timing of alarms issued by alarm device 5 to an appropriate timing, a moderate level of alertness among the crew can be maintained.

[0131] In the second embodiment, instead of determining whether the duration of a particular behavior is equal to or greater than the reference time, it may be determined whether the frequency of a particular behavior is equal to or greater than the reference frequency, as in the modified example.

[0132] In the second embodiment, the notification device 1 may be used as at least a part of the functions of the alarm device 5, or it may be integrated with the alarm device 5.

[0133] The embodiments disclosed above are illustrative in all respects and not restrictive. The scope of the invention is defined by the claims and includes all modifications in the meaning and scope equivalent to the claims.

[0134] Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. Moreover, while the claims use a multi-claim format in which claims refer to two or more other claims (multi-claim format), this is not the only option. A multi-claim format in which at least one multi-claim is referenced (multi-multi-claim format) may also be used.

[0135] The following additional information is disclosed regarding the embodiments described above.

[0136] (Note 1) A camera installed on the bridge of a ship to capture the crew inside the bridge, A behavior determination unit that determines the behavior of the crew members in the image based on the image acquired from the camera, An acquisition unit that acquires navigation data relating to the navigation of the vessel at the time corresponding to the image, A necessity determination unit determines whether or not to notify the crew member of the crew member's behavior based on the behavior determined by the behavior determination unit and the navigation data acquired by the acquisition unit, If it is determined that the aforementioned notification is necessary, the notification unit will notify the crew. A notification device equipped with the following features.

[0137] (Note 2) The behavior determination unit is, Given the aforementioned image, a trained model is used that is trained to output data on the orientation of the crew member's face. The direction of the sailor's gaze at the time corresponding to the image is determined by the face orientation data obtained by feeding the image acquired from the camera to the learning model. The notification device described in Appendix 1.

[0138] (Note 3) The behavior determination unit is, Given the aforementioned image, a head detection model trained to output the position of the crew member's head is used. In addition to the facial orientation data, the position of the face obtained by providing the image to the head detection model is used to determine the direction of the crew member's gaze and the crew member's position within the bridge at the time corresponding to the image. The notification device described in Appendix 2.

[0139] (Note 4) The acquisition unit acquires the ship's navigation position, the weather, wind speed and direction of the sea area in which the ship is navigating, the degree of congestion in the sea area, and the navigation status, whether or not the ship is navigating. An alarm device as described in any one of the appendices 1 to 3.

[0140] (Note 5) The necessity determination unit raises the criteria for determining whether notification is necessary if the navigation data acquired by the acquisition unit indicates that the vessel is at anchor. An alarm device as described in any one of the appendices 1 to 4.

[0141] (Note 6) The necessity determination unit lowers the criteria for determining whether notification is necessary if the navigation data acquired by the acquisition unit indicates that the vessel is underway and visibility is poor. An alarm device as described in any one of the appendices 1 to 5.

[0142] (Note 7) The necessity determination unit lowers the criteria for determining whether notification is necessary if the navigation data acquired by the acquisition unit indicates a high degree of congestion in the sea area where the vessel is navigating. An alarm device as described in any one of the appendices 1 to 6.

[0143] (Note 8) The necessity determination unit changes the criteria for determining whether notification is necessary based on the navigation position of the vessel and the presence or severity of past accident cases in the surrounding area, as indicated by the navigation data acquired by the acquisition unit. An alarm device as described in any one of the appendices 1 through 7.

[0144] (Note 9) If the aforementioned criteria are raised, the necessity determination unit increases the criterion time for determining that notification is required for the duration of the identified specific behavior of the crew member. A notification device as described in any one of the appendices 5 to 8.

[0145] (Note 10) If the aforementioned criteria are raised, the necessity determination unit raises the threshold frequency for determining whether notification is required for the frequency of the specific behavior of the identified crew member. A notification device as described in any one of the appendices 5 to 8.

[0146] (Note 11) A computer installed on the bridge of a ship, which acquires images from a camera that captures the crew inside the bridge, Based on the image acquired from the aforementioned camera, the behavior of the crew members shown in the image is determined. Navigation data relating to the navigation of the vessel at the time corresponding to the aforementioned image is obtained. Based on the identified behavior and the acquired navigation data, a determination is made as to whether or not to notify the crew member of the behavior. If it is determined that notification is necessary, the aforementioned crew members will be notified. Notification method.

[0147] (Note 12) A computer that acquires images from a camera installed on the bridge of a ship, which captures the crew inside the bridge, Based on the image acquired from the aforementioned camera, the behavior of the crew members shown in the image is determined. Navigation data relating to the navigation of the vessel at the time corresponding to the aforementioned image is obtained. Based on the identified behavior and the acquired navigation data, a determination is made as to whether or not to notify the crew member of the behavior. If it is determined that notification is necessary, the aforementioned crew members will be notified. A computer program that executes a process. [Explanation of symbols]

[0148] 100 Funabashi Duty Management System 1. Notification device 10 Processing Unit 101 Image acquisition unit 102 Behavior determination unit 103 Navigation Data Acquisition Unit 104 Business Discrimination Department 105 Reporting and Judgment Department M1 Learning Model M11 Head Detection Model M12 Head Direction Discrimination Model P1 Information Processing Program (Computer Program) 2 cameras 31 Navigation control system 32 GPS receivers 33 Anemometer 34 AIS communication device 35 Chart Display Systems 4 Alarm device 5. Data Server

Claims

1. A camera installed on the bridge of a ship to capture the crew inside the bridge, A behavior determination unit that determines the behavior of the crew members in the image based on the image acquired from the camera, An acquisition unit that acquires navigation data relating to the navigation of the vessel at the time corresponding to the image, A necessity determination unit determines whether or not to notify the crew member of the crew member's behavior based on the behavior determined by the behavior determination unit and the navigation data acquired by the acquisition unit, If it is determined that the aforementioned notification is necessary, the notification unit will notify the crew. A notification device equipped with the following features.

2. The behavior determination unit is, Given the aforementioned image, a trained model is used that is trained to output data on the orientation of the crew member's face. The direction of the sailor's gaze at the time corresponding to the image is determined by the face orientation data obtained by feeding the image acquired from the camera to the learning model. The notification device according to claim 1.

3. The behavior determination unit is, Given the aforementioned image, a head detection model trained to output the position of the crew member's head is used. In addition to the facial orientation data, the position of the face obtained by providing the image to the head detection model is used to determine the direction of the crew member's gaze and the crew member's position within the bridge at the time corresponding to the image. The notification device according to claim 2.

4. The acquisition unit acquires the ship's navigation position, the weather, wind speed and direction of the sea area in which the ship is navigating, the degree of congestion in the sea area, and the navigation status, whether or not the ship is navigating. The notification device according to any one of claims 1 to 3.

5. The necessity determination unit raises the criteria for determining whether notification is necessary if the navigation data acquired by the acquisition unit indicates that the vessel is at anchor. The notification device according to claim 1.

6. The necessity determination unit lowers the criteria for determining whether notification is necessary if the navigation data acquired by the acquisition unit indicates that the vessel is underway and visibility is poor. The notification device according to claim 1.

7. The necessity determination unit lowers the criteria for determining whether notification is necessary if the navigation data acquired by the acquisition unit indicates a high degree of congestion in the sea area where the vessel is navigating. The notification device according to claim 1.

8. The necessity determination unit changes the criteria for determining whether notification is necessary based on the navigation position of the vessel and the presence or severity of past accident cases in the surrounding area, as indicated by the navigation data acquired by the acquisition unit. The notification device according to claim 1.

9. If the aforementioned criteria are raised, the necessity determination unit increases the criterion time for determining that notification is required for the duration of the identified specific behavior of the crew member. The notification device according to any one of claims 5 to 8.

10. If the aforementioned criteria are raised, the necessity determination unit raises the threshold frequency for determining whether notification is required for the frequency of the specific behavior of the identified crew member. The notification device according to any one of claims 5 to 8.

11. A computer installed on the bridge of a ship, which acquires images from a camera that captures the crew inside the bridge, Based on the image acquired from the aforementioned camera, the behavior of the crew members shown in the image is determined. Navigation data relating to the navigation of the vessel at the time corresponding to the aforementioned image is obtained. Based on the identified behavior and the acquired navigation data, a determination is made as to whether or not to notify the crew member of the behavior. If it is determined that notification is necessary, the aforementioned crew members will be notified. Notification method.

12. A computer that acquires images from a camera installed on the bridge of a ship, which captures the crew inside the bridge, Based on the image acquired from the aforementioned camera, the behavior of the crew members shown in the image is determined. Navigation data relating to the navigation of the vessel at the time corresponding to the aforementioned image is obtained. Based on the identified behavior and the acquired navigation data, a determination is made as to whether or not to notify the crew member of the behavior. If it is determined that notification is necessary, the aforementioned crew members will be notified. A computer program that executes a process.

Images