Marine accident monitoring system and method for preventing marine passenger accident, and computer program therefor

Abstract

This marine accident monitoring system may comprise: a reception module configured to receive image information of a space including a ship; an evaluation module configured to store a machine learning-based detection model and apply the image information to the detection model, so as to calculate a risk level related to one or more areas partitioning the space corresponding to the image information or passengers in the space; and an output module configured to generate dashboard content by matching the risk level calculated by the evaluation module to the space, and display the dashboard content through a user interface on the marine accident monitoring system or a terminal device communicatively connected to the marine accident monitoring system.

Description

Marine accident monitoring system and method for preventing marine passenger accidents and computer program therefor

[0001] The embodiments relate to a marine accident monitoring system and method for preventing marine passenger accidents, and a computer program therefor. More specifically, the embodiments relate to a technology that utilizes machine learning technology to assess the risk of each area on a ship, such as an exposed deck, or the risk to passengers related to safety accidents, and to display the risk on a control system via a dashboard-style user interface, thereby preventing accidents in advance or enabling a rapid response to them.

[0002] With the quality of life improving compared to the past and the growing preference for personal leisure, marine leisure activities and marine tourism are experiencing a surge in popularity. In particular, after the COVID-19 pandemic, demand for marine recreation is expected to increase, leading to a surge in passenger ship demand. Furthermore, maritime passenger traffic is also expected to increase, driven by technological advancements that lead to larger vessels and the emergence of future-oriented vessels.

[0003] Meanwhile, maritime activities inevitably entail the possibility of safety accidents. In a broad sense, maritime accidents refer to (1) accidents resulting in death, disappearance, or injury related to the structure, equipment, or operation of a vessel; (2) accidents resulting in damage to a vessel, land-based facilities, or offshore facilities related to vessel operation; (3) accidents resulting in the loss, abandonment, or disappearance of a vessel; (4) accidents resulting in a vessel colliding, running aground, capsizing, sinking, or becoming unmaneuverable; and (5) accidents resulting in marine pollution damage related to vessel operation.

[0004] In particular, accidents related to human safety at sea (also referred to herein as "maritime safety accidents," "maritime passenger accidents," or "maritime accidents") can result in fatal harm to individuals. However, despite the efforts of relevant government agencies to promote maritime safety, the public's overall safety awareness remains low, and the maritime environment and the structural characteristics of passenger ships present challenges in preventing passengers from falling or jumping, as well as in rescue operations. Consequently, casualties continue to occur.

[0005] In Korea, the rate of 17.3% of maritime passenger accidents resulting in fatalities, such as falls or jumping, makes rescue impossible. This is a very high rate compared to other maritime accidents. In most cases, rescue is impossible due to the missed golden window immediately following the accident. Therefore, an intelligent process is needed to predict potential accident locations on passenger vessels and to identify areas where search for survivors is necessary immediately after an accident. However, such technology currently lacks any such technology.

[0006] According to one aspect of the present invention, a marine accident monitoring system and method, and a computer program therefor, can be provided that utilizes machine learning technology to evaluate the risk level of each area of ​​a ship, such as an exposed deck, or the risk level of passengers related to a safety accident, and display the risk level on a control system through a user interface in the form of a dashboard, thereby preventing accidents in advance or enabling a quick response to accidents.

[0007] A marine accident monitoring system according to one aspect of the present invention comprises: a receiving module configured to receive image information of a space including a vessel; an evaluation module configured to store a machine learning-based detection model and calculate a risk level related to one or more areas defining the space corresponding to the image information or passengers within the space by applying the image information to the detection model; and an output module configured to match the risk level calculated by the evaluation module to the space to generate dashboard content and display the dashboard content through a user interface on the marine accident monitoring system or a terminal device communicatively connected to the marine accident monitoring system.

[0008] In one embodiment, the output module is further configured to divide the space into a plurality of regions based on boundary structures within the ship or one or more passenger locations, and dynamically change one or more of whether to display the region, the display form of the region, whether to display image information corresponding to the region, and the display form of the image information corresponding to the region in the dashboard content based on the risk level calculated for the plurality of divided regions.

[0009] A marine accident monitoring system according to one embodiment further includes a database configured to store location information of boundary structures within a ship.

[0010] In one embodiment, the receiving module is further configured to receive operational status information of the vessel from a vessel control system. Furthermore, the evaluation module is further configured to calculate the risk level for the one or more areas using the distance to the boundary structure, the visibility of the image information, and the operational status information as input values ​​for the detection model.

[0011] In one embodiment, the evaluation module comprises a preprocessing unit configured to extract image data representing one or more preset behavioral patterns of the passenger from the image information; and a learning unit configured to calculate the risk level related to the passenger by using one or more of the duration of the behavioral pattern shown in the image data, an additional behavior following the behavioral pattern, and a movement of an area in which the behavioral pattern is detected as input values ​​to the detection model.

[0012] A method for monitoring marine accidents according to one aspect of the present invention comprises: a step in which a marine accident monitoring system storing a machine learning-based detection model receives image information of a space including a vessel; a step in which the marine accident monitoring system calculates a risk level related to one or more areas dividing the space corresponding to the image information or passengers within the space by applying the image information to the detection model; a step in which the marine accident monitoring system generates dashboard content by matching the calculated risk level to the space; and a step in which the marine accident monitoring system transmits the dashboard content for display through a user interface on the marine accident monitoring system or a terminal device communicatively connected to the marine accident monitoring system.

[0013] In one embodiment, the step of generating the dashboard content includes the step of the marine accident monitoring system dividing the space into a plurality of the zones based on boundary structures within the ship or the locations of one or more passengers; and the step of the marine accident monitoring system dynamically changing one or more of whether to display the zones, the display form of the zones, whether to display image information corresponding to the zones, and the display form of the image information corresponding to the zones in the dashboard content based on the risk level calculated for the plurality of the zones.

[0014] A marine accident monitoring method according to one embodiment further includes, before the step of calculating the risk level, a step in which the marine accident monitoring system stores location information of boundary structures within the ship in the marine accident monitoring system; and a step in which the marine accident monitoring system receives operating status information of the ship from a ship control system.

[0015] At this time, the step of calculating the risk level includes a step in which the marine accident monitoring system calculates the risk level for the one or more areas by using the distance to the boundary structure, the visibility of the image information, and the operation status information as input values ​​for the detection model.

[0016] In one embodiment, the step of calculating the risk level includes the step of the marine accident monitoring system extracting image data representing one or more preset behavioral patterns of the passenger from the image information; and the step of the marine accident monitoring system calculating the risk level related to the passenger by using one or more of the duration of the behavioral pattern shown in the image data, an additional behavior following the behavioral pattern, and a movement in an area where the behavioral pattern was detected as input values ​​to the detection model.

[0017] A computer program according to one aspect of the present invention is stored in a computer-readable recording medium so as to be combined with hardware and execute other marine accident monitoring methods according to the embodiments described above.

[0018] According to a marine accident monitoring system and method according to one aspect of the present invention, by evaluating the risk of one or more areas dividing a space within a ship and / or the risk related to passengers within such space and providing the results to a control system through a user interface in the form of a dashboard, the risk of an accident can be reduced by intelligently predicting a point where a marine passenger accident may occur or a point where a search for a victim is required immediately after an accident.

[0019] In addition, according to the marine accident monitoring system and method according to one aspect of the present invention, when a risk level exceeding a certain level is predicted or detected, a notification can be transmitted to a manager (e.g., captain, crew, helmsman, etc.), a relevant organization (e.g., coast guard, etc.) and / or a nearby vessel, and there is an advantage in that other people can be immediately made aware of a dangerous situation through a mixed notification using sound, screen, and / or mobile device.

[0020] Figure 1 is a conceptual diagram showing the operating environment of a marine accident monitoring system according to one embodiment.

[0021] Figure 2a is a schematic block diagram of a marine accident monitoring system according to one embodiment.

[0022] FIG. 2b is a schematic block diagram showing the hardware configuration of a marine accident monitoring system according to one embodiment.

[0023] Figure 3 is a flowchart showing each step of a marine accident monitoring method according to one embodiment.

[0024] Figure 4 is a plan view showing the location of the camera device on board the ship and the target area for calculating the risk level on the exposed deck as an example.

[0025] Figures 5a and 5b are conceptual diagrams showing the risk level calculated for each area within a ship by a marine accident monitoring method according to one embodiment.

[0026] Figures 6a and 6b are further conceptual diagrams showing the risk level calculated for each area within a ship by a marine accident monitoring method according to one embodiment.

[0027] FIG. 7 is a conceptual diagram illustrating an exemplary user interface of a marine accident monitoring system according to one embodiment.

[0028] FIG. 8 is a conceptual diagram illustrating an exemplary user interface of a marine accident monitoring system according to another embodiment.

[0029] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0030] Figure 1 is a conceptual diagram showing the operating environment of a marine accident monitoring system according to one embodiment.

[0031] Referring to FIG. 1, a marine accident monitoring system (2) according to embodiments is operated to receive image information captured from the interior and exterior spaces of a ship from one or more photographing devices (100) installed on a passenger ship, cargo ship, or any other type of ship (1). For example, the photographing devices (100) may be CCTVs installed in any location, such as a wheelhouse, cabin, or deck of a passenger ship. In addition, in one embodiment, one or more photographing devices (100) may be installed on an exposed deck of the ship (1). However, the location of the photographing devices (100) is not limited thereto.

[0032] The marine accident monitoring system (2) stores a detection model based on machine learning, and by inputting feature values ​​extracted from image information into the detection model, the risk level related to each area within the ship (1) or passengers on board the ship (1) included in the image information can be calculated.

[0033] Here, machine learning in this specification refers to all arbitrary processing methods that, in addition to machine learning in the traditional sense, include deep learning and the like, to classify and analyze data and apply what has been learned to make decisions based on information learned through the data.

[0034] In one embodiment, in addition to image information, information received by the marine accident monitoring system (2) from other external devices or other information previously stored in the marine accident monitoring system (2) may be used to calculate the risk level related to an area or passengers within the ship (1).

[0035] For example, the marine accident monitoring system (2) can store the location information of one or more photographing devices (100) installed on the ship (1) and the location information of boundary structures that serve as reference points for determining whether an accident has occurred, such as a railing installed on the deck of the ship (1). In addition, the marine accident monitoring system (2) can receive image information from the photographing device (100) and receive information on the operation status of the ship (1) from the control system (110) of the ship (1). The marine accident monitoring system (2) can calculate the risk level related to each area or passenger within the ship (1) by using a combination of information stored or received in the marine accident monitoring system (2) as an input value for a machine learning-based detection model.

[0036] Next, the marine accident monitoring system (2) can configure content and a user interface (UI) to provide the calculated risk level through content in the form of a dashboard displayed on a user terminal. For example, the dashboard content may display the calculated risk level for an area or passenger within the vessel (1) by matching it to the corresponding location. The dashboard content may be provided from the marine accident monitoring system (2) to the control system (110) and displayed to the user, or may be displayed to the user through a UI on any other terminal device communicatively connected to the marine accident monitoring system (2). Alternatively, in another embodiment, the marine accident monitoring system (2) itself may be implemented as a terminal device used by the user.

[0037] In one embodiment, the marine accident monitoring system (2) outputs notification information in the form of video and / or sound when the calculated risk level for an area or passengers within the ship (1) is higher than a preset threshold (in this specification, this is also referred to as first notification information), or / and may transmit notification information on the risk of marine passenger accidents to the control system (110) of the ship (1), a server (3) of a relevant organization such as the coast guard, and / or a ship (4) located nearby (in this specification, this is also referred to as second notification information).

[0038] For the above operation, the marine accident monitoring system (2) can be communicatively connected to one or more photographing devices (100), a control system (110), a related organization server (3), and / or a surrounding vessel (4) via a communication method via a wired and / or wireless network. In the present specification, the communication method via a wired and / or wireless network can be implemented using any communication method that allows objects to network with each other, and is not limited to wired communication, wireless communication, 3G, 4G, 5G communication, or other methods.

[0039] Figure 2a is a schematic block diagram of a marine accident monitoring system according to one embodiment.

[0040] Referring to FIG. 2A, in one embodiment, the marine accident monitoring system (2) includes a receiving module (21), an evaluation module (24), and an output module (22). In one embodiment, the marine accident monitoring system (2) further includes a database (DB) (23). Furthermore, in one embodiment, the evaluation module (24) includes a preprocessing unit (241) and a learning unit (242). Furthermore, each of these functional units of the marine accident monitoring system (2) can be implemented at least partially using the hardware (200) of the marine accident monitoring system (2).

[0041] Systems, devices, and servers according to the embodiments may be entirely hardware, or may have aspects that are partially hardware and partially software. For example, the systems, devices, or servers described herein and each unit included therein may collectively refer to hardware and related software for processing data of a specific format and content and / or exchanging data electronically. As used herein, terms such as "unit," "module," "device," "terminal," "server," or "system" are intended to refer to a combination of hardware and software driven by the hardware. For example, hardware may be a data processing device including a CPU or other processor. In addition, software driven by hardware may refer to a running process, object, executable, thread of execution, program, etc.

[0042] Furthermore, each element constituting the marine accident monitoring system (2) is not necessarily intended to refer to a physically distinct and separate device. That is, each functional unit of the marine accident monitoring system (2) illustrated in Fig. 2a is merely functionally distinguished based on the operations performed by the hardware constituting the marine accident monitoring system (2) and / or the software implemented thereby, and each component does not necessarily need to be provided independently of one another. Of course, depending on the embodiment, one or more of the aforementioned functional units may be implemented as a physically distinct and separate device.

[0043] Referring to FIGS. 1 and 2A, the receiving module (21) is configured to receive image information of a space within a ship (1) from one or more photographing devices (100). In addition, the receiving module (21) may also receive ship operation status information from the ship's control system (110). At this time, the operation status information may include information indicating whether the ship (1) is at anchor or in operation, and / or time information indicating when the ship (1) is in operation. At this time, the time information may be defined as a predetermined section (e.g., day / night, etc.) to which the current time belongs.

[0044] DB (23) can store location information of one or more boundary structures located on the ship (1), such as a deck railing of the ship (1). In addition, DB (23) can store location information of one or more photographing devices (100), such as CCTVs, placed on the ship (1).

[0045] The evaluation module (24) is configured to calculate the risk level for each area that divides the space within the ship (1) or for passengers (which may include crew members) on board the ship (1) based on the information received by the receiving module (21) and / or the information previously stored in the DB (23). The evaluation module (24) can calculate the risk level by using the feature values ​​extracted from the information received by the receiving module (21) and / or the information previously stored in the DB (23) as input values ​​for a machine learning-based detection model.

[0046] In one embodiment, the evaluation module (24) can set up a partition that divides the space within the ship (1) into a plurality of areas, and calculate the risk level for each area based on the information received by the receiving module (21) and the information stored in the DB (23) for each divided area. The evaluation module (24) can calculate the risk level for each area by inputting a first value determined based on the distance between each area within the ship (1) and a boundary structure, a second value determined based on the visibility for each area in the image information received by the receiving module (21), and a third value determined based on the operating status information of the ship (1) as feature values ​​into a machine learning-based detection model.

[0047] In another embodiment, the evaluation module (24) may recognize passenger behavior patterns from the video information received by the receiving module (21) and use a learned machine learning-based detection model to classify passenger behaviors to determine risk levels based on passenger behavior patterns within the video information. For example, the marine accident monitoring system (2) may classify passenger behaviors within the video as normal or abnormal behavior, and determine a high risk level if abnormal behavior is detected.

[0048] To this end, the marine accident monitoring system (2) may receive a learning data set containing image data representing normal and abnormal behaviors, respectively. The learning data set may refer to data in which image data by type is labeled according to pre-established classification rules based on accident cases. This learning data set is intended to be used for the creation and training of a machine learning-based detection model of the marine accident monitoring system (2).

[0049] The preprocessing unit (241) of the evaluation module (24) can extract image data representing one or more preset behavioral patterns of passengers from the image information received by the receiving module (21). In addition, the learning unit of the evaluation module (24) can determine the risk level related to passengers by using the duration of the behavioral pattern shown in the image data, additional actions following the behavioral pattern, and / or movement of the area where the behavioral pattern was detected within the vessel (1) as input values ​​for the detection model. This will be described in detail later with reference to FIG. 3.

[0050] The output module (22) can generate dashboard content that visually represents the risk level of a space within the ship (1) by matching the risk level for an area or passenger calculated by the evaluation module (24) to the space within the ship (1). In addition, the output module (22) can output the generated dashboard content on a display means (not shown) of the marine accident monitoring system (2), or transmit it for display through a UI on a terminal device (e.g., a control system (110)) that is communicatively connected to the marine accident monitoring system (2).

[0051] In one embodiment, the dashboard content provided by the output module (22) may divide the space within the ship (1) into a plurality of areas based on the location of boundary structures, such as railings, within the ship (1) and / or the locations of passengers on board the ship (1), and dynamically change the display format of the content based on the calculated risk level for each divided area. For example, based on the calculated risk level for each area, the display or format (e.g., size, color, etc.) of the area on the dashboard content may be changed, or the display or format (e.g., size, color, etc.) of image information (e.g., CCTV image) corresponding to the area may be changed. This will be described in detail later with reference to FIGS. 7 and 8.

[0052] In one embodiment, if, as a result of calculating the risk level through the evaluation module (24), there is an area within the ship (1) whose risk level is higher than a preset threshold or a risk level of passengers is detected, the output module (22) can output the result of calculating the risk level as first notification information through a sound and / or video output device (not shown) equipped on the ship (1).

[0053] Alternatively, the output module (22) may transmit the risk calculation result as second notification information to one or more servers or terminal devices associated with the operation of the ship (1). In this case, the server or terminal device associated with the operation of the ship (1) may include, but is not limited to, a control system (110) which is an operation system of the ship (1) accessible from the wheelhouse of the ship (1), a server (3) of a related organization such as the Coast Guard, a nearby ship (4), and / or a mobile device (not shown) of a manager (e.g., captain, crew member, helmsman, etc.) associated with the operation of the ship (1).

[0054] In one embodiment, the output module (22) can automatically release the buoy device (130) onto the water by transmitting a control command to the control system (110) of the ship (1) to release the buoy device (130) in the direction of the corresponding area from the ship (1) when the risk calculated by the evaluation module (24) is above a certain level. The buoy device (130) is equipped with a location transmission function based on a Global Positioning System (GPS) and is a device configured to float and remain on the water, and is intended to provide priority assistance until rescue personnel arrive in the event of a passenger's fall or suicide.

[0055] Furthermore, in one embodiment, the output module (22) may further transmit the identification information of the released buoy device (130) to one or more external servers or terminal devices, such as a related organization server (3) or a nearby vessel (1), thereby enabling the related organization or other nearby vessels to accurately recognize the location where a dangerous situation has occurred through the location information of the buoy (130).

[0056] FIG. 2b is a schematic block diagram showing the hardware configuration of a marine accident monitoring system according to one embodiment.

[0057] Referring to FIG. 2b, the marine accident monitoring system according to the embodiments is implemented in the form of a computing device including hardware (200), and may include a memory (210), a processor (220), a communication module (230), and an input / output unit (240).

[0058] The memory (210) is a non-transitory computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. Here, the non-permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the above-described device or server as a separate permanent storage device distinct from the memory (210).

[0059] Additionally, the memory (210) may store an operating system and at least one program code (e.g., code for a security module or an application installed to provide a specific service). These software components may be loaded from a computer-readable recording medium separate from the memory (210). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card.

[0060] In another embodiment, the software components may be loaded into the memory (210) via a communication module (230) rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory (210) based on a computer program that is installed by files provided over a network by developers or a file distribution system (e.g., an application store service server) that distributes installation files for applications.

[0061] The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (220) by the memory (210) or the communication module (230). For example, the processor (220) may be configured to execute instructions received according to program code stored in a storage device such as the memory (210).

[0062] The communication module (230) may provide a function for the marine accident monitoring system to communicate with the photographing device (100), the control system (110), the relevant organization server (3), and / or the surrounding vessels (4) via a network. In addition, the communication module (230) may provide a function for the marine accident monitoring system to communicate with one or more other devices via a wired and / or wireless network. That is, the communication module (230) is a part that realizes each functional module described above with reference to FIG. 1 by having its function controlled by the processor (220) referencing the memory (210).

[0063] The input / output unit (240) may be a means for interfacing with an external input / output device (not shown). For example, external input devices may include devices such as a keyboard, mouse, microphone, camera, etc., and external output devices may include devices such as a display, speaker, haptic feedback device, etc. As another example, the input / output unit (240) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen.

[0064] In addition, in other embodiments, the marine accident monitoring system may include more hardware components than those illustrated in FIG. 2B, depending on the nature of the device to which it is applied. For example, when the marine accident monitoring system is applied to a terminal device used by a user, it may be implemented to include at least some of the above-described input / output devices, or may further include other components such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, a database, etc. As a more specific example, when the terminal device is a smartphone, it may be implemented to further include various components that are generally included in a smartphone, such as an acceleration sensor or a gyro sensor, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration.

[0065] The marine accident monitoring method described below can be performed by a marine accident monitoring system implemented in the form of a computing device including the configuration described above with reference to FIGS. 2A and 2B. For example, the marine accident monitoring method can be provided to a user in the form of a service based on at least one of an application, software, or other program operating on a user device and / or a server.

[0066] Figure 3 is a flowchart illustrating each step of a marine accident monitoring method according to one embodiment. For convenience of explanation, the marine accident monitoring method according to this embodiment will be described below with reference to Figures 1 and 3.

[0067] In one embodiment, the location information of boundary structures of a vessel (1) and the location information of one or more photographing devices (100) within the vessel (1) may be stored in the DB (23) of the marine accident monitoring system (2) (S1). In this specification, boundary structures are described using a railing on an exposed deck as an example, but the type of boundary structure is not limited thereto, and any structure that serves as a standard for determining that there is a high risk of a marine safety accident, such as a fall or jumping, when a passenger is located beyond the location of the structure may be considered a boundary structure.

[0068] Next, the receiving module (21) of the marine accident monitoring system (2) can receive image information from one or more photographing devices (100) installed on the ship (1). In addition, the receiving module (21) can receive information on the operational status of the ship (1) from the control system (110) of the ship (1) (S2). This can be performed through communication between the marine accident monitoring system (2) and the control system (110), or in another embodiment, the marine accident monitoring system (2) itself can be implemented as a part of the control system (110).

[0069] In one embodiment, the marine accident monitoring system (2) can divide the space within the vessel (1) into multiple zones based on the distance from the boundary structure (S3). Expert research and statistics indicate that the risk of marine safety accidents varies depending on the distance from the railing. In this specification, the distance from each point within the vessel (1) to the boundary structure is also referred to as the "boundary distance."

[0070] For example, when the height of the railing is 95 cm, the risk of an accident varies depending on which section the distance to the railing falls into: (i) less than 60 cm (a person can stand and touch the railing and step over it), (ii) 60 cm or more and less than 1 m (a person can lean over and touch the railing and put one foot on it), (iii) 1 m or more and less than 1 m 10 cm (a person can touch the railing if they lean over completely, but it is difficult to step over it), or (iv) 1 m 10 cm or more and less than 1 m 20 cm (a person cannot touch the railing and is difficult to step over it even if they lean over it).

[0071] Therefore, in this embodiment, the evaluation module (24) of the marine accident monitoring system (2) can divide the space within the ship (1) into multiple areas depending on which of the above-mentioned sections (i) to (iv) the distance to the railing of the ship (1) belongs to. In addition, in one embodiment, the evaluation module (24) can also set each area within the ship (1) by further reflecting whether an image of the area can be viewed by the photographing device (100) installed on the ship (1).

[0072] However, in other embodiments, when the risk is determined only based on the behavior of passengers and not on the structure, the step (S1) of storing the aforementioned structure information, etc. in the DB (23) of the marine accident monitoring system (2) and the step (S3) of dividing the space within the ship (1) based on the same may be omitted.

[0073] Next, the evaluation module (24) of the marine accident monitoring system (2) can calculate a risk value corresponding to an area or passenger based on the characteristics of each area within the ship (1) or the behavioral patterns of passengers within the image information (S4). In the following description, determining the risk based on each area that divides the space within the ship (1) is referred to as the first embodiment, and determining the risk based on the behavioral patterns of passengers is also referred to as the second embodiment.

[0074]

[0075] Example 1

[0076] In this embodiment, the evaluation module (24) determines a risk value according to each of the distance from each point to the boundary structure (boundary distance), whether or not the point is visible from the photographing device (100) or the degree to which it is visible (visibility), and the operating status of the ship (1) (e.g., anchoring / operating, day / night), and can use the three risk values ​​described above as input values ​​for a machine learning-based detection model to calculate a risk value for each area within the ship (1).

[0077] For example, the feature value matrix input to the detection model produced by the present embodiment can be expressed as shown in the following mathematical expression 1.

[0078] [Mathematical Formula 1]

[0079] Feature value = [(d×w d ), (c×w c ), (s×w s )]

[0080]

[0081] In the above mathematical expression 1, d represents the first risk value (accident probability) according to the boundary distance, c represents the second risk value (accident probability) according to the visibility in the image information, and s represents the third risk value (accident severity) according to the ship's operating status. In addition, w d , w c and w s represents the weight applied to each risk value.

[0082] In one embodiment, the first risk value may be determined as shown in Table 1 below, depending on the interval to which the boundary distance belongs.

[0083] Boundary distance (m) 1st risk value Over 1.0 3.00 Over 0.6 Less than 1.0 3.73 Less than 0.6 4.27

[0084] Also, in one embodiment, the second risk value may be determined as shown in Table 2 below depending on whether the corresponding area is within or outside the visible range of the image information. In Table 2 below, being outside the visible range is intended to include not only the area being located far from the photographing device (100) and thus invisible, but also the area becoming invisible in the image information due to any cause such as movement in the direction of the photographing device (100)'s gaze or a foreign substance adhering to the photographing device (100).

[0085] Visibility 2 Risk Value Within Visibility Range 2.93 Outside Visibility Range 4.27

[0086] Additionally, in one embodiment, the third risk value may be determined according to the operating status of the vessel as shown in Table 3 below.

[0087] Operational Status 3rd Risk Value: Operating / Daytime 4.27 Operating / Nighttime 4.93 Anchored / Daytime 3.40 Anchored / Night 4.27

[0088] Furthermore, in one embodiment, the weight for each risk value can be determined as shown in Table 4 below.

[0089] Value weighting: 1st risk value 0.29, 2nd risk value 0.23, 3rd risk value 0.48

[0090] By applying the weights described in Table 4 to the first to third risk values ​​determined by each numerical interval described in Tables 1 to 3 above, a feature value matrix to be input into the detection model for risk determination can be determined in the form of mathematical expression 1.

[0091] Each risk value and its weighting value described in Tables 1 to 4 above were determined by averaging the risk values ​​assigned by experts for each situation through multiple expert interviews conducted by the inventors of the present invention. However, the numerical range and weighting values ​​for determining the risk value in the marine accident monitoring system (2) according to the embodiments are not limited to the examples described in this specification, and may be appropriately set depending on the purpose for which the marine accident monitoring system (2) according to the embodiments is applied.

[0092] For example, the upper and lower limits of the interval of the boundary distance for determining the first risk value may be appropriately determined as 60 cm, 1 m, 1 m 10 cm, 1 m 20 cm, or other numerical values ​​not described herein. As another example, the visibility for determining the second risk value is determined in Table 2 based on whether the corresponding area in the image information is within the visible range (i.e., whether the corresponding area is visible in the image information) or outside the visible range (i.e., whether the corresponding area is not visible in the image information), but in other embodiments, the second risk value may be determined by visibility defined in another way, such as how often the corresponding point in the image information becomes invisible.

[0093] Fig. 4 is a plan view exemplarily showing the locations of onboard camera devices and the target areas for calculating risk levels within an exposed deck. Fig. 4 shows exemplary locations of one or more camera devices (101 to 103) installed on the deck area of ​​a ship (1). In this specification, risk calculation through a risk area evaluation method according to embodiments will be described, taking as an example the exposed deck space (A) within the shooting range of the camera device (102).

[0094] Figures 5a and 5b are conceptual diagrams showing the risk level calculated for each area by a marine accident monitoring method according to one embodiment for the exposed deck space (A) of Figure 4.

[0095] Referring to FIGS. 5a and 5b, in the risk area evaluation method according to the embodiments, the width (W1, W2) of each area (500) can be defined according to the distance from the longitudinal (ship length direction) railing of the ship, and the length (L1, L2) of each area (500) can be defined according to the distance from the transverse (ship length direction) railing of the ship.

[0096] In the ship on which the inventors conducted the research, the exposed deck area (A) has a length of 19.4 m and a width of 15.6 m, and when the distance from the railing, which is a boundary structure, is classified into three sections of more than 1 m, more than 60 cm, less than 1 m, and less than 60 cm, the exposed deck area (A) can be divided into a plurality of sections (500) as illustrated in FIG. 5a.

[0097] In addition, the virtual lines (501, 502) in FIGS. 5a and 5b are intended to indicate the visible range of the photographing device (102), and the spaces (510, 520) located behind the lines (501, 502) with respect to the photographing device (102) cannot be imaged by the photographing device (102). Therefore, in calculating the risk value, the areas located in the corresponding spaces (510, 520) must be classified as being outside the visible range.

[0098] By applying the risk calculation process described above with reference to the above area division criteria and Tables 1 to 4, the final risk for each area (500) shown in FIG. 5a can be calculated as shown in Table 5 below.

[0099] Boundary distance visibility During operation / day During operation / night During anchorage / day During anchorage / night Over 1m Within visibility range 2.5 2.9 2.0 2.5 Over 0.6m Less than 1m Within visibility range 3.1 3.6 2.5 Less than 3.10 6m Within visibility range 3.6 4.12 8 3.6 Over 1m Outside visibility range 3.6 4.2 2.9 3.6 Over 0.6m Less than 1m Outside visibility range 4.5 5.2 3.6 4.5 Less than 0.6m Outside visibility range 5.2 6.0 4.15.2

[0100] The numbers displayed in each area (500) in FIGS. 5a and 5b represent the final risk level of the area (500), and FIG. 5a represents the risk level when the operating status is [in operation / daytime], while FIG. 5b represents the risk level when the operating status is [in operation / daytime].

[0101] In addition, FIGS. 6a and 6b are further conceptual diagrams showing the risk calculated for each area by a marine accident monitoring method according to an embodiment for the exposed deck space (A) of FIG. 4. FIG. 6a shows the risk when the operating status is [At anchor / daytime], and FIG. 6b shows the risk when the operating status is [At anchor / daytime].

[0102]

[0103] Second Example

[0104] Referring again to FIG. 3, in the second embodiment of the present invention, the evaluation module (24) may determine the risk level corresponding to the passenger (or corresponding to the area in which the passenger is located) based on the result of detecting the behavioral pattern of the passenger located within the ship (1) from the image information instead of the characteristics of each area within the ship (1) (S4).

[0105] More specifically, the preprocessing unit (241) of the evaluation module (24) can detect an object corresponding to a passenger in each image data (e.g., each frame) constituting the image information, thereby determining a bounding box, which is a rectangular frame corresponding to the edge of the object, and annotate the image data with this.

[0106] Next, the preprocessing unit (241) may perform keypoint annotation to add interpretation information about the shape taken by one or more feature points of a pre-set object in order to recognize the behavior of the object within the bounding box. The feature points for behavior recognition may be, for example, joints or extremities of the body, such as a person's nose, lower part of the head, upper part of the head, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right pelvises, left and right knees, and left and right ankles, but the types of feature points are not limited thereto. In addition, the preprocessing unit (241) may perform polygon annotation to indicate pixels corresponding to passengers in the image data in the form of polygons.

[0107] At this time, the risk level can be determined by using as input the duration of the pattern, the additional actions that follow, and / or the movement of the area in which the action is performed based on normal behavior patterns.

[0108] For example, Table 6 below provides examples of behavioral patterns that could be classified as abnormal behavior based on pre-constructed accident cases, and that could be considered precursors to suicide at sea.

[0109] Behavior types above the order of occurrence1st behavior pattern2nd behavior pattern definition1WalkingWalk (1st direction) -> Gaze (2nd direction) -> Gaze (3rd direction) / Repeat a certain number of timesGaze or look around2Stay for a long timeWaitWait -> Move to an area (outside some railings) -> Last for a certain timeLie downLie down -> Last for a certain time3Lean down and then sitWaitStand, lean -> Sit (in the same location)StandingSit4Smoking and wanderingSmoking -> Lean, use a device -> Last for a certain time -> WalkWaitUse a deviceWalk5Gaze for a long timeGaze -> Last for a certain time -> Use a device -> Last for a certain timeUse a device6Organize itemsGaze -> Take off shoes -> Stand -> GazeTake off shoesStand7Cross the railingWaitWait -> Move to an area (outside the railing)

[0110] As shown in Table 6 above, based on general behavior patterns (first behavior patterns) such as walking, standing, leaning, and gazing, a second behavior pattern corresponding to an abnormal behavior can be defined based on the duration of the general behavior pattern or the additional behavior that follows it and the area in which the behavior occurs. The definition of the second behavior pattern shown in Table 1 above is merely exemplary, and the combination of the first behavior patterns that constitute the second behavior pattern and its duration and area can be set in various ways depending on the embodiment.

[0111] In addition, the preprocessing unit (241) detects movement in the area where the passenger's behavior pattern is detected, for example, whether the area where the passenger's behavior pattern is detected moves toward the boundary structure, through a change in the position of the boundary box, and if this behavior pattern continues for a predetermined period of time or longer, it can further include this in the feature values ​​input to the detection model to classify the risk.

[0112]

[0113] When the risk level according to the characteristics of each area or passenger behavior pattern within the ship (1) is calculated through a detection model, the output module (22) of the marine accident monitoring system (2) can configure dashboard content to display the calculated risk level by matching it to a space section within the ship (1) (S5).

[0114] For example, the dashboard content may be dynamic content in which the display and / or display form (e.g., size, color, etc.) of at least some areas of the dashboard content changes in real time according to the risk level determined in real time for each area or passenger within the ship (1), and the specific form of the dashboard content is described in detail later with reference to FIGS. 7 and 8.

[0115] Next, the output module (22) can provide the dashboard content configured in real time to be displayed on the UI of the marine accident monitoring system (2) or a terminal device (e.g., control system (110)) that is communicatively connected to the marine accident monitoring system (2) in order to display the content to management personnel operating the ship (1), such as the captain, crew, and helmsman (S6).

[0116] In one embodiment, the output module (22) of the marine accident monitoring system (2) can generate a notification when the calculated risk for a specific area or specific passenger within the ship (1) is higher than a preset threshold based on the risk calculation result (S7). For example, the output module (22) can generate first notification information to provide an alarm in the form of video and / or audio to management personnel operating the ship (1), such as the captain, crew, and helmsman, and transmit the first notification information to one or more output devices (not shown), such as a monitor or speaker, installed on the ship (1) for outputting the first notification information. For example, the output device for outputting the first notification information may refer to a device equipped in the control system (110) corresponding to the operation system of the ship (1), but is not limited thereto.

[0117] In addition, the output module (22) may transmit a notification indicating that the risk level is above a threshold value as second notification information to one or more servers or terminal devices. At this time, the server or terminal device receiving the second notification information may be the control system (110) of the ship (1) or a mobile device such as a smartphone of management personnel operating the ship (1), such as the captain, crew, or helmsman. In addition, the server or terminal device receiving the second notification information may be a server (3) of a related organization such as the Coast Guard, or a control system of one or more other ships (4) located geographically adjacent to the ship (1). Furthermore, the second notification information may include information on the type of abnormal behavior detected.

[0118] In the past, even when an alarm system was established to detect passenger falls or suicides, there was a problem that it could be difficult to recognize alarms because management personnel who understand the ship's operational structure did not constantly check the alarms. On the other hand, in the marine accident monitoring method according to embodiments of the present invention, the level of risk can be easily recognized through dashboard content that changes in real time. In addition, in the marine accident monitoring method according to embodiments, notifications are transmitted in a mixed manner using multiple methods among sound and / or screen output using an output device, notifications to relevant organizations or nearby vessels, and notifications sent to the mobile devices of each management personnel (text messages, push notifications via applications, etc.), so that the relevant personnel can immediately recognize dangerous situations.

[0119] In one embodiment, the output module (22) of the marine accident monitoring system (2) may transmit a control command to the control system (110) to release a buoy device (130) toward an area where a high risk level is calculated from the vessel (1). This control command may be automatically executed by the control system (110) or upon confirmation by the crew, thereby automatically executing rapid life-saving measures when necessary.

[0120] In addition, in one embodiment, the output module (22) may transmit identification information about the released buoy device (130) to the relevant authorities and / or surrounding vessels (4). The buoy device (130) has its own GPS function and can transmit location information, but the relevant authorities or surrounding vessels (4) cannot identify which buoy device (130) was released in relation to which crisis situation. In this case, the output module (22) transmits identification information about the automatically or manually released buoy device (130) to the relevant authorities server (3) and / or surrounding vessels (4), thereby allowing personnel of the relevant authorities or surrounding vessels to identify the location of a safety accident through the GPS signal of the identified buoy device (130) and provide assistance.

[0121] Figure 7 is a conceptual diagram showing an exemplary UI of a marine accident monitoring system according to one embodiment.

[0122] FIG. 7 illustrates an exemplary form in which a marine accident monitoring system according to one embodiment provides dashboard content to be displayed on a screen of a ship's control system (110). The dashboard content includes a UI element (700) that divides the space within the ship into one or more areas (701 to 703) and displays them. Whether each area (701 to 703) is displayed within the UI element (700) and / or the display form (e.g., color change, etc.) of each area (701 to 703) can be used to display characteristics (e.g., distance from boundary structures, visibility, etc.) of the area (701 to 703) or the risk level due to the behavioral pattern of passengers located in the area (701 to 703) on the dashboard content.

[0123] In addition, in one embodiment, the dashboard content may display information necessary for safety control of the vessel, such as the operational status of one or more Search and Rescue Radar Transponders (SART) installed on the vessel, location information of one or more CCTVs deployed on the vessel, location information of boundary structures, operation status information of the vessel, release information of buoy devices installed on the vessel, or GPS information, in the form of UI elements (710, 720). The information displayed through the UI elements (710, 720) may be changed in real time to information corresponding to a specific area (701 to 703) of the vessel when the user selects the area.

[0124] Figure 8 is a conceptual diagram showing an exemplary UI of a marine accident monitoring system according to another embodiment.

[0125] FIG. 8 illustrates an exemplary form in which image information (e.g., CCTV image) captured by a capturing device is included in the dashboard content provided to a terminal device (120) by a marine accident monitoring system according to one embodiment. In this case, the dashboard content includes a UI element (800) that displays image information (801, 802) corresponding to one or more areas of a vessel, and whether or not the image information (801, 802) is displayed or the display form (e.g., size, blinking, etc.) may be dynamically changed depending on the risk level determined for each area.

[0126] In addition, in one embodiment, the dashboard content may be configured to further display, through UI elements (810, 820), the detection results of the behavioral patterns of passengers on board the ship, or the change history of the risk value for an area or passenger on board the ship. Furthermore, in one embodiment, the terminal device (120) further includes an output device (140) configured to display alarm information in the form of light and / or sound, so that the marine accident monitoring system may further transmit a control command to the terminal device (120) to output alarm information through the output device (140) when the calculated risk level for a specific area or specific passenger on board the ship is higher than a preset threshold value, in addition to the dashboard content.

[0127] The operations of the marine accident monitoring method according to the embodiments described above can be implemented at least partially as a computer program and recorded on a computer-readable recording medium. The computer-readable recording medium on which the program for implementing the operations of the method according to the embodiments is recorded includes all types of recording devices that store data that can be read by a computer. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium can be distributed across network-connected computer systems, so that the computer-readable code can be stored and executed in a distributed manner. In addition, the functional programs, codes, and code segments for implementing the present embodiment will be readily understood by those skilled in the art to which the present embodiment pertains.

[0128] While the present invention has been described above with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will appreciate that various modifications and variations of the embodiments are possible. However, such modifications should be considered within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

[0129] The embodiments relate to a marine accident monitoring system and method for preventing marine passenger accidents, and a computer program therefor. More specifically, the embodiments relate to a technology that utilizes machine learning technology to assess the risk of each area on a ship, such as an exposed deck, or the risk to passengers related to safety accidents, and to display the risk on a control system via a dashboard-style user interface, thereby preventing accidents in advance or enabling a rapid response to them.

Claims

1. As a marine accident monitoring system, A receiving module configured to receive image information of a space including a ship; An evaluation module configured to store a machine learning-based detection model and calculate a risk level related to one or more areas or passengers within the space that divide the space corresponding to the image information by applying the image information to the detection model; and A marine accident monitoring system including an output module configured to match the risk level calculated by the evaluation module to the space to generate dashboard content and display the dashboard content through a user interface on the marine accident monitoring system or a terminal device that is communicatively connected to the marine accident monitoring system.

2. In paragraph 1, The above output module, Dividing said space into a plurality of said zones based on boundary structures within the ship or on one or more passenger locations, A marine accident monitoring system further configured to dynamically change at least one of whether to display the area, the display form of the area, whether to display image information corresponding to the area, and the display form of the image information corresponding to the area in the dashboard content based on the risk level calculated for the divided plurality of areas.

3. In paragraph 1, Further comprising a database configured to store location information of boundary structures within the ship; The above receiving module is further configured to receive operating status information of the vessel from the vessel control system, A marine accident monitoring system wherein the evaluation module is further configured to calculate the risk level for the one or more areas by using the distance to the boundary structure, the visibility of the image information, and the operation status information as input values ​​for the detection model.

4. In paragraph 1, The above evaluation module, A preprocessing unit configured to extract image data representing one or more preset behavioral patterns of the passenger from the image information; and A marine accident monitoring system including a learning unit configured to calculate the risk level related to the passenger by using one or more of the duration of the behavior pattern shown in the image data, an additional behavior following the behavior pattern, and a movement of the area where the behavior pattern was detected as input values ​​to the detection model.

5. A marine accident monitoring system storing a machine learning-based detection model, the step of receiving image information of a space including a ship; The step of the above marine accident monitoring system calculating a risk level related to one or more areas defining the space corresponding to the image information or passengers within the space by applying the image information to the detection model; The above marine accident monitoring system generates dashboard content by matching the calculated risk level to the above space; and A method for monitoring marine accidents, comprising the step of transmitting the dashboard content to the marine accident monitoring system for display via a user interface on the marine accident monitoring system or a terminal device communicatively connected to the marine accident monitoring system.

6. In paragraph 5, The steps to create the above dashboard content are: The above marine accident monitoring system comprises a step of dividing the space into a plurality of said areas based on boundary structures within the ship or the locations of one or more passengers; and A marine accident monitoring method, wherein the marine accident monitoring system dynamically changes one or more of whether to display the area, the display form of the area, whether to display image information corresponding to the area, and the display form of the image information corresponding to the area in the dashboard content based on the risk level calculated for the plurality of divided areas.

7. In paragraph 5, Before calculating the above risk level, The above marine accident monitoring system stores the location information of the boundary structure inside the ship in the marine accident monitoring system; and The above marine accident monitoring system further includes a step of receiving operating status information of the vessel from a vessel control system, The steps for calculating the above risk are: A marine accident monitoring method comprising a step of calculating the risk level for one or more areas by using the distance to the boundary structure, the visibility of the image information, and the operation status information as input values ​​for the detection model, wherein the marine accident monitoring system comprises:

8. In paragraph 5, The steps for calculating the above risk are: The above marine accident monitoring system extracts image data representing one or more preset behavioral patterns of the passenger from the image information; and A marine accident monitoring method comprising a step of calculating the risk level related to the passenger by using one or more of the duration of the behavior pattern shown in the image data, an additional behavior following the behavior pattern, and a movement of the area where the behavior pattern was detected as input values ​​to the detection model.

9. A computer program stored on a computer-readable recording medium that is combined with hardware to execute a marine accident monitoring method according to any one of clauses 5 to 8.

Images