Seaman behavior perception and intelligent early warning method based on deep learning

Abstract

A sailor behavior perception and intelligent early warning method based on deep learning comprises the following steps: acquiring real-time video data by using a video monitoring system on a ship, accessing ship AIS data, acquiring a ship navigation area through an electronic fence, accessing a ship end behavior perception system, and performing early warning on the ship end behavior perception system. According to different navigation areas, nonstandard behaviors related to safety in ship navigation are analyzed and detected in real time, and a detection result is transmitted to a shore-end platform through a satellite signal for shore-end personnel to check, so that ship-shore cooperation is realized. According to the invention, the functions of active navigation monitoring, ship-shore early warning monitoring, cooperative closed-loop management and rapid emergency disposal can be realized. And meanwhile, a certain management mechanism is matched to supervise and urge standardization of behaviors of sailors, so that the navigation safety management level is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of ship artificial intelligence, and particularly relates to a crew behavior perception and intelligent early warning method based on deep learning. Background Art

[0002] In ocean voyages, human factors always come first. Ocean-going ships and cargo are of great value. During long-term voyages in the sea area, various external factors such as meteorology, hydrology, and neighboring ships in the ocean pose threats to safe navigation. Once an accident occurs, rescue is very difficult and often leads to huge losses. It is a prerequisite for making reasonable responses and ensuring safe navigation that crew members abide by behavioral norms and maintain sufficient vigilance against threat factors during navigation.

[0003] Despite the high risks of ocean navigation and the unimaginable consequences of safety accidents, crew members will still become complacent due to the temporary safe situation and neglect behavioral norms, thus leading to serious safety accidents.

[0004] Existing design solutions for personnel behavior supervision and the disadvantages of this solution:

[0005] Currently, the shipping industry mainly installs monitoring systems in positions such as the ship's engine room and centralized control room to record the driving behavior of crew members. After the ship docks at the shore, the video records on the ship are copied to the shore end for relevant shore-end personnel to view. Due to the long navigation time and poor network signal of ocean-going ships, the shore end cannot obtain real-time video from the ship end through the network and can only view the video historical records when the ship docks at the shore. This results in the inability to analyze the non-standard behaviors of the driver during the navigation process in real time, with poor detection real-time performance; a large amount of historical data needs to be viewed, and the detection efficiency is very low; due to the need for manual participation and the influence of subjective factors of the viewing personnel, there will also be problems of missed reports. Summary of the Invention

[0006] To solve the above problems of the existing technology, the present invention provides a crew behavior perception and intelligent early warning method based on deep learning.

[0007] The technical solution of the present invention is as follows:

[0008] A crew behavior perception and intelligent early warning method based on deep learning, characterized by including the following steps:

[0009] Utilize the on-board video monitoring system to collect real-time video data, access the ship's AIS data, obtain the ship's navigation area through an electronic fence, access the ship-end behavior perception system, analyze and detect in real time the non-standard behaviors related to safety during ship navigation according to different navigation areas, and transmit the detection results to the shore-end platform through satellite signals for shore-end personnel to view, thereby realizing ship-shore collaboration.

[0010] Preferably, the AIS data includes the speed, position, and navigation status, where

[0011] Speed: To determine whether the ship is in a stall. When the ship suddenly stalls, the requirements for the crew's driving shall be increased.

[0012] Position: Based on the ship's current position, determine the navigation area it is in, activate different algorithm logic modules, and execute different driving requirements.

[0013] Navigation status: According to the different navigation statuses of the ship, activate different detection algorithms and enable different detection functions.

[0014] Preferably, the electronic fence divides the navigation waters into pirate areas, narrow channels, and dense areas. Combining with the ship's current position, it determines whether the ship is sailing in a special area. If so, the algorithm module activates the driving rules for special areas for detection; otherwise, it conducts detection according to the driving rules for ordinary areas.

[0015] Further preferably, the division method for each area is as follows:

[0016] 1) Pirate area:

[0017] Based on the alerts and reports on pirate activities issued by the International Maritime Organization and the International Maritime Bureau, determine the areas with frequent pirate activities; and / or

[0018] Based on historical data and statistical analysis, determine the geographical areas with high incidences of pirate activities; and / or

[0019] Based on the international community's attention to pirate activities and safety risk assessments, determine the high-risk areas for pirate activities;

[0020] 2) Narrow channel area:

[0021] Based on considerations of navigation safety, divide the narrow channels according to factors such as the width, depth, geographical conditions, and navigation difficulty of the waterways; and / or

[0022] Based on the guidance and recommendations on narrow channels issued by the International Maritime Organization and the International Maritime Bureau according to the requirements of navigation safety;

[0023] 3) Dense area:

[0024] Based on factors such as ship traffic volume, ship density, and the concentration of navigation activities, divide the dense navigation area.

[0025] Preferably, it also includes obtaining port information. To determine whether the ship is in the port area, when the ship is at berth, the detection of the crew's non-standard driving behavior is turned off.

[0026] Further preferably, the deep learning object recognition of the ship-end behavior perception system includes:

[0027] 1) Algorithm functions

[0028] ① Captain on-duty monitoring

[0029] According to the requirement that the captain needs to be on the bridge for duty in key areas, the system analyzes the GPS position, and uses the electronic fence information of narrow channels and complex waters set in advance, and uses face recognition technology to obtain the picture information of the captain on duty, forming a logical closed loop; the system can save alarm information according to requirements and conduct ship-shore communication.

[0030] ② Regular lookout monitoring

[0031] The system analyzes the GPS position to obtain the current navigation area and navigation status of the ship, and obtains the lookout information of the crew, forming a logical closed loop; the system can save alarm information according to requirements and conduct ship-shore communication.

[0032] ③ Number of people on the bridge on-duty monitoring

[0033] The system obtains the area information according to the GPS position analysis and the communication method with the shipping middle platform, and uses face positioning and personnel recognition technology to obtain the information of the number of crew members on duty, forming a logical closed loop. The system can save alarm information according to requirements and conduct ship-shore communication;

[0034] ④ Monitoring of the on-duty status on the bridge

[0035] Identify drowsy phenomena including closing eyes, lying prone, and leaning during duty. When the system monitors that the suspected drowsy time of the on-duty personnel exceeds 10 minutes, an alarm for long-term stillness of the on-duty personnel on the bridge is generated, and ship-shore communication is conducted.

[0036] 2) Deep learning module

[0037] A) Object recognition

[0038] The deep learning algorithm for person and telescope picture recognition in the deep learning algorithm module of the present invention refers to the Yolov5 algorithm for single-stage object detection. This algorithm consists of a network input, intermediate layers, fully connected layers, and network outputs. The usage process is as follows:

[0039] a) Dataset collection

[0040] Collect relevant target datasets for people and telescope targets in different postures under different scenarios;

[0041] b) Data augmentation

[0042] Adopt the mixup method, stuffing-style image processing, adding negative samples and automatically generating xml files to achieve data augmentation, reducing the diversity of the light perturbation dataset, suppressing the overfitting of the model, and improving the robustness of the model;

[0043] c) Image enhancement

[0044] Image enhancement is performed on some of the pictures in the training dataset through white balance algorithms, morphological methods, and Laplacian operator image enhancement methods;

[0045] d) Data training

[0046] The pictures after annotation and image enhancement are used as the training dataset, the parameters are tuned, and then input into the deep learning network for training to obtain weights;

[0047] e) Target recognition

[0048] Feeding the real-time video stream into the trained weights can achieve the target recognition of people and telescopes;

[0049] B) Face recognition

[0050] The face recognition module includes a collection module, a picture processing module, and a face feature extraction and matching module.

[0051] Further preferably, the algorithm logic judgment includes

[0052] 1) Day and night judgment

[0053] Since the ship's sailing route is random, to ensure the unity of time, the system uses UTC time for statistics. To ensure the accuracy of time, the present invention calculates the local time according to the longitude difference between the ship's location and the 0-degree longitude;

[0054] 2) Each functional algorithm logic

[0055] A) Captain on-duty monitoring

[0056] When the ship sails to dense waters and narrow channels, or when the ship's speed suddenly drops below 6 knots during normal navigation, the captain on-duty detection function is activated. Currently, in the system, if the captain has not been on the bridge for 15 minutes, it is considered that the captain is not in port and an alarm is issued. The time thresholds and speed thresholds for the captain on-duty can be adjusted and remotely set;

[0057] The specific algorithm process is as follows: when the deep learning module recognizes a human target, the face recognition module is used to detect this target to determine whether it is the captain. If the captain cannot be found continuously for 15 minutes in the video, an alarm is issued;

[0058] B) Proper lookout monitoring

[0059] Proper lookout actions include: telescope lookout, patrol area lookout, and key equipment patrol lookout. When the ship is sailing normally, if there is a continuous lack of proper lookout for 12 minutes or the crew member remains stationary for more than 12 minutes, an alarm is issued. Among them, the time thresholds for the lack of proper lookout and the crew member's long-term stillness can be adjusted and remotely set;

[0060] The specific algorithm process is as follows: When the deep learning module recognizes a human or telescope target, it determines whether the target is in the lookout area or in front of key equipment. If so, it is considered that there is a regular lookout. At the same time, the target is tracked to prevent misidentification caused by abnormal targets, so as to improve the detection accuracy of the algorithm. When the target trajectory remains stationary for a long time, an alarm for lack of regular lookout is issued;

[0061] C) Monitoring the number of officers on the bridge

[0062] The judgment condition for insufficient number of officers on the bridge is as follows: During the navigation of the ship, if there is no one on the bridge for 5 consecutive minutes, an alarm will be issued. Taking the time from 8 pm to 5 am in the local time zone as night, if the number of officers on the bridge at night is less than 2 and exceeds 5 minutes, an alarm will be issued. Among them, the alarm time and the number threshold can be adjusted and set remotely;

[0063] The specific algorithm process is as follows: When the deep learning module recognizes a human target, it tracks the target to determine that it is a human target. At the same time, according to the tracking ID, it prevents repeated recognition of the same target, counts the number of officers on the bridge, and issues an alarm according to the alarm conditions.

[0064] Preferably, when an irregular behavior of a crew member is recognized, the alarm result will be uploaded to the shore-end platform for relevant personnel to view. An offline message processing mechanism is adopted. When the network signal is weak and the message sending fails, the message is stored in the database. After the network signal returns to normal, the corresponding message is processed again to ensure the consistency of the data between the ship end and the shore end.

[0065] The present invention discloses a method for crew behavior perception and intelligent early warning based on deep learning. By using the existing video monitoring system on the ship, real-time video data is collected and connected to the ship-end behavior perception system. The navigation area of the ship is obtained through the existing electronic fence. For irregular behaviors involving safety during ship navigation in different navigation areas, such as irregular captain's duty, irregular bridge control duty, irregular lookout, etc., behaviors that seriously affect navigation safety and personal safety are analyzed and detected in real time. The detection results are transmitted to the shore-end platform through satellite signals for shore-end personnel to view, so as to achieve ship-shore collaboration, play the roles of active navigation monitoring, ship-shore early warning monitoring, collaborative closed-loop management, and rapid emergency response. At the same time, with a certain management mechanism, it urges the standardization of crew behaviors, thereby effectively improving the level of navigation safety management.

[0066] The crew behavior perception system mainly analyzes the collected video image data by sending it into a pre-trained deep learning model for detecting targets such as humans and telescopes. According to the recognized targets and combined with the algorithm logic, it detects the irregular behaviors of the crew during navigation. Compared with the prior art, the present invention has the following technical effects:

[0067] 1) Advancedness

[0068] The present invention integrates deep learning technology into the crew behavior perception system, and the concepts and technologies adopted should be leading in the industry. The video detection method based on deep learning has a higher detection rate, a simpler algorithm, easier implementation and higher detection accuracy.

[0069] 2) Efficiency

[0070] The use of crew behavior perception and intelligent early warning systems can realize real-time detection of irregular behaviors of crew members during driving. When the ship is sailing in an area with weak satellite signals on the sea surface, offline message transmission can be resumed after the network is restored, reducing the workload of manual review and improving detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a flow chart of the method of the present invention;

[0072] Figure 2 It is the algorithm detection flow chart;

[0073] Figure 3 Provide the captain's duty alarm logic diagram;

[0074] Figure 4 Lookout logic diagram for Fan;

[0075] Figure 5 This is the logic diagram for bridge watch detection. DETAILED DESCRIPTION

[0076] In order to better understand the present invention, the present invention is further explained below in conjunction with the accompanying drawings and specific embodiments.

[0077] Example

[0078] In this embodiment, Figure 1 , 2 As shown in the figure, by analyzing the real-time video of the high-definition camera, combined with the deep learning target recognition method and logic algorithm of the crew behavior perception and intelligent early warning system, the detection of irregular crew behavior is realized, and the alarm results are uploaded to the shore platform for relevant personnel to view, realizing shore-based collaborative management. At the same time, the system accesses the ship's AIS data, and according to the ship's speed, longitude and latitude and other information analyzed by AIS, the algorithm logic rules are continuously adjusted to change the driving requirements for crew members in different areas. The specific work processes are as follows:

[0079] 1. AIS data access

[0080] AIS data is widely used in navigation safety, ship traffic management and navigation monitoring. Ships can obtain the location and operating status of nearby ships by receiving AIS data to avoid collisions and conflicts.

[0081] AIS data includes the basic information and operating status of vessels. The basic information includes vessel name, country of registry, call sign, vessel type, dimensions, etc., and the operating status includes position, course, speed, navigation status, etc. In addition, AIS data also includes the static information (such as vessel name, country of registry, etc.) and dynamic information (such as position, speed, etc.) of vessels.

[0082] The AIS data mainly used in this system are speed, position (longitude, latitude) and navigation status, and the main uses of each parameter are

[0083] · Speed: To determine whether a vessel is in a stall. When a vessel suddenly stalls, the requirements for the crew's navigation shall be increased.

[0084] · Position: To determine the navigation area where the vessel is located based on its current position, activate different algorithm logic modules, and execute different navigation requirements.

[0085] · Navigation status: Based on the different navigation statuses of vessels, activate different detection algorithms and enable different detection functions

[0086]

[0087] 2. Acquisition of electronic fence

[0088] During the navigation of vessels, they may pass through different areas. According to geographical conditions, navigation safety requirements, international standards, etc., the navigation waters are divided into areas such as pirate areas, narrow channels, and dense areas. The purpose of the division is to provide relevant safety information and suggestions to vessels and crew so that they can take corresponding measures and precautions during navigation.

[0089] Special area division method:

[0090] 1) Pirate area

[0091] · Agencies such as the International Maritime Organization and the International Maritime Bureau will issue alerts and reports on pirate activities. Based on these reports, the areas where pirate activities occur frequently can be determined.

[0092] · Based on historical data and statistical analysis, the geographical areas with high incidence of pirate activities can be determined, such as the waters of Somalia, the Niger Delta, etc.

[0093] · According to the degree of international community's attention to pirate activities and safety risk assessment, the high-risk areas for pirates can be identified.

[0094] 2) Narrow channel area

[0095] · Considering navigation safety, narrow channels can be divided according to factors such as the width, depth, geographical conditions and navigation difficulty of the waterway.

[0096] · Organizations such as the International Maritime Organization and the International Maritime Bureau will issue guidance and recommendations on narrow channels according to the needs of navigation safety.

[0097] 3) Dense area

[0098] · Dense navigation areas can be demarcated based on factors such as ship traffic volume, ship density, and the concentration of navigation activities.

[0099] · Usually, corresponding navigation rules and traffic diversion measures will be formulated according to the needs of navigation safety to ensure the safe navigation of ships in dense areas.

[0100] According to the above demarcation method of special areas, the invention demarcates pirate areas, narrow channel areas, and dense areas, records the corresponding longitude and latitude ranges of each area, and organizes and records them into an xls table. When the system runs, combined with the current position of the ship, it judges whether it is sailing in a special area. If so, the algorithm module starts to detect according to the driving rules of the special area; otherwise, it detects according to the driving rules of the ordinary area.

[0101] 3. Acquisition of port information

[0102] Since there are no requirements for the driving behavior of crew members during ship berthing, it is necessary to judge whether the ship is in the port area. When the ship is berthing, the detection of non-standard driving behavior of crew members is turned off. The method for judging the port in the invention is: record the longitude and latitude of each port, count them into an xls table, and judge whether it is within the port area range according to the current position of the ship in the system, so as to determine the functions to be detected by the current algorithm.

[0103] 4. Deep learning target recognition

[0104] 1) Algorithm function

[0105] The main detection functions of the behavior perception system are:

[0106] · Monitoring of the master's watch

[0107] The master is the first person responsible for the safe navigation of the ship. According to the requirement that the master needs to be on the bridge for watchkeeping in key areas such as narrow channels and complex waters, the system analyzes the GPS position and the pre-set electronic fence information of narrow channels and complex waters, and uses face recognition technology to obtain the master's watchkeeping picture information to form a logical closed loop. The system can save alarm information according to requirements and conduct ship-shore communication.

[0108] · Monitoring of proper lookout

[0109] Regular lookout is of utmost importance in the safety management of the bridge. The system analyzes the GPS position to obtain the current navigation area and status of the ship, and uses technologies such as face positioning, telescope recognition, area recognition, and position recognition to obtain the lookout information of the crew, forming a logical closed loop. The system can save the regular lookout records and alarm information according to requirements and conduct ship-shore communication. Regular lookout actions include: telescope lookout, patrol lookout in the lookout area, observation of key equipment (radar, compass), etc. Any of the above actions can be regarded as regular lookout.

[0110] · Monitoring of the number of people on duty on the bridge

[0111] Identifying the number of people on duty on the bridge is a basic detection function for the safety management of the bridge. The system obtains regional information through methods such as GPS position analysis and communication with the shipping center, and uses technologies such as face positioning and personnel recognition to obtain the information on the number of crew members on duty, forming a logical closed loop. The system can save the duty records of the officers and alarm information according to requirements and conduct ship-shore communication.

[0112] · Monitoring of the duty status on the bridge

[0113] Identify drowsy phenomena such as closing eyes, lying prone, or leaning on during the duty. When the system monitors that the suspected drowsy time of the duty personnel exceeds 10 minutes, an alarm for long-term stillness of the duty personnel on the bridge will be generated, and ship-shore communication will be conducted.

[0114] 2) Deep learning module

[0115] A) Target recognition

[0116] The deep learning algorithm module of the present invention for person and telescope image recognition refers to the Yolov5 algorithm of single-stage object detection. This algorithm consists of network input, intermediate layer, fully connected layer, and network output. The Yolov5 algorithm has four network models, which have higher flexibility and target recognition accuracy compared to Yolov3. Its detection speed has also been significantly improved compared to the Yolov3 algorithm. In the scenario of the present invention, the more flexible Yolov5 small model is selected, which can not only achieve real-time recognition of people and telescopes,

[0117] but also ensure a certain recognition accuracy of people and telescopes. By evenly dividing an image into i*i grids,

[0118] each grid is responsible for predicting the target whose center point falls within the grid, thus achieving efficient target recognition.

[0119] The usage process of the Yolov5 algorithm in this project is as follows:

[0120] a) Dataset collection

[0121] Collect relevant target datasets for people and telescope targets in different postures in different scenarios.

[0122] b) Data augmentation

[0123] Adopt the mixup method, force-feeding image processing, add negative samples and automatically generate xml files to achieve data augmentation, reduce the diversity of the light perturbation dataset, suppress the overfitting of the model, and improve the robustness of the model.

[0124] c) Image enhancement

[0125] In the yolov5 training dataset, some pictures are also used for image enhancement processing, which can not only enrich the dataset but also improve the subsequent recognition of the enhanced images. Image enhancement is performed on a certain proportion of the training dataset pictures through the above white balance algorithm, morphological method and Laplacian operator image enhancement method.

[0126] d) Data training

[0127] Use the pictures after annotation and image enhancement as the training dataset, tune the parameters, and input them into the deep learning network for training to obtain the weights.

[0128] e) Target recognition

[0129] Send the real-time video stream into the trained weights to achieve the target recognition of people and telescopes.

[0130] B) Face recognition

[0131] This method needs to monitor the requirement that the captain needs to be on the bridge for duty in key areas such as narrow channels and complex waters. Therefore, it is first necessary to identify the captain, and thus introduce the face recognition module.

[0132] Before startup, it is necessary to collect picture data of the captain in different orientations such as the front, looking down, looking up, left side, and right side as sampling samples.

[0133] The face recognition module includes a collection module, an image processing module and a face feature extraction and matching module.

[0134] · Data collection module, obtain the natural monitoring video stream to be detected, identify multiple face pictures of multiple people in the natural monitoring video stream according to the face detection algorithm, and extract key feature points from each face picture;

[0135] · The image processing module, based on key feature points, uses the SVM (Support Vector Machine) algorithm to distinguish frontal face images and skewed face images in face images. It compares the face skewing angle in the skewed face images with a preset skewing angle threshold. If the face skewing angle is greater than or equal to the preset skewing angle threshold, the skewed face images with a face skewing angle greater than or equal to the preset threshold are excluded. If the face skewing angle is less than the preset skewing angle threshold, the skewed face images and frontal face images with a skewing degree less than the preset threshold are corrected according to the reference feature points set for the standard face to obtain corrected face images, and the corrected face images are preprocessed.

[0136] · The face feature extraction and matching module, for the preprocessed face images, uses the face recognition model Arcface to extract the multi-dimensional feature vectors of each face in each face image, obtaining the multi-dimensional feature vectors of multiple faces. And it calculates the minimum Euclidean distance between the multi-dimensional feature vectors of each person in each face image in the natural surveillance video stream and the multi-dimensional feature vectors of all face images in the face information database respectively. It compares the calculated minimum Euclidean distance with a preset distance threshold. If the minimum Euclidean distance is less than or equal to the preset distance threshold, the identities of each person in the natural surveillance video stream are recognized according to the face images in the face information database corresponding to the minimum Euclidean distance.

[0137] 5. Specific algorithm logic judgment

[0138] 1) Night and day judgment

[0139] Since the ship's sailing route is random, to ensure the unity of time, the system uses UTC time for statistics. During the driving process, since the requirements for the number of people on duty on the bridge are different at different times (day and night), to ensure the accuracy of time, the present invention calculates the local time according to the longitude difference between the ship's location and the 0-degree longitude (the longitude itself contains positive and negative values). The specific calculation principle is as follows:

[0140] · The earth rotates 360 degrees in a day and there are 24 hours in a day, so for every 15-degree difference in longitude, the time difference is one hour;

[0141] · Since the longitude of UTC is 0 degrees, therefore, the current longitude / 15 is the number of hours by which the local time differs from the UTC time. However, there will be a 1-hour deviation between the time calculated by this method and the actual time. To ensure the accuracy of the calculated time, the present invention improves the time calculation method:

[0142] · Calculate the time difference between the current longitude and the UTC time, quotient = longitude / 15

[0143] · Divide the current time zone by 7.5 degrees to check whether the current longitude is within this time zone or exceeds one time zone, remainder = current longitude - the absolute value of 15 * quotient

[0144] · Calculate the number of time zones that the current longitude location differs from UTC time.

[0145] When the remainder <= 7.5, the difference in time zones = the quotient

[0146] When the remainder > 7.5, when the current longitude is greater than 0, the difference in time zones = the quotient + 1; when the current longitude is less than 0, the difference in time zones = the quotient - 1

[0147] · Obtain the current time of the computer. The number of hours of time = the current number of hours + the difference in time zones

[0148] · When the number of hours of time is greater than 24, the number of hours of time = the number of hours of time - 24, and the date of time = the current date + 1

[0149] · When the number of hours of time is less than 0, the number of hours of time = the number of hours of time + 24, and the date of time = the current date - 1

[0150] 2) Logic of each functional algorithm

[0151] A) Captain's duty monitoring

[0152] When the ship sails to dense waters and narrow channel areas, or when the ship's speed suddenly drops below 6 knots during normal navigation, the captain's duty detection function is activated. Currently, in the system, if the captain has not been on the bridge for 15 minutes, it is considered that the captain is not in port and an alarm is issued. The time thresholds and speed thresholds for the captain's duty can all be adjusted and remotely set.

[0153] The specific algorithm process is as follows: After the deep learning module identifies a human target, the face recognition module detects the target to determine whether it is the captain. If the captain is not found in the video for 15 consecutive minutes, an alarm is issued.

[0154] B) Proper lookout monitoring

[0155] Proper lookout actions include: lookout with binoculars, lookout in the patrol area, and patrol lookout of key equipment (radar, compass), etc. When the ship is sailing normally, if there is a continuous lack of proper lookout for 12 minutes or the crew remains stationary for more than 12 minutes, an alarm is issued. Among them, the time thresholds for lack of proper lookout and long-term crew immobility can be adjusted and remotely set.

[0156] The specific algorithm process is as follows: After the deep learning module identifies a human or binoculars target, it determines whether the target is in the lookout area or in front of key equipment. If so, it is considered that there is proper lookout. At the same time, the target is tracked to prevent misidentification caused by abnormal targets, so as to improve the detection accuracy of the algorithm. When the target trajectory remains stationary for a long time, an alarm for lack of proper lookout is issued.

[0157] C) Monitoring of the number of people on the bridge

[0158] The identification of the number of people on duty on the bridge is a basic detection function for bridge safety management. The system analyzes the area where the ship is located based on the GPS position parsed from AIS data, and uses technologies such as face positioning and face recognition to obtain the information on the number of crew members on duty, forming a logical closed loop. The system can save the duty records and alarm information of the officers according to requirements and conduct ship-shore communication.

[0159] In this system, the judgment condition for insufficient number of people on duty on the bridge is as follows: during the navigation of the ship, if there is no one on the bridge for 5 consecutive minutes, an alarm will be issued. Taking the time from 8 pm to 5 am in the local time zone as night, if the number of people on the bridge at night is less than 2 and exceeds 5 minutes, an alarm will be issued. Among them, the alarm time and the number threshold can be adjusted and set remotely.

[0160] The specific algorithm process is as follows: when the deep learning module identifies a human target, it tracks it to determine that it is a human target. At the same time, according to the tracking ID, it prevents the same target from being repeatedly identified, counts the number of people on the bridge, and issues an alarm according to the alarm conditions.

[0161] 6. Upload of Alarm Data

[0162] When this system identifies the non-standard behaviors of the crew members, it will upload the alarm results to the shore platform for relevant personnel to view. Due to the particularity of ship navigation, the network signal in the ocean is extremely poor, and it is impossible to ensure the real-time transmission of messages. Considering the above problems, this embodiment provides a message offline processing mechanism. When the network signal is weak and the message sending fails, the message is stored in the database, and when the network signal returns to normal, the corresponding message is reprocessed to ensure that the data will not be lost and to ensure the consistency of the data between the ship end and the shore end.

Claims

1. Crew behavior perception and intelligent early warning method based on deep learning, characterized by The following steps are involved: Utilize the ship's video surveillance system to collect real-time video data, access the ship's AIS data, obtain the ship's navigation area through electronic fences, access the ship's behavior perception system, and conduct real-time analysis and detection of irregular behaviors involving safety during the ship's navigation according to different navigation areas. The detection results are transmitted to the shore platform via satellite signals for shore personnel to review, thereby achieving ship-shore collaboration.

2. The method according to claim 1, characterized in that AIS data includes speed, position and navigation status. Speed: Determine whether the ship is stalled. When the ship suddenly stalls, the crew's driving requirements must be improved; Position: According to the current position of the ship, determine its navigation area, start different algorithm logic modules, and execute different driving requirements; Navigation status: According to the different navigation status of the ship, different detection algorithms are activated and different detection functions are enabled.

3. The method according to claim 1, characterized in that The electronic fence is used to divide the navigation waters into pirate areas, narrow waterways, and dense areas. Combined with the current location of the ship, it is determined whether it is sailing in a special area. If so, the algorithm module starts the special area driving rules for detection, otherwise it is detected according to the ordinary area driving rules.

4. The method according to claim 3, characterized in that The division of each area is as follows: 1) Pirate Area: Identify areas of high incidence of piracy activity based on alerts and reports of piracy issued by the International Maritime Organization and the International Maritime Bureau; and / or Identify geographic areas of high incidence of piracy based on historical data and statistical analysis; and / or Identify high-risk areas for piracy activities based on the international community's attention to piracy activities and security risk assessment; 2) Narrow waterway area: Based on navigation safety considerations, narrow waterways are classified according to their width, depth, geographical conditions and navigation difficulty; and / or In accordance with the guidance and recommendations on narrow waterways issued by the International Maritime Organization and the International Maritime Bureau based on the needs of navigation safety; 3) Dense areas: Intensive navigation areas are divided based on factors such as ship traffic volume, ship density and concentration of navigation activities.

5. The method according to claim 1, characterized in that It also includes port information acquisition to determine whether the ship is in the port area. When the ship is berthed, the detection of irregular driving behavior of the crew is turned off.

6. The method according to claim 4, characterized in that The deep learning target recognition of the ship-side behavior perception system includes: 1) Algorithm function ①Captain on duty monitoring According to the requirement that the captain should be on duty on the bridge in key areas, the system analyzes the GPS position and the pre-set narrow waterway and complex water electronic fence information, and uses face recognition technology to obtain the captain's duty picture information to form a logical closed loop; the system can save alarm information as needed and conduct ship-shore communication. ; ②Regular surveillance and monitoring The system obtains the current navigation area and navigation status of the ship based on GPS position analysis, and obtains the crew's lookout information to form a logical closed loop; the system can save alarm information as needed and conduct ship-shore communication. ; ③ Monitoring of the number of people on duty at the bridge The system obtains regional information based on GPS location analysis and communication with the shipping center, and uses facial positioning and personnel recognition technology to obtain information on the number of crew members on duty, forming a logical closed loop. The system can save alarm information as needed and conduct ship-shore communication; ④Driving station duty status monitoring Identify drowsiness on duty, including closing eyes, lying prone, and lying down. When the system detects that the duty personnel are suspected of drowsiness for more than 10 minutes, it will generate an alarm for long-term inactivity of duty personnel on the bridge and conduct ship-to-shore communication. 2) Deep Learning Module A) Target Identification The deep learning algorithm used by the deep learning algorithm module of the present invention for human and telescope image recognition refers to the Yolov5 algorithm for single-stage target detection. The algorithm consists of a network input, an intermediate layer, a fully connected layer, and a network output. The use process is as follows: a) Dataset Collection Collect relevant target data sets for people and telescope targets in different postures in different scenes; b) Data Augmentation The mixup method is used to process images by duck-feeding, add negative samples and automatically generate XML files to achieve data enhancement, reduce the diversity of light perturbation data sets, suppress overfitting of the model, and improve the robustness of the model; c) Image Enhancement Image enhancement was performed on some training dataset images using white balance algorithm, morphological method and Laplacian operator image enhancement method; d) Data training The annotated and enhanced images are used as training data sets, the parameters are tuned, and the images are input into the deep learning network for training to obtain weights. e) Target recognition By feeding the real-time video stream into the trained weights, target recognition of people and telescopes can be achieved; B) Face Recognition The face recognition module includes an acquisition module, an image processing module and a face feature extraction and matching module.

7. The method according to claim 6, characterized in that Algorithm logic judgment includes 1) Night and day judgment Since the ship's navigation route is random, in order to ensure the uniformity of time, the system uses UTC time for statistics. To ensure the accuracy of time, the present invention calculates the local time based on the longitude difference between the ship's location and 0 degrees longitude; 2) Algorithm logic of each function A) Captain's watch monitoring When the ship is sailing to dense waters and narrow waterways, or when the ship's speed suddenly drops below 6 knots during normal navigation, the captain's duty detection function is activated. The system currently sets that if the captain has not been on the bridge for 15 minutes, it is considered that the captain is not at the port and an alarm is triggered. The captain's duty time thresholds and speed thresholds can be adjusted and set remotely; The specific algorithm process is as follows: when the deep learning module recognizes a human target, the face recognition module detects the target to determine whether it is the captain. If the captain is not found in the video for 15 consecutive minutes, an alarm is issued; B) Regular surveillance Regular lookout actions include: telescope lookout, patrol area lookout, and key equipment patrol lookout. When the ship lacks regular lookout for 12 consecutive minutes or the driver remains stationary for more than 12 minutes during normal navigation, an alarm will be issued. The time thresholds for lack of regular lookout and long-term stationary of the crew can be adjusted and set remotely; The specific algorithm process is as follows: When the deep learning module recognizes a person or telescope target, it determines whether it is in the observation area or in front of key equipment. If so, it is considered that there is a formal observation. At the same time, the target is tracked to prevent abnormal targets from causing misidentification, so as to improve the accuracy of algorithm detection. If the target trajectory remains stationary for a long time, an alarm for lack of formal observation is issued; C) Monitoring of the number of people on duty at the bridge The judgment condition for insufficient number of people on duty on the bridge is: during the voyage of the ship, if there is no one on the bridge for 5 consecutive minutes, an alarm will be issued. The local time zone is 8 pm to 5 am at night. If the number of people on the bridge at night is less than 2 for more than 5 minutes, an alarm will be issued. The alarm time and number threshold can be adjusted and set remotely; The specific algorithm process is as follows: when the deep learning module recognizes a human target, it tracks it and determines it as a human target. At the same time, based on the tracking ID, it prevents repeated identification of the same target, counts the number of people on the bridge, and issues an alarm based on the alarm conditions.

8. The method according to claim 1, characterized in that When irregular behavior of the crew is identified, the alarm result will be uploaded to the shore platform for relevant personnel to view. The message offline processing mechanism is adopted. When the network signal is weak and the message is not sent successfully, the message will be stored in the database. After the network signal returns to normal, the corresponding message will be reprocessed to ensure the consistency of the ship-side and shore-side data.

Images