Intelligent ship safety monitoring method based on multi-scale fusion graph convolutional network

Abstract

The invention relates to a ship intelligent safety monitoring method based on a multi-scale fusion graph convolutional network. The method comprises the following steps: constructing a crew non-standard behavior data set; modeling a multi-scale simplified skeleton sequence; performing data preprocessing; constructing a behavior recognition algorithm based on a multi-scale fusion graph convolutional network; carrying out model pre-training; and carrying out model training and verification. Skeleton data is adopted as input of a behavior recognition algorithm, and interference caused by redundant information such as environment background and illumination change is effectively avoided. By focusing on the joint information of the human body, the accuracy of behavior recognition can be improved, and especially in a complex dynamic scene, the algorithm can detect illegal behaviors more accurately and steadily, so that the intelligent level of ship safety management is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of ship safety monitoring, and in particular relates to a ship intelligent safety monitoring method based on a multi-scale fusion graph convolutional network. Background Art

[0002] Safety has always been the top priority of ship navigation. With the rapid development of the global shipping industry, the number of ships and the frequency of navigation have increased significantly, and the resulting safety hazards have become increasingly prominent. Non-standard behavior of crew members, equipment failure, bad weather, and the increase in ocean traffic density are all important factors leading to navigation accidents. Crew safety awareness is directly related to navigation safety, and improper crew behavior is one of the most direct and major factors affecting ship safety. Strengthening safety training, improving safety monitoring mechanisms, and realizing intelligent safety management are key issues that need to be addressed urgently.

[0003] Traditional safety monitoring strategies rely on human monitoring mechanisms, using monitoring equipment such as cameras and sensors to monitor crew behavior, troubleshoot equipment failures, and ensure safe navigation of ships. However, this approach has limitations. First, manual monitoring is easily affected by fatigue and distraction, resulting in the omission of important information. Second, the data processing capacity of traditional equipment is limited, and it is difficult to analyze large amounts of data in real time, thus missing the opportunity to identify potential safety hazards. Therefore, monitoring solutions based on deep learning technology have emerged. Deep learning can automatically identify and classify various abnormal situations through big data analysis, which not only improves the accuracy of monitoring, but also responds to potential threats in real time, greatly improving the safety and intelligence level of ships. It has been widely used. Chinese Patent Publication No. CN114419607A segments the collected video data, divides the driving area and the driver, and uses deep learning algorithms to monitor non-standard behaviors in the cockpit. Chinese Patent Publication No. CN113486843A uses an improved YOLOv3 algorithm to learn a database containing six behaviors such as leaving the post, smoking, and playing with mobile phones, in order to achieve the purpose of real-time monitoring of non-standard behaviors. Chinese patent publication number CN115661766A proposes an intelligent ship safety monitoring system based on deep learning, which uses algorithms such as target detection, behavior recognition, and face recognition to achieve multi-category event monitoring and establish a complete monitoring system. The ship safety monitoring systems and methods disclosed in the above patents are helpful for ship safety detection, but there are still the following three problems:

[0004] Low accuracy in behavior recognition. The above patent relies on convolutional neural networks for human behavior recognition. Although it performs well in image classification, the accuracy of behavior recognition is still insufficient in dynamic scenes, especially in ship operation environments. This is mainly because the algorithm is prone to misjudgment when dealing with fast-moving or overlapping objects, and fails to fully consider the inherent joint connection of the human skeleton, thus affecting the accurate recognition of non-standard behaviors.

[0005] Poor environmental adaptability. The recognition network model used in the above patent directly processes the video data collected by the camera. However, in a complex marine environment, factors such as lighting changes, weather conditions, water surface reflections, and cab background may affect the recognition accuracy, resulting in unstable performance of the algorithm in different scenarios.

[0006] Strong data dependence. A good safety monitoring system often relies on a large amount of high-quality data for training, but in the ship driving environment, relevant database resources are relatively scarce. If image data from other fields is directly used, it may lead to environmental background mismatch, thus affecting the generalization ability and detection effect of the algorithm and increasing the risk of misidentification.

[0007] Therefore, combining multi-scale fusion technology with the graph convolutional network model to build an efficient human behavior safety detection algorithm is of great significance for improving the accuracy of non-standard behavior recognition and promoting the development of intelligent safety management of ships. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention proposes a ship intelligent safety monitoring method based on a multi-scale fusion graph convolutional network, which is used to solve the problems of low behavior recognition accuracy, poor environmental adaptability, and strong data dependence in the existing methods.

[0009] The above-mentioned object of the present invention is achieved by the following technical solutions:

[0010] A ship intelligent safety monitoring method based on a multi-scale fusion graph convolutional network comprises the following steps:

[0011] Step 1: Construct an initial dataset of crew members' non-standard behavior, including the collected video data of non-standard behavior cases and the main joint coordinate information data of the human skeleton extracted from the video data, and save the data;

[0012] Step 2: Based on the main joint coordinate information data of the human skeleton in step 1, multi-scale simplified skeleton sequence modeling is performed to form multi-scale simplified skeleton sequence data;

[0013] Step 3, preprocessing the multi-scale simplified skeleton sequence data obtained in step 2 to achieve data denoising and unify the size, forming a non-standardized row processed data set, and dividing the processed data set into a training set and a test set;

[0014] Step 4: Build a behavior recognition algorithm based on a multi-scale fusion graph convolutional network;

[0015] Step 5: Select NTU RGB+D 120 as the pre-training dataset for model pre-training;

[0016] Step 6: Based on the pre-training, the data divided into the training set in step 3 is used as the input data for model training, and the network model is trained again for parameter optimization; the data divided into the validation set in step 3 is input into the trained network model, and the model recognition accuracy is statistically analyzed.

[0017] Furthermore, step 1 includes:

[0018] Step 1.1 Determine the types of non-standard behaviors of crew members based on the type of ship, navigation scenario, etc., including smoking, drinking, playing with mobile phones, sleeping and all other possible non-standard behaviors;

[0019] Step 1.2 Use the surveillance camera on the ship to collect video cases of non-standard behaviors of each crew member, and number the crew members and behavior categories before collection. During the collection process, each crew member stands at different angles of the surveillance camera to complete the action, and each time needs to repeat the same behavior three times in full;

[0020] Step 1.3 stores the collected data samples in the form of MP4 video files, and the naming format follows the naming convention of "crew number_behavior number_monitoring angle.mp4".

[0021] Step 1.4 uses the Openpose algorithm to extract the coordinate information of the 25 main joints of the human skeleton in the video data, and stores the corresponding joint coordinate information separately in a text file in a fixed order. The text file uses the same naming format as the video file, that is, "Crew number_Behavior number_Monitoring angle.txt".

[0022] Furthermore, step 2 includes:

[0023] Step 2.1 Determine the skeleton simplification rules based on the connection relationship between human joints: any joint in the simplification is merged from n closely adjacent joints in the original, where closely adjacent means that n joints and their connection relationship can form a connected graph; joint merging is achieved by channel splicing, and the merging direction satisfies the position distribution from hand to foot and from head to foot; two directly connected joints in the simplified skeleton share the same original joint; in the simplification process, all joints except the shared sub-joints are used and can only be used once;

[0024] Step 2.2: select 3 and 5 as the simplification scales to be used, and perform third-order simplification and fifth-order simplification on the original skeleton sequence, respectively, to form a third-order simplified skeleton sequence and a fifth-order simplified skeleton sequence;

[0025] Step 2.3 concatenates and fuses the original skeleton sequence with the above two simplified skeleton sequences to form a multi-scale simplified skeleton sequence.

[0026] Furthermore, step 3 includes:

[0027] Step 3.1: Eliminate data samples with large joint information errors based on the characteristics of human skeleton structure, and sort all skeletons appearing in the data samples based on skeleton motion momentum. When the sample category is a single-person action, retain the first skeleton information; when the sample category is a two-person action, retain the first two skeleton information;

[0028] Step 3.2: fill the joint point coordinates of the second person in the single-person action with 0, and scale all skeleton sequence samples to 64 frames by interpolation alignment to unify the data size;

[0029] Step 3.3: Divide the training set and test set: The dataset after non-standard row processing is divided into training set and test set in a ratio of 8:2.

[0030] Furthermore, step 4 includes:

[0031] Step 4.1 Select the adaptive graph convolution block CTR-GC in the CTR-GCN network as the graph convolution block, which contains two trainable adjacency matrices and Tanh activation function;

[0032] Step 4.2: Temporal convolution uses a single-layer convolution with a convolution kernel of K*1, where the value of K is consistent with the skeleton simplification scale;

[0033] Step 4.3 uses grouped convolution to fuse different joints by channel and fuse different channels of a single joint to form a grouped convolution block;

[0034] Step 4.4 concatenates the grouped convolution block with the temporal convolution block to build a graph convolution network model containing 10 layers of sub-convolution blocks, where the output channels of each layer are 64, 64, 64, 64, 128, 128, 128, 256, 256, 256 respectively.

[0035] Furthermore, step 5 includes:

[0036] Step 5.1: Select NTU RGB+D 120 as the pre-training dataset and use the data preprocessing method in step 3 to achieve data denoising and uniform size.

[0037] In step 5.2, the processed data is input into the graph convolutional network model for training. The number of iterations of network training is 100, and the stochastic gradient descent optimization algorithm with momentum of 0.9 and weight decay of 0.0004 is adopted. In terms of learning rate adjustment, the initial learning rate is 0.1, 5 rounds of warm-up training are adopted, and it is decayed by 0.1 times at the 35th and 85th epochs. The batch size of the input data is 64.

[0038] The advantages and positive effects of the present invention are:

[0039] 1. The present invention uses a multi-scale fused graph convolutional network model to realize behavior recognition. The use of graph convolution can better model the interdependence between joints, so as to more effectively process skeleton data. This method can not only capture the motion characteristics at different scales, but also improve the understanding and recognition ability of complex behaviors. By comprehensively considering multi-level information, the performance of the algorithm in a dynamic environment will be significantly improved, and the recognition accuracy of non-standard behaviors will be enhanced, providing strong support for intelligent safety management of ships.

[0040] 2. The present invention uses skeleton data as the input of the behavior recognition algorithm, effectively avoiding interference caused by redundant information such as environmental background and lighting changes. By focusing on the joint information of the human body, the accuracy of behavior recognition can be improved, especially in complex dynamic scenes, making the algorithm more accurate and robust in detecting illegal behaviors, thereby enhancing the intelligent level of ship safety management.

[0041] 3. The present invention adopts multi-scale fusion technology to derive the original input into a third-order simplified and fifth-order simplified skeleton sequence, which greatly enriches the original input information. This method not only retains important motion features, but also reduces the demand for data scale, thereby effectively saving the manpower and time cost required for data set construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a framework diagram of the multi-scale fusion graph convolutional network model of the present invention;

[0043] Figure 2 It is a skeleton diagram of different simplified scales of the present invention;

[0044] Figure 3 It is a structural diagram of the graph convolution block of the present invention;

[0045] Figure 4 This is a structural diagram of the grouped convolution block of the present invention. DETAILED DESCRIPTION

[0046] The structure of the present invention is further described below with reference to the accompanying drawings and by way of examples. It should be noted that the present examples are descriptive rather than restrictive.

[0047] From the perspective of safe navigation of ships, the present invention aims to use deep learning technology to realize intelligent safety monitoring of crew behavior and ensure the safety of ships and people on board. In order to effectively improve the accuracy of intelligent recognition and the efficiency of safety monitoring, the present invention proposes a ship intelligent safety monitoring algorithm based on a multi-scale fusion graph convolutional network, including the construction of a crew non-standard behavior dataset, multi-scale simplified skeleton sequence modeling, the construction of a behavior recognition algorithm based on a multi-scale fusion graph convolutional network, data preprocessing, model pre-training, and model training and verification. The specific steps are as follows:

[0048] Step 1: Construct a dataset of crew non-standard behaviors

[0049] Step 1.1 Determine the types of non-standard behaviors of crew members based on the type of ship, navigation scene, etc., including all possible non-standard behaviors such as smoking, drinking, playing with mobile phones, sleeping, etc.;

[0050] Step 1.2 Use the surveillance camera on board to collect video cases of non-standard behaviors of each crew member. Before collecting, the crew members and behavior categories are numbered for file naming. The collection process requires each crew member to stand at different angles of the surveillance camera to complete the action, and each time the same action needs to be repeated three times in full;

[0051] Step 1.3 stores the collected data samples in the form of MP4 video files, and the naming format follows the naming convention of "crew number_behavior number_monitoring angle.mp4";

[0052] Step 1.4 uses the Openpose algorithm to extract the coordinate information of the 25 main joints of the human skeleton in the video data, and stores the corresponding joint coordinate information separately in a text file in a fixed order. The text file uses the same naming format as the video file, that is, "Crew number_Behavior number_Monitoring angle.txt".

[0053] Step 2: Multi-scale simplified skeleton modeling

[0054] Step 2.1 Determine the skeleton simplification rules based on the connection relationship between human joints: any joint in the simplification is merged from n closely adjacent joints in the original, where closely adjacent means that n joints and their connection relationship can form a connected graph; joint merging is achieved by channel splicing, and the merging direction satisfies the position distribution from hand to foot and from head to foot; two directly connected joints in the simplified skeleton share the same original joint; in the simplification process, the remaining joints except the shared sub-joints must be used and can only be used once;

[0055] Step 2.2: select 3 and 5 as the simplification scales to be used, and perform third-order simplification and fifth-order simplification on the original skeleton sequence, respectively, to form a third-order simplified skeleton sequence and a fifth-order simplified skeleton sequence;

[0056] Step 2.3 concatenates and fuses the original skeleton sequence with the two simplified skeleton sequences to form a multi-scale simplified skeleton sequence containing multi-dimensional skeleton information.

[0057] Step 3: Data preprocessing

[0058] Step 3.1: According to the characteristics of human skeleton structure, multi-scale simplified skeleton sequence samples with incorrect or missing joint information are eliminated, and all skeletons appearing in the data sample are sorted according to the skeleton motion momentum. When the sample category is a single-person action, the skeleton information of the first one is retained, and when the sample category is a two-person action, the skeleton information of the first two is retained;

[0059] In step 3.2, the joint point coordinates of the second person in the single-person action are filled with 0, and all skeleton sequence samples are scaled to 64 frames by interpolation alignment to unify the data size.

[0060] 3.3. Divide the training set and test set: The dataset after non-standard row processing is divided into training set and test set in the ratio of 8:2.

[0061] Step 4: Build a behavior recognition algorithm based on a multi-scale fusion graph convolutional network.

[0062] Step 4.1 Select the adaptive graph convolution block CTR-GC in the CTR-GCN network as the graph convolution block, which contains two trainable adjacency matrices and Tanh activation function;

[0063] Step 4.2: Temporal convolution uses a single-layer convolution with a convolution kernel of K*1, where the value of K is consistent with the skeleton simplification scale;

[0064] Step 4.3 To fully model and simplify the connections between the sub-joints within a joint, group convolution is used to fuse different joints by channel and fuse different channels of a single joint to form a group convolution block. The specific structure is as follows: Figure 3 As shown;

[0065] Step 4.4 concatenates the group convolution block with the graph convolution block to build a graph convolution network model containing 10 layers of sub-convolution blocks, where the output channels of each layer are 64, 64, 64, 64, 128, 128, 128, 256, 256, 256 respectively.

[0066] Step 5: Model pre-training

[0067] Step 5.1: In order to enable the model to learn more general knowledge and semantic features and improve its robustness, model pre-training is usually performed. The NTU RGB+D 120 dataset contains 120 behavior samples including non-standard behaviors such as jumping, running, and pushing, which can be regarded as a supplementary dataset for multi-scale skeleton sequence samples. Therefore, it is selected as the pre-training dataset, and the data preprocessing method in step 3 is used to achieve data denoising and uniform size;

[0068] Step 5.2 inputs the processed data into the graph convolutional network model for training, where the number of iterations of network training is 100, and the stochastic gradient descent optimization algorithm with momentum of 0.9 and weight decay of 0.0004 is used. In terms of learning rate adjustment, the initial learning rate is 0.1, 5 rounds of warm-up training are used, and the decay is 0.1 times at the 35th and 85th epochs. In addition, the batch size of the input data is 64.

[0069] Step 6: Model training and validation

[0070] Step 6.1: Use the data divided into training sets in step 3 as input data for model training, and keep the number of iterations, optimization algorithm, and batch size settings consistent with those in step 5.2, and train the network model twice for parameter optimization;

[0071] Step 6.2 inputs the data divided into the validation set in step 3 into the trained network model and calculates the recognition accuracy of the model.

[0072] Although the embodiments and drawings of the present invention are disclosed for illustrative purposes, those skilled in the art will appreciate that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A ship intelligent safety monitoring method based on multi-scale fusion graph convolutional network, characterized in that: The steps include: Step 1: Construct an initial dataset of crew members' non-standard behavior, including the collected video data of non-standard behavior cases and the main joint coordinate information data of the human skeleton extracted from the video data, and save the data; Step 2: Based on the main joint coordinate information data of the human skeleton in step 1, multi-scale simplified skeleton sequence modeling is performed to form multi-scale simplified skeleton sequence data; Step 3, preprocessing the multi-scale simplified skeleton sequence data obtained in step 2 to achieve data denoising and unify the size, forming a non-standardized row processed data set, and dividing the processed data set into a training set and a test set; Step 4: Build a behavior recognition algorithm based on a multi-scale fusion graph convolutional network; Step 5: Select NTU RGB+D 120 as the pre-training dataset for model pre-training; Step 6: Based on the pre-training, the data divided into the training set in step 3 is used as the input data for model training, and the network model is trained again for parameter optimization; the data divided into the validation set in step 3 is input into the trained network model, and the model recognition accuracy is statistically analyzed.

2. According to claim 1, the method for intelligent safety monitoring of ships based on multi-scale fusion graph convolutional network is characterized in that: Step 1 includes: Step 1.1 Determine the types of non-standard behaviors of crew members based on the type of ship, navigation scenario, etc., including smoking, drinking, playing with mobile phones, sleeping and all other possible non-standard behaviors; Step 1.2 Use the surveillance camera on the ship to collect video cases of non-standard behaviors of each crew member, and number the crew members and behavior categories before collection. During the collection process, each crew member stands at different angles of the surveillance camera to complete the action, and each time needs to repeat the same behavior three times in full; Step 1.3 stores the collected data samples in the form of MP4 video files, and the naming format follows the naming convention of "crew number_behavior number_monitoring angle.mp4"; Step 1.4 uses the Openpose algorithm to extract the coordinate information of the 25 main joints of the human skeleton in the video data, and stores the corresponding joint coordinate information separately in a text file in a fixed order. The text file uses the same naming format as the video file, that is, "Crew number_Behavior number_Monitoring angle.txt".

3. The method for intelligent safety monitoring of ships based on multi-scale fusion graph convolutional network according to claim 1 is characterized in that: Step 2 includes: Step 2.1 Determine the skeleton simplification rules based on the connection relationship between human joints: any joint in the simplification is merged from n closely adjacent joints in the original, where closely adjacent means that n joints and their connection relationship can form a connected graph; joint merging is achieved by channel splicing, and the merging direction satisfies the position distribution from hand to foot and from head to foot; two directly connected joints in the simplified skeleton share the same original joint; in the simplification process, all joints except the shared sub-joints are used and can only be used once; Step 2.2: select 3 and 5 as the simplification scales to be used, and perform third-order simplification and fifth-order simplification on the original skeleton sequence, respectively, to form a third-order simplified skeleton sequence and a fifth-order simplified skeleton sequence; Step 2.3 concatenates and fuses the original skeleton sequence with the two simplified skeleton sequences to form a multi-scale simplified skeleton sequence.

4. The method for intelligent safety monitoring of ships based on a multi-scale fusion graph convolutional network according to claim 1 is characterized in that: Step 3 includes: Step 3.1: Eliminate data samples with large joint information errors based on the characteristics of human skeleton structure, and sort all skeletons appearing in the data samples based on skeleton motion momentum. When the sample category is a single-person action, retain the first skeleton information; when the sample category is a two-person action, retain the first two skeleton information; Step 3.2: fill the joint point coordinates of the second person in the single-person action with 0, and scale all skeleton sequence samples to 64 frames by interpolation alignment to unify the data size; Step 3.3 divides the training set and the test set. The data set after non-standard row processing is divided into training set and test set in a ratio of 8:

2.

5. The method for intelligent safety monitoring of ships based on multi-scale fusion graph convolutional network according to claim 1 is characterized in that: Step 4 includes: Step 4.1 Select the adaptive graph convolution block CTR-GC in the CTR-GCN network as the graph convolution block, which contains two trainable adjacency matrices and Tanh activation function; Step 4.2: Temporal convolution uses a single-layer convolution with a convolution kernel of K*1, where the value of K is consistent with the skeleton simplification scale; Step 4.3 uses grouped convolution to fuse different joints by channel and fuse different channels of a single joint to form a grouped convolution block; Step 4.4 concatenates the grouped convolution block with the temporal convolution block to build a graph convolution network model containing 10 layers of sub-convolution blocks, where the output channels of each layer are 64, 64, 64, 64, 128, 128, 128, 256, 256, 256 respectively.

6. The method for intelligent safety monitoring of ships based on multi-scale fusion graph convolutional network according to claim 1 is characterized in that: Step 5 includes: Step 5.1: Select NTU RGB+D 120 as the pre-training dataset and use the data preprocessing method in step 3 to achieve data denoising and uniform size. In step 5.2, the processed data is input into the graph convolutional network model for training. The number of iterations of network training is 100, and the stochastic gradient descent optimization algorithm with momentum of 0.9 and weight decay of 0.0004 is adopted. In terms of learning rate adjustment, the initial learning rate is 0.1, 5 rounds of warm-up training are adopted, and it is decayed by 0.1 times at the 35th and 85th epochs. The batch size of the input data is 64.

Images