A cruise ship safety video monitoring method and system based on video classification

Abstract

The invention discloses a cruise ship safety video monitoring method and system based on video classification, and relates to the field of video monitoring technology. The method comprises: a cruise ship-side server collects monitoring videos of various areas on the cruise ship through a front-end camera; the cruise ship-side server performs picture enhancement and defogging processing on the monitoring videos of various areas to obtain a real-time video stream; the cruise ship-side server uses a video perception algorithm and a background reconstruction algorithm to perform high-definition restoration of the real-time video stream; the cruise ship-side server transmits the high-definition restored real-time video stream to a shore-side server based on real-time video encoding parameters; the cruise ship-side server uses the time sequence characteristics, spatiotemporal characteristics and scene characteristics of the high-definition restored real-time video stream to classify various behaviors in the high-definition restored real-time video stream, and identifies abnormal behaviors in the high-definition restored real-time video stream; the application realizes automatic monitoring of the safety of the cruise ship.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a cruise ship safety video surveillance method and system based on video classification. Background Art

[0002] The original meaning of cruise refers to a large passenger ship that sails regularly on the ocean. The word "mail" itself has the meaning of transportation, and in the past, transoceanic mail was always carried by such large and fast passenger ships, hence the name. Now it usually refers to tourist passenger ships sailing in the ocean. With the vigorous development of ocean and marine tourism business, the safety demand for cruise transportation is growing. As a complex comprehensive facility, cruise ships require diversified systems to ensure their safe operation. As an important part of it, video surveillance technology needs to continuously improve its scientific and technological content and innovation to meet the development needs of the cruise industry.

[0003] Traditional video surveillance systems mainly rely on manual intervention, and have problems such as poor recognition effect and high false positive rate.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a cruise ship safety video monitoring method and system based on video classification to solve the above-mentioned technical problems.

[0006] The present application provides a cruise ship security video monitoring method based on video classification, comprising: a cruise ship-side server collects monitoring videos of various areas on the cruise ship through a front-end camera;

[0007] The cruise ship server performs image enhancement and defogging processing on the surveillance videos of each area to obtain a real-time video stream;

[0008] The cruise ship server uses a video perception algorithm and a background reconstruction algorithm to restore the real-time video stream to high definition;

[0009] The cruise ship server transmits the high-definition restored real-time video stream to the shore server based on the real-time video encoding parameters; wherein the real-time video encoding parameters are adjusted by real-time analysis of the high-definition restored real-time video stream and real-time network conditions; the cruise ship server receives the command and dispatch instructions of the shore server in real time;

[0010] The cruise ship server utilizes the timing features, spatiotemporal features and scene features of the high-definition restored real-time video stream to classify various behaviors in the high-definition restored real-time video stream, and identifies abnormal behaviors in the high-definition restored real-time video stream. Further, the timing features, spatiotemporal features and scene features of the high-definition restored real-time video stream are utilized to classify various behaviors in the high-definition restored real-time video stream, and identify abnormal behaviors in the high-definition restored real-time video stream, including:

[0011] Extracting the temporal features, the spatiotemporal features and the scene features from the high-definition restored real-time video stream;

[0012] Inputting the temporal features, the spatiotemporal features and the scene features into a behavior classification model to obtain a behavior classification result; wherein the behavior classification model is trained and optimized based on a deep learning algorithm; and the behavior classification result is a strong temporal dependency behavior type or a weak temporal dependency behavior type;

[0013] According to the behavior classification result, determining the first abnormal behavior recognition model or the second abnormal behavior recognition model as the target recognition model; wherein the first abnormal behavior recognition model corresponds to the strong timing-dependent behavior type; and the second abnormal behavior recognition model corresponds to the weak timing-dependent behavior type;

[0014] The high-definition restored real-time video stream is input into the target recognition model to identify the abnormal behavior.

[0015] Furthermore, the behavior classification model is trained and optimized based on the following steps:

[0016] Obtaining a sample video stream including a sample strong temporal dependency behavior label and a sample weak temporal dependency behavior label;

[0017] Extracting a first sample temporal feature, a first sample spatiotemporal feature, a first sample scene feature, a second sample temporal feature, a second sample spatiotemporal feature, and a second sample scene feature of the sample video stream; wherein the first sample temporal feature, the first sample spatiotemporal feature, and the first sample scene feature all correspond to the sample strong temporal dependency behavior label; the second sample temporal feature, the second sample spatiotemporal feature, and the second sample scene feature all correspond to the sample weak temporal dependency behavior label;

[0018] Inputting the first sample time series feature, the first sample spatiotemporal feature, the first sample scene feature, the second sample time series feature, the second sample spatiotemporal feature, and the second sample scene feature into a convolutional neural network to obtain a first convolutional feature map and a second convolutional feature map;

[0019] Generate a fusion weighted representation of the first convolutional feature map and the second convolutional feature map using a channel and spatiotemporal interest point attention model;

[0020]

[0021]

[0022] in, is the fusion weight representation of the first convolution feature map and the second convolution feature map; is the sigmoid activation function; is the ReLu activation function; To reduce the ratio to The weight matrix of the channel downsampling convolutional layer; To enlarge the ratio The weight matrix of the channel upsampling convolutional layer; is the first convolution feature map; is the second convolution feature map;

[0023] Performing a product operation on the first convolution feature map, the second convolution feature map, and the fusion weight representation according to channels to obtain a channel attention weighted feature map;

[0024] The channel attention weighted feature map is used to optimize and adjust the network parameters of the convolutional neural network to obtain the behavior classification model.

[0025] Furthermore, before determining the first abnormal behavior recognition model or the second abnormal behavior recognition model as the target recognition model according to the behavior classification result, the method further includes:

[0026] defining a first behavior identification method corresponding to the strong timing-dependent behavior type and a second behavior identification method corresponding to the weak timing-dependent behavior type;

[0027] The first abnormal behavior recognition model is determined based on the following steps:

[0028] Get samples of behavior fragments with strong timing dependency;

[0029] Dividing the strongly time-dependent behavior segment sample into a plurality of segments of equal duration, and randomly sampling a frame from each segment to obtain an input sequence having a plurality of frames;

[0030] Inputting the features corresponding to the input sequence having a plurality of frames into the CTTransformer model, and outputting the spatiotemporal information change features of the behavior state;

[0031] Constructing a multi-head time-domain adjacency matrix with time-series dependency based on the state spatiotemporal information change characteristics;

[0032] Constructing a three-dimensional convolutional neural network model according to the multi-head time-domain adjacency matrix;

[0033] The parameters of the three-dimensional convolutional neural network model are optimized based on cross entropy loss and smooth L1 loss to obtain the first abnormal behavior recognition model.

[0034] Furthermore, the three-dimensional convolutional neural network model is optimized based on cross entropy loss and smooth L1 loss, including:

[0035] Based on the loss function Optimizing parameters of the three-dimensional convolutional neural network model;

[0036]

[0037] in, is the true probability distribution; is the predicted probability distribution; is the smoothed L1 loss.

[0038] Furthermore, before the cruise ship server collects surveillance videos of various areas on the cruise ship through the front-end camera, the method further includes:

[0039] Based on the camera dimension and the area dimension, batch addition operations are performed on the camera configuration algorithm, the video stream access and processing algorithm, and the monitoring area division algorithm; wherein the camera configuration algorithm includes an automatic exposure algorithm, an automatic white balance algorithm, and an optical zoom and digital zoom algorithm.

[0040] Furthermore, the abnormal behaviors include fighting, climbing, jumping into the sea, gathering, and crew members’ improper dressing behavior;

[0041] When the cruise ship server identifies the abnormal behavior, the cruise ship server sends an alarm message to the shore server and synchronizes the abnormal behavior identification result to the shore server in real time;

[0042] The alarm message carries the location frame and timing information of the abnormal behavior.

[0043] The present application provides a cruise ship safety video monitoring system based on video classification, including a front-end video acquisition unit, a monitoring video processing unit, a video stream restoration unit, a real-time video stream transmission unit, and a behavior classification and abnormality recognition unit; wherein,

[0044] The front-end video acquisition unit is used for the cruise ship server to collect monitoring videos of various areas on the cruise ship through the front-end camera;

[0045] The monitoring video processing unit is used for the cruise ship server to perform image enhancement and defogging on the monitoring videos of each area to obtain a real-time video stream;

[0046] The video stream restoration unit is used for the cruise server to restore the real-time video stream to high definition using a video perception algorithm and a background reconstruction algorithm;

[0047] The real-time video stream transmission unit is used for the cruise ship server to transmit the high-definition restored real-time video stream to the shore server based on the real-time video encoding parameters; wherein the real-time video encoding parameters are adjusted by real-time analysis of the high-definition restored real-time video stream and real-time network conditions; the cruise ship server receives the command and dispatch instructions of the shore server in real time;

[0048] The behavior classification and abnormality identification unit is used for the cruise ship server to classify various behaviors in the real-time video stream restored by high-definition using the timing characteristics, spatiotemporal characteristics and scene characteristics of the real-time video stream restored by high-definition, and to identify abnormal behaviors in the real-time video stream restored by high-definition. Furthermore, before the cruise ship server collects surveillance videos of various areas on the cruise ship through the front-end camera, it also includes:

[0049] Based on the camera dimension and the area dimension, batch addition operations are performed on the camera configuration algorithm, the video stream access and processing algorithm, and the monitoring area division algorithm; wherein the camera configuration algorithm includes an automatic exposure algorithm, an automatic white balance algorithm, and an optical zoom and digital zoom algorithm.

[0050] Furthermore, the abnormal behaviors include fighting, climbing, jumping into the sea, gathering, and crew members’ improper dressing behavior;

[0051] When the cruise ship server identifies the abnormal behavior, the cruise ship server sends an alarm message to the shore server and synchronizes the abnormal behavior identification result to the shore server in real time;

[0052] Wherein, the alarm message carries the location frame and timing information of the abnormal behavior. Based on the technical solution provided by the present application, the cruise end server collects the monitoring video of each area on the cruise ship through the front-end camera; the cruise end server performs picture enhancement and defogging on the monitoring video of each area to obtain a real-time video stream; the cruise end server uses the video perception algorithm and the background reconstruction algorithm to restore the real-time video stream in high definition; the cruise end server transmits the real-time video stream restored in high definition to the shore end server based on the real-time video encoding parameters; wherein, by real-time analysis of the real-time video stream restored in high definition and the real-time network conditions, the real-time video encoding parameters are adjusted; the cruise end server receives the command and dispatch instructions of the shore end server in real time; the cruise end server uses the timing characteristics, spatiotemporal characteristics and scene characteristics of the real-time video stream restored in high definition to classify various behaviors in the real-time video stream restored in high definition, and to identify abnormal behaviors in the real-time video stream restored in high definition. Thereby, the safety of the cruise ship is automatically monitored, and the efficiency of identifying abnormal behaviors on the cruise ship is improved.

[0053] The present invention has the following beneficial effects: the video collected by the front-end camera is enhanced and defogged to improve the clarity and visibility of the monitoring screen, making the monitoring video clearer and facilitating the observation of details; based on the adjustment of real-time video encoding parameters, the transmission of the video stream can be dynamically optimized according to the network conditions, reducing the video freeze and delay caused by network fluctuations, and ensuring the smoothness of the video stream; real-time analysis of the video stream and network conditions can reasonably allocate network bandwidth resources, ensure the transmission of high-quality video streams under limited bandwidth, and improve network utilization efficiency; the cruise-side server uses the video perception algorithm and the background reconstruction algorithm to perform high-definition restoration of the video stream, which can further improve the quality and readability of the video content and make the video analysis more accurate; the temporal characteristics, spatiotemporal characteristics and scene characteristics of the video stream are used to classify the behavior The classification and abnormal behavior identification of behaviors in real-time video streams can improve the intelligence level of the monitoring system and detect potential security threats in a timely manner; the classification of behaviors and abnormal behavior identification in real-time video streams can help to detect and respond to emergencies in a timely manner, and improve the safety monitoring capabilities of cruise ships; through real-time monitoring and rapid identification of abnormal behaviors, the response to emergencies can be accelerated and the efficiency of handling emergencies can be improved; automated video analysis reduces reliance on manual monitoring, reduces labor costs, and allows monitoring personnel to focus on events that require more judgment and intervention; through intelligent analysis of passenger behavior and cruise operations, more personalized and efficient services can be provided, improving passenger satisfaction and the overall service quality of cruise ships; recording and analyzing video data can facilitate long-term safety trend analysis and risk assessment, and provide data support for cruise safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0055] Figure 1 is a flowchart of an optional cruise ship safety video monitoring method based on video classification according to an embodiment of the present application;

[0056] Figure 2 The structure diagram of an optional cruise ship security video surveillance system based on video classification according to an embodiment of the present application.

[0057] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0059] Alternatively, if Figure 1 As shown, the present application provides a cruise ship security video monitoring method based on video classification, comprising:

[0060] S101, the cruise ship server collects surveillance videos of various areas on the cruise ship through the front-end camera;

[0061] S102, the cruise ship server performs image enhancement and defogging processing on the surveillance videos of each area to obtain a real-time video stream;

[0062] S103, the cruise server uses a video perception algorithm and a background reconstruction algorithm to restore the real-time video stream to high definition;

[0063] S104, the cruise ship server transmits the high-definition restored real-time video stream to the shore server based on the real-time video encoding parameters; wherein the real-time video encoding parameters are adjusted by real-time analysis of the high-definition restored real-time video stream and real-time network conditions; the cruise ship server receives the command and dispatch instructions of the shore server in real time;

[0064] S105, the cruise server uses the timing characteristics, spatiotemporal characteristics and scene characteristics of the high-definition restored real-time video stream to classify various behaviors in the high-definition restored real-time video stream, and identifies abnormal behaviors in the high-definition restored real-time video stream.

[0065] Based on the technical solution provided by this application, the cruise ship server collects surveillance videos of various areas on the cruise ship through the front-end camera; the cruise ship server performs picture enhancement and defogging on the surveillance videos of various areas to obtain real-time video streams; the cruise ship server uses video perception algorithm and background reconstruction algorithm to restore the real-time video stream to high definition; the cruise ship server transmits the real-time video stream restored to the shore server based on the real-time video encoding parameters; wherein, the real-time video encoding parameters are adjusted by real-time analysis of the real-time video stream restored to high definition and the real-time network conditions; the cruise ship server receives the command and dispatch instructions of the shore server in real time; the cruise ship server uses the time sequence characteristics, spatiotemporal characteristics and scene characteristics of the real-time video stream restored to high definition to classify various behaviors in the real-time video stream restored to high definition, and to identify abnormal behaviors in the real-time video stream restored to high definition. Thus, the safety of the cruise ship is automatically monitored, and the efficiency of identifying abnormal behaviors on the cruise ship is improved.

[0066] The present invention has the following beneficial effects: the video collected by the front-end camera is enhanced and defogged to improve the clarity and visibility of the monitoring screen, making the monitoring video clearer and facilitating the observation of details; based on the adjustment of real-time video encoding parameters, the transmission of the video stream can be dynamically optimized according to the network conditions, reducing the video freeze and delay caused by network fluctuations, and ensuring the smoothness of the video stream; real-time analysis of the video stream and network conditions can reasonably allocate network bandwidth resources, ensure the transmission of high-quality video streams under limited bandwidth, and improve network utilization efficiency; the cruise-side server uses the video perception algorithm and the background reconstruction algorithm to perform high-definition restoration of the video stream, which can further improve the quality and readability of the video content and make the video analysis more accurate; the temporal characteristics, spatiotemporal characteristics and scene characteristics of the video stream are used to classify the behavior The classification and abnormal behavior identification of behaviors in real-time video streams can improve the intelligence level of the monitoring system and detect potential security threats in a timely manner; the classification of behaviors and abnormal behavior identification in real-time video streams can help to detect and respond to emergencies in a timely manner, and improve the safety monitoring capabilities of cruise ships; through real-time monitoring and rapid identification of abnormal behaviors, the response to emergencies can be accelerated and the efficiency of handling emergencies can be improved; automated video analysis reduces reliance on manual monitoring, reduces labor costs, and allows monitoring personnel to focus on events that require more judgment and intervention; through intelligent analysis of passenger behavior and cruise operations, more personalized and efficient services can be provided, improving passenger satisfaction and the overall service quality of cruise ships; recording and analyzing video data can facilitate long-term safety trend analysis and risk assessment, and provide data support for cruise safety management.

[0067] Furthermore, the timing features, spatiotemporal features and scene features of the high-definition restored real-time video stream are used to classify various behaviors in the high-definition restored real-time video stream, and to identify abnormal behaviors in the high-definition restored real-time video stream, including:

[0068] Extract temporal features, spatiotemporal features and scene features from the real-time video stream restored with high definition;

[0069] Inputting the temporal features, spatiotemporal features and scene features into the behavior classification model to obtain the behavior classification result; wherein the behavior classification model is trained and optimized based on a deep learning algorithm; the behavior classification result is a strong temporal dependency behavior type or a weak temporal dependency behavior type;

[0070] According to the behavior classification result, the first abnormal behavior recognition model or the second abnormal behavior recognition model is determined as the target recognition model; wherein the first abnormal behavior recognition model corresponds to the strong timing-dependent behavior type; and the second abnormal behavior recognition model corresponds to the weak timing-dependent behavior type;

[0071] The high-definition restored real-time video stream is input into the target recognition model to identify abnormal behavior.

[0072] Based on the embodiments provided in the present application, real-time analysis of video streams and behavior classification can quickly respond to dynamic changes in the video and identify abnormal behaviors in a timely manner; the model optimized by deep learning algorithm training has good generalization ability and can adapt to different video scenes and conditions; by distinguishing between strong timing-dependent behaviors and weak timing-dependent behaviors, different processing strategies can be adopted for different types of behaviors to improve recognition efficiency; selecting the corresponding abnormal behavior recognition model based on the behavior classification results can perform more accurate anomaly detection for specific types of behaviors; quickly determining the target recognition model and inputting the real-time video stream can shorten the time from detection to response and improve the response speed of the system; accurate behavior classification and targeted abnormal behavior recognition can reduce false alarms and missed alarms and improve the reliability of the monitoring system; automated behavior classification and abnormal behavior recognition reduce the need for manual intervention and improve the automation level of the monitoring system; timely identification of abnormal behavior helps prevent potential security issues and improve the safety of the environment; providing accurate recognition results provides valuable information for security personnel and decision makers to support them in making better decisions.

[0073] Furthermore, the behavior classification model is trained and optimized based on the following steps:

[0074] Obtaining a sample video stream including a sample strong temporal dependency behavior label and a sample weak temporal dependency behavior label;

[0075] Extracting the first sample temporal feature, the first sample spatiotemporal feature, the first sample scene feature, the second sample temporal feature, the second sample spatiotemporal feature, and the second sample scene feature of the sample video stream; wherein the first sample temporal feature, the first sample spatiotemporal feature, and the first sample scene feature all correspond to the sample strong temporal dependency behavior label; the second sample temporal feature, the second sample spatiotemporal feature, and the second sample scene feature all correspond to the sample weak temporal dependency behavior label;

[0076] Inputting the first sample time series feature, the first sample spatiotemporal feature, the first sample scene feature, the second sample time series feature, the second sample spatiotemporal feature, and the second sample scene feature into a convolutional neural network to obtain a first convolutional feature map and a second convolutional feature map;

[0077] Generate a fusion weight representation of the first convolutional feature map and the second convolutional feature map using the channel and spatiotemporal interest point attention model;

[0078]

[0079]

[0080] in, It is the fusion weight representation of the first convolution feature map and the second convolution feature map; is the sigmoid activation function; is the ReLu activation function; To reduce the ratio to The weight matrix of the channel downsampling convolutional layer; To enlarge the ratio The weight matrix of the channel upsampling convolutional layer; is the first convolution feature map; is the second convolution feature map;

[0081] The first convolution feature map, the second convolution feature map and the fusion weight representation are multiplied according to the channel to obtain a channel attention weighted feature map;

[0082] The channel attention weighted feature map is used to optimize and adjust the network parameters of the convolutional neural network to obtain a behavior classification model.

[0083] Furthermore, before determining the first abnormal behavior recognition model or the second abnormal behavior recognition model as the target recognition model according to the behavior classification result, the method further includes:

[0084] Define a first behavior identification method corresponding to a strong timing-dependent behavior type and a second behavior identification method corresponding to a weak timing-dependent behavior type;

[0085] Determine the first abnormal behavior recognition model based on the following steps:

[0086] Get samples of behavior fragments with strong timing dependency;

[0087] Divide the strongly temporally dependent behavior segment samples into a number of segments of equal duration, and randomly sample a frame from each segment to obtain an input sequence with a number of frames;

[0088] Input the features corresponding to the input sequence with several frames into the CTTransformer model, and output the spatiotemporal information change features of the behavior state;

[0089] Based on the state spatiotemporal information change characteristics, a multi-head time domain adjacency matrix with time series dependency is constructed;

[0090] Construct a three-dimensional convolutional neural network model based on the multi-head time-domain adjacency matrix;

[0091] The parameters of the three-dimensional convolutional neural network model were optimized based on cross entropy loss and smooth L1 loss to obtain the first abnormal behavior recognition model.

[0092] Furthermore, the parameters of the 3D convolutional neural network model are optimized based on the cross entropy loss and the smooth L1 loss, including:

[0093] Based on the loss function Optimize parameters of 3D convolutional neural network models;

[0094]

[0095] in, is the true probability distribution; is the predicted probability distribution; is the smoothed L1 loss.

[0096] Furthermore, before the cruise ship server collects surveillance videos of various areas on the cruise ship through the front-end camera, the method also includes:

[0097] Based on the camera dimension and the area dimension, batch addition operations are performed on the camera configuration algorithm, the video stream access and processing algorithm, and the monitoring area division algorithm; among them, the camera configuration algorithm includes the automatic exposure algorithm, the automatic white balance algorithm, and the optical zoom and digital zoom algorithms.

[0098] Furthermore, abnormal behaviors include fighting, climbing, jumping into the sea, gathering, and crew members not wearing the correct attire;

[0099] When the cruise server identifies abnormal behavior, it sends an alarm message to the shore server and synchronizes the abnormal behavior identification results to the shore server in real time;

[0100] The alarm message carries the location frame and timing information of the abnormal behavior.

[0101] Alternatively, if Figure 2 As shown, the present application provides a cruise ship safety video monitoring system based on video classification, including a front-end video acquisition unit 201, a monitoring video processing unit 202, a video stream restoration unit 203, a real-time video stream transmission unit 204, and a behavior classification and abnormality recognition unit 205; wherein,

[0102] The front-end video acquisition unit 201 is used for the cruise ship server to collect monitoring videos of various areas on the cruise ship through the front-end camera;

[0103] The monitoring video processing unit 202 is used for the cruise ship server to perform image enhancement and defogging on the monitoring video of each area to obtain a real-time video stream;

[0104] The video stream restoration unit 203 is used for the cruise server to restore the real-time video stream to high definition by using the video perception algorithm and the background reconstruction algorithm;

[0105] The real-time video stream transmission unit 204 is used for the cruise ship server to transmit the high-definition restored real-time video stream to the shore server based on the real-time video encoding parameters; wherein the real-time video encoding parameters are adjusted by real-time analysis of the high-definition restored real-time video stream and the real-time network conditions; the cruise ship server receives the command and dispatch instructions of the shore server in real time;

[0106] The behavior classification and abnormality identification unit 205 is used by the cruise server to classify various behaviors in the real-time video stream restored by high-definition using the timing characteristics, spatiotemporal characteristics and scene characteristics of the real-time video stream restored by high-definition, and to identify abnormal behaviors in the real-time video stream restored by high-definition.

[0107] Furthermore, before the cruise ship server collects surveillance videos of various areas on the cruise ship through the front-end camera, the method also includes:

[0108] Based on the camera dimension and the area dimension, batch addition operations are performed on the camera configuration algorithm, the video stream access and processing algorithm, and the monitoring area division algorithm; among them, the camera configuration algorithm includes the automatic exposure algorithm, the automatic white balance algorithm, and the optical zoom and digital zoom algorithms.

[0109] Furthermore, abnormal behaviors include fighting, climbing, jumping into the sea, gathering, and crew members not wearing the correct attire;

[0110] When the cruise server identifies abnormal behavior, it sends an alarm message to the shore server and synchronizes the abnormal behavior identification results to the shore server in real time;

[0111] The alarm message carries the location frame and timing information of the abnormal behavior.

[0112] It should be noted that in the present application, the embodiments implemented by the cruise ship safety video surveillance system based on video classification can be cross-referenced with the embodiments implemented by the cruise ship safety video surveillance method based on video classification, and the present application will not repeat them one by one.

[0113] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A cruise ship safety video monitoring method based on video classification, characterized in that: include: The cruise ship server collects surveillance videos of various areas on the cruise ship through front-end cameras; The cruise ship server performs image enhancement and defogging processing on the surveillance videos of each area to obtain a real-time video stream; The cruise ship server uses a video perception algorithm and a background reconstruction algorithm to restore the real-time video stream to high definition; The cruise ship server transmits the high-definition restored real-time video stream to the shore server based on the real-time video encoding parameters; wherein the real-time video encoding parameters are adjusted by real-time analysis of the high-definition restored real-time video stream and real-time network conditions; the cruise ship server receives the command and dispatch instructions of the shore server in real time; The cruise ship server uses the timing characteristics, spatiotemporal characteristics and scene characteristics of the high-definition restored real-time video stream to classify various behaviors in the high-definition restored real-time video stream, and identifies abnormal behaviors in the high-definition restored real-time video stream; The method of utilizing the timing features, spatiotemporal features and scene features of the high-definition restored real-time video stream to classify various behaviors in the high-definition restored real-time video stream and to identify abnormal behaviors in the high-definition restored real-time video stream includes: Extracting the temporal features, the spatiotemporal features and the scene features from the high-definition restored real-time video stream; Inputting the temporal features, the spatiotemporal features and the scene features into a behavior classification model to obtain a behavior classification result; wherein the behavior classification model is trained and optimized based on a deep learning algorithm; and the behavior classification result is a strong temporal dependency behavior type or a weak temporal dependency behavior type; According to the behavior classification result, determining the first abnormal behavior recognition model or the second abnormal behavior recognition model as the target recognition model; wherein the first abnormal behavior recognition model corresponds to the strong timing-dependent behavior type; and the second abnormal behavior recognition model corresponds to the weak timing-dependent behavior type; Inputting the high-definition restored real-time video stream into the target recognition model to identify the abnormal behavior; The behavior classification model is trained and optimized based on the following steps: Obtaining a sample video stream including a sample strong temporal dependency behavior label and a sample weak temporal dependency behavior label; Extracting a first sample temporal feature, a first sample spatiotemporal feature, a first sample scene feature, a second sample temporal feature, a second sample spatiotemporal feature, and a second sample scene feature of the sample video stream; wherein the first sample temporal feature, the first sample spatiotemporal feature, and the first sample scene feature all correspond to the sample strong temporal dependency behavior label; the second sample temporal feature, the second sample spatiotemporal feature, and the second sample scene feature all correspond to the sample weak temporal dependency behavior label; Inputting the first sample time series feature, the first sample spatiotemporal feature, the first sample scene feature, the second sample time series feature, the second sample spatiotemporal feature, and the second sample scene feature into a convolutional neural network to obtain a first convolutional feature map and a second convolutional feature map; Generate a fusion weighted representation of the first convolutional feature map and the second convolutional feature map using a channel and spatiotemporal interest point attention model; ； ； in, is the fusion weight representation of the first convolution feature map and the second convolution feature map; is the sigmoid activation function; is the ReLu activation function; To reduce the ratio to The weight matrix of the channel downsampling convolutional layer; To enlarge the ratio The weight matrix of the channel upsampling convolutional layer; is the first convolution feature map; is the second convolution feature map; Performing a product operation on the first convolution feature map, the second convolution feature map, and the fusion weight representation according to channels to obtain a channel attention weighted feature map; Optimizing and adjusting the network parameters of the convolutional neural network using the channel attention weighted feature map to obtain the behavior classification model; Wherein, before determining the first abnormal behavior recognition model or the second abnormal behavior recognition model as the target recognition model according to the behavior classification result, the method further includes: defining a first behavior identification method corresponding to the strong timing-dependent behavior type and a second behavior identification method corresponding to the weak timing-dependent behavior type; The first abnormal behavior recognition model is determined based on the following steps: Get samples of behavior fragments with strong timing dependency; Dividing the strongly time-dependent behavior segment sample into a plurality of segments of equal duration, and randomly sampling a frame from each segment to obtain an input sequence having a plurality of frames; Inputting the features corresponding to the input sequence having a plurality of frames into the CTTransformer model, and outputting the spatiotemporal information change features of the behavior state; Constructing a multi-head time-domain adjacency matrix with time-series dependency based on the state spatiotemporal information change characteristics; Constructing a three-dimensional convolutional neural network model according to the multi-head time-domain adjacency matrix; The parameters of the three-dimensional convolutional neural network model are optimized based on cross entropy loss and smooth L1 loss to obtain the first abnormal behavior recognition model.

2. The cruise ship safety video monitoring method based on video classification according to claim 1 is characterized in that: The method of performing parameter optimization on the three-dimensional convolutional neural network model based on cross entropy loss and smooth L1 loss includes: Based on the loss function Optimizing parameters of the three-dimensional convolutional neural network model; ； in, is the true probability distribution; is the predicted probability distribution; is the smoothed L1 loss.

3. The cruise ship safety video monitoring method based on video classification according to claim 1 is characterized in that: Before the cruise ship server collects surveillance videos of various areas on the cruise ship through the front-end camera, the method further includes: Based on the camera dimension and the area dimension, batch addition operations are performed on the camera configuration algorithm, the video stream access and processing algorithm, and the monitoring area division algorithm; wherein the camera configuration algorithm includes an automatic exposure algorithm, an automatic white balance algorithm, and an optical zoom and digital zoom algorithm.

4. The cruise ship safety video monitoring method based on video classification according to claim 1 is characterized in that: The abnormal behaviors include fighting, climbing, jumping into the sea, gathering, and crew members not wearing the correct attire; When the cruise ship server identifies the abnormal behavior, the cruise ship server sends an alarm message to the shore server and synchronizes the abnormal behavior identification result to the shore server in real time; The alarm message carries the location frame and timing information of the abnormal behavior.

5. A cruise ship security video monitoring system based on video classification, the system implementing the method according to claim 1, characterized in that: It includes a front-end video acquisition unit, a monitoring video processing unit, a video stream restoration unit, a real-time video stream transmission unit, and a behavior classification and abnormality recognition unit; wherein, The front-end video acquisition unit is used for the cruise ship server to collect monitoring videos of various areas on the cruise ship through the front-end camera; The monitoring video processing unit is used for the cruise ship server to perform image enhancement and defogging on the monitoring videos of each area to obtain a real-time video stream; The video stream restoration unit is used for the cruise server to restore the real-time video stream to high definition using a video perception algorithm and a background reconstruction algorithm; The real-time video stream transmission unit is used for the cruise ship server to transmit the high-definition restored real-time video stream to the shore server based on the real-time video encoding parameters; wherein the real-time video encoding parameters are adjusted by real-time analysis of the high-definition restored real-time video stream and real-time network conditions; the cruise ship server receives the command and dispatch instructions of the shore server in real time; The behavior classification and abnormality identification unit is used for the cruise server to classify various behaviors in the high-definition restored real-time video stream by using the time sequence characteristics, spatiotemporal characteristics and scene characteristics of the high-definition restored real-time video stream, and to identify abnormal behaviors in the high-definition restored real-time video stream; The method of utilizing the timing features, spatiotemporal features and scene features of the high-definition restored real-time video stream to classify various behaviors in the high-definition restored real-time video stream and to identify abnormal behaviors in the high-definition restored real-time video stream includes: Extracting the temporal features, the spatiotemporal features and the scene features from the high-definition restored real-time video stream; Inputting the temporal features, the spatiotemporal features and the scene features into a behavior classification model to obtain a behavior classification result; wherein the behavior classification model is trained and optimized based on a deep learning algorithm; and the behavior classification result is a strong temporal dependency behavior type or a weak temporal dependency behavior type; According to the behavior classification result, determining the first abnormal behavior recognition model or the second abnormal behavior recognition model as the target recognition model; wherein the first abnormal behavior recognition model corresponds to the strong timing-dependent behavior type; and the second abnormal behavior recognition model corresponds to the weak timing-dependent behavior type; Inputting the high-definition restored real-time video stream into the target recognition model to identify the abnormal behavior; The behavior classification model is trained and optimized based on the following steps: Obtaining a sample video stream including a sample strong temporal dependency behavior label and a sample weak temporal dependency behavior label; Extracting a first sample temporal feature, a first sample spatiotemporal feature, a first sample scene feature, a second sample temporal feature, a second sample spatiotemporal feature, and a second sample scene feature of the sample video stream; wherein the first sample temporal feature, the first sample spatiotemporal feature, and the first sample scene feature all correspond to the sample strong temporal dependency behavior label; the second sample temporal feature, the second sample spatiotemporal feature, and the second sample scene feature all correspond to the sample weak temporal dependency behavior label; Inputting the first sample time series feature, the first sample spatiotemporal feature, the first sample scene feature, the second sample time series feature, the second sample spatiotemporal feature, and the second sample scene feature into a convolutional neural network to obtain a first convolutional feature map and a second convolutional feature map; Generate a fusion weighted representation of the first convolutional feature map and the second convolutional feature map using a channel and spatiotemporal interest point attention model; ； ； in, is the fusion weight representation of the first convolution feature map and the second convolution feature map; is the sigmoid activation function; is the ReLu activation function; To reduce the ratio to The weight matrix of the channel downsampling convolutional layer; To enlarge the ratio The weight matrix of the channel upsampling convolutional layer; is the first convolution feature map; is the second convolution feature map; Performing a product operation on the first convolution feature map, the second convolution feature map, and the fusion weight representation according to channels to obtain a channel attention weighted feature map; Optimizing and adjusting the network parameters of the convolutional neural network using the channel attention weighted feature map to obtain the behavior classification model; Wherein, before determining the first abnormal behavior recognition model or the second abnormal behavior recognition model as the target recognition model according to the behavior classification result, a first behavior recognition method corresponding to the strong timing dependent behavior type and a second behavior recognition method corresponding to the weak timing dependent behavior type are defined; the first abnormal behavior recognition model is determined based on the following steps: obtaining a sample of strongly timing dependent behavior segments; dividing the sample of strongly timing dependent behavior segments into a plurality of segments of equal length, and randomly sampling a frame from each segment to obtain an input sequence having a plurality of frames; inputting the features corresponding to the input sequence having a plurality of frames into the CTTransformer model, and outputting the characteristics of the spatiotemporal information change of the behavior state; constructing a multi-head time domain adjacency matrix with time dependency based on the state spatiotemporal information change characteristics; constructing a three-dimensional convolutional neural network model based on the multi-head time domain adjacency matrix; and optimizing the parameters of the three-dimensional convolutional neural network model based on the cross entropy loss and the smooth L1 loss to obtain the first abnormal behavior recognition model.

6. The cruise ship security video monitoring system based on video classification according to claim 5 is characterized in that: Before the cruise ship server collects surveillance videos of various areas on the cruise ship through the front-end camera, it also includes: Based on the camera dimension and the area dimension, batch addition operations are performed on the camera configuration algorithm, the video stream access and processing algorithm, and the monitoring area division algorithm; wherein the camera configuration algorithm includes an automatic exposure algorithm, an automatic white balance algorithm, and an optical zoom and digital zoom algorithm.

7. The cruise ship security video monitoring system based on video classification according to claim 5 is characterized in that: The abnormal behaviors include fighting, climbing, jumping into the sea, gathering, and crew members not wearing the correct attire; When the cruise ship server identifies the abnormal behavior, the cruise ship server sends an alarm message to the shore server and synchronizes the abnormal behavior identification result to the shore server in real time; The alarm message carries the location frame and timing information of the abnormal behavior.

Images