Unattended ship monitoring method and system
By using feature extraction and cross-modal correlation learning modules in the cabin monitoring system, combined with the YOLOv4 model and Fourier transform, the problem of insufficient precise positioning and early warning capabilities of cabin water inlet monitoring in the existing technology is solved, and efficient , accurate water inlet detection and early warning.
Patent Information
- Application Number
- CN202411072658.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-12
AI Technical Summary
The existing technology lacks the ability to accurately locate leakage locations and early warning capabilities in cabin water inlet monitoring. In complex environments, sensor data is easily misleading and has a high false alarm rate. Image monitoring cannot effectively identify subtle water level changes and seepage. The signs of missing are not formed, and a comprehensive judgment cannot be formed.
An unattended ship monitoring method is adopted to extract image features and signal features through feature extraction module, and input them into the cross-modal correlation learning module through feature fusion module, combining YOLOv4 model and Fourier transform to perform image features Multimodal correlation learning and optimization with signal characteristics enhance the ability to judge water inlet in the cabin.
Realize timely and accurate monitoring of water inlet in the cabin, reduce the false alarm rate, improve the reliability and adaptability of the monitoring system, and can quickly identify the water inlet location and estimate the water inlet volume in complex environments, providing key emergency response information .
Smart Images

Figure CN118942039A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of engine room monitoring, and in particular relates to an unmanned ship monitoring method and system. Background Art
[0002] With the increasing volume of international shipping business and the complexity of the maritime operating environment, ship safety management has become a key link in ensuring the smooth flow of global trade. Among them, cabin flooding incidents pose a major threat to ship safety due to their suddenness, concealment and harmfulness. Traditional monitoring methods mainly rely on manual inspections and basic sensor alarms. These methods are limited by the limitations of human resources and the insensitivity of technology. It is often difficult to detect and accurately judge small cabin leaks in a timely manner, especially under unattended or extreme weather conditions. The monitoring efficiency and accuracy are greatly reduced.
[0003] In recent years, with the rapid development of artificial intelligence and Internet of Things technologies, intelligent ship monitoring systems have gradually emerged, but the application of existing technologies in monitoring cabin water ingress is still immature. Most systems only achieve direct measurement of water levels, lack the ability to accurately locate leakage locations and provide early warnings, and under the interference of complex and changing environmental factors, sensor data is easily misled and has a high false alarm rate. In addition, although image monitoring has been used for ship safety monitoring, image analysis alone often cannot effectively identify subtle water level changes and signs of leakage, and is not effectively integrated with sensor data to form a comprehensive judgment.
[0004] The patent application number is CN206638306U, and the towed ship cargo hold water ingress alarm monitoring system includes a towing ship side subsystem and a towed ship side subsystem. The towed ship side subsystem includes a switch output type liquid level gauge, a wireless transmitting unit connected to the output of the liquid level gauge, and the towing ship side subsystem includes a receiver communicating with the wireless transmitting unit, and an alarm controllably connected to the receiver. The above device monitors whether the cabin is flooded through the liquid level gauge and the alarm, and the monitoring efficiency is low. Therefore, it is urgent for those skilled in the art to solve the above technical problems. Summary of the invention
[0005] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide an unmanned ship monitoring method and system. The unmanned ship monitoring method and system extract image features and signal features through a feature extraction module, fuse the image features and signal features through a feature fusion module and input them into a cross-modal association learning module, and enhance the model's ability to judge whether the cabin is flooded through the cross-modal association learning module.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0007] An unmanned ship monitoring method is used to timely monitor water ingress into the engine room under extreme conditions of the ship, the method comprising:
[0008] S1: Establish a ship monitoring model including feature extraction module, feature fusion module and cross-modal association learning module;
[0009] S2: extracting the ship image features and the signal features measured by the sensor through the feature extraction module;
[0010] S3: importing image features and signal features into feature fusion module;
[0011] S4: Strengthen the fusion of image features and signal features through cross-modal association learning module, and in the feedback process
[0012] Online learning and reinforcement learning are used to optimize the model;
[0013] S5: Get the monitoring results.
[0014] Furthermore, in step S2, the ship image features and the signal features measured by the sensor are extracted by a feature extraction module, specifically:
[0015] S21: Install surveillance cameras and cabin water ingress alarm sensors in the ship cabin;
[0016] S211: The image obtained by the surveillance camera of the YOLOv4 model is resized to match the input requirement of the YOLOv4 model;
[0017] The YOLOv4 model includes input layer, Backbone network, Neck network, PANet and output layer;
[0018] The input layer preprocesses the image by normalization;
[0019] The Backbone network includes a residual structure, a 3×3 convolution layer and a 1×1 pooling layer. The residual structure, the 3×3 convolution layer and the 1×1 pooling layer are used to reduce computational redundancy and improve network efficiency. Image features are processed in two ways, one part is subjected to conventional convolution, and the other part is subjected to a large number of dimensionality reduction and dimensionality increase operations, and then merged to share features and reduce the amount of computation.
[0020] The PANetPANet achieves efficient integration of multi-scale image features through image feature fusion of top-down and bottom-up paths;
[0021] The output layer outputs an image feature vector;
[0022] S212: The signal characteristics detected by the cabin water ingress alarm sensor are converted into frequency domain data using Fourier transform within a fixed time window to extract frequency characteristics.
[0023] YOLOv4 is selected as the core of image processing. It is an efficient object detection model that is particularly suitable for real-time monitoring scenarios. The image size is adjusted and normalized to ensure the consistency and effectiveness of the model input. The residual structure is used to reduce the gradient vanishing problem. The 3×3 convolution layer and 1×1 pooling layer are used to balance the computational efficiency and feature expression ability. Residual connections allow deeper network layers to learn more advanced features without worrying about gradient propagation problems. At the same time, through dimensionality reduction and dimensionality increase operations, the model can maintain rich feature expressions at a lower computational cost. Although the Neck part is not described in detail here, it usually involves feature pyramid networks (FPN) or similar mechanisms in YOLOv4, which aims to integrate features of different scales to improve detection accuracy. Through top-down and bottom-up path fusion strategies, effective combination of features of different scales is achieved, which is crucial for detecting targets of different sizes, especially in complex and changing cabin environments. The final output image feature vector contains rich information about potential signs of water ingress in the image, such as water level position, flow morphology, etc. The sensor monitors in real time. Once water enters, the relevant signals are captured immediately. The sensor signals within a fixed time window are converted to frequency, and frequency features are mined from the time domain data. This conversion helps to identify specific frequencies or frequency bands related to water ingress, because vibrations, pressure changes, etc. caused by water ingress may present characteristic peaks or patterns on the spectrum, making the signal features more obvious and easier to analyze. The combination of visual and signal data ensures multi-angle monitoring of water ingress events and improves the reliability of the monitoring system. The high efficiency of YOLOv4 supports real-time image analysis, while Fourier transform quickly identifies signal features. The combination of the two can quickly trigger an alarm. Multimodal feature fusion helps filter out false alarms not caused by water ingress, such as visual interference caused by normal fluctuations or light changes, and sensor misreading caused by environmental noise. Through multi-scale image features and frequency domain analysis, the water ingress location can be more accurately located and the amount of water ingress can be estimated, providing key information for emergency response.
[0024] Furthermore, in step S3, the image features and signal features are imported into the feature fusion module, specifically: the image feature vector is directly input into the feature fusion module;
[0025] The signal features monitored by the cabin water inflow alarm sensor use the Pearson correlation coefficient to select signal features, and use linear discriminant analysis to reduce the dimension of signal features. The feature vectors of each sample are arranged in rows to form a two-dimensional array or matrix. Each row represents a sample and each column represents a feature. The generated signal feature vector matrix is input into the feature fusion module.
[0026] The Pearson correlation coefficient measures the degree of linear correlation between two variables. This statistical method can be used to identify signal features that are highly correlated with water inflow events, eliminate irrelevant or weakly correlated noise features, and focus on the information that is most indicative of the water inflow status. Through screening, the number of unnecessary features in subsequent processing is reduced, the computational complexity is reduced, and the processing speed is accelerated. LDA is a supervised dimensionality reduction technique that aims to find a new feature space that can maximize the distance between categories while minimizing the distance within categories. Applied to the characteristics of water inflow signals, it helps to extract the key features that can best distinguish different water inflow conditions (such as normal state and water inflow state), reduce the feature dimensions, not only simplify the model complexity, but also avoid the "dimensionality disaster", making model training more efficient, and also improving the model's interpretability.
[0027] The construction and fusion of the feature vector matrix integrates the image features and the signal features after selection and dimensionality reduction into a unified feature matrix, which is convenient for subsequent analysis and model processing. This organizational form facilitates the algorithm to understand and utilize the complementarity between information from different sources. The image features provide intuitive visual information, such as water level height, water flow pattern, etc.; the signal features reveal the physical changes caused by water ingress, such as vibration frequency, pressure fluctuations, etc. The fusion of the two realizes the complementarity of multimodal information and significantly improves the accuracy and robustness of the detection system. By integrating multiple types of information, the system can capture the characteristics of water ingress events from multiple dimensions and reduce the misjudgment that may be caused by a single detection method. In the face of complex cabin environments and changeable water ingress conditions, multi-feature fusion improves the adaptability and stability of the system. Feature selection and dimensionality reduction effectively reduce the burden of data processing, allowing the system to operate efficiently under limited computing resources. The high-quality feature fusion results provide a solid data foundation for subsequent automatic alarms or preventive measures, which helps to achieve rapid response and intelligent management.
[0028] Furthermore, in step S4, the fusion of image features and signal features is strengthened through a cross-modal association learning module, specifically, the recurrent neural network and the convolutional neural network are fused:
[0029] S41: First, use a recurrent neural network to process the time series data to obtain a series of hidden state vectors Hrnn=[h1,h2,...,hT], where T is the length of the sequence. At the same time, use a convolutional neural network to process the image data and extract the image feature vector Fcnn. Then, these two feature vectors are concatenated together to form a new feature vector as the input of the subsequent fully connected layer or other models.
[0030] S42: Assign weights to the outputs of the recurrent neural network and the convolutional neural network, and then perform weighted summation to dynamically adjust the contribution of the two according to task requirements, and set a weight vector α to adjust the weights of Hrnn and Fcnn;
[0031] This is achieved by designing a mathematical model based on Additive Attention:
[0032] The additive attention mechanism calculates the weighted sum of the query and key and then generates an attention score through a nonlinear activation function and a fully connected layer.
[0033] Q is the output from the RNN (Query), with a shape of (B, Tq, dk)(B, Tq, dk), where B is the batch size, Tq is the sequence length, dk is the feature dimension, V is the feature map of the CNN (Values), with a shape of (B, H, W, C), where H and W are the height and width respectively, and C is the number of channels;
[0034] First, convert V to the shape of (B, H*W, C) to interact with Q.
[0035] The mathematical expression of additive attention is as follows:
[0036] Calculate weights:
[0037] Transform V and Q respectively through the fully connected layer (weight matrices Wk and Wq):
[0038] TK=VW K T,Q′=QWq
[0039] Expand Q′ to (B, Tq, 1, H*W), broadcast and add it to K, and then use the tanh function to get the matching metric M:
[0040] M=tanh(Q′[B;Tq;1;H*W,None]+K[None,B;Tq;1;H*W]);
[0041] Calculate the attention score:
[0042] The attention weight α is calculated through another fully connected layer (weight matrix Wv) and the softmax function:
[0043] α=softmax(MWv);
[0044] Weighted sum:
[0045] Use the attention weight α to perform a weighted summation on the original V to get the output O:
[0046] O=α·V.
[0047] It promotes the effective fusion of different modal features, and through adaptive weight allocation and attention mechanism, it achieves accurate capture of key information, providing strong technical support for the cabin water ingress alarm system.
[0048] Furthermore, in step S4, the model is optimized through online learning and reinforcement learning during the feedback process, specifically:
[0049] The reinforcement learning is defined by a five-tuple through an MDP: S (state set), A (action set), P(s′, s, a) (state transition probability), R(s, a, s′) (reward function), γ (discount factor) imported into the state value function of the Bellman equation:
[0050] V(s)=∑a∈Aπ(a∣s)∑s′∈SP(s′∣s,a)[R(s,a,s′)+γV(s′)]
[0051] Among them, π(a|s) is the probability of taking action a in state s;
[0052] The update rule of the state value function is:
[0053] Q(st,at)←Q(st,at)+α[rt+γa′maxQ(st+1,a′)-Q(st,at)]
[0054] Among them, α is the learning rate and rt is the immediate reward;
[0055] The online learning combines Q-Learning and deep learning through DQN, uses a neural network to approximate the Q function, and reduces the bias and variance of the estimation through the experience replay buffer (Experience Replay) and the fixed target network (Target Network);
[0056] The core idea of the experience replay buffer is to store historical transition experiences, i.e., et = (st, at, rt, st+1), where st is the current state, at is the action taken, rt is the immediate reward, and st+1 is the next state after the action is taken. These experiences are stored in an experience pool (replay buffer), and the size of the experience pool (replaybuffer) is fixed.
[0057] In the learning step, instead of updating directly based on the most recent single experience, a mini-batch of data is randomly extracted from the experience pool for learning. Such a mini-batch update can be expressed as:
[0058] L(θi)=E(s,a,r,s′)~U(D)[(yi-Q(s,a;θi))2];
[0059] Among them, L(θi) loss function is used to update the network parameters θi, yi is the target value, usually defined as yi=r+γmaxa′Q(s′,a′;θi-), where θi- is the parameter of the target network, γ is the discount factor, a′ is the best action selected from the state s′, and U(D) represents the distribution of randomly drawn samples from the experience pool D;
[0060] The concept of the fixed target network is to maintain two networks: one is the online network being trained with parameters θi, and the other is the target network with parameters θi-. The online network is used to select actions, while the target network is used to evaluate the value of these actions and reduce the volatility of the target. The parameters of the target network are regularly copied (hard update) from the online network, and θi←θi- is directly copied at regular intervals (for example, every C steps).
[0061] This method combines DQN with reinforcement learning, which not only enhances the autonomous learning and adaptability of the unmanned ship monitoring system, but also significantly improves the accuracy and reliability of monitoring.
[0062] In addition, the present invention also includes an unmanned ship monitoring method, which is implemented by the unmanned ship monitoring method, including a feature extraction module, a feature fusion module and a cross-modal association learning module; the feature extraction module is used to extract the image features of the monitoring camera and the signal features of the cabin water inflow alarm sensor, and the feature fusion module realizes the conversion of image features and signal features into feature vectors and fuses the image feature vectors and the signal feature vectors, and the two verify each other to ensure the accuracy of the monitoring results; the cross-modal association learning module enhances the accuracy of the model's judgment on the data through deep learning.
[0063] The present invention has the following beneficial effects:
[0064] 1. By integrating deep learning, reinforcement learning, and advanced computer vision technology, the system can automatically complete complex monitoring tasks without human intervention, significantly reducing labor costs and human errors. Combining image features with sensor signals, it realizes multi-dimensional monitoring of ship cabin water ingress and improves the accuracy and completeness of monitoring. Image data captures visual clues, while sensor data supplements physical change information. The combination of the two enhances the system's comprehensive judgment ability. Through online learning and reinforcement learning, the system can continuously adjust strategies and optimize models based on real-time feedback, adapt to various complex and changing environmental conditions, and improve flexibility and response speed in dealing with emergencies;
[0065] 2. The use of the replay mechanism and fixed target network effectively reduces the deviation and fluctuation in the learning process, ensures the robustness of the model, and accelerates the learning process, enabling the model to achieve good performance faster. By using the additive attention mechanism and state-value function, the system can more accurately evaluate the current state and make the best decision, reducing false positives and negatives, ensuring the timeliness and accuracy of early warnings. The application of feature selection and dimensionality reduction techniques, as well as effective data processing strategies such as mini-batch updates, enable the system to run efficiently with limited computing resources, reducing energy consumption and hardware requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 The present invention is a flow chart of the unmanned ship monitoring method.
[0067] Figure 2 It is a system framework diagram of the present invention.
[0068] Figure 3 It is a flow chart of the feature fusion module of the present invention.
[0069] Figure 4 It is a flow chart of the cross-modal association learning module of the present invention. DETAILED DESCRIPTION
[0070] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific preferred embodiments.
[0071] In the description of the present invention, it should be understood that the terms "left side", "right side", "upper part", "lower part" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. "First", "second" and the like do not indicate the importance of the components, and therefore cannot be understood as limiting the present invention. The specific dimensions used in this embodiment are only for illustrating the technical solution, and do not limit the scope of protection of the present invention.
[0072] Reference Figure 1 , it can be seen that an unmanned ship monitoring method and system,
[0073] S1: Establish a ship monitoring model including feature extraction module, feature fusion module and cross-modal association learning module;
[0074] S2: extracting the ship image features and the signal features measured by the sensor through the feature extraction module;
[0075] S3: importing image features and signal features into feature fusion module;
[0076] S4: Strengthen the fusion of image features and signal features through cross-modal association learning module, and in the feedback process
[0077] Online learning and reinforcement learning are used to optimize the model;
[0078] S5: Get the monitoring results.
[0079] Reference Figure 2 ,It can be seen that the monitoring model is established: First, a ,comprehensive monitoring model is constructed, which contains three core ,modules: feature extraction module, feature fusion module and cross-modal association learning ,module.
[0080] In one embodiment, image feature extraction: surveillance cameras are arranged in the ship cabin, and the images obtained by the cameras are processed using the YOLOv4 model. The images are resized to meet the YOLOv4 input standard, and the key information in the images is extracted through a series of processing steps (including normalization, residual structure, convolution layer, pooling layer, etc.), and multi-scale feature fusion is performed through PANet, and finally the image feature vector is output.
[0081] Signal feature extraction: At the same time, cabin flooding alarm sensors are installed to collect real-time signal data. Within a fixed time window, Fourier transform is used to convert the signal from the time domain to the frequency domain to extract the frequency features related to flooding.
[0082] Feature fusion: In the feature fusion module, the image feature vector is directly input, and the signal features are first screened for important features through the Pearson correlation coefficient, and the dimension is reduced through linear discriminant analysis (LDA), and then integrated into the fusion module in the form of a matrix to achieve the initial fusion of the two features, providing a comprehensive data basis for subsequent analysis.
[0083] Feature fusion enhancement: Through the cross-modal association learning module (S4), the recurrent neural network (RNN) is used to process the signal data of the time series, and the convolutional neural network (CNN) is used to process the image data. After the feature vectors output by the two are concatenated, the weights of the two are dynamically adjusted according to the task requirements through weighted fusion to achieve a deeper level of fusion.
[0084] Online learning and reinforcement learning: In this stage (S4), DQN (Deep Q-Network) combines Q-Learning with deep learning, introduces experience replay mechanism and fixed target network to reduce learning bias and variance, and improve the adaptability and stability of the model. The model optimizes itself based on real-time feedback to achieve rapid learning and adaptive adjustment.
[0085] Monitoring result output: After the above steps are processed, the system comprehensively analyzes and obtains the monitoring result (S5), which can timely and accurately determine whether there is water ingress in the ship's engine room to ensure the safety of the ship.
[0086] Reference Figure 3 ,It can be seen that the feature fusion module is located in the center of the ,unmanned ship monitoring system, following the feature extraction module and ,before the cross-modal association learning module.,Its main responsibility is to effectively integrate the feature vectors ,obtained from the surveillance camera images and sensors, and provide a ,comprehensive data basis for subsequent analysis.
[0087] In one embodiment, the image feature vector: the image feature directly processed by the YOLOv4 model is output after multi-scale fusion (PANet), retaining rich image feature information.
[0088] Signal characteristics: The signal data of the cabin water ingress alarm sensor captures the frequency characteristics related to water ingress through the frequency domain features extracted by Fourier transform.
[0089] Feature selection: Use the Pearson correlation coefficient to screen signal features that are strongly correlated with water inflow events and remove irrelevant noise.
[0090] Dimensionality reduction: Linear discriminant analysis (LDA) is used to reduce the dimensionality of signal features and extract the features that can best distinguish the water inflow status.
[0091] Direct input: The image feature vector is directly fed into the fusion module.
[0092] Signal integration: The filtered and reduced signal features are arranged into a matrix by rows and input into the fusion module.
[0093] Preliminary fusion: Combine the two features and initially integrate them into a unified feature representation to provide a comprehensive perspective for subsequent analysis.
[0094] Cross-modal association learning: In the subsequent steps, the signal time series data is processed by the recurrent neural network (RNN), and the image data is processed by the convolutional neural network (CNN). The outputs of the two are further strengthened through weighted fusion to integrate the features.
[0095] Reference Figure 4 It can be seen that the cross-modal association learning module is a key component of multimodal learning, which aims to process and fuse data from different perceptual modalities, such as images, text, sound, video or sensor signals, to enhance the understanding and prediction capabilities of complex scenes. In the unmanned ship monitoring method and system, the cross-modal association learning module plays a vital role. By integrating image features and signal features, the accuracy and reliability of cabin water ingress monitoring are improved. Feature fusion: Features extracted from different modalities (for example, image feature vectors and signal feature vectors) are combined to achieve information complementarity.
[0096] Modeling inter-modal relationships: By learning the intrinsic correlations between different modal data, we can establish cross-modal correspondences. For example, in a ship monitoring scenario, image features (such as the position of the water level and flow pattern) are combined with sensor signal features (such as the vibration frequency caused by water ingress) to reveal the water ingress status.
[0097] Adaptive weight allocation: Use attention mechanisms (such as additive attention) to dynamically adjust the weights of different modal features to ensure that key information is given priority. This mechanism allows the model to flexibly adjust its reliance on each modal information in different situations to most effectively guide decision-making.
[0098] Online learning and reinforcement learning: Through the feedback mechanism, the model continuously optimizes itself during operation, and uses online learning (such as the DQN method) and reinforcement learning to adjust model parameters based on real-time data to optimize monitoring performance. This includes using experience replay and fixed target networks to reduce bias and variance in the learning process and accelerate convergence.
[0099] In one embodiment, a convolutional neural network (CNN) is combined to process image data and a recurrent neural network (RNN) is combined to process time series signal data. Through feature concatenation and weighted fusion, deep integration between modalities is achieved. By calculating additive attention scores, different weights are assigned to features from different modalities to enhance attention to important information and reduce noise interference. The optimization goal of reinforcement learning is defined by the Bellman equation and the state value function. The discount factor and immediate reward are used to guide the model learning process so that the system achieves a balance between exploration and utilization, including preprocessing of image features (image resizing, normalization, feature extraction, etc. of the YOLOv4 model) and time-frequency domain conversion (Fourier transform) of signal features, etc., to prepare for fusion.
[0100] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.
Claims
1. An unmanned ship monitoring method for timely monitoring of water ingress into the engine room under extreme conditions of the ship, characterized in that: The method comprises: S1: Establish a ship monitoring model including feature extraction module, feature fusion module and cross-modal association learning module; S2: The ship image features and the signal features measured by the sensor are extracted through the feature extraction module; S3: importing image features and signal features into feature fusion module; S4: Strengthen the fusion of image features and signal features through the cross-modal association learning module, and optimize the model through online learning and reinforcement learning in the feedback process; S5: Get the monitoring results.
2. The unmanned ship monitoring method according to claim 1, characterized in that: In step S2, the ship image features and the signal features measured by the sensor are extracted by the feature extraction module, specifically: S21: Install surveillance cameras and cabin water ingress alarm sensors in the ship cabin; S211: The image obtained by the surveillance camera of the YOLOv4 model is resized to match the input requirement of the YOLOv4 model; The YOLOv4 model includes input layer, Backbone network, Neck network, PANet and output layer; The input layer preprocesses the image by normalization; The Backbone network includes a residual structure, a 3×3 convolution layer and a 1×1 pooling layer. The residual structure, the 3×3 convolution layer and the 1×1 pooling layer are used to reduce computational redundancy and improve network efficiency. Image features are processed in two ways, one part is subjected to conventional convolution, and the other part is subjected to a large number of dimensionality reduction and dimensionality increase operations, and then merged to share features and reduce the amount of computation. The PANetPANet achieves efficient integration of multi-scale image features through image feature fusion of top-down and bottom-up paths; The output layer outputs an image feature vector; S212: The signal characteristics detected by the cabin water ingress alarm sensor are converted into frequency domain data using Fourier transform within a fixed time window to extract frequency characteristics.
3. The unmanned ship monitoring method according to claim 1, characterized in that: In step S3, the image features and signal features are imported into the feature fusion module, specifically: the image feature vector is directly input into the feature fusion module; The signal features monitored by the cabin water ingress alarm sensor use the Pearson correlation coefficient to select signal features, and use linear discriminant analysis to reduce the dimension of signal features. The feature vectors of each sample are arranged in rows to form a two-dimensional array or matrix. Each row represents a sample and each column represents a feature. The generated signal feature vector matrix is input into the feature fusion module.
4. The unmanned ship monitoring method according to claim 1, characterized in that: In step S4, the fusion of image features and signal features is strengthened through the cross-modal association learning module, specifically, the recurrent neural network and the convolutional neural network are fused: S41: First, use a recurrent neural network to process the time series data to obtain a series of hidden state vectors Hrnn=[h1,h2,...,hT], where T is the length of the sequence. At the same time, use a convolutional neural network to process the image data and extract the image feature vector Fcnn. Then, these two feature vectors are concatenated together to form a new feature vector as the input of the subsequent fully connected layer or other models. S42: Assign weights to the outputs of the recurrent neural network and the convolutional neural network, and then perform weighted summation to dynamically adjust the contribution of the two according to task requirements, and set a weight vector α to adjust the weights of Hrnn and Fcnn; This is achieved by designing a mathematical model based on Additive Attention: The additive attention mechanism calculates the weighted sum of the query and key and then generates an attention score through a nonlinear activation function and a fully connected layer. Q is the output from the RNN (Query), with a shape of (B, Tq, dk)(B, Tq, dk), where B is the batch size, Tq is the sequence length, dk is the feature dimension, V is the feature map of the CNN (Values), with a shape of (B, H, W, C), where H and W are the height and width respectively, and C is the number of channels; First, convert V to the shape of (B, H*W, C) to interact with Q. The mathematical expression of additive attention is as follows: Calculate weights: Transform V and Q respectively through the fully connected layer (weight matrices Wk and Wq): Q′=QWq Expand Q′ to (B, Tq, 1, H*W), broadcast and add it to K, and then use the tanh function to get the matching metric M: M=tanh(Q′[B;Tq;1;H*W,None]+K[None,B;Tq;1;H*W]); Calculate the attention score: The attention weight α is calculated through another fully connected layer (weight matrix Wv) and the softmax function: α=softmax(MWv); Weighted sum: Use the attention weight α to perform a weighted summation on the original V to get the output O: O = α·V.
5. The unmanned ship monitoring method according to claim 1, characterized in that: In step S4, the model is optimized through online learning and reinforcement learning during the feedback process, specifically: The reinforcement learning is defined by a five-tuple through an MDP: S (state set), A (action set), P(s′, s, a) (state transition probability), R(s, a, s′) (reward function), γ (discount factor) imported into the state value function of the Bellman equation: V(s)=∑a∈Aπ(a∣s)∑s′∈SP(s′∣s,a)[R(s,a,s′)+γV(s′)] Among them, π(a|s) is the probability of taking action a in state s; The update rule of the state value function is: Q(st,at)←Q(st,at)+α[rt+γa′maxQ(st+1,a′)-Q(st,at)] Among them, α is the learning rate and rt is the immediate reward; The online learning combines Q-Learning and deep learning through DQN, uses a neural network to approximate the Q function, and reduces the bias and variance of the estimation through the experience replay buffer (Experience Replay) and the fixed target network (Target Network); The core idea of the experience replay buffer is to store historical transition experiences, i.e., et = (st, at, rt, st+1), where st is the current state, at is the action taken, rt is the immediate reward, and st+1 is the next state after the action is taken. These experiences are stored in an experience pool (replay buffer), and the size of the experience pool (replay buffer) is fixed. In the learning step, instead of updating directly based on the most recent single experience, a mini-batch of data is randomly extracted from the experience pool for learning. Such a mini-batch update can be expressed as: L(θi)=E(s,a,r,s′)~U(D)[(yi-Q(s,a;θi))2]; Among them, L(θi) loss function is used to update the network parameters θi, yi is the target value, usually defined as yi=r+γmaxa′Q(s′,a′;θi-), where θi- is the parameter of the target network, γ is the discount factor, a′ is the best action selected from the state s′, and U(D) represents the distribution of randomly drawn samples from the experience pool D; The concept of the fixed target network is to maintain two networks: one is the online network being trained with parameters θi, and the other is the target network with parameters θi-. The online network is used to select actions, while the target network is used to evaluate the value of these actions and reduce the volatility of the target. The parameters of the target network are regularly copied (hard update) from the online network, and θi←θi- is directly copied at regular intervals (for example, every C steps).
6. A method for monitoring an unmanned ship, implemented based on the method for monitoring an unmanned ship according to any one of claims 1 to 5, characterized in that: Feature extraction module, feature fusion module and cross-modal association learning module; the feature extraction module is used to extract the image features of the monitoring camera and the signal features of the cabin water ingress alarm sensor, the feature fusion module realizes the conversion of image features and signal features into feature vectors and fuses the image feature vectors and the signal feature vectors, the two of which verify each other to ensure the accuracy of the monitoring results; the cross-modal association learning module enhances the accuracy of the model's judgment on the data through deep learning.
Citation Information
Cited By
Ship power system fault monitoring method
CN119846358A
Mooring cable monitoring method and device
CN120013874A
