A multi-sensor fusion monitoring method for ship berthing and unberthing status
Through the multi-sensor fusion monitoring method, combined with adaptive weighted fusion and BP neural network, the accuracy and stability problems of ship off-beat measurement are solved, and efficient and safe ship off-beat operation is achieved.
Patent Information
- Application Number
- CN202410895533.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-05
AI Technical Summary
The existing ship off-beat measurement methods have problems such as measurement error, limited measurement range, low measurement frequency, and unstable data, which are difficult to meet the requirements of high accuracy, high efficiency and high safety for large ships.
Multi-sensor fusion monitoring methods are adopted, including multi-source data acquisition, preprocessing, adaptive weighted fusion, BP neural network mapping and fitting, real-time data processing, real-time map generation and human-computer interaction control analysis, through GPS, gyroscope, acceleration The meter, magnetometer and laser rangefinder collect data, use the least squares method and Kalman filtering method to calculate the weight coefficient, and the BP neural network performs nonlinear mapping and fitting, generates a real-time three-dimensional map, and provides path planning suggestions.
It improves the accuracy and stability of the ship's mooring, enhances the adaptability and robustness of the system, reduces the risk of collision, improves the level of coordinated operations and efficiency, and reduces operating errors and time-consuming.
Smart Images

Figure CN118424273B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of ship berthing and unberthing, and in particular to a multi-sensor fusion monitoring method for ship berthing and unberthing status. Background Art
[0002] The berthing and unberthing of ships is an important part of ship transportation, which directly affects the safety and efficiency of ships. When a ship berths, it is necessary to accurately obtain information such as the relative position and posture between the ship and the dock in order to perform precise control and adjustment. However, due to factors such as the size, weight, inertia of the ship, as well as environmental factors such as water flow, wind, and waves, ship berthing is a complex dynamic process that is difficult to achieve accurate measurement and control.
[0003] At present, the main means of measurement for ship berthing are manual observation and single sensor measurement. These methods have disadvantages such as measurement errors, limited measurement range, low measurement frequency, and unstable data. They are difficult to meet the requirements of high precision, high efficiency, and high safety for large ship berthing. Summary of the invention
[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a multi-sensor fusion monitoring method for ship berthing and unberthing status to solve the problems raised in the above-mentioned background technology.
[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a multi-sensor fusion monitoring method for ship berthing and unberthing status, comprising the following steps: step S1: collecting multi-source data, step S2: preprocessing multi-source data, step S3: calculating the weight coefficient of each data source, step S4: BP neural network mapping and fitting, step S5: data display processing, step S6: processing real-time data, step S7: generating a real-time map, and step S8: human-computer interaction control analysis.
[0006] Step S1: Collecting multi-source data: installing multiple sensors including GPS, gyroscope, accelerometer, magnetometer, and laser rangefinder on the unmanned ship to collect multi-source data of the ship, respectively. The multi-source data includes the latitude, longitude, heading, pitch angle, roll angle, and distance of the ship, and the sampling frequency is 10 Hz;
[0007] Step S2: multi-source data preprocessing: filtering, calibrating, and aligning the multi-source data collected in step S1;
[0008] Step S3: Calculate the weight coefficient of the data source: According to the data preprocessed in step S2, a method combining the least square method and the Kalman filter method is used to calculate the weight coefficient of each data source to achieve adaptive weighted fusion;
[0009] Step S4: BP neural network mapping and fitting: performing nonlinear mapping and fitting on the fused data in step S3;
[0010] Step S5: Data display processing: displaying the data output in step S4 in the form of graphics or numbers on a computer or other terminal;
[0011] Step S6: Processing real-time data: using the data acquired by the laser radar and other data for real-time processing and fusion, the other data including the ship's draft, berthing equipment, water flow, and tidal current;
[0012] Step S7: Generate a real-time map: Generate a real-time three-dimensional map based on the data processed in step S6, wherein the three-dimensional map includes the distance between the ship and the dock, the angle, the draft of the ship, the water flow state, and the speed of the ship;
[0013] Step S8: Human-computer interaction control analysis: Through tablets, mobile phones, and computer devices, drivers and dispatchers can observe the ship's posture in real time. At the same time, users can operate the interface in real time to provide path planning suggestions for the ship.
[0014] Preferably, it is characterized in that: the specific method of multi-source data preprocessing is: Kalman filtering is performed on GPS data to eliminate positioning errors; gyroscope, accelerometer, and magnetometer data are calibrated to eliminate zero bias and drift; laser rangefinder data is filtered to eliminate noise and outliers; all data are time-aligned to ensure data synchronization.
[0015] Preferably, it is characterized in that: the adaptive weighted fusion adopts a method combining the least square method and the Kalman filter method to calculate the weight coefficient of each data source, and the specific steps are as follows:
[0016] Step S31: According to the least squares method, the following objective function is established:
[0017] , where n represents the number of data, m represents the number of data sources, and w j Expressed as the weight coefficient of the j-th data source, x ij Represents the value of the jth data source of the i-th data, y i Represented as the true value of the i-th data;
[0018] Step S32: According to the Kalman filter method, the following state equation and observation equation are established:
[0019] ,
[0020] ,
[0021] Among them, w k+1 Represented as the weight coefficient vector at the k+1th moment, w k Represented as the weight coefficient vector at the kth moment, v k Expressed as system noise, z k It is represented as the observation value at the kth moment, and H is represented as the observation matrix;
[0022] Step S33: According to the recursive formula of Kalman filtering, the estimated value of the weight coefficient and the covariance matrix are updated to obtain the optimal weight coefficient vector; through the above steps, the fusion results of data such as longitude, latitude, heading angle, pitch angle, roll angle, distance and angle can be obtained.
[0023] Preferably, it is characterized in that: the BP neural network adopts a three-layer structure, including an input layer, a hidden layer, and an output layer; the input layer receives data output by the adaptive weighted fusion module, the output layer outputs data such as the position and posture of the ship, and the hidden layer is an S-type activation function.
[0024] Preferably, it is characterized in that: the BP neural network mapping and fitting adjusts the weights and thresholds of the network through the back propagation algorithm, and the specific steps are as follows:
[0025] Step S41: Network structure: This embodiment adopts a three-layer BP neural network, the input layer has 7 neurons, corresponding to longitude, latitude, heading angle, pitch angle, roll angle, distance and angle data; the output layer has 6 neurons, corresponding to the position, attitude and other data of the ship; the number of neurons in the hidden layer is 10, which is an S-type activation function, and its output is:
[0026] , where y j Represented as the output of the jth hidden layer neuron, z j is represented as the input of the jth hidden layer neuron, and e is represented as a natural constant, which is calculated by the following formula:
[0027] , where z j Represented as the input of the jth hidden layer neuron, w ij It is represented as the connection weight from the i-th input layer neuron to the j-th hidden layer neuron, x i is the input of the i-th input layer neuron, b j is the threshold of the jth hidden layer neuron;
[0028] Step S42: Network training: This embodiment adopts the back propagation algorithm to train the weights and thresholds of the network according to the known input and output data pairs, so that the error between the network output and the expected output is minimized; let the kth data pair be (xk, dk), where xk=(x1, x2, ..., x7) is the input vector, dk=(d1, d2, ..., d6) is the expected output vector, yk=(y1, y2, ..., y6) is the actual output vector, E k is the error function, calculated by the following formula:
[0029] , where E k is the error function, d i Denote the desired output, y i Represented as actual output;
[0030] Forward propagation: For the kth data pair (xk, dk), the input vector xk is sent to the network, and after calculations in the hidden layer and the output layer, the actual output vector yk is obtained;
[0031] Back propagation: Starting from the output layer, the error signal of each neuron is calculated layer by layer, and the weight and threshold of each neuron are adjusted according to the error signal; specifically, for the i-th neuron in the output layer, its error signal is:
[0032] ,in, Represented as the error signal of the i-th neuron in the output layer, d i Denote the desired output, y i Represented as the actual output; according to the error signal, the weight and threshold of the i-th output layer neuron are adjusted and updated by the following formula:
[0033] ,
[0034] ,
[0035] Termination condition: Repeat the above forward propagation and back propagation process until the error function Ek is less than the preset threshold or reaches the maximum number of iterations, stop training, and obtain the optimal weights and thresholds of the network.
[0036] Preferably, the feature is that the ship-shore collaborative augmented reality technology includes the following steps:
[0037] Step S01: using multi-sensor fusion monitoring technology for ship berthing and unberthing status to obtain the relative distance and angle information between the ship and the wharf in real time;
[0038] Step S02: Generate graphical augmented reality information based on the expected route of the ship and the operating area information of the dock;
[0039] Step S03: Displaying the augmented reality information in real time in the field of vision of the ship driver and the onshore dispatcher.
[0040] Preferably, it is characterized in that: the augmented reality information includes the relative distance and angle between the ship and the dock, the expected route of the ship, and the operating area of the dock.
[0041] Technical effects and advantages of the present invention:
[0042] The adaptive weighted fusion algorithm of the present invention can dynamically adjust the weight coefficient of each sensor according to the data quality and credibility, thereby enhancing the adaptability and robustness of the system;
[0043] In the present invention, the BP neural network module can perform nonlinear mapping and fitting on the fused data to improve the accuracy and stability of the data and overcome the nonlinearity and uncertainty of the data;
[0044] Through this system, real-time information display is realized, helping drivers and dispatchers to quickly understand the operating environment, thereby improving the collision avoidance ability of ships and reducing collision risks; at the same time, through collaborative enhancement, relevant information of ships and docks is graphically superimposed in the field of vision, which is conducive to improving the overall collaborative operation level and operation efficiency, and reducing operation errors and time consumption;
[0045] The present invention uses deep reinforcement learning to complete ship path planning, assist pilots in decision-making, reduce the pressure of manual observation and has high safety.
[0046] In the present invention, the system structure is simple and flexible, easy to implement and apply, and can be applicable to different ship types and berthing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.
[0048] Figure 1 It is a schematic diagram of the overall structure of the system of the present invention;
[0049] Figure 2 It is a schematic diagram of the method flow of the present invention;
[0050] Figure 3 Schematic diagram of the overall structure of the three-dimensional augmented reality system of the present invention;
[0051] Figure 4 Schematic diagram of the process flow of the ship-shore collaborative augmented reality technology method for ship berthing and unberthing operations of the present invention;
[0052] Figure 5 This is a schematic diagram of the path planning principle of the present invention. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example
[0054] See also Figure 1 As shown, the present invention provides a system for a multi-sensor fusion monitoring method for ship berthing and unberthing status, including a multi-source data acquisition module, a data preprocessing module, an adaptive weighted fusion module, a BP neural network module, a data display module, a real-time data processing module, a real-time map generation module, and a user interaction control module.
[0055] The output end of the multi-source data acquisition module is connected to the input end of the data preprocessing module, the output end of the data preprocessing module is connected to the input end of the adaptive weighted fusion module, the output end of the adaptive weighted fusion module is connected to the input end of the BP neural network module, the output end of the BP neural network module is connected to the input end of the data display module, the output end of the data display module is connected to the input end of the real-time data processing module, the output end of the real-time data processing module is connected to the input end of the real-time map generation module, and the real-time map generation module is connected to the input end of the user interaction control module.
[0056] The multi-source data acquisition module is used to collect multi-source data such as the position and attitude of the ship from multiple sensors or devices, including the latitude, longitude, heading, pitch angle, roll angle, and distance of the ship, and transmit the multi-source data to the data preprocessing module;
[0057] In this embodiment, it should be specifically explained that the sensor includes a GPS, a gyroscope, an accelerometer, a magnetometer, and a laser rangefinder.
[0058] The data preprocessing module is used to preprocess the multi-source data collected by the multi-source data collection module, and transmit the preprocessed multi-source data to the adaptive weighted fusion module;
[0059] In this embodiment, it should be specifically explained that the preprocessing method includes filtering, calibration, and alignment operations to eliminate noise, errors and deviations;
[0060] The adaptive weighted fusion module dynamically adjusts the weight of each data source according to the credibility and consistency of multi-source data to achieve optimal data fusion;
[0061] In this embodiment, it should be specifically explained that the adaptive weighted fusion module uses a method combining the least square method and the Kalman filter method to calculate the weight coefficient of each data source so that the mean square error between the fused data and the real data is minimized;
[0062] The BP neural network module performs nonlinear mapping and fitting on the fused data to improve the accuracy and stability of the data;
[0063] In this embodiment, it should be specifically explained that the BP neural network module is a three-layer structure, including an input layer, a hidden layer and an output layer; the input layer receives the data output by the adaptive weighted fusion module, the output layer outputs the position, posture and other data of the ship, and the hidden layer is an S-type activation function. The weights and thresholds of the network are adjusted by the back propagation algorithm so that the error between the output of the network and the expected output is minimized; this system adopts a method combining adaptive weighted fusion and BP neural network to fuse the multi-source heterogeneous ship berthing status information to obtain more accurate and higher precision ship comprehensive data information;
[0064] The data display module displays the data output by the BP neural network module on a computer or other terminal in the form of graphics or numbers;
[0065] The real-time data processing module uses the data acquired by the laser radar and other data for real-time processing and fusion;
[0066] In this embodiment, it should be specifically explained that the other data include ship draft, berthing equipment, water flow, and tidal current; its data processing involves point cloud data processing and sensor data fusion;
[0067] The real-time map generation module is used to generate a real-time three-dimensional map with the data processed by the real-time data processing module, which includes the distance between the ship and the dock, the angle, the draft of the ship, the water flow state, and the speed of the ship;
[0068] The user interaction control module enables the driver and dispatcher to observe the ship's posture in real time through tablet computers and mobile phones. At the same time, the user can operate the interface in real time to provide path planning suggestions for the ship.
[0069] See also Figure 2As shown, the present invention provides a multi-sensor fusion monitoring technology method for ship berthing and unberthing status, comprising the following steps: step S1: collecting multi-source data, step S2: preprocessing multi-source data, step S3: calculating the weight coefficient of the data source, step S4: BP neural network mapping and fitting, step S5: data display processing, step S6: processing real-time data, step S7: generating a real-time map, and step S8: human-computer interaction control analysis.
[0070] Step S1: Collecting multi-source data: installing multiple sensors including GPS, gyroscope, accelerometer, magnetometer, and laser rangefinder on the unmanned ship to collect multi-source data of the ship, respectively. The multi-source data includes the latitude, longitude, heading, pitch angle, roll angle, and distance of the ship, and the sampling frequency is 10 Hz;
[0071] Step S2: multi-source data preprocessing: filtering, calibrating, and aligning the multi-source data collected in step S1;
[0072] In this embodiment, it should be specifically explained that the specific method of multi-source data preprocessing is: Kalman filtering is performed on GPS data to eliminate positioning errors; gyroscope, accelerometer, and magnetometer data are calibrated to eliminate zero bias and drift; laser rangefinder data is filtered to eliminate noise and outliers; all data are time-aligned to ensure data synchronization;
[0073] Step S3: Calculate the weight coefficient of the data source: According to the data preprocessed in step S2, a method combining the least square method and the Kalman filter method is used to calculate the weight coefficient of each data source to achieve adaptive weighted fusion;
[0074] In this embodiment, a method combining the least square method and the Kalman filter method is used to calculate the weight coefficient of each data source so that the mean square error between the fused data and the real data is minimized. The specific steps are as follows:
[0075] Step S31: According to the least squares method, the following objective function is established:
[0076] , where n represents the number of data, m represents the number of data sources, and w j Expressed as the weight coefficient of the j-th data source, x ij Represents the value of the jth data source of the i-th data, y i Represented as the true value of the i-th data;
[0077] Step S32: According to the Kalman filter method, the following state equation and observation equation are established:
[0078] ,
[0079] ,
[0080] Among them, w k+1 Represented as the weight coefficient vector at the k+1th moment, w k Represented as the weight coefficient vector at the kth moment, v k Expressed as system noise, z k It is represented as the observation value at the kth moment, and H is represented as the observation matrix;
[0081] Step S33: according to the recursive formula of Kalman filtering, the estimated value of the weight coefficient and the covariance matrix are updated to obtain the optimal weight coefficient vector; through the above steps, the fusion results of the longitude, latitude, heading angle, pitch angle, roll angle, distance and angle data can be obtained;
[0082] Step S4: BP neural network mapping and fitting: performing nonlinear mapping and fitting on the fused data in step S3;
[0083] In this embodiment, the implementation method performs nonlinear mapping and fitting on the fused data to improve the accuracy and stability of the data; specifically, the implementation method adopts a three-layer BP neural network, including an input layer, a hidden layer, and an output layer; the input layer receives the data output by the adaptive weighted fusion module, the output layer outputs the position, posture and other data of the ship, and the hidden layer is an S-type activation function. The weights and thresholds of the network are adjusted by the back propagation algorithm to minimize the error between the output of the network and the expected output; the specific steps are as follows:
[0084] Step S41: Network structure: This embodiment adopts a three-layer BP neural network, the input layer has 7 neurons, corresponding to longitude, latitude, heading angle, pitch angle, roll angle, distance and angle data; the output layer has 6 neurons, corresponding to the position, attitude and other data of the ship; the number of neurons in the hidden layer is 10, which is an S-type activation function, and its output is:
[0085] , where y j Represented as the output of the jth hidden layer neuron, z j is represented as the input of the jth hidden layer neuron, and e is represented as a natural constant, which is calculated by the following formula:
[0086] , where z j Represented as the input of the jth hidden layer neuron, w ij It is represented as the connection weight from the i-th input layer neuron to the j-th hidden layer neuron, x i is the input of the i-th input layer neuron, b jis the threshold of the jth hidden layer neuron;
[0087] Step S42: Network training: This embodiment adopts the back propagation algorithm to train the weights and thresholds of the network according to the known input and output data pairs, so that the error between the network output and the expected output is minimized; let the kth data pair be (xk, dk), where xk=(x1, x2, ..., x7) is the input vector, dk=(d1, d2, ..., d6) is the expected output vector, yk=(y1, y2, ..., y6) is the actual output vector, E k is the error function, calculated by the following formula:
[0088] , where E k is the error function, d i Denote the desired output, y i Represented as actual output;
[0089] Forward propagation: For the kth data pair (xk, dk), the input vector xk is sent to the network, and after calculations in the hidden layer and the output layer, the actual output vector yk is obtained;
[0090] Back propagation: Starting from the output layer, the error signal of each neuron is calculated layer by layer, and the weight and threshold of each neuron are adjusted according to the error signal; specifically, for the i-th neuron in the output layer, its error signal is:
[0091] ,in, Represented as the error signal of the i-th neuron in the output layer, d i Denote the desired output, y i Represented as the actual output; according to the error signal, the weight and threshold of the i-th output layer neuron are adjusted and updated by the following formula:
[0092] ,
[0093] ,
[0094] Termination condition: Repeat the above forward propagation and back propagation process until the error function Ek is less than the preset threshold or reaches the maximum number of iterations, stop training, and obtain the optimal weights and thresholds of the network;
[0095] Step S5: Data display processing: displaying the data output in step S4 in the form of graphics or numbers on a computer or other terminal;
[0096] Step S6: Processing real-time data: using the data acquired by the laser radar and other data for real-time processing and fusion, the other data including the ship's draft, berthing equipment, water flow, and tidal current;
[0097] Step S7: Generate a real-time map: Generate a real-time three-dimensional map based on the data processed in step S6, wherein the three-dimensional map includes the distance between the ship and the dock, the angle, the draft of the ship, the water flow state, and the speed of the ship;
[0098] Step S8: Human-computer interaction control analysis: Through tablets, mobile phones, and computer devices, drivers and dispatchers can observe the ship's posture in real time. At the same time, users can operate the interface in real time to provide path planning suggestions for the ship. Example
[0099] See also Figure 3 As shown, the present invention provides a three-dimensional enhanced real-scene system for ship berthing based on three-dimensional laser radar, including a laser radar scanning module, a data processing and fusion module, a real-time map generation module, and a user interaction control module.
[0100] The output end of the laser radar scanning module is connected to the input end of the data processing and fusion module, the output end of the data processing and fusion module is connected to the input end of the real-time map generation module, and the output end of the real-time map generation module is connected to the input end of the user interaction control module.
[0101] The laser radar scanning module is used to obtain high-precision point cloud information of the ship and its surrounding environment, and the cloud information includes the shape of the ship, the relative distance and angle between the ship and the dock, and the speed of the ship;
[0102] The data processing and fusion module is used to convert the cloud information data obtained by the laser radar scanning module into
[0103] Real-time processing and fusion of data and other sensor data;
[0104] In this embodiment, it should be specifically noted that the other sensor data include ship draft, berthing equipment, water flow, tidal current, and ship image;
[0105] The data processing involves point cloud data processing and sensor data fusion; the original point cloud data contains noise points, which are denoised by Gaussian filtering; the sensor data needs to be pre-processed by data alignment, denoising, coordinate conversion, etc.
[0106] The real-time map generation module is used to generate a real-time three-dimensional map using the data processed by the real-time data processing and fusion module, including the distance between the ship and the dock, the angle, the ship's draft, the water flow state, and the ship's speed display;
[0107] In this embodiment, it should be specifically explained that the point cloud data of the ship is combined with the ship image data to perform multimodal data fusion, information complementation, and generate a comprehensive ship model;
[0108] The user interaction control module uses the model obtained by the reinforcement learning module to control and select the optimal action according to the real-time state of the ship to improve the collision avoidance capability;
[0109] In this embodiment, it should be specifically explained that the user interaction control module specifically includes the following steps:
[0110] Step 01: Design appropriate state space, action space, and reward function according to the ship berthing process; the state space includes information such as the ship's position, posture, and speed, and the current state is denoted as s, which serves as the input of the algorithm; the action space is designed according to the operations that can be taken during the berthing operation, and the executed action is denoted as a; the reward function is used to evaluate the behavior of the intelligent agent, which is designed according to the ship's collision avoidance capability and corresponds to the executed action, denoted as r.
[0111] Step 02: Collect historical experience of the agent’s interaction with the environment, replay it into the memory, and obtain the training sample set D.
[0112] Step 03: Based on the mainstream deep reinforcement learning network model DQN, adjust the structure and parameters as needed, perform training, and the predicted value of the corresponding action is recorded as Q.
[0113] Step 04: Input the real-time status into the network to obtain the predicted value of the feasible action. The action corresponding to the maximum Q value output by the network is the optimal action.
[0114] See also Figure 4 As shown, the present invention provides a ship-shore collaborative augmented reality technology for ship berthing and unberthing operations, comprising the following steps: step S01: using a multi-sensor fusion fusion monitoring technology for ship berthing and unberthing status to obtain real-time relative distance and angle information between the ship and the dock, step S02: generating graphical augmented reality information based on the expected route of the ship and the operating area information of the dock, and step S03: displaying the augmented reality information in real time in the field of vision of the ship driver and the onshore dispatcher.
[0115] Step S01: using multi-sensor fusion monitoring technology for ship berthing and unberthing status to obtain the relative distance and angle information between the ship and the wharf in real time;
[0116] Step S02: Generate graphical augmented reality information based on the expected route of the ship and the operating area information of the dock;
[0117] In this embodiment, it should be specifically explained that the augmented reality information includes the relative distance and angle between the ship and the dock, the expected route of the ship, and the operating area of the dock;
[0118] Step S03: Displaying the augmented reality information in real time in the field of vision of the ship driver and the onshore dispatcher.
[0119] In this embodiment, it should be specifically explained that the method of step S03 realizes ship-shore collaborative perception and improves the positioning accuracy and collision avoidance capability of the ship; the berthing navigation decision reinforcement learning method adopts a value function-based method, using the state, action and reward value of the ship to update the value function and select the action that maximizes the value function.
[0120] The present invention fuses the relative position, posture and other information between the ship and the dock collected by multiple sensors by using algorithms such as adaptive weighted fusion and BP neural network, so as to realize real-time monitoring of the berthing status of large ships; by using augmented reality technology, the information such as the relative distance and angle between the ship and the dock, as well as the expected route of the ship, the operation area of the dock and other information are displayed in real time in the field of vision of the ship driver and the onshore dispatcher in a graphical manner, so as to realize ship-shore collaborative perception and improve the positioning accuracy and collision avoidance ability of the ship; by adopting the berthing navigation decision reinforcement learning method, according to the current state of the ship, the target position, the environmental conditions and other factors, the optimal berthing navigation strategy is generated to realize the auxiliary decision of the berthing operation of large ships. The beneficial effects of the present invention are that it can effectively improve the efficiency and safety of large ships berthing and unberthing, reduce the workload of ship drivers and onshore dispatchers, and promote the operation and management of ports.
[0121] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A multi-sensor fusion monitoring method for ship berthing and unberthing status, It is characterized in that The following steps are involved: Step S1: Collecting multi-source data: installing multiple sensors including GPS, gyroscope, accelerometer, magnetometer, and laser rangefinder on the unmanned ship to collect multi-source data of the ship, respectively. The multi-source data includes the latitude, longitude, heading, pitch angle, roll angle, and distance of the ship, and the sampling frequency is 10 Hz; Step S2: multi-source data preprocessing: filtering, calibrating, and aligning the multi-source data collected in step S1; Step S3: Calculate the weight coefficient of the data source: According to the data preprocessed in step S2, a method combining the least square method and the Kalman filter method is used to calculate the weight coefficient of each data source to achieve adaptive weighted fusion; The adaptive weighted fusion adopts a method combining the least square method and the Kalman filter method to calculate the weight coefficient of each data source. The specific steps are as follows: Step S31: According to the least squares method, the following objective function is established: , where n represents the number of data, m represents the number of data sources, and w i Expressed as the weight coefficient of the i-th data source, x ij Represents the value of the jth data source of the i-th data, y i Represents the true value of the i-th data; Step S32: According to the Kalman filter method, the following state equation and observation equation are established: , , Among them, w k+1 Represented as the weight coefficient vector at the k+1th moment, w k Represented as the weight coefficient vector at the kth moment, v k Expressed as system noise, z k It is represented as the observation value at the kth moment, and H is represented as the observation matrix; Step S33: according to the recursive formula of Kalman filtering, the estimated value of the weight coefficient and the covariance matrix are updated to obtain the optimal weight coefficient vector; through the above steps, the fusion results of the longitude, latitude, heading angle, pitch angle, roll angle, distance and angle data can be obtained; Step S4: BP neural network mapping and fitting: performing nonlinear mapping and fitting on the fused data in step S3; The BP neural network mapping and fitting adjusts the weights and thresholds of the network through the back propagation algorithm. The specific steps are as follows: Step S41: Network structure: This embodiment adopts a three-layer BP neural network, the input layer has 7 neurons, corresponding to longitude, latitude, heading angle, pitch angle, roll angle, distance and angle data; the output layer has 6 neurons, corresponding to the position, attitude and other data of the ship; the number of neurons in the hidden layer is 10, which is an S-type activation function, and its output is: , where y i Represented as the output of the i-th hidden layer neuron, z j is represented as the input of the jth hidden layer neuron, and e is represented as a natural constant, which is calculated by the following formula: , where z j Represented as the input of the jth hidden layer neuron, w ij It is represented as the connection weight from the i-th input layer neuron to the j-th hidden layer neuron, x i is the input of the i-th input layer neuron, b j is the threshold of the jth hidden layer neuron; Step S42: Network training: This embodiment adopts the back propagation algorithm to train the weights and thresholds of the network according to the known input and output data pairs, so that the error between the network output and the expected output is minimized; let the kth data pair be (xk, dk), where xk=(x1, x2, ..., x7) is the input vector, dk=(d1, d2, ..., d6) is the expected output vector, yk=(y1, y2, ..., y6) is the actual output vector, E k is the error function, calculated by the following formula: , where E k is the error function, d i Denote the desired output, y i Represented as actual output; Forward propagation: For the kth data pair (xk, dk), the input vector xk is sent to the network, and after calculations in the hidden layer and the output layer, the actual output vector yk is obtained; Back propagation: Starting from the output layer, the error signal of each neuron is calculated layer by layer, and the weight and threshold of each neuron are adjusted according to the error signal; specifically, for the i-th neuron in the output layer, its error signal is: ,in, Represented as the error signal of the i-th neuron in the output layer, d i Denote the desired output, y i Represented as the actual output; according to the error signal, the weight and threshold of the i-th output layer neuron are adjusted and updated by the following formula: , , Termination condition: Repeat the above forward propagation and back propagation process until the error function Ek is less than the preset threshold or reaches the maximum number of iterations, stop training, and obtain the optimal weights and thresholds of the network; Step S5: Data display processing: displaying the data output in step S4 in the form of graphics or numbers on a computer or other terminal; Step S6: Processing real-time data: using the data acquired by the laser radar and other data for real-time processing and fusion, the other data including the ship's draft, berthing equipment, water flow, and tidal current; Step S7: Generate a real-time map: Generate a real-time three-dimensional map based on the data processed in step S6, wherein the three-dimensional map includes the distance between the ship and the dock, the angle, the draft of the ship, the water flow state, and the speed of the ship; Step S8: Human-computer interaction control analysis: Through tablets, mobile phones, and computer devices, drivers and dispatchers can observe the ship's posture in real time. At the same time, users can operate the interface in real time to provide path planning suggestions for the ship.
2. According to the multi-sensor fusion monitoring method for ship berthing and unberthing status according to claim 1, Features: The specific method of multi-source data preprocessing is: performing Kalman filtering on GPS data; calibrating gyroscope, accelerometer, and magnetometer data; filtering laser rangefinder data; and time aligning all data.
3. According to the multi-sensor fusion monitoring method for ship berthing and unberthing status according to claim 1, Features: The BP neural network adopts a three-layer structure, including an input layer, a hidden layer, and an output layer; the input layer receives data output by the adaptive weighted fusion module, the output layer outputs data such as the position and posture of the ship, and the hidden layer is an S-type activation function.
4. According to the multi-sensor fusion monitoring method for ship berthing and unberthing status according to claim 1, Features: The ship-shore collaborative augmented reality technology includes the following steps: Step S01: using multi-sensor fusion monitoring technology for ship berthing and unberthing status to obtain the relative distance and angle information between the ship and the wharf in real time; Step S02: Generate graphical augmented reality information based on the expected route of the ship and the operating area information of the dock; Step S03: Displaying the augmented reality information in real time in the field of vision of the ship driver and the onshore dispatcher.
5. According to the multi-sensor fusion monitoring method for ship berthing and unberthing status according to claim 4, Features: The augmented reality information includes the relative distance and angle between the ship and the dock, the expected route of the ship, and the operating area of the dock.
