Berthing early warning method for unattended screw ship unloader

Abstract

The embodiment of the invention relates to a berthing early warning method for an unattended screw ship unloader. The berthing early warning method comprises the steps that environment monitoring data of a port of the screw ship unloader and state monitoring data of a ship body are obtained; generating a 3D environment map with temperature attributes from the laser radar point cloud data and the infrared thermal imaging data through a pixel-level fusion algorithm; generating an environment sensing feature vector and a ship body state feature vector based on the environment monitoring data and the state monitoring data; inputting the 3D environment map, the environment perception feature vector and the ship state feature vector into a preset dynamic risk prediction model for prediction; and carrying out berthing early warning according to the prediction result. According to the technical scheme provided by the embodiment of the invention, the environment data and the ship state data are monitored, and the dynamic change feature data and the cross-dimensional correlation features are generated according to the environment data and the ship state data to serve as the input vectors of the prediction model, so that the environment sensing precision and the modeling precision of the model are effectively improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of unmanned screw ship unloaders, and in particular to a berthing warning method for unmanned screw ship unloaders. Background Art

[0002] In the existing technology, the berthing warning of unmanned screw unloaders usually relies on a single sensor. However, the berthing scene is relatively complex and affected by many factors. A single sensor will lead to insufficient perception of the berthing scene, resulting in inaccurate warning results.

[0003] In addition, the risk prediction models used in existing technologies are usually constructed using a single model. A single model usually revolves around a specific data type or task objective and cannot fully cover complex needs. As a result, there are problems such as limited predictive ability of the model, poor environmental adaptability, inefficient data utilization, and insufficient robustness. Summary of the Invention

[0004] Based on the above situation of the prior art, the purpose of an embodiment of the present invention is to provide a berthing warning method for an unmanned screw ship unloader, which effectively improves the reliability and accuracy of the berthing warning of the unmanned screw ship unloader by monitoring multidimensional data and constructing a hybrid model for dynamic risk prediction.

[0005] To achieve the above object, according to one aspect of the present invention, a berthing warning method for an unmanned screw ship unloader is provided, comprising the steps of: Obtaining environmental monitoring data of the screw unloader port and status monitoring data of the hull; the environmental monitoring data includes at least laser radar point cloud data and infrared thermal imaging data; Generate a 3D environmental map with temperature attributes by using a pixel-level fusion algorithm to combine lidar point cloud data and infrared thermal imaging data; Generate an environmental perception feature vector and a hull state feature vector based on the above environmental monitoring data and state monitoring data; Input the 3D environmental map, environmental perception feature vector and hull state feature vector into the preset dynamic risk prediction model for prediction; Providing berthing warnings based on the prediction results; Among them, the dynamic risk prediction model adopts an LSTM-Transformer hybrid model, and the Transformer encoder in the LSTM-Transformer hybrid model includes an intra-group self-attention module and an inter-group cross-attention module.

[0006] Furthermore, the LiDAR point cloud data and infrared thermal imaging data are fused through a pixel-level algorithm to generate a 3D environmental map with temperature attributes, including the following steps: Projecting the LiDAR point cloud data onto the infrared thermal imaging data image plane; For each laser point corresponding to the lidar point cloud data, the temperature value at the coordinate of the infrared thermal imaging data image plane is obtained to generate a point cloud with temperature attributes; The port space is divided into voxels of a predetermined size. Each voxel stores the point cloud density as a geometric attribute and the mean temperature within the voxel as a temperature attribute to form a 3D environmental map. Obstacles are marked on the 3D environment map to generate a marked 3D environment map.

[0007] Furthermore, the 3D environment map is marked with obstacles, including the following steps: If the point cloud density is greater than the first density threshold and the temperature mean is greater than the first temperature threshold, the voxel is marked as a ship metal structure obstacle; If the point cloud density is greater than a first density threshold and the temperature mean is less than a second temperature threshold, the voxel is marked as a static obstacle.

[0008] Furthermore, the environmental monitoring data also includes meteorological data; the status monitoring data includes vibration frequency, attitude angle, dock displacement deviation and structural stress value.

[0009] Furthermore, the environmental perception characteristic vector includes the wind speed mutation gradient, the visibility attenuation coefficient and the obstacle distance-temperature correlation vector; the hull state characteristic vector includes the vibration energy entropy, the attitude angle change rate and the strain energy density of the stress concentration area.

[0010] Furthermore, the 3D environmental map, the environmental perception feature vector, and the hull state feature vector are input into a preset dynamic risk prediction model for prediction to obtain two types of risk values, including an environmental risk value and a structural risk value; A berthing warning is issued based on the environmental risk value and the structural risk value.

[0011] Furthermore, the self-attention module within the group performs the first self-attention calculation and the second self-attention calculation in parallel; The first self-attention calculation is used to interact between environmental perception features and learn the relationship between environmental perception features; the second self-attention calculation is used to interact between hull state features and learn the relationship between hull state features.

[0012] Furthermore, the inter-group cross attention module performs the first cross attention calculation and the second cross attention calculation in parallel; The first cross-attention calculation uses the hull state as a query and searches for relevant clues and information in the environmental perception features; the second cross-attention calculation uses environmental perception as a query and searches for relevant clues and information in the hull state features.

[0013] Furthermore, the Transformer encoder further includes an information fusion output module; The information fusion output module is used to fuse and output the environmental perception features updated by the first self-attention calculation with the output of the second cross-attention calculation; it is also used to fuse and output the hull state features updated by the second self-attention calculation with the output of the first cross-attention calculation.

[0014] Furthermore, the information fusion output module is also used to splice the two fused sets of features and input them into the feedforward neural network.

[0015] In summary, an embodiment of the present invention provides a berthing warning method for an unmanned screw unloader, comprising the steps of: acquiring environmental monitoring data of the screw unloader port and status monitoring data of the hull; generating a 3D environmental map with temperature attributes using a pixel-level fusion algorithm for lidar point cloud data and infrared thermal imaging data; generating an environmental perception feature vector and a hull status feature vector based on the above-mentioned environmental monitoring data and status monitoring data; inputting the 3D environmental map, environmental perception feature vector and hull status feature vector into a preset dynamic risk prediction model for prediction; and performing a berthing warning based on the prediction results. The technical solution provided by the embodiment of the present invention monitors environmental data and hull status data, and generates dynamically changing feature data and cross-dimensional correlation features as input vectors of the prediction model based on them, thereby effectively improving the environmental perception accuracy and modeling accuracy of the model; adopts the LSTM-Transformer hybrid model and combines it with the grouping attention mechanism, which not only ensures the capture of real-time time series risks, but also strengthens the modeling of long-distance lag risks. By reducing unnecessary calculations through grouping, the model's attention is more focused on feature interactions that are strongly correlated physically, thereby achieving higher accuracy with fewer parameters and data and reducing the risk of overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention provides a flowchart of a method for early warning of berthing of an unmanned screw ship unloader. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0018] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in one or more embodiments of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprising" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0019] The technical solution of the present invention is described in detail below with reference to the accompanying drawings. The embodiment of the present invention provides a berthing warning method for an unmanned screw unloader. Figure 1 FIG. 4 is a flow chart of a method for early warning of berthing of an unmanned screw ship unloader according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: S202. Obtain environmental monitoring data for the screw unloader port and ship status monitoring data. The environmental monitoring data includes laser radar point cloud data, infrared thermal imaging data, and meteorological data. In this embodiment of the present invention, the ship hull refers to the hull of the transport vessel used by the screw unloader. The status monitoring data includes vibration frequency, attitude angle, dock displacement deviation, and structural stress values. This environmental and status monitoring data can be obtained using laser radar, infrared thermal imaging cameras, gyroscopes, and various other sensors installed at the port and / or on the ship hull. Meteorological data can be obtained from a weather station.

[0020] The acquired monitoring data is appended with GPS timestamps and spatial coordinates to construct a spatiotemporally aligned data matrix. In this embodiment, the IEEE 1588 Precision Time Protocol (PTP) is used for time synchronization to minimize time synchronization errors among monitoring devices and sensors. Spatial data from each sensor is uniformly converted using a dock reference coordinate system (e.g., the origin is the berthing target).

[0021] S204: Generate a 3D environment map with temperature attributes by combining the LiDAR point cloud data with the infrared thermal imaging data through a pixel-level fusion algorithm. The 3D environment map can be generated based on the following steps: S2041. Project the laser radar point cloud data onto the infrared thermal imaging data image plane.

[0022] S2042. For each laser point corresponding to the lidar point cloud data, obtain the temperature value at the infrared thermal imaging data image plane coordinate to generate a point cloud with temperature attributes. For laser points without infrared thermal imaging data, radial basis function interpolation can be used to obtain the temperature attribute.

[0023] S2043. Divide the port space into voxels of a predetermined size. The predetermined size can be set according to the size of the port space and the detection accuracy required by the lock. Each voxel stores the point cloud density as a geometric attribute and the mean temperature within the voxel as a temperature attribute to form a 3D environment map.

[0024] S2044: Obstacle marking is performed on the 3D environment map to generate a marked 3D environment map. In this embodiment of the present invention, marking is performed using the following steps: if the point cloud density is greater than a first density threshold and the average temperature is greater than a first temperature threshold, the voxel is marked as a ship metal structure obstacle; if the point cloud density is greater than the first density threshold and the average temperature is less than a second temperature threshold, the voxel is marked as a static obstacle.

[0025] S206. Generate environmental perception feature vectors and hull state feature vectors based on the above environmental monitoring data and state monitoring data. The environmental perception feature vector includes wind speed mutation gradient, visibility attenuation coefficient and obstacle distance-temperature correlation vector; the hull state feature vector includes vibration energy entropy, attitude angle change rate and stress concentration area strain energy density. The wind speed mutation gradient represents the rate of change of wind speed in unit time, which can reflect the degree of wind mutation. The greater the degree of mutation, the more drastic the change in the hull power disturbance, which increases the risk of berthing. It can be calculated using the following formula: ; in, represents the wind speed vector at the current moment, represents the wind speed vector at the previous moment, Represents the time interval between two measurements, and the wind speed vector can be obtained from the weather station.

[0026] Visibility attenuation coefficient It can be calculated using the following formula: ; in, Indicates the maximum range of the visibility sensor of the weather station, Indicates the current visibility. represents the attenuation coefficient, which can be adjusted according to the actual environment. In the embodiment of the present invention, it is set to 0.02.

[0027] Obstacles are detected using LiDAR point cloud data. The obstacle distance-temperature correlation vector is generated for the 5 closest obstacles detected. : ; in, Indicates the distance from the obstacle to the hull, represents the surface temperature of the obstacle. The obstacle distance-temperature association vector considers both the distance and temperature properties of the obstacle. It can distinguish different types of obstacles (such as ships, buoys, and dock equipment) and assess their threat levels (high-temperature metal obstacles are usually ships, which have a high collision risk).

[0028] The vibration energy entropy can be calculated by performing FFT (Fast Fourier Transform) on the vibration signal collected by the acceleration sensor to obtain the spectrum and calculate the energy ratio of each frequency component; then the entropy is calculated based on the energy ratio. : ; in, It represents the energy ratio of each frequency component and is calculated using the following formula: ; Represents frequency components The amplitude of the vibration is represented by N, where N represents the number of frequency components. A more even distribution of vibration energy results in a higher entropy value, indicating multi-source vibration or abnormal vibration (such as a mechanical failure). A more concentrated distribution of vibration energy results in a lower entropy value, indicating normal vibration.

[0029] The attitude angle change rate can reflect the severity of the change in the hull attitude. A large change rate indicates that the hull is in an unstable state and there is a risk of capsizing or collision. The attitude angle change rate is characterized by three attitude angles for the hull attitude, namely roll angle, pitch angle and bow angle. The change rates of the three attitude angles are calculated separately: ; in, represents the rate of change of the roll angle, represents the rate of change of pitch angle, represents the rate of change of the heading angle, Indicates the roll angle at the current moment, Indicates the roll angle at the previous moment, The time interval between two measurements, the parameters in the formulas for the pitch angle change rate and the yaw angle change rate have similar meanings. The second norm of each of the above change rates is taken as the total attitude angle change rate: ; The strain energy density of the stress concentration area can reflect the energy accumulation in the key structural area of ​​the hull. A high value indicates the risk of structural damage. To obtain the strain energy density of the stress concentration area, it is necessary to first identify the stress concentration area. In the embodiment of the present invention, the area where the strain value exceeds two standard deviations of the average strain is identified as the stress concentration area. For the identified stress concentration area, the strain energy density of each strain gauge in the area is calculated. : ; in, represents stress, represents the strain, according to Hooke's law , E represents the elastic modulus. By summing and normalizing all strain gauges in the stress concentration area, the strain energy density in the stress concentration area can be obtained: ; in, represents the stress concentration area, Indicates the area or weight represented by the strain gauge.

[0030] S208: Input the 3D environment map, environment perception feature vector, and hull state feature vector into a preset dynamic risk prediction model for prediction. In this embodiment of the present invention, the dynamic risk prediction model utilizes a Long Short-Term Memory (LSTM)-Transformer hybrid model, which can be trained using historical data. Inputting the input feature vector into the model yields two risk values: an environmental risk value and a structural risk value. In the LSTM-Transformer hybrid model, the LSTM network is used to capture the temporal evolution of the environment and hull data. The LSTM network includes three gating mechanisms: a forget gate, an input gate, and an output gate, enabling effective learning of long-range dependencies. The Transformer encoder receives the environment and hull context vectors output by the LSTM network and utilizes a self-attention mechanism and cross-feature interactions to explicitly model the coupling effects between the environment and hull state. In reality, some interactions between environment perception features and hull state features may be weakly correlated, and forcing full connectivity in the Transformer encoder may introduce noise and increase computational overhead.

[0031] According to certain optional embodiments, the Transformer encoder includes a first attention module, a second attention module, and an information fusion output module. The first attention module is an intra-group self-attention module that performs two self-attention calculations in parallel: a first self-attention calculation and a second self-attention calculation. The first self-attention calculation is used to interact with environmental perception features and learn the relationships between them; the second self-attention calculation is used to interact with ship state features and learn the relationships between them. The second attention module is an inter-group cross-attention module that performs two cross-attention calculations in parallel: a first cross-attention calculation and a second cross-attention calculation. The first cross-attention calculation is a cross-attention calculation that learns environmental perception features and ship state features, using the ship state as the "query" and searching for relevant "keys" and "values" in the environmental perception features; the second cross-attention calculation is a cross-attention calculation that learns ship state features and environmental perception features, using the environmental perception as the "query" and searching for relevant "keys" and "values" in the ship state features. The information fusion output module is used to fuse and output the updated environmental perception features from the first self-attention calculation with the output from the second cross-attention calculation, and to fuse and output the updated hull state features from the second self-attention calculation with the output from the first cross-attention calculation. Specifically, the updated environmental perception features from the first attention module (i.e., the intra-group self-attention module) are fused with the output from the cross-attention calculation of the hull state features in the second attention module (i.e., the inter-group cross-attention module) → the environmental perception features; and the updated hull state features from the first attention module (i.e., the intra-group self-attention module) are fused with the output from the cross-attention calculation of the environmental perception features in the second attention module (i.e., the inter-group cross-attention module) → the hull state features. The output vectors are fused by addition or concatenation, and the two fused feature sets are concatenated and input into a subsequent feedforward neural network (FFN). In this embodiment of the present invention, grouping attention reduces unnecessary computation, allowing the model to focus more on physically strongly correlated feature interactions, achieving higher accuracy with fewer parameters and data, and reducing the risk of overfitting.

[0032] S210. Perform berthing warning based on the prediction results. Using the above dynamic risk prediction model, two types of risk values ​​can be obtained: environmental risk value and structural risk value. Set multiple warning levels for each risk value, and the setting of the warning level can be calibrated based on historical data, simulation and expert experience. For example, the warning level can be set to low, medium, high and emergency according to the value range of the risk value. Generate the final global warning level based on the combined warning level of the environmental risk value and the structural risk value. The generation of the global warning level is based on the first principle and the second principle. In the first principle, the global warning level is determined by the higher level of the environmental risk value and the structural risk value. For example, if the warning level of the environmental risk value is medium and the warning level of the structural risk value is high, the global warning level is high. In the second principle, if any warning level reaches emergency, the global warning level is emergency.

[0033] In summary, an embodiment of the present invention relates to a berthing warning method for an unmanned screw unloader, comprising the steps of: acquiring environmental monitoring data of the screw unloader port and status monitoring data of the hull; generating a 3D environmental map with temperature attributes using a pixel-level fusion algorithm for lidar point cloud data and infrared thermal imaging data; generating an environmental perception feature vector and a hull status feature vector based on the above-mentioned environmental monitoring data and status monitoring data; inputting the 3D environmental map, environmental perception feature vector and hull status feature vector into a preset dynamic risk prediction model for prediction; and performing a berthing warning based on the prediction results. The technical solution provided by the embodiment of the present invention monitors environmental data and hull status data, and generates dynamically changing feature data and cross-dimensional correlation features as input vectors of the prediction model based on them, thereby effectively improving the environmental perception accuracy and modeling accuracy of the model; adopts the LSTM-Transformer hybrid model and combines it with the grouping attention mechanism, which not only ensures the capture of real-time time series risks, but also strengthens the modeling of long-distance lag risks. By reducing unnecessary calculations through grouping, the model's attention is more focused on feature interactions that are strongly correlated physically, thereby achieving higher accuracy with fewer parameters and data and reducing the risk of overfitting.

[0034] It should be understood that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention (including the claims) is limited to these examples. Within the spirit of the present invention, the technical features of the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of one or more embodiments of the present invention described above, which are not provided in detail for the sake of clarity. The above specific embodiments of the present invention are merely intended to illustrate or explain the principles of the present invention and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included within the scope of protection of the present invention. In addition, the appended claims of the present invention are intended to cover all variations and modifications that fall within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A berthing warning method for an unmanned screw unloader, characterized in that: Including steps: Obtaining environmental monitoring data of the screw unloader port and status monitoring data of the hull; the environmental monitoring data includes at least laser radar point cloud data and infrared thermal imaging data; Generate a 3D environmental map with temperature attributes by using a pixel-level fusion algorithm to combine lidar point cloud data and infrared thermal imaging data; Generate an environmental perception feature vector and a hull state feature vector based on the above environmental monitoring data and state monitoring data; Input the 3D environmental map, environmental perception feature vector and hull state feature vector into the preset dynamic risk prediction model for prediction; Providing berthing warnings based on the prediction results; Among them, the dynamic risk prediction model adopts an LSTM-Transformer hybrid model, and the Transformer encoder in the LSTM-Transformer hybrid model includes an intra-group self-attention module and an inter-group cross-attention module.

2. The method according to claim 1, characterized in that The laser radar point cloud data and infrared thermal imaging data are fused through a pixel-level algorithm to generate a 3D environmental map with temperature attributes, including the following steps: Projecting the LiDAR point cloud data onto the infrared thermal imaging data image plane; For each laser point corresponding to the lidar point cloud data, the temperature value at the coordinate of the infrared thermal imaging data image plane is obtained to generate a point cloud with temperature attributes; The port space is divided into voxels of a predetermined size. Each voxel stores the point cloud density as a geometric attribute and the mean temperature within the voxel as a temperature attribute to form a 3D environmental map. Obstacles are marked on the 3D environment map to generate a marked 3D environment map.

3. The method according to claim 2, characterized in that Marking obstacles on a 3D environment map includes the following steps: If the point cloud density is greater than the first density threshold and the temperature mean is greater than the first temperature threshold, the voxel is marked as a ship metal structure obstacle; If the point cloud density is greater than a first density threshold and the temperature mean is less than a second temperature threshold, the voxel is marked as a static obstacle.

4. The method according to claim 1, wherein The environmental monitoring data also includes meteorological data; the status monitoring data includes vibration frequency, attitude angle, dock displacement deviation and structural stress value.

5. The method according to claim 4, characterized in that The environmental perception feature vector includes the wind speed mutation gradient, the visibility attenuation coefficient and the obstacle distance-temperature correlation vector; the hull state feature vector includes the vibration energy entropy, the attitude angle change rate and the strain energy density of the stress concentration area.

6. The method according to any one of claims 1 to 5, characterized in that Inputting the 3D environmental map, the environmental perception feature vector, and the hull state feature vector into a preset dynamic risk prediction model for prediction, thereby obtaining two types of risk values, including an environmental risk value and a structural risk value; A berthing warning is issued based on the environmental risk value and the structural risk value.

7. The method according to claim 6, characterized in that The self-attention module within the group performs a first self-attention calculation and a second self-attention calculation in parallel; The first self-attention calculation is used to interact between environmental perception features and learn the relationship between environmental perception features; the second self-attention calculation is used to interact between hull state features and learn the relationship between hull state features.

8. The method according to claim 7, characterized in that The inter-group cross attention module performs a first cross attention calculation and a second cross attention calculation in parallel; The first cross-attention calculation uses the hull state as a query and searches for relevant clues and information in the environmental perception features; the second cross-attention calculation uses environmental perception as a query and searches for relevant clues and information in the hull state features.

9. The method according to claim 8, characterized in that The Transformer encoder also includes an information fusion output module; The information fusion output module is used to fuse the environmental perception features updated by the first self-attention calculation with the output of the second cross-attention calculation and output them; It is also used to fuse the hull state features updated by the second self-attention calculation with the output of the first cross-attention calculation and output it.

10. The method according to claim 9, characterized in that The information fusion output module is further used to splice the two fused sets of features and input them into the feedforward neural network.

Images