Mooring early warning method for unattended screw ship unloader

Abstract

The embodiment of the present application relates to a kind of unmanned spiral ship unloader berthing early warning method, comprising the following steps: obtaining the environmental monitoring data and the state monitoring data of ship body of spiral ship unloader port;Laser radar point cloud data and infrared thermal imaging data are generated 3D environment map with temperature attribute by pixel-level fusion algorithm;Environment perception feature vector and ship body state feature vector are generated based on the above environmental monitoring data and state monitoring data;3D environment map, environment perception feature vector and ship body state feature vector are input into the preset dynamic risk prediction model for prediction;According to the result of the prediction, berthing early warning is carried out.The technical scheme provided by the embodiment of the present application, by monitoring environmental data and ship body state data, and generating dynamic change feature data and cross-dimension associated features as input vectors of prediction model, effectively improves the environment perception accuracy and modeling accuracy of the model.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the unmanned spiral ship unloader technical field, and particularly relates to a berthing early warning method of an unmanned spiral ship unloader. BACKGROUND

[0002] In the prior art, the berthing early warning of the unmanned spiral ship unloader usually depends on a single sensor, and the berthing scene is relatively complex and is affected by many factors. The single sensor can cause insufficient perception of the berthing scene, thereby causing inaccurate early warning results.

[0003] In addition, the risk prediction model used in the prior art is usually constructed by a single model. The single model usually focuses on a certain specific data type or task target, and cannot comprehensively cover complex requirements, thereby having problems such as limited prediction ability of the model, poor environmental adaptability, low data utilization efficiency, and insufficient robustness. SUMMARY

[0004] Based on the above problems of the prior art, the purpose of the embodiment of the present application is to provide a berthing early warning method of an unmanned spiral ship unloader, which can effectively improve the reliability and accuracy of the berthing early warning of the unmanned spiral ship unloader by monitoring multi-dimensional data and constructing a hybrid model for dynamic risk prediction.

[0005] To achieve the above purpose, according to one aspect of the present application, a berthing early warning method of an unmanned spiral ship unloader is provided, which comprises the following steps:

[0006] obtaining environment monitoring data of a spiral ship unloader port and state monitoring data of a ship body; the environment monitoring data at least includes laser radar point cloud data and infrared thermal imaging data;

[0007] generating a 3D environment map with temperature attributes by pixel-level fusion algorithm from the laser radar point cloud data and the infrared thermal imaging data;

[0008] generating an environment perception feature vector and a ship body state feature vector based on the above environment monitoring data and state monitoring data;

[0009] inputting the 3D environment map, the environment perception feature vector and the ship body state feature vector into a preset dynamic risk prediction model for prediction;

[0010] performing berthing early warning according to the prediction result;

[0011] The dynamic risk prediction model adopts an LSTM-Transformer hybrid model, and the Transformer encoder in the LSTM-Transformer hybrid model includes an intra-group self-attention module and an inter-group cross-attention module.

[0012] Further, the laser radar point cloud data and the infrared thermal imaging data are fused by a pixel-level fusion algorithm to generate a 3D environment map with temperature attributes, including the steps of:

[0013] projecting the laser radar point cloud data onto the image plane of the infrared thermal imaging data;

[0014] for each laser radar point cloud data corresponding to a laser point, obtaining the temperature value at the image plane coordinate of the infrared thermal imaging data to generate a point cloud with temperature attributes;

[0015] dividing the port space into voxels of a predetermined size, each voxel storing the point cloud density as a geometric attribute and storing the temperature mean value in the voxel as a temperature attribute to form a 3D environment map;

[0016] marking obstacles on the 3D environment map to generate a marked 3D environment map.

[0017] Further, marking obstacles on the 3D environment map includes the steps of:

[0018] if the point cloud density is greater than a first density threshold and the temperature mean value is greater than a first temperature threshold, the voxel is marked as a ship metal structure obstacle;

[0019] if the point cloud density is greater than a first density threshold and the temperature mean value is less than a second temperature threshold, the voxel is marked as a static obstacle.

[0020] Further, the environment monitoring data further includes weather data; and the state monitoring data includes vibration frequency, attitude angle, wharf displacement deviation, and structure stress value.

[0021] Further, the environment perception feature vector includes wind speed mutation gradient, visibility attenuation coefficient, and obstacle distance-temperature correlation vector; and the ship state feature vector includes vibration energy entropy, attitude angle change rate, and stress concentration area strain energy density.

[0022] Further, the 3D environment map, the environment perception feature vector, and the ship state feature vector are input into a preset dynamic risk prediction model for prediction to obtain two types of risk values, including an environmental risk value and a structure risk value;

[0023] performing berthing warning according to the environmental risk value and the structure risk value.

[0024] Further, the intra-group self-attention module performs first self-attention calculation and second self-attention calculation in parallel;

[0025] The first self-attention calculation is used for interaction between the environment perception features to learn the relationship between the environment perception features, and the second self-attention calculation is used for interaction between the hull state features to learn the relationship between the hull state features.

[0026] Further, the inter-group cross-attention module performs the first cross-attention calculation and the second cross-attention calculation in parallel.

[0027] The first cross-attention calculation takes the hull state as a query to find relevant clue information in the environment perception features, and the second cross-attention calculation takes the environment perception as a query to find relevant clues and information in the hull state features.

[0028] Further, the Transformer encoder further comprises an information fusion output module.

[0029] The information fusion output module is configured to output the environment perception features updated by the first self-attention calculation after being fused with the output of the second cross-attention calculation, and output the hull state features updated by the second self-attention calculation after being fused with the output of the first cross-attention calculation.

[0030] Further, the information fusion output module is further configured to splice the two groups of features after fusion and input them into a feedforward neural network.

[0031] In summary, the embodiment of the present application provides an unattended spiral ship unloader berthing early warning method, comprising the steps of: acquiring environment monitoring data and ship body state monitoring data of a spiral ship unloader port; generating a 3D environment map with temperature attributes by pixel-level fusion algorithm based on laser radar point cloud data and infrared thermal imaging data; generating environment perception feature vectors and hull state feature vectors based on the above-mentioned environment monitoring data and state monitoring data; inputting the 3D environment map, the environment perception feature vectors and the hull state feature vectors into a preset dynamic risk prediction model for prediction; and performing berthing early warning according to the prediction result. The technical scheme provided by the embodiment of the present application effectively improves the environment perception accuracy and modeling accuracy of the model by monitoring the environment data and ship body state data, and generating dynamic change feature data and cross-dimension correlation features as input vectors of the prediction model. The LSTM-Transformer hybrid model combined with the grouping attention mechanism not only ensures the capture of real-time time series risks, but also strengthens the modeling of long-distance lag risks. By grouping, unnecessary calculations are reduced, so that the model attention is more focused on the physical interaction of features with strong correlation, thereby using fewer parameters and data to achieve higher accuracy and reduce the risk of overfitting. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1is a flow chart of the berthing early warning method of the unattended screw ship unloader provided by the embodiment of the present application. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and advantages of the present application clearer, further detailed description will be made to the present application with reference to the specific embodiments and the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of the well-known structures and technologies will be omitted to avoid unnecessary confusion of the concept of the present application.

[0034] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should be understood as the common meanings understood by those skilled in the art to which the present application belongs. The terms "first", "second", and similar terms used in one or more embodiments of the present application do not represent any order, number, or importance, but are only used to distinguish different components. The terms "include" or "contain" and similar terms mean that the elements or objects listed before the terms cover the elements or objects listed after the terms and their equivalents, without excluding other elements or objects. The terms "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0035] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings. The embodiments of the present application provide a berthing early warning method of an unattended screw ship unloader, Figure 1 The flow chart of the berthing early warning method of the unattended screw ship unloader provided by the embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the method comprises the following steps:

[0036] S202, acquiring environmental monitoring data and state monitoring data of the screw ship unloader port, the environmental monitoring data comprising laser radar point cloud data, infrared thermal imaging data, and meteorological data, etc. In the embodiment of the present application, the ship body refers to the ship body of the transport ship which is the operation object of the screw ship unloader, and the state monitoring data comprises vibration frequency, attitude angle, wharf displacement deviation, and structure stress value, etc. The above environmental monitoring data and state monitoring data can be acquired through laser radars, infrared thermal imagers, gyroscopes, and various sensors arranged at the port and / or the ship body, and the meteorological data can be acquired from a meteorological station.

[0037] The acquired monitoring data is attached with a GPS time stamp and a space coordinate, and a time-space aligned data matrix is constructed. In the embodiment of the present application, the IEEE 1588 Precision Time Protocol (PTP) is used for time synchronization to minimize the time synchronization error of each monitoring device and sensor. The space data of each sensor is uniformly converted through a wharf reference coordinate system (for example, the origin is the berthing target point).

[0038] S204, generating a 3D environment map with temperature attribute by pixel-level fusion algorithm of the lidar point cloud data and the infrared thermal imaging data. The 3D environment map can be generated based on the following steps:

[0039] S2041, projecting the lidar point cloud data to the image plane of the infrared thermal imaging data.

[0040] S2042, for each laser point corresponding to the lidar point cloud data, obtaining the temperature value at the image plane coordinate of the infrared thermal imaging data to generate a point cloud with temperature attribute. For the laser points without infrared thermal imaging data, radial basis function interpolation can be used to obtain.

[0041] S2043, dividing the port space into voxels of a predetermined size, which can be set according to the size of the port space and the required detection accuracy of the lock, each voxel storing the point cloud density as a geometric attribute and the temperature mean value in the voxel as a temperature attribute, to form a 3D environment map.

[0042] S2044, marking obstacles on the 3D environment map to generate a marked 3D environment map. In the embodiment of the present application, the following steps are used for marking: if the point cloud density is greater than the first density threshold and the temperature mean value is greater than the first temperature threshold, the voxel is marked as a ship metal structure obstacle; if the point cloud density is greater than the first density threshold and the temperature mean value is less than the second temperature threshold, the voxel is marked as a static obstacle.

[0043] S206, generating an environment perception feature vector and a ship body state feature vector based on the above environment monitoring data and state monitoring data. The environment perception feature vector includes wind speed mutation gradient, visibility attenuation coefficient and obstacle distance-temperature correlation vector; the ship body state feature vector includes vibration energy entropy, attitude angle change rate and stress concentration area strain energy density. The wind speed mutation gradient represents the change rate of wind speed in unit time, which can reflect the mutation degree of wind, and the greater the mutation degree, the more violent the disturbance force change of the ship body, increasing the berthing risk. The wind speed mutation gradient can be calculated by the following formula:

[0044] ;

[0045] wherein, represents the wind speed vector at the current time, represents the wind speed vector at the previous time, represents the time interval between two measurements, and the wind speed vector can be obtained from the weather station.

[0046] The visibility attenuation coefficient can be calculated by the following formula:

[0047] ;

[0048] wherein, represents the maximum range of the weather station visibility sensor, represents the current visibility, represents the attenuation coefficient, which can be adjusted according to the actual environment, and is taken as 0.02 in the embodiment of the present application.

[0049] The laser radar point cloud data is used to detect obstacles, and for the detected obstacles, the five closest obstacles are used to generate an obstacle distance-temperature correlation vector :

[0050] ;

[0051] wherein, represents the distance of the obstacle to the ship body, represents the surface temperature of the obstacle, and the obstacle distance-temperature correlation vector considers both the distance and temperature properties of the obstacle, so that different types of obstacles (such as ships, buoys, and terminal equipment) can be distinguished, and the threat level can be evaluated (high-temperature metal obstacles are usually ships, and the collision risk is high).

[0052] The vibration energy entropy can be obtained by performing FFT (Fast Fourier Transform) processing on the vibration signal collected by the acceleration sensor to obtain a frequency spectrum, and calculating the energy proportion of each frequency component; then the entropy is calculated based on the energy proportion :

[0053] ;

[0054] wherein, represents the energy proportion of each frequency component, which is calculated using the following formula:

[0055] ;

[0056] represents the amplitude of the frequency component , and N represents the number of frequency components. The more uniform the vibration energy distribution, the higher the entropy value, indicating that there is multi-source vibration or abnormal vibration (such as mechanical failure); the more concentrated the vibration energy distribution, the lower the entropy value, indicating normal vibration.

[0057] The attitude angle change rate can reflect the degree of change of the ship body attitude, and a large change rate indicates that the ship body is in an unstable state, and there is a risk of capsizing or collision. Three attitude angles are used to represent the attitude angle change rate of the ship body attitude, which are the roll angle, the pitch angle, and the yaw angle. The change rates of the three attitude angles are calculated respectively:

[0058] ;

[0059] where, denotes the roll angle rate, denotes the pitch angle rate, denotes the yaw angle rate, denotes the current roll angle, denotes the previous roll angle, denotes the time interval between two measurements, the parameter meanings in the formulas of the pitch angle rate and the yaw angle rate are similar. Take the two-norm of the above rates as the total attitude angle rate:

[0060] ;

[0061] The stress concentration area strain energy density can reflect the energy accumulation of the key structure area of the ship body, and a high value indicates that there is a risk of structural damage. The acquisition of the stress concentration area strain energy density needs to identify the stress concentration area first, and in the embodiment of the present application, the area with a strain value exceeding twice the standard deviation of the average strain is identified as the stress concentration area. For the identified stress concentration area, the strain energy density of each strain gauge in the area is calculated :

[0062] ;

[0063] where, denotes the stress, denotes the strain, according to Hooke's law , E denotes the elastic modulus. Summing up all the strain gauges in the stress concentration area and normalizing, the stress concentration area strain energy density can be obtained:

[0064] ;

[0065] where, denotes the stress concentration area, denotes the area or weight represented by the strain gauge.

[0066] S208, input the 3D environment map, the environment perception feature vector and the hull state feature vector into a preset dynamic risk prediction model for prediction. In the embodiment of the application, the dynamic risk prediction model adopts a Long Short-Term Memory (LSTM for short in the following) -Transformer hybrid model, which can be obtained by training historical data. The input feature vector is input into the model, and two types of risk values can be obtained: an environmental risk value and a structural risk value. In the LSTM-Transformer hybrid model, the LSTM network is used to capture the time evolution law in the environment and hull data, and the LSTM network includes three gating mechanisms: a forgetting gate, an input gate and an output gate, which can effectively learn long-distance dependencies. The Transformer encoder receives the environment and hull context vectors output by the LSTM network, uses the self-attention mechanism and cross-feature interaction, and explicitly models the coupling effect of the environment and the hull state. In fact, some interactions between the environment perception features and the hull state features may have weak correlations, and forced full connection in the Transformer encoder may introduce noise and increase the computational burden.

[0067] According to certain optional embodiments, the Transformer encoder comprises a first attention module, a second attention module and an information fusion output module. The first attention module is an intra-group self-attention module, and two self-attention calculations are performed in parallel, which are a first self-attention calculation and a second self-attention calculation. The first self-attention calculation is used to interact between the environmental perception features to learn the relationship between the environmental perception features. The second self-attention calculation is used to interact between the ship state features to learn the relationship between the ship state features. The second attention module is an inter-group cross-attention module, and two directions of cross-attention calculation are performed in parallel, which are a first cross-attention calculation and a second cross-attention calculation. The first cross-attention calculation is a cross-attention calculation for learning environmental perception features ship state features, taking the ship state as the “query (Query)”, and finding the “clue” (Key) and “information” (Value) related to it in the environmental perception features. The second cross-attention calculation is a cross-attention calculation for learning ship state features environmental perception features, taking the environmental perception as the “query (Query)”, and finding the “clue” (Key) and “information” (Value) related to it in the ship state features. The information fusion output module is used to fuse and output the environmental perception features updated by the first self-attention calculation and the output of the second cross-attention calculation, and fuse and output the ship state features updated by the second self-attention calculation and the output of the first cross-attention calculation, that is, fuse and output the environmental perception features updated in the first attention module, i.e., the intra-group self-attention module, and the output of the cross-attention calculation of the ship state features environmental perception features in the second attention module, i.e., the inter-group cross-attention module. Fuse and output the ship state features updated in the first attention module, i.e., the intra-group self-attention module, and the output of the cross-attention calculation of the environmental perception features ship state features in the second attention module, i.e., the inter-group cross-attention module. Among them, the output vectors are fused in an additive or splicing manner, and the two fused feature groups are spliced and input to the subsequent feedforward neural network (FFN). In the embodiment of the application, unnecessary calculations are reduced by grouping attention, so that the model attention is more focused on the physical interaction of features with strong correlation, achieving higher precision with fewer parameters and data, and reducing the risk of overfitting.

[0068] S210, berthing early warning is performed according to the predicted result. By using the above dynamic risk prediction model, two types of risk values can be obtained: environmental risk value and structural risk value. A plurality of early warning levels are set for each risk value, and the setting of the early warning level can be calibrated according to historical data, simulation and expert experience. For example, the early warning level can be set to low, medium, high and urgent according to the value range of the risk value. According to the combination of the early warning level of the environmental risk value and the structural risk value, the final global early warning level is generated. The generation of the global early warning level is based on the first principle and the second principle. In the first principle, the global early warning level is determined by the higher level of the environmental risk value and the structural risk value. For example, the early warning level of the environmental risk value is medium, and the early warning level of the structural risk value is high, and the global early warning is high. In the second principle, if any one early warning level reaches urgent, the global early warning level is urgent.

[0069] In summary, the embodiment of the present application relates to a berthing early warning method of an unattended screw ship unloader, comprising the steps of: obtaining environmental monitoring data and ship body state monitoring data of the screw ship unloader port; generating a 3D environment map with temperature attribute by pixel-level fusion algorithm from laser radar point cloud data and infrared thermal imaging data; generating an environment perception feature vector and a ship body state feature vector based on the above-mentioned environmental monitoring data and state monitoring data; inputting the 3D environment map, the environment perception feature vector and the ship body state feature vector into a preset dynamic risk prediction model for prediction; and performing berthing early warning according to the predicted result. The technical scheme provided by the embodiment of the present application effectively improves the environment perception accuracy and modeling accuracy of the model by monitoring the environmental data and ship body state data, and generating dynamic change feature data and cross-dimension correlation features as input vectors of the prediction model; by using the LSTM-Transformer hybrid model combined with the grouping attention mechanism, both real-time time series risk capture and long-distance lag risk modeling are ensured, unnecessary calculation is reduced by grouping, and model attention is more focused on the physical interaction of strongly correlated features, so that fewer parameters and data can be used to achieve higher accuracy and reduce the risk of overfitting.

[0070] It should be understood that all the above-described embodiments are merely exemplary and are not intended to limit the scope of the present application (including claims) to these examples; the technical features among the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of one or more embodiments of the present application as described above, which are not provided in details for the sake of simplicity. The above detailed description of the present application merely serves to illustrate or explain the principles of the present application, and does not constitute a limitation on the present application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present application shall be included in the protection scope of the present application. In addition, the claims of the present application are intended to cover all changes and modifications falling within the scope and boundary of the appended claims, or the equivalent forms of such scope and boundary.

Claims

1. A berthing early warning method for an unattended screw-type ship unloader, characterized by, The method comprises the steps of: acquiring environmental monitoring data and hull state monitoring data of a spiral ship unloader port; the environmental monitoring data at least includes laser radar point cloud data and infrared thermal imaging data; generating a 3D environmental map with temperature attributes by pixel-level fusion algorithm of the laser radar point cloud data and the infrared thermal imaging data, comprising: projecting the laser radar point cloud data to the image plane of the infrared thermal imaging data; for each laser radar point cloud data corresponding to a laser point, acquiring the temperature value at the image plane coordinate of the infrared thermal imaging data to generate a point cloud with temperature attributes; dividing the port space into voxels of a predetermined size, each voxel storing the point cloud density as a geometric attribute and storing the temperature mean value in the voxel as a temperature attribute to form a 3D environmental map; marking obstacles on the 3D environmental map to generate a marked 3D environmental map; generating an environmental perception feature vector and a hull state feature vector based on the above environmental monitoring data and state monitoring data; inputting the 3D environmental map, the environmental perception feature vector and the hull state feature vector into a preset dynamic risk prediction model for prediction; performing berthing pre-warning according to the prediction result; wherein the dynamic risk prediction model adopts an LSTM-Transformer hybrid model, and the Transformer encoder in the LSTM-Transformer hybrid model includes an intra-group self-attention module and an inter-group cross-attention module.

2. The method of claim 1, wherein, The method of marking obstacles on the 3D environmental map comprises the steps of: if the point cloud density is greater than a first density threshold and the temperature mean value is greater than a first temperature threshold, the voxel is marked as a ship metal structure obstacle; if the point cloud density is greater than the first density threshold and the temperature mean value is less than a second temperature threshold, the voxel is marked as a static obstacle.

3. The method of claim 1, wherein, The environmental monitoring data further includes meteorological data; the state monitoring data includes vibration frequency, attitude angle, wharf displacement deviation and structural stress value.

4. The method of claim 3, wherein, The environmental perception feature vector includes wind speed mutation gradient, visibility attenuation coefficient and obstacle distance-temperature correlation vector; the hull state feature vector includes vibration energy entropy, attitude angle change rate and stress concentration area strain energy density.

5. The method according to any one of claims 1 to 4, characterized in that, The 3D environmental map, the environmental perception feature vector and the hull state feature vector are input into a preset dynamic risk prediction model for prediction to obtain two types of risk values, including environmental risk value and structural risk value; performing berthing pre-warning according to the environmental risk value and the structural risk value.

6. The method of claim 5, wherein, The intra-group self-attention module performs first self-attention calculation and second self-attention calculation in parallel; The first self-attention calculation is used for interaction between environmental perception features to learn the relationship between environmental perception features; the second self-attention calculation is used for interaction between hull state features to learn the relationship between hull state features.

7. The method of claim 6, wherein, The inter-group cross-attention module performs first cross-attention calculation and second cross-attention calculation in parallel; The first cross-attention calculation takes the hull state as a query and looks for clue information related to the query in the environment perception features; the second cross-attention calculation takes the environment perception as a query and looks for clue information related to the query in the hull state features.

8. The method of claim 7, wherein, The Transformer encoder further comprises an information fusion output module; The information fusion output module is configured to output the environment perception features updated by the first self-attention calculation after being fused with the output of the second cross-attention calculation. The information fusion output module is further configured to output the hull state features updated by the second self-attention calculation after being fused with the output of the first cross-attention calculation.

9. The method of claim 8, wherein, The information fusion output module is further configured to splice the two groups of fused features and input the spliced features into a feedforward neural network.

Images