Intelligent port safety early warning method based on multi-source perception

Abstract

The invention discloses an intelligent port safety early warning method based on multi-source perception. The method comprises the steps that S1, three-dimensional point cloud data and sensor data of a to-be-detected material pile are acquired; s2, constructing a corresponding material pile three-dimensional model based on the three-dimensional point cloud data of the to-be-detected material pile; s3, extracting appearance structure features of the to-be-detected material pile based on the material pile three-dimensional model; s4, extracting sensor sensing features of the to-be-detected material pile based on the sensor data; s5, performing feature fusion on the appearance structure features of the to-be-detected material pile and the sensor sensing features to obtain fusion features; s6, inputting the fusion features of the to-be-detected material pile into the trained collapse prediction model, and outputting a corresponding predicted collapse probability; and S7, realizing safety early warning of the to-be-detected material pile based on the predicted collapse probability output by the collapse prediction model. According to the invention, high-precision and real-time monitoring of the safety state of the material pile is realized in the technical level, and the safety management level of a port is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of safety management of intelligent ports, and particularly relates to a safety early warning method for intelligent ports based on multi-source perception. Background Art

[0002] With the rapid development of globalization and informatization, traditional ports are gradually transforming into intelligent ports. The construction of intelligent ports covers all aspects of port operation, from cargo handling, warehousing management to logistics tracking, safety monitoring, etc. Through the Internet of Things technology, various devices, vehicles and containers in the port can be monitored and scheduled in real time, greatly improving the operation efficiency. At the same time, big data analysis and artificial intelligence technologies are widely used in predicting port throughput, optimizing route planning, improving customer service quality, etc., providing a scientific basis for port managers to make decisions.

[0003] In terms of safety management, intelligent ports use video monitoring, sensor networks and intelligent analysis systems to monitor and early warn of potential safety hazards in the port area in real time. However, although intelligent ports have achieved remarkable results in improving the overall operation efficiency and management level, there are still some problems to be solved urgently in the safety management of material piles.

[0004] Among them, the material pile is the main form of storing bulk goods in the port, and its stability and safety are directly related to the safety and efficiency of port operation. However, the traditional safety management method of material piles mainly relies on manual inspection and empirical judgment. This method is not only time-consuming and laborious, but also difficult to achieve real-time monitoring and accurate early warning of the state of material piles. With the continuous increase of port cargo throughput, the scale of material piles is also constantly expanding, which brings greater challenges to safety management. On the one hand, factors such as the height, shape and stacking density of the material pile directly affect its stability. Once the pile body tilts or collapses, it will pose a serious threat to the surrounding personnel and equipment. On the other hand, the port operation environment is complex and changeable, and factors such as weather and geological conditions may also affect the stability of the material pile, increasing the difficulty of safety management.

[0005] Although modern information technologies such as video surveillance and sensor networks have been introduced in smart ports, their application in the safety management of material piles is still insufficient. The existing monitoring systems mainly rely on two-dimensional image information and are difficult to accurately reflect the three-dimensional shape and internal structure of the material piles. At the same time, although the sensor network can monitor some physical parameters (such as temperature, humidity, etc.) of the material piles in real time, it lacks the ability to comprehensively analyze these parameters and give early warnings, making it difficult to detect potential safety hazards in a timely manner. In addition, due to the diverse types and stacking methods of material piles, the assessment of their stability and safety requires comprehensive consideration of various factors, including material properties, stacking methods, environmental conditions, etc. Traditional empirical judgment methods are often difficult to accurately evaluate the stability of material piles, and the existing intelligent analysis systems also lack dedicated models and algorithms for the safety management of material piles, resulting in limited early warning effects. In summary, there are still obvious defects in the safety management of material piles in smart ports, and an efficient and accurate safety early warning method is urgently needed to solve this problem. Summary of the Invention

[0006] Aiming at the deficiencies of the above-mentioned existing technologies, the technical problem to be solved by the present invention is: how to provide a safety early warning method for smart ports based on multi-source perception, which realizes high-precision and real-time monitoring of the safety status of material piles at the technical level, effectively improves the safety management level of ports, and provides strong technical support for the construction and operation of smart ports.

[0007] To solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A safety early warning method for smart ports based on multi-source perception, comprising:

[0009] S1: Obtain the three-dimensional point cloud data and sensor data of the material pile to be detected;

[0010] S2: Construct a corresponding three-dimensional model of the material pile based on the three-dimensional point cloud data of the material pile to be detected;

[0011] S3: Extract the external structure features of the material pile to be detected based on the three-dimensional model of the material pile;

[0012] S4: Extract the sensor perception features of the material pile to be detected based on the sensor data;

[0013] S5: Perform feature fusion on the external structure features and sensor perception features of the material pile to be detected to obtain fusion features;

[0014] S6: Input the fusion features of the material pile to be detected into the trained collapse prediction model, and output the corresponding predicted collapse probability;

[0015] S7: Realize the safety early warning of the material pile to be detected based on the predicted collapse probability output by the collapse prediction model.

[0016] Preferably, in step S1, the three-dimensional point cloud data of the material pile to be detected includes: ground point cloud data obtained by a ground laser scanning device on the ground where the material pile to be detected is located, and aerial point cloud data obtained by an unmanned aerial vehicle laser scanning device.

[0017] Preferably, in step S1, the sensor data of the material pile to be detected includes any one or more of tilt sensor data, pressure sensor data, wind speed and direction sensor data, and displacement sensor data.

[0018] Preferably, in step S2, the three-dimensional model of the material pile is constructed through the following steps:

[0019] S201: Perform preprocessing including denoising and downsampling on the ground point cloud data and the aerial point cloud data;

[0020] S202: Align the preprocessed ground point cloud data and aerial point cloud data through a point cloud registration algorithm to obtain a point cloud data set;

[0021] S203: Perform filtering processing on the point cloud data in the point cloud data set to remove noise and irrelevant points and retain the effective point cloud data of the material pile;

[0022] S204: Convert the point cloud data in the point cloud data set into a three-dimensional mesh model through a three-dimensional reconstruction algorithm to obtain the three-dimensional model of the material pile.

[0023] Preferably, in step S3, the external shape structure features of the material pile to be detected include any one or more of height features, slope features, volume features, surface area features, stacking angle features, and center of gravity position features.

[0024] Preferably, in step S4, the sensor perception features of the material pile to be detected include any one or more of tilt angle and direction features, pressure distribution features, wind speed and direction features, and displacement features. Each feature is obtained through a corresponding sensor.

[0025] Preferably, in step S5, the feature fusion is realized through the following steps:

[0026] S501: Perform preprocessing including data standardization and spatio-temporal alignment on the external shape structure features and the sensor perception features;

[0027] S502: Encode the preprocessed external shape structure features through a fully connected neural network to obtain an embedded representation of the external shape structure features;

[0028] The formula is expressed as:

[0029] f w = FCN(W);

[0030] In the formula: f w represents the embedded representation of the external shape structure features; FCN represents the fully connected neural network; W represents the external shape structure features;

[0031] S503: Encode the preprocessed sensor perception features through the LSTM network to obtain the embedded representation of the sensor perception features;

[0032] The formula is expressed as:

[0033] f c = LSTM(A 1 , A 2 ,...., A n );

[0034] In the formula: f c represents the embedded representation of the sensor perception features; LSTM represents the LSTM network; A 1 , A 2 ,...., A n represents the time series data of the sensor perception features;

[0035] S504: Dynamically fuse the attention weights of the embedded representation f w of the external shape structure features and the embedded representation f c of the sensor perception features through the attention mechanism to obtain the attention features;

[0036] The formula is expressed as:

[0037]

[0038] In the formula: Attention(Q, K, V) represents the attention features; Q represents the Query of the attention mechanism, which is the embedded representation of the external shape structure features; K and V represent the Key and Value of the attention mechanism, which are the embedded representations of the sensor perception features here; d k represents the feature dimension; softmax represents the normalization function;

[0039] S505: Concatenate the attention features Attention(Q, K, V) with the embedded representation f w of the external shape structure features to obtain the fused features;

[0040] The formula is expressed as:

[0041] f r = Concat(f w , Attention(Q, K, V))

[0042] In the formula: f rIt represents the fused feature; Concat represents the concatenation operation.

[0043] Preferably, in step S6, the collapse prediction model is constructed based on a fully connected neural network model.

[0044] Preferably, in step S6, the collapse prediction model is trained with the following loss function;

[0045] The formula is expressed as:

[0046]

[0047] In the formula: y i represents the true label; represents the predicted collapse probability; N represents the number of samples.

[0048] Preferably, in step S7, when the predicted collapse probability output by the collapse prediction model is greater than a preset safety threshold, it is determined that there is a collapse risk for the material pile to be detected, and a corresponding safety warning signal is issued.

[0049] Compared with the prior art, the intelligent port safety warning method based on multi-source perception in the present invention has the following beneficial effects:

[0050] By combining the 3D point cloud data obtained from ground laser scanning and UAV laser scanning, the present invention realizes high-precision 3D reconstruction of the material pile morphology in the intelligent port, covering not only the surface morphology of the material pile but also deeply capturing its spatial three-dimensional structure, greatly improving the accuracy of the material pile stability analysis. Subsequently, by combining the real-time perception features extracted from sensor data (such as tilt sensors, stress sensors, etc.), the sensitivity and specificity of the warning system are further enhanced, making the safety warning more accurate and timely.

[0051] The present invention effectively integrates the 3D morphological features of the material pile and sensor data through feature fusion. This multi-source information fusion technology can comprehensively reflect the physical state and potential risks of the material pile, overcoming the problems of false alarms or missed alarms that may be caused by a single data source. By processing the fused features with a machine learning model, the system can automatically identify the complex and variable stability states of the material pile, improving the robustness and reliability of the warning system.

[0052] Using the trained collapse prediction model, the present invention can issue a warning signal in advance according to the predicted collapse probability before the material pile actually collapses. This not only provides a valuable time window for port managers to take emergency measures, such as strengthening the structure, adjusting the stacking method, or evacuating surrounding personnel and equipment, but also significantly reduces the probability of accidents and the possible economic losses and casualties caused by them, realizing the transformation from a passive response to an active prevention safety management mode.

[0053] The entire early warning process of the present invention is highly automated. From data collection, model construction to feature extraction and predictive analysis, it reduces manual intervention and improves work efficiency. At the same time, with the accumulation of more data, the collapse prediction model can continuously self-optimize, continuously improve the early warning accuracy and adaptability, and lay a solid technical foundation for the long-term development of intelligent ports. Moreover, by monitoring and managing the safety of material piles through intelligent means, the present invention reduces the resource consumption and environmental pollution caused by traditional manual inspections, which is in line with the development trend of green ports. In addition, the efficient early warning system helps to maintain the stability of port operations, ensure the smooth flow of logistics, and has positive significance for promoting the sustainable development of the regional economy. Brief Description of the Drawings

[0054] In order to make the objectives, technical solutions and advantages of the invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings, where:

[0055] Figure 1 It is a logic block diagram of a smart port safety early warning method based on multi-source perception. Detailed Embodiments

[0056] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Components of the embodiments of the present invention usually described and illustrated in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0057] It should be noted that similar reference numerals and letters denote similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship in which the product of the present invention is usually placed when in use. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present invention. In addition, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance. In addition, terms such as "horizontal" and "vertical" do not mean that the components are required to be absolutely horizontal or hanging vertically, but can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined. In the description of the present invention, it should also be noted that unless otherwise clearly specified and defined, the terms "set", "install", "connect", "couple" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0058] The following will be further described in detail through specific embodiments:

[0059] Embodiment:

[0060] A method for intelligent port safety early warning based on multi-source perception is disclosed in this embodiment.

[0061] As Figure 1 shown, a method for intelligent port safety early warning based on multi-source perception includes:

[0062] S1: Obtain the three-dimensional point cloud data of the material pile to be detected, as well as the sensor data;

[0063] S2: Construct a corresponding three-dimensional model of the material pile based on the three-dimensional point cloud data of the material pile to be detected;

[0064] S3: Extract the external structure features of the material pile to be detected based on the three-dimensional model of the material pile;

[0065] S4: Extract the sensor perception features of the material pile to be detected based on the sensor data;

[0066] S5: Fuse the shape structure features and sensor perception features of the material pile to be detected to obtain fused features;

[0067] S6: Input the fused features of the material pile to be detected into the trained collapse prediction model, and output the corresponding predicted collapse probability;

[0068] S7: Implement safety warning for the material pile to be detected based on the predicted collapse probability output by the collapse prediction model.

[0069] Through the combination of 3D point cloud data obtained by terrestrial laser scanning and UAV laser scanning, the present invention realizes high-precision 3D reconstruction of the shape of material piles in intelligent ports, covering not only the surface shape of the material piles but also deeply capturing their spatial three-dimensional structures, greatly improving the accuracy of the stability analysis of the material piles. Subsequently, combined with real-time perception features extracted from sensor data (such as tilt sensors, stress sensors, etc.), the sensitivity and specificity of the warning system are further enhanced, making the safety warning more accurate and timely.

[0070] The present invention effectively integrates the 3D shape features and sensor data of the material pile through feature fusion. This multi-source information fusion technology can comprehensively reflect the physical state and potential risks of the material pile, overcoming the problems of false alarms or missed alarms that may be caused by a single data source. By processing the fused features through a machine learning model, the system can automatically identify the complex and changeable stability states of the material pile, improving the robustness and reliability of the warning system.

[0071] Using the trained collapse prediction model, the present invention can send out warning signals in advance according to the predicted collapse probability before the actual collapse of the material pile. This not only provides a valuable time window for port managers to take emergency measures, such as strengthening the structure, adjusting the stacking method, or evacuating surrounding personnel and equipment, but also significantly reduces the probability of accidents and the potential economic losses and casualties caused by them, realizing the transformation from passive response to active prevention in the safety management mode.

[0072] The entire warning process of the present invention is highly automated. From data acquisition, model construction to feature extraction and prediction analysis, it reduces manual intervention and improves work efficiency. At the same time, with the accumulation of more data, the collapse prediction model can continuously self-optimize, continuously improving the warning accuracy and adaptability, laying a solid technical foundation for the long-term development of intelligent ports. Moreover, by monitoring and managing the safety of material piles through intelligent means, the present invention reduces the resource consumption and environmental pollution caused by traditional manual inspections, conforming to the development trend of green ports. In addition, the efficient warning system helps to maintain the stability of port operations, ensure the smooth flow of logistics, and has a positive significance for promoting the sustainable development of the regional economy.

[0073] To better introduce the technical solution of the present invention, this embodiment will be described through the following several parts.

[0074] I. Outer shape structure features

[0075] 1. Three-dimensional point cloud data

[0076] In this embodiment, the three-dimensional point cloud data of the material pile to be detected includes: the ground point cloud data obtained by the ground laser scanning device of the ground where the material pile to be detected is located, and the aerial point cloud data obtained by the unmanned aerial vehicle laser scanning device.

[0077] 2. Three-dimensional model of the material pile

[0078] In this embodiment, the three-dimensional model of the material pile is constructed through the following steps:

[0079] S201: Perform preprocessing including denoising and downsampling on the ground point cloud data and the aerial point cloud data;

[0080] S202: Align the preprocessed ground point cloud data and aerial point cloud data through a point cloud registration algorithm (in this embodiment, the ICP algorithm) to obtain a point cloud data set;

[0081] S203: Perform filtering processing on the point cloud data in the point cloud data set to remove noise and irrelevant points, and retain the effective point cloud data of the material pile;

[0082] S204: Convert the point cloud data in the point cloud data set into a three-dimensional mesh model through a three-dimensional reconstruction algorithm (in this embodiment, Poisson reconstruction) to obtain the three-dimensional model of the material pile.

[0083] 3. Extraction of outer shape structure features

[0084] In this embodiment, the outer shape structure features of the material pile to be detected include any one or more of height features, slope features, volume features, surface area features, stacking angle features, and center of gravity position features.

[0085] Each feature is extracted through the following methods respectively:

[0086] (1) Height feature

[0087] 1) Determine the reference plane of the material pile (usually the ground or the stacking plane).

[0088] 2) Calculate the vertical distance from each point to the reference plane.

[0089] 3) Find the maximum vertical distance among all points, which is the height of the material pile.

[0090] (2) Slope feature

[0091] 1) Select the surface point cloud data of the material pile.

[0092] 2) Mesh the surface point cloud data to generate a triangular mesh model.

[0093] 3) Calculate the normal vector of each triangular face.

[0094] 4) Calculate the slope of each triangular face according to the angle between the normal vector and the vertical direction.

[0095] 5) Conduct statistical analysis on the slopes of all triangular faces to obtain the average slope or the maximum slope of the material pile.

[0096] (3) Volume characteristics

[0097] 1) Determine the reference plane of the material pile.

[0098] 2) Perform a Boolean operation on the three-dimensional model of the material pile and the reference plane to obtain the volume of the material pile.

[0099] 3) Calculate the volume of the material pile using numerical integration methods (such as voxelization or Monte Carlo method).

[0100] (4) Surface area characteristics

[0101] 1) Select the surface point cloud data of the material pile.

[0102] 2) Mesh the surface point cloud data to generate a triangular mesh model.

[0103] 3) Calculate the area of each triangular face.

[0104] 4) Add up the areas of all triangular faces to obtain the surface area of the material pile.

[0105] (5) Angle of repose characteristics

[0106] 1) Select the surface point cloud data of the material pile.

[0107] 2) Mesh the surface point cloud data to generate a triangular mesh model.

[0108] 3) Calculate the normal vector of each triangular face.

[0109] 4) Calculate the slope of each triangular face according to the angle between the normal vector and the vertical direction.

[0110] 5) Conduct statistical analysis on the slopes of all triangular faces to find the maximum slope of the material pile, which is the angle of repose.

[0111] (6) Center of gravity position characteristics

[0112] 1) Divide the three-dimensional model of the material pile into several voxels or small units.

[0113] 2) Calculate the mass of each voxel or small unit (assuming uniform density, the mass is proportional to the volume).

[0114] 3) Calculate the centroid position of each voxel or small unit.

[0115] 4) Perform a weighted average of the centroid positions of all voxels or small units to obtain the overall centroid position of the material pile.

[0116] II. Sensor Perception Features

[0117] 1. Sensor Data

[0118] In this embodiment, the sensor data of the material pile to be detected includes any one or more of tilt sensor data, pressure sensor data, wind speed and direction sensor data, and displacement sensor data.

[0119] Specifically:

[0120] The tilt sensor data includes the tilt angle and tilt direction of the material pile, which are used to reflect the tilt change of the material pile and identify potential collapse risks.

[0121] The pressure sensor data is the pressure distribution at the bottom of the material pile, which is used to reflect the pressure anomaly at the bottom of the material pile and predict possible local collapses.

[0122] The wind speed and direction sensor data includes the wind speed and direction around the material pile, because strong winds may exert additional forces on the material pile and affect its stability.

[0123] The displacement sensor data includes the displacement change of the material pile, which is used to reflect the displacement of the material pile and identify potential collapse risks.

[0124] 2. Sensor Perception Feature Extraction

[0125] In this embodiment, the sensor perception features of the material pile to be detected include any one or more of tilt angle and direction features, pressure distribution features, wind speed and direction features, and displacement features. Each feature is obtained through the corresponding sensor.

[0126] III. Feature Fusion

[0127] In this embodiment, feature fusion is achieved through the following steps:

[0128] S501: Perform preprocessing on the external shape structure features and sensor perception features, including data standardization and spatio-temporal alignment;

[0129] In this embodiment, the external shape structure features adopt the Z-score standardization method to eliminate the dimension difference. The sensor perception features are time series features and adopt the sliding window standardization method (dynamically adapting to time series changes).

[0130] The external structure features are static features (single scan), while the sensor data is a time series stream. It is necessary to align the external features with the sensor data in the time dimension. For example, associate the external features of the most recent scan with each sensor data window. The sensors are distributed at different positions of the material pile, and it is necessary to associate the sensor data with the spatial coordinates in the 3D model. For example, the pressure sensor data corresponds to a specific area at the bottom of the material pile.

[0131] S502: Encode the preprocessed external structure features through a fully connected neural network to obtain the embedded representation of the external structure features;

[0132] The formula is expressed as:

[0133] f w = FCN(W) = FCN(g, p, t, b, d, z);

[0134] In the formula: f w represents the embedded representation of the external structure features; FCN represents the fully connected neural network; W represents the external structure features, where g, p, t, b, d, and z represent height feature, slope feature, volume feature, surface area feature, stacking angle feature, and center of gravity position feature respectively;

[0135] S503: Encode the preprocessed sensor perception features (time series features) through an LSTM network to obtain the embedded representation of the sensor perception features;

[0136] The formula is expressed as:

[0137] f c = LSTM(A 1 , A 2 ,...., A n );

[0138] In the formula: f c represents the embedded representation of the sensor perception features; LSTM represents the LSTM network; A 1 , A 2 ,...., A n represents the time series data of the sensor perception features;

[0139] In this embodiment, the sensor perception features include tilt angle and direction features, pressure distribution features, wind speed and direction features, and displacement features. The tilt angle and direction features, pressure distribution features, wind speed and direction features, and displacement features are respectively input into the LSTM network to obtain the embedded representations of the corresponding features, and then the embedded representations of all features are put into a set as the final embedded representation of the sensor perception features.

[0140] S504: Dynamically fuse the attention weights of the embedded representation f of the external shape structure features w and the embedded representation f of the sensor perception features c to obtain attention features;

[0141] The formula is expressed as:

[0142]

[0143] In the formula: Attention(Q, K, V) represents the attention features; Q represents the Query of the attention mechanism, which is the embedded representation of the external shape structure features; K and V represent the Key and Value of the attention mechanism, which are the embedded representations of the sensor perception features here; d k represents the feature dimension; softmax represents the normalization function;

[0144] S505: Concatenate the attention features Attention(Q, K, V) with the embedded representation f of the external shape structure features w to obtain fused features;

[0145] The formula is expressed as:

[0146] f r = Concat(f w , Attention(Q, K, V))

[0147] In the formula: f r represents the fused features; Concat represents the concatenation operation.

[0148] IV. Collapse Prediction Model

[0149] In this embodiment, the collapse prediction model is constructed based on a fully connected neural network model. The fused features are input into the collapse prediction model, and the corresponding predicted collapse probability is output, where the predicted collapse probability is a value between 0 and 1.

[0150] Train the collapse prediction model through the following loss function;

[0151] The formula is expressed as:

[0152]

[0153] In the formula: y i represents the true label (0 or 1); represents the predicted collapse probability; N represents the number of samples.

[0154] V. Safety Warning

[0155] In this embodiment, when the predicted collapse probability output by the collapse prediction model is greater than a preset safety threshold, it is determined that there is a collapse risk for the material pile to be detected, and a corresponding safety warning signal is issued.

[0156] In this embodiment, when it is determined that there is a collapse risk for the material pile to be detected, the collapse direction and the collapse danger area of the material pile to be detected can be further predicted, and then targeted safety warnings can be carried out based on the collapse direction and the collapse danger area, such as warning the personnel present in the collapse direction and within the collapse danger area.

[0157] 1. Collapse direction prediction

[0158] Use finite element analysis (FEA) or discrete element method (DEM) to simulate the stress distribution of the material pile and identify potential weak areas. At the same time, analyze the stress changes of the material pile under different external forces (such as wind force, vibration) to predict the possible collapse direction.

[0159] 2. Danger area identification

[0160] By comparing three-dimensional models at different time points, detect abnormal conditions such as local deformation and cracks of the material pile, and then use image processing and computer vision techniques to identify the danger area.

[0161] Based on the stress analysis and local deformation detection results, a risk map of the material pile can be further generated to mark high-risk areas. Use a heat map or contour map to visualize the risk distribution.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and do not limit the technical solutions. Those of ordinary skill in the art should understand that any modifications or equivalent replacements made to the technical solutions of the present invention without departing from the purpose and scope of the present technical solution should be covered within the scope of the claims of the present invention.

Claims

1. A smart port safety early warning method based on multi-source perception, characterized in that: include: S1: Obtain the three-dimensional point cloud data of the material pile to be inspected and the sensor data; S2: constructing a corresponding three-dimensional model of the material pile based on the three-dimensional point cloud data of the material pile to be inspected; S3: extracting the shape and structure features of the material pile to be inspected based on the three-dimensional model of the material pile; S4: extracting sensor perception features of the material pile to be detected based on the sensor data; S5: Fusing the appearance and structure features of the material pile to be detected and the sensor perception features to obtain fusion features; S6: inputting the fusion features of the material pile to be detected into the trained collapse prediction model, and outputting the corresponding predicted collapse probability; S7: A safety warning of the material pile to be inspected is realized based on the predicted collapse probability output by the collapse prediction model.

2. The smart port safety early warning method based on multi-source perception as claimed in claim 1 is characterized by: In step S1, the three-dimensional point cloud data of the material pile to be inspected includes: ground point cloud data obtained by a ground laser scanning device on the ground where the material pile to be inspected is located, and aerial point cloud data obtained by an unmanned aerial vehicle laser scanning device.

3. The smart port safety early warning method based on multi-source perception as claimed in claim 1 is characterized by: In step S1, the sensor data of the material pile to be detected includes any one or more of tilt sensor data, pressure sensor data, wind speed and direction sensor data, and displacement sensor data.

4. The smart port safety early warning method based on multi-source perception as claimed in claim 2 is characterized by: In step S2, a three-dimensional model of the material pile is constructed by the following steps: S201: Preprocessing the ground point cloud data and the aerial point cloud data including denoising and downsampling; S202: aligning the pre-processed ground point cloud data and the aerial point cloud data by using a point cloud registration algorithm to obtain a point cloud data set; S203: filtering the point cloud data in the point cloud data set to remove noise and irrelevant points, and retaining valid point cloud data of the material pile; S204: Convert the point cloud data in the point cloud data set into a three-dimensional mesh model by using a three-dimensional reconstruction algorithm to obtain a three-dimensional model of the material pile.

5. The smart port safety early warning method based on multi-source perception as claimed in claim 1 is characterized by: In step S3, the external structural features of the material pile to be detected include any one or more of height features, slope features, volume features, surface area features, stacking angle features and center of gravity position features.

6. The smart port safety early warning method based on multi-source perception as claimed in claim 5 is characterized by: In step S4, the sensor sensing characteristics of the material pile to be detected include any one or more of the tilt angle and direction characteristics, pressure distribution characteristics, wind speed and wind direction characteristics, and displacement characteristics. Each feature is obtained through the corresponding sensor.

7. The smart port safety early warning method based on multi-source perception as claimed in claim 6 is characterized by: In step S5, feature fusion is achieved through the following steps: S501: Preprocessing the appearance structure features and sensor perception features including data standardization and spatiotemporal alignment; S502: encoding the preprocessed shape structure features through a fully connected neural network to obtain an embedded representation of the shape structure features; The formula is: favorite w = FCN(W); Where: f w represents the embedded representation of shape structure features; FCN represents the fully connected neural network; W represents the shape structure features; S503: Encode the preprocessed sensor perception features through an LSTM network to obtain an embedded representation of the sensor perception features; The formula is: f c =LSTM(A1,A2,....,A n ); Where: f c represents the embedded representation of sensor perception features; LSTM represents the LSTM network; A1, A2, ..., A n Time series data representing sensor-sensed features; S504: Embedding the shape structure features through the attention mechanism w and sensor-aware feature embedding representation f c Perform dynamic fusion of attention weights to obtain attention features; The formula is: Where: Attention(Q,K,V) represents the attention feature; Q represents the Query of the attention mechanism, which is the embedded representation of the appearance structure feature; K and V represent the Key and Value of the attention mechanism, which are the embedded representation of the sensor perception feature here; d k Represents feature dimension; Softmax represents the normalized function; S505: embed the attention feature Attention (Q, K, V) and the appearance structure feature into representation f w Perform splicing to obtain fusion features; The formula is: f r =Concat(f w ,Attention(Q,K,V)) Where: f r Represents fusion features; Concat represents concatenation operation.

8. The smart port safety early warning method based on multi-source perception as claimed in claim 1 is characterized by: In step S6, the collapse prediction model is constructed based on a fully connected neural network model.

9. The smart port safety early warning method based on multi-source perception as claimed in claim 8 is characterized by: In step S6, the collapse prediction model is trained by the following loss function; The formula is: Where: y i represents the true label; represents the predicted collapse probability; N represents the number of samples.

10. The smart port safety early warning method based on multi-source perception according to claim 1 is characterized in that: In step S7, when the predicted collapse probability output by the collapse prediction model is greater than a preset safety threshold, it is determined that the material pile to be detected has a risk of collapse, and a corresponding safety warning signal is issued.

Images