A port panoramic operation safety AI monitoring method and system based on big data

Abstract

The application provides a port panoramic operation safety AI monitoring method and system based on big data, relates to the field of port unmanned aerial vehicle inspection, and comprises the following steps: based on standard operation specifications and preset port operation specifications, a port operation safety rule library with consequence severity levels is constructed; the server is used to acquire multi-modal inspection data collected by an unmanned aerial vehicle flying along a preset flight path and environment sensor data; abnormal state calculation is performed, and an abnormal set containing abnormal numbers and corresponding consequence severity levels is output; and the server triggers corresponding early warning actions based on the consequence severity levels. The above method effectively makes up for the limited coverage of fixed cameras. At the same time, intelligent identification of port operation abnormalities is realized, the dependence on manual inspection and manual monitoring is reduced, and the automation level and intelligent level of port panoramic operation safety monitoring are improved.

Description

Technical Field

[0001] This invention relates to the field of port drone inspection, specifically to a port panoramic operation safety AI monitoring method and system based on big data. Background Technology

[0002] Current port operation safety monitoring mainly relies on a combination of fixed cameras and manually operated drone inspections. Fixed cameras are usually deployed at key nodes in the port for manual monitoring of local areas; for areas that are difficult for fixed cameras to cover, drones are used for temporary, supplementary aerial patrols. Operators determine the flight path and shooting angle based on the on-site situation and experience, and the collected images or videos are transmitted back to the monitoring center for manual review and judgment.

[0003] However, the above methods have significant shortcomings in practical applications. First, the coverage of fixed cameras is limited, making it difficult to achieve a panoramic view of port operations, especially stacked containers. Second, the flight altitude, flight path, hovering position, and data collection process of manually operated drones lack standardization and automation, and the inspection results are greatly influenced by the operator's experience and subjective judgment. Third, existing video surveillance and drone inspections mainly rely on manual observation and post-event playback for anomaly detection, lacking real-time automatic analysis capabilities and failing to identify property damage or environmental anomalies in a timely and accurate manner. Furthermore, the criteria for judging abnormal events are inconsistent, and the severity levels of consequences are difficult to quantify, resulting in a lack of targeted early warning and response, and an inability to automatically trigger differentiated early warning actions based on the severity level of consequences.

[0004] Therefore, there is an urgent need for a port panoramic operation safety AI monitoring method and system based on big data. Summary of the Invention

[0005] To overcome existing technical problems, this invention provides a port panoramic operation safety AI monitoring method and system based on big data.

[0006] The present invention adopts the following technical solution.

[0007] A big data-based AI-powered method for monitoring port operations and safety includes the following steps: S1. Based on standard operating procedures and pre-set port operation specifications, construct a port operation safety rule base with severity levels of consequences; S2. Obtain multimodal inspection data collected by the UAV flying along a preset route through the server, as well as environmental sensor data collected by environmental sensors located in the port. S3. The server calculates abnormal states based on the port operation safety rule base on multimodal inspection data and environmental sensor data, and outputs an anomaly set containing anomaly numbers and the severity level of the consequences of the corresponding anomaly numbers. S4. The server triggers a corresponding warning action based on the severity level of the consequences.

[0008] As a further improvement to the present invention, the specific steps of S1 include: S11. Select multiple candidate sentences from multiple standard operating procedures and preset port operation procedures through pre-trained language models and preset trigger word rules, or through manual retrieval, and extract and generate structured rules containing causal features, result features and rule sources through pre-trained language models or manual extraction. S12. Based on the preset severity levels of consequences and the corresponding meanings of severity levels, generate the severity level of consequences and anomaly number corresponding to each result feature according to the pre-trained language model or manual judgment, and integrate the severity levels of consequences, anomaly numbers and structured rules to obtain the port operation safety rule base.

[0009] As a further improvement of the present invention, the preset route includes multiple hovering points arranged in sequence, multiple interval routes respectively connecting two hovering points, flight altitude, flight speed, gimbal angle and hovering duration; While flying along a preset route, the drone performs automatic obstacle avoidance detection using its onboard obstacle avoidance sensors to determine whether there are obstacles within the preset safe obstacle avoidance range. If obstacles are found, the drone performs one or more of the following actions: hovering and waiting, horizontal detour, or vertical detour, and records the obstacle avoidance behavior and uploads it to the server.

[0010] As a further improvement of the present invention, the gimbal of the UAV is equipped with a visible light camera and an infrared thermal imager for joint calibration and acquisition frequency adjustment. The multimodal inspection data includes visible light image sequences and infrared thermal image sequences with UAV pose data, gimbal attitude data and timestamps. The environmental sensor data includes gas concentration sequences and wind speed values ​​with environmental sensing location data and timestamps.

[0011] As a further improvement of the present invention, the server can obtain all container slot numbers, container slot coordinates corresponding to the container slot numbers in the port coordinate system, current container slot status corresponding to the container slot numbers, and container type size corresponding to the container slot numbers through the port TOS system, and construct a theoretical container slot table containing the theoretical number of layers. The specific steps for calculating abnormal states include: generating an actual container location detection table based on the container identification method and multimodal inspection data; and outputting a geometric anomaly set based on the actual container location detection table and the theoretical container location table. Infrared temperature anomaly detection is performed using visible light image sequences and infrared thermal image sequences, and a set of temperature anomalies is output. Anomalies in concentration and wind speed are detected by combining gas concentration sequences and wind speed values, and an environmental anomaly set is output.

[0012] As a further improvement to the present invention, the specific steps for generating the actual container location detection table include: S301. Run the box detection model on the visible light image sequence in the multimodal inspection data and combine it with the theoretical box position table to output the center pixel coordinates of the top and bottom surfaces of each vertically stacked box. S302. Based on the visible light camera intrinsic parameter calibration data and visible light camera installation parameter calibration data obtained from the joint calibration, combined with the UAV pose data and gimbal attitude data, the top surface center pixel coordinates are projected to obtain the ray starting point and top surface ray direction in the port coordinate system, and the bottom surface center pixel coordinates are projected to obtain the bottom surface ray direction in the port coordinate system. S303. Based on the port ground plane equation, the intersection of the ray starting point and the bottom ray direction, the coordinates of the bottom center of the stacked box in the port coordinate system are obtained. S304. Construct the vertical line parametric equation using the center coordinates of the bottom of the stacked container, construct the top ray parametric equation based on the ray starting point and the ray direction of the top surface, solve for the shortest distance between the vertical line parametric equation and the top ray parametric equation, obtain the measured top height, and output the center coordinates of the top of the stacked container in the port coordinate system. S305. Select the box type and size of each layer from the theoretical box position table according to the center coordinates of the bottom of the stacked boxes, and calculate the number of measured layers by combining the measured top height. S306. Integrate the measured number of layers corresponding to each vertically stacked box, the center coordinates of the top of the stacked box, the center coordinates of the bottom of the stacked box, and the timestamps of the visible light image sequence to obtain the actual box position detection table.

[0013] As a further improvement of the present invention, the specific steps of outputting a geometric anomaly set based on the actual container location detection table and the theoretical container location table include: if the actual number of measured layers in the actual container location detection table is less than the theoretical number of layers, it is determined that the goods are lost, and geometric anomaly data containing the anomaly number of the corresponding lost goods, the corresponding container location number, and the severity level of the consequences of the corresponding anomaly number is output. If the actual number of stacking layers in the actual container location detection table is greater than the theoretical number of stacking layers, it is judged as an abnormal stacking height. The output includes the abnormality number of the corresponding stacking height abnormality, the corresponding container location number, and the severity level of the consequences of the corresponding abnormality number. If the difference between the center coordinates of the bottom of the stacked boxes and the coordinates of the corresponding box position is greater than the preset maximum tolerance threshold, it is judged as a positional anomaly, and geometric anomaly data containing the anomaly number of the corresponding positional anomaly, the corresponding box position number, and the severity level of the consequences of the corresponding anomaly number is output. All geometric anomaly data are integrated to obtain a geometric anomaly set.

[0014] As a further improvement of the present invention, the specific steps for detecting and outputting a temperature anomaly set using visible light image sequences and infrared thermal image sequences include: S311. Perform timestamp alignment processing on the visible light image sequence and the infrared thermal image sequence; S312. Generate the original temperature matrix based on the infrared images of the infrared thermogram sequence; S313. Detect high reflectance regions in visible light images at the same timestamp and generate a reflectance binary mask characterizing unreliable infrared regions. S314. Adjust the original temperature matrix according to the reflection binary mask to generate an effective temperature matrix. Calculate the median based on the effective temperature matrix. Calculate the robust standard deviation based on the median. Calculate and generate candidate hotspots based on the median and robust standard deviation to obtain a candidate hotspot set. If there are no candidate hotspots, skip the infrared temperature anomaly detection. S315. Adjust the effective temperature matrix according to the candidate hotspot set, generate the background temperature matrix, calculate the background mean value according to the background temperature matrix, judge and confirm each candidate hotspot according to the background mean value and the preset temperature difference threshold. If confirmed, it is regarded as a hotspot pixel, and the hotspot pixels are integrated to obtain the hotspot pixel set. S316. Perform connected component analysis on the hot spot pixel set to obtain the hot spot region set. Perform multi-frame consistency verification on each hot spot region in the hot spot region set. If the verification fails, delete the corresponding hot spot region. The remaining hot spot regions are recorded as temperature abnormal regions and judged as temperature abnormal. S317. Identify the abnormal center point of each temperature abnormal region through visible light images, generate an abnormal ray equation based on the abnormal center point and calculate the abnormal coordinates, integrate the abnormal coordinates and the corresponding temperature abnormal region to obtain a temperature abnormal region set, and output a temperature abnormal set containing the abnormal number of the corresponding temperature abnormality, the corresponding temperature abnormal region set and the severity level of the consequences of the corresponding abnormal number.

[0015] This invention also proposes a port panoramic operation safety AI monitoring system based on big data, which adopts a port panoramic operation safety AI monitoring method based on big data as described above, including: drones, environmental sensors and servers; The drone is used to fly along a preset route and collect multimodal inspection data, which is then transmitted to the server. Environmental sensors are fixedly installed within the port to collect environmental sensor data and transmit it to the server; The servers include: The rule base construction module is used to build a port operation safety rule base with severity levels of consequences based on standard operating procedures and preset port operation procedures. The data acquisition module is used to acquire multimodal inspection data collected by the UAV and environmental sensor data collected by the environmental sensors. The abnormal state calculation module is used to calculate abnormal states based on multimodal inspection data and environmental sensor data based on the port operation safety rule base, and output an abnormal set containing anomaly numbers and the severity level of the consequences of the corresponding anomaly numbers. The early warning module is used to trigger corresponding early warning actions based on the severity level of the consequences.

[0016] The beneficial effects of this invention are as follows: it effectively compensates for the limited coverage and susceptibility to obstruction of fixed cameras, as well as the arbitrary flight paths and poor data consistency of manually controlled drones. Simultaneously, based on standard operating procedures and pre-set port operation specifications, a port operation safety rule base with severity levels is constructed, providing a unified and quantitative judgment standard for anomaly identification, avoiding the subjectivity and inconsistency of traditional manual judgment. The server calculates anomaly states based on multimodal inspection data and environmental sensor data using this port operation safety rule base, and outputs an anomaly set containing anomaly numbers and corresponding severity levels in real time, achieving intelligent identification of port operation anomalies and overcoming the shortcomings of poor timeliness and high missed detection rate of traditional manual observation and post-event playback. Corresponding early warning actions are automatically triggered according to the severity level of the consequences, realizing graded early warning and differentiated handling of abnormal events, ensuring that events with different severity levels receive matching response measures; overall, it reduces the reliance on manual inspection and monitoring, and improves the automation and intelligence levels of port panoramic operation safety monitoring. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the present invention; Figure 2 This is a schematic diagram illustrating the generation process of the actual container position detection table of the present invention. Detailed Implementation

[0019] It is understandable for those skilled in the art that some well-known structures and descriptions thereof in the accompanying drawings may be omitted. The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] Reference Figures 1 to 2 It can be seen that the present invention proposes a port panoramic operation safety AI monitoring method based on big data, comprising the following steps: S1. Construct a port operation safety rule base with consequence severity levels based on standard operation specifications and preset port operation specifications; The standard operation specifications include, but are not limited to, Safety Requirements for Port Operations (GB16994), Safety Technical Requirements for Port Steel Loading and Unloading Operations (JT / T 245-2011) and Safety Operation Regulations for Port Container Yard Areas (GB / T 35551-2017).

[0021] Wherein, Safety Requirements for Port Operations (GB16994) includes eight parts, namely oil and gas chemical terminals, petrochemical storage areas, dangerous goods containers, general cargo containers, general breakbulk cargo, solid bulk dangerous goods, cement, and passenger terminals under technical review. It should be noted that since various provisions are time-sensitive, the higher authorities are currently promoting the incorporation of Safety Technical Requirements for Port Steel Loading and Unloading Operations (JT / T 245-2011) into Safety Requirements for Port Operations (GB16994). Therefore, the standard operation specifications mentioned herein in the present invention are not limited to the contents mentioned above, and may also include newly added or revised specifications.

[0022] The preset port operation specifications specifically refer to the contents that are not included in the standard operation specifications but need to be inspected in the port's own patrol inspection, serving as a supplement to the specifications.

[0023] As a further improvement of the present invention, the specific steps of S1 include: S11. Select a plurality of candidate sentences from a plurality of standard operation specifications and preset port operation specifications through a pre-trained language model and preset trigger word rules, or through manual retrieval, and extract and generate structured rules including cause features, result features and rule sources through the pre-trained language model or manually. Specifically, a plurality of candidate sentences can be recalled by a commonly used pre-trained language model based on preset trigger word rules. The trigger word rules refer to words such as "suitable", "shall", "less than", "greater than", "equal to" and "not", etc., and the corresponding entire paragraph of text is recalled and selected. Of course, as a preferred embodiment of the present invention, it is recommended to perform the selection through manual retrieval, which can reduce the pre-training cost of the language model and improve the accuracy of recall and selection.

[0024] Pre-trained language models or manual analysis can be used to generate corresponding result features based on candidate sentences and label the source of the corresponding rules. For example, the "Safety Operation Procedures for Port Container Areas" (GB / T 35551-2017) states that "the first and last rows of each container space should be lowered by one container height" and "containers shorter than their length should not be stacked on top of containers, such as stacking two 20ft containers on top of a 40ft or 45ft container." The former can generate result features of incorrect container placement (risk of container falling and injuring people) and label them with the corresponding "Safety Operation Procedures for Port Container Areas" (GB / T 35551-2017); the latter can generate result features of incorrect container placement (container damage) and label them with the corresponding "Safety Operation Procedures for Port Container Areas" (GB / T 35551-2017).

[0025] By annotating the source of the rules, it is possible to quickly locate the source of false alarms and manually adjust the structured rules, thereby improving the accuracy of subsequent warnings. Furthermore, when the corresponding standard operating procedure or preset port operation specifications become obsolete, the corresponding structured rules can be uniformly filtered and deleted.

[0026] It's important to note that structured rules are not created once and then left unchanged. For cause characteristics, their descriptions will be adjusted to better reflect the needs of subsequent exception sets generated by the server, making them more targeted. Existing cause characteristics will also be moved to the rule source to facilitate subsequent manual verification.

[0027] S12. Based on the preset severity levels of consequences and the corresponding meanings of severity levels, generate the severity level of consequences and anomaly number corresponding to each result feature according to the pre-trained language model or manual judgment, and integrate the severity levels of consequences, anomaly numbers and structured rules to obtain the port operation safety rule base.

[0028] The severity level of the consequences has three levels, and the meanings of each severity level are as follows: Level 3: May result in minor injury or general property damage, such as damage to goods, small obstructions, or misalignment of storage locations.

[0029] Level 2: May result in a single death or significant property damage, such as stack collapse, loss of goods, or personnel injury.

[0030] Level 1: May cause multiple deaths, significant property damage, or environmental disasters, such as fires, explosions, or mass dumping of containers.

[0031] It should be noted that the above severity levels are only a general definition, not a hard and fast rule. In practice, it is very common for the server to detect lost goods as due to various factors such as the container being moved and not being updated to the server in time. False alarms (not actual lost goods) are quite frequent. If a level two warning is issued and the entire control room is activated with audible and visual alarms and SMS notifications, it would lead to a significant waste of manpower and long-term staff strain. A level three warning with flashing alerts, requiring manual verification, is sufficient. Therefore, this invention preferably uses manual voting for decision-making.

[0032] A severity level for each outcome feature is generated. Specifically, referring to the outcome feature of the risk of a box falling and hitting someone, it can be judged as Level 2, and the outcome feature of box damage can be judged as Level 3. As a preferred embodiment of the present invention, the severity level of the outcome can be judged and voted on by three or more people. If there are different severity levels of the outcome, opinions can be exchanged and a vote can be taken again. If there are still different severity levels of the outcome, the severity level of the outcome with the higher severity can be selected according to the principle of conservative safety.

[0033] As an example of the port operation safety rule base, such as anomaly number: G001, result characteristic: single container missing / lost, cause characteristic: the actual number of layers measured in the actual container location inspection table is less than the theoretical number of layers, consequence severity level: level three, rule source: Port Operation Specifications Part 2, Article 2.1.3 + "Container areas shall be inspected regularly, and the container location inspection table records shall be checked against the actual stacking situation in the container area to confirm whether any containers are missing or lost; if any missing or lost containers are found, they shall be registered in a timely manner, the cause shall be investigated, and a report shall be made in accordance with regulations." S2. Obtain multimodal inspection data collected by the UAV flying along a preset route through the server, as well as environmental sensor data collected by environmental sensors located in the port. Regarding the type of environmental sensors, the appropriate environmental sensors can be selected according to the requirements of Clause 4.1.4 of Part 2 of the "Port Operation Safety Requirements" (GB16994) for petrochemical storage areas: combustible gas and toxic gas detectors and alarms, as well as online monitoring and control of temperature, pressure, liquid level, flow rate, etc., of major hazard sources to ensure normal operation. And according to the requirements of Clause 8.2 of Part 3 for dangerous goods containers, the appropriate environmental sensors should be selected: the inspection area should be equipped with toxic gas and combustible gas detection equipment.

[0034] As a further improvement of the present invention, the preset route includes multiple hovering points arranged in sequence, multiple interval routes respectively connecting two hovering points, flight altitude, flight speed, gimbal angle and hovering duration; By pre-setting flight paths, the drone inspection process is given clear spatial positioning and data collection parameter constraints, ensuring that highly consistent multimodal inspection data is obtained for the same target area each time. Furthermore, since different ports have different layouts, pre-setting flight paths can effectively and directly avoid fixed obstacles within the port.

[0035] It should be noted that for each section of the route, the flight altitude, flight speed, gimbal angle, and hovering duration can be set independently. Specifically, the flight altitude can be 30-60 meters to ensure that the effective field of view of the visible light camera and infrared thermal imager avoids tall equipment such as quay cranes and gantry cranes. The flight speed can be 2-5 meters per second, and the hovering duration is 10-30 seconds to ensure that the visible light camera and infrared thermal imager capture a sufficient number of images. Furthermore, it should be noted that, as a preferred embodiment of this invention, the hovering point should not be placed above the container position; it is recommended to place it on the passageway to avoid the inability to solve for the shortest distance between the vertical line parametric equation and the top ray parametric equation laterally.

[0036] While flying along a preset route, the drone performs automatic obstacle avoidance detection using its onboard obstacle avoidance sensors to determine whether there are obstacles within the preset safe obstacle avoidance range. If obstacles are found, the drone performs one or more of the following actions: hovering and waiting, horizontal detour, or vertical detour, and records the obstacle avoidance behavior and uploads it to the server.

[0037] Effectively avoids collisions with obstacles such as tower cranes, container stacks, and moving vehicles within the port, ensuring flight safety and the continuity of inspection missions. The airborne obstacle avoidance sensors include one or more of the following: forward-facing binocular vision or millimeter-wave radar, downward-facing ultrasonic or laser ranging modules, and optional lateral obstacle avoidance sensors. The preset safe obstacle avoidance range can be determined based on flight speed, such as a 10-meter range. When an obstacle is detected entering the safe obstacle avoidance range, the UAV first decelerates and hovers. If the obstacle does not leave within the preset waiting time, a temporary detour path is generated horizontally or vertically based on the current obstacle position (offsetting by 2 to 5 meters while keeping the obstacle outside the safe obstacle avoidance range). The UAV then performs a low-speed horizontal or vertical detour, not exceeding 2 meters per second. After the detour, the shortest path is automatically calculated to return to the next waypoint on the original route to continue the mission.

[0038] As a further improvement of the present invention, when the UAV flies in the port according to the preset route, the server can obtain the attitude data of the port gantry crane through the ECS device control system or the CMS crane management system and continuously send it to the UAV, so that the UAV can avoid or wait for the gantry crane in advance.

[0039] The technical solution of this invention combines manual route setting with automatic obstacle avoidance. During the manual route setting phase, it fully utilizes human experience with port operations to precisely set hovering point positions and shooting parameters, ensuring coverage of key areas. In the flight phase, the onboard obstacle avoidance system automatically handles sudden obstacles without real-time human intervention, ensuring both route targeting and data quality while effectively addressing uncertainties in a dynamic environment.

[0040] S3. The server calculates abnormal states based on the port operation safety rule base on multimodal inspection data and environmental sensor data, and outputs an anomaly set containing anomaly numbers and the severity level of the consequences of the corresponding anomaly numbers. As a further improvement of the present invention, the gimbal of the UAV is equipped with a visible light camera and an infrared thermal imager for joint calibration and acquisition frequency adjustment. The multimodal inspection data includes visible light image sequences and infrared thermal image sequences with UAV pose data, gimbal attitude data and timestamps. The environmental sensor data includes gas concentration sequences and wind speed values ​​with environmental sensing location data and timestamps.

[0041] The visible light camera and infrared thermal imager undergo joint calibration and acquisition frequency adjustment on the gimbal, enabling the server to obtain high-quality data with time synchronization, spatial alignment, and complete attitude when performing abnormal state calculations. It should be noted that acquisition frequency adjustment does not mean that the two frequencies are the same; they can be multiples of each other, such as 50Hz for the visible light camera and 25Hz for the infrared thermal imager.

[0042] As a further improvement of the present invention, the server can obtain all container slot numbers, container slot coordinates corresponding to the container slot numbers in the port coordinate system, current container slot status corresponding to the container slot numbers, and container type size corresponding to the container slot numbers through the port TOS system, and construct a theoretical container slot table containing the theoretical number of layers. The specific steps for calculating abnormal states include: generating an actual container location detection table based on the container identification method and multimodal inspection data; and outputting a geometric anomaly set based on the actual container location detection table and the theoretical container location table. Infrared temperature anomaly detection is performed using visible light image sequences and infrared thermal image sequences, and a set of temperature anomalies is output. Anomalies in concentration and wind speed are detected by combining gas concentration sequences and wind speed values, and an environmental anomaly set is output.

[0043] An anomaly set is obtained by integrating geometric anomalies, temperature anomalies, and environmental anomalies.

[0044] The abnormal state calculation is divided into three independent parts: geometric anomaly set, temperature anomaly set, and environmental anomaly set. Each independent part uses different types of data and calculation methods, and finally they are summarized into an anomaly set containing anomaly number and consequence severity level. This architecture of independent calculation in parts allows the system to flexibly expand or replace a certain part without affecting the operation of other parts. At the same time, the output of each independent part is unified as anomaly number and consequence severity level, which facilitates the subsequent execution of early warning actions.

[0045] Reference Figure 2 The specific steps for generating the actual container location detection table include: S301. Run the box detection model on the visible light image sequence in the multimodal inspection data and combine it with the theoretical box position table to output the center pixel coordinates of the top and bottom surfaces of each vertically stacked box. Because the existing port TOS system clearly shows the current status of the container location corresponding to the location number, and the container type and size corresponding to the location number, and because the "Safety Operation Procedures for Port Container Areas" (GB / T 35551-2017) stipulates that the container type and size of different layers must be consistent except for height, the container detection model can be calibrated and trained by using the same hovering point to measure the container type and size of different heights on different layers. When using a drone to identify the top and bottom center pixel coordinates of each vertically stacked container, it is not necessary to identify all edges or corner pieces to obtain the top and bottom center pixel coordinates. It is only necessary to identify at least one corner piece and the extension direction of two edges, combined with the corresponding container type and size, to output the top and bottom center pixel coordinates through the container detection model.

[0046] Container berth numbering varies by port, but can be categorized into zone-block-shell-column-layer naming rules. For example, a zone generally corresponds to a specific berth; berth number two corresponds to zone two. Blocks are determined sequentially from the sea side to the road side. Shells are ordered sequentially, with some ports using odd numbers for smaller containers and even numbers for larger containers. Columns are ordered from the side closest to the driveway. Layers are the number of vertically stacked containers.

[0047] S302. Based on the visible light camera intrinsic parameter calibration data and visible light camera installation parameter calibration data obtained from the joint calibration, combined with the UAV pose data and gimbal attitude data, the top surface center pixel coordinates are projected to obtain the ray starting point and top surface ray direction in the port coordinate system, and the bottom surface center pixel coordinates are projected to obtain the bottom surface ray direction in the port coordinate system. S303. Based on the port ground plane equation, the intersection of the ray starting point and the bottom ray direction, the coordinates of the bottom center of the stacked box in the port coordinate system are obtained. Since visible light cameras do not have depth data, and port TOS systems typically use port coordinates, and the ground can be considered essentially zero, knowing the ray origin and ray direction allows us to calculate the coordinates in the port coordinate system by solving the plane equations.

[0048] As a specific calculation example of the present invention, taking the center pixel coordinates of the top surface as an example, the center pixel coordinates of the bottom surface are calculated similarly. The center pixel coordinates of the top surface are mapped to the camera coordinate system through camera intrinsic parameters. A gimbal rotation matrix is ​​constructed based on the gimbal attitude data. The rotation of the camera from the body is calculated based on the camera mounting angle obtained from joint calibration. , It is a gimbal rotation matrix. It's the camera mounting angle. (Through...) The ray direction of the top surface center pixel coordinates on the aircraft body is calculated, and the camera optical center is calculated using the camera arm obtained from joint calibration combined with the gimbal rotation matrix. The UAV pose data is then mapped to the world coordinate system, and then to the port coordinate system; alternatively, it can be directly mapped to the port coordinate system. The port coordinate system generally uses a corner of the port as the origin of the Cartesian coordinate system.

[0049] Since obtaining a ray in the real coordinate system by mapping a point in an image is a conventional technique for UAVs, this invention will only be briefly described here.

[0050] In the port coordinate system, the port ground plane equation is Z=0, which allows us to solve for the coordinates of the bottom center of the stacked containers in the port coordinate system. Of course, if there are certain differences in height or slope on the port ground, the port ground plane equation for each block or each shell can be calculated. The port ground plane equation Z=0 in this invention is only used as an example for illustrative purposes.

[0051] S304. Construct the vertical line parametric equation using the center coordinates of the bottom of the stacked container, construct the top ray parametric equation based on the ray starting point and the ray direction of the top surface, solve for the shortest distance between the vertical line parametric equation and the top ray parametric equation, obtain the measured top height, and output the center coordinates of the top of the stacked container in the port coordinate system. The expression for the parametric equation of a vertical line: 'b' refers to the coordinates of the center point at the bottom of the stacked box. The expression for the parametric equation of the top ray is as follows: The shortest distance between the parametric equations of the vertical line and the parametric equations of the top ray can be found by constructing a distance minimization function: By taking the partial derivatives with respect to s and t and setting them to zero, we can solve the system of linear equations to obtain the optimal s and t, where t is the measured top height. This method avoids the problem of non-intersecting rays due to various errors, which would render the solution unsolvable.

[0052] S305. Select the box type and size of each layer from the theoretical box position table according to the center coordinates of the bottom of the stacked boxes, and calculate the number of measured layers by combining the measured top height. Specifically, in the theoretical container location table, select the container type and size that are currently "present" in each layer. Starting from the bottom layer, subtract the height of the corresponding container type and size from the measured top height one by one. If the remaining measured top height after subtraction is greater than the preset container height difference (which can be two-thirds of the shortest container as the preset value), then the measured number of layers is greater than the theoretical number of layers. If the remaining measured top height is negative and the absolute value is greater than the preset container height difference, then the measured number of layers is less than the theoretical number of layers.

[0053] S306. Integrate the measured number of layers corresponding to each vertically stacked box, the center coordinates of the top of the stacked box, the center coordinates of the bottom of the stacked box, and the timestamps of the visible light image sequence to obtain the actual box position detection table.

[0054] As another embodiment of the present invention, LiDAR can be used for processing. However, due to the influence of dust, rain, fog, metal surface reflection, and vibration of large equipment in the port environment, the point cloud quality of LiDAR is often unstable, resulting in large plane fitting errors and unreliable segmentation results. At the same time, high-precision LiDAR is expensive and has a large computational load for point cloud data processing, making it difficult to meet the needs of real-time inspection. In addition, near dangerous goods storage yards, the active light source of LiDAR may affect certain sensitive goods.

[0055] As another embodiment of the present invention, a depth map can be obtained by using a depth camera, but the effective range of existing consumer and industrial RGB-D cameras is much shorter than that of visible light cameras.

[0056] As a further improvement of the present invention, the specific steps of outputting a geometric anomaly set based on the actual container location detection table and the theoretical container location table include: if the actual number of measured layers in the actual container location detection table is less than the theoretical number of layers, it is determined that the goods are lost, and geometric anomaly data containing the anomaly number of the corresponding lost goods, the corresponding container location number, and the severity level of the consequences of the corresponding anomaly number is output. If the actual number of stacking layers in the actual container location detection table is greater than the theoretical number of stacking layers, it is judged as an abnormal stacking height. The output includes the abnormality number of the corresponding stacking height abnormality, the corresponding container location number, and the severity level of the consequences of the corresponding abnormality number. If the difference between the center coordinates of the bottom of the stacked boxes and the coordinates of the corresponding box position is greater than the preset maximum tolerance threshold, it is judged as a positional anomaly. The output is geometric anomaly data containing the anomaly number of the corresponding positional anomaly, the corresponding box position number, and the severity level of the consequences of the corresponding anomaly number. The maximum tolerance threshold can be set to half of the corresponding length of the corresponding box type size.

[0057] All geometric anomaly data are integrated to obtain a geometric anomaly set.

[0058] It is important to note that the judgment techniques described here, whether for geometric anomalies or environmental anomalies, are merely for the purpose of assisting those skilled in the art and do not imply that the invention can only output the geometric anomaly data mentioned above. For example, if the measured number of layers in adjacent columns is less than the theoretical number of layers, it can be judged as a stack collapse, and a severity level of one or two consequences can be output, rather than the corresponding level three cargo loss.

[0059] If the actual number of layers in the container location detection table is greater than the theoretical number of layers, it may be because a container location has been reserved, and the gantry crane placed the goods in the reserved location, but the system has not updated the data. In the examples of this invention, lost goods, abnormal stacking height, and abnormal location all correspond to a level three severity of consequences.

[0060] As a further improvement of the present invention, the specific steps for detecting and outputting a temperature anomaly set using visible light image sequences and infrared thermal image sequences include: S311. Perform timestamp alignment processing on the visible light image sequence and the infrared thermal image sequence; The visible light camera and infrared thermal imager undergo joint calibration and acquisition frequency adjustment on the gimbal, ensuring high synchronization between the visible light image sequence and the infrared thermal image sequence in terms of timestamps. It's important to reiterate that frequency adjustment does not mean the two frequencies are identical; they can be multiples of each other, such as 50Hz for the visible light camera and 25Hz for the infrared thermal imager. This ensures that the subsequent reflection binary mask generated from the visible light images accurately corresponds to the infrared temperature matrix at the same moment, avoiding mask misalignment caused by altitude discrepancies between the two sensors during UAV flight. This significantly improves accuracy. Furthermore, the hovering point setting also aims to improve the image alignment accuracy between the visible light image sequence and the infrared thermal image sequence.

[0061] S312. Generate the original temperature matrix based on the infrared images of the infrared thermogram sequence; S313. Detect high reflectance regions in visible light images at the same timestamp and generate a reflectance binary mask characterizing unreliable infrared regions. Since visible light images can clearly reflect unreliable infrared regions such as specular reflections on metal surfaces, water glare, and oil stains, by detecting pixel regions with extremely high grayscale values ​​and low saturation, and projecting these regions onto the infrared image coordinate system to form a reflection binary mask, the locations of pixels with unreliable radiation temperature data can be accurately marked. This step effectively overcomes the limitation of relying solely on infrared images to distinguish between real hotspots and reflective pseudo-hotspots, because reflective pseudo-hotspots also appear as localized high temperatures in infrared images, but their root cause is ambient light reflection rather than the target's own heat generation. For example, if the brightness channel threshold is set to 245 and the saturation threshold to 50, areas with a brightness channel greater than 245 and a saturation lower than 50 can be considered as strongly reflective regions. A pre-trained semantic segmentation model can also be used to detect oil stains and water glare. Subsequently, the reflection binary mask detected in the visible light image is mapped onto the infrared image coordinate system using the calibration rotation matrix and translation vector obtained during joint calibration.

[0062] S314. Adjust the original temperature matrix according to the reflection binary mask to generate an effective temperature matrix. Calculate the median based on the effective temperature matrix. Calculate the robust standard deviation based on the median. Calculate and generate candidate hotspots based on the median and robust standard deviation to obtain a candidate hotspot set. If there are no candidate hotspots, skip the infrared temperature anomaly detection. After excluding unreliable pixels, only the remaining pixels participate in statistical calculations, avoiding contamination of the background temperature distribution estimation by high-reflectivity areas. The median and robust standard deviation are used instead of the mean and ordinary standard deviation because infrared temperature data in port scenes may occasionally contain a small number of outliers or non-Gaussian distributions. Robust statistics can resist the interference of these outliers, making the hotspot detection threshold more reasonable. If no candidate hotspots exist, infrared temperature anomaly detection is skipped. This conditional judgment avoids unnecessary subsequent calculations in scenes without hotspots and reduces false hotspot detections.

[0063] Since robust standard deviation is a readily available mathematical calculation method, this invention will not elaborate on it further. However, for calculating and generating candidate hotspots, specifically, k=5 can be set, and the median can be used. A robust standard deviation of 1.8℃ yields 40.2℃, and pixels in the original temperature matrix above 40.2℃ are marked as candidate hotspots.

[0064] S315. Adjust the effective temperature matrix according to the candidate hotspot set, generate the background temperature matrix, calculate the background mean value according to the background temperature matrix, judge and confirm each candidate hotspot according to the background mean value and the preset temperature difference threshold. If confirmed, it is regarded as a hotspot pixel, and the hotspot pixels are integrated to obtain the hotspot pixel set. Because the temperature difference between the ignition point and the background temperature can be quite large, the background mean is recalculated after eliminating candidate hotspots. Each candidate hotspot is then re-verified, requiring its temperature difference from the background mean to exceed a preset threshold. This design avoids the influence of the high temperature of the candidate hotspots themselves on the background statistics, making the background estimation purer and thus improving the reliability of hotspot confirmation.

[0065] For example, the average background temperature calculated from the remaining background pixels is 30.8℃, and the temperature difference threshold is set to 15℃. Candidate hotspots are evaluated one by one, and those with a temperature difference below 15℃ are removed. It should be noted that due to the different geographical locations and climates of different ports, the temperature difference threshold cannot be generalized and can be adjusted through actual measurement calibration or by considering false alarms and missed alarms.

[0066] S316. Perform connected component analysis on the hot spot pixel set to obtain the hot spot region set. Perform multi-frame consistency verification on each hot spot region in the hot spot region set. If the verification fails, delete the corresponding hot spot region. The remaining hot spot regions are recorded as temperature abnormal regions and judged as temperature abnormal. Since connected component analysis is a standard technique in image recognition, it will not be elaborated upon further in this invention. Furthermore, the multi-frame consistency check requires that the same hotspot region be detected in multiple frames, allowing for positional offsets of multiple pixels and partial overlap of hotspot regions, such as a positional offset of twenty pixels with only 10% of the region within the offset range. This is because real fires exhibit temporal continuity and stability. Even under high wind speeds, some hotspot regions may overlap within the acceptable range of positional offsets.

[0067] S317. Identify the abnormal center point of each temperature abnormal region through visible light images, generate an abnormal ray equation based on the abnormal center point and calculate the abnormal coordinates, integrate the abnormal coordinates and the corresponding temperature abnormal region to obtain a temperature abnormal region set, and output a temperature abnormal set containing the abnormal number of the corresponding temperature abnormality, the corresponding temperature abnormal region set and the severity level of the consequences of the corresponding abnormal number.

[0068] The steps for generating the anomalous ray equation are the same as those for generating the top surface ray parametric equation. The calculated anomalous coordinates can be obtained by solving the port ground plane equation or by matching a close top or bottom surface ray parametric equation. When there are no nearby containers, the solution calculated using the port ground plane equation is more accurate. When there are nearby containers, matching a close top or bottom surface ray parametric equation and selecting the corresponding top or bottom center coordinates of the stacked containers is more accurate. This is to prevent the fire from originating in containers on the second or higher level.

[0069] The specific steps for detecting concentration and wind speed anomalies using gas concentration sequences and wind speed values ​​are as follows: conventional numerical threshold comparison processing is directly adopted. If the value exceeds the preset threshold, the corresponding anomaly number, the port installation coordinates of the corresponding environmental sensor, and the severity level of the consequences of the corresponding anomaly number are directly output.

[0070] S4. The server triggers a corresponding warning action based on the severity level of the consequences.

[0071] For Level 3 warnings, a flashing alert can be displayed on the system monitoring terminal, along with information about the corresponding cluster of anomalies.

[0072] For a Level 2 warning, the system can be activated by an audible and visual alarm in the central control room, along with an SMS notification to the safety administrator and the person in charge of the corresponding area, displaying information about the concentration of anomalies.

[0073] For a Level 1 warning, the central control room can trigger an audible and visual alarm, and a telephone call can be made to the safety administrator, the person in charge of the corresponding area, and the duty leader to express the information about the abnormal concentration, and the port's pre-set emergency plan can be activated manually or automatically.

[0074] This invention also proposes a port panoramic operation safety AI monitoring system based on big data, which adopts a port panoramic operation safety AI monitoring method based on big data as described above, including: drones, environmental sensors and servers; The drone is used to fly along a preset route and collect multimodal inspection data, which is then transmitted to the server. Environmental sensors are fixedly installed within the port to collect environmental sensor data and transmit it to the server; The servers include: The rule base construction module is used to build a port operation safety rule base with severity levels of consequences based on standard operating procedures and preset port operation procedures. The data acquisition module is used to acquire multimodal inspection data collected by the UAV and environmental sensor data collected by the environmental sensors. The abnormal state calculation module is used to calculate abnormal states based on multimodal inspection data and environmental sensor data based on the port operation safety rule base, and output an abnormal set containing anomaly numbers and the severity level of the consequences of the corresponding anomaly numbers. The early warning module is used to trigger corresponding early warning actions based on the severity level of the consequences.

[0075] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A port panoramic operation safety AI monitoring method based on big data, characterized in that, Includes the following steps: S1. Based on standard operating procedures and pre-set port operation specifications, construct a port operation safety rule base with severity levels of consequences; S2. Obtain multimodal inspection data collected by the UAV flying along a preset route through the server, as well as environmental sensor data collected by environmental sensors located in the port. S3. The server performs anomaly calculations on the multimodal inspection data and the environmental sensor data based on the port operation safety rule base, and outputs an anomaly set containing anomaly numbers and the severity level of the consequences of the corresponding anomaly numbers. S4. The server triggers a corresponding warning action based on the severity level of the consequences.

2. The port panoramic operation safety AI monitoring method based on big data according to claim 1, characterized in that, The specific steps of S1 include: S11. Select multiple candidate sentences from multiple standard operating procedures and preset port operation procedures through pre-trained language models and preset trigger word rules, or through manual retrieval, and extract and generate structured rules containing causal features, result features and rule sources through pre-trained language models or manual extraction. S12. Based on the preset severity levels of consequences and the corresponding meanings of severity levels, generate the severity level of consequences and anomaly number corresponding to each result feature according to the pre-trained language model or manual judgment, and integrate the severity levels of consequences, anomaly numbers and structured rules to obtain the port operation safety rule base.

3. The port panoramic operation safety AI monitoring method based on big data according to claim 1, characterized in that, The preset flight path includes multiple hovering points set in sequence, multiple interval routes connecting two hovering points respectively, flight altitude, flight speed, gimbal angle and hovering duration; While flying along a preset route, the UAV performs automatic obstacle avoidance detection using its onboard obstacle avoidance sensors to determine whether there are obstacles within a preset safe obstacle avoidance range. If obstacles are found, the UAV performs one or more of the following actions: hovering and waiting, horizontal detour, or vertical detour, and records the obstacle avoidance behavior and uploads it to the server.

4. The port panoramic operation safety AI monitoring method based on big data according to claim 1, characterized in that, The UAV's gimbal is equipped with a visible light camera and an infrared thermal imager for joint calibration and acquisition frequency adjustment. The multimodal inspection data includes visible light image sequences and infrared thermal image sequences with UAV pose data, gimbal attitude data, and timestamps. The environmental sensor data includes gas concentration sequences and wind speed values ​​with environmental sensing location data and timestamps.

5. The port panoramic operation safety AI monitoring method based on big data according to claim 4, characterized in that, The server can obtain all container slot numbers, container coordinates corresponding to the container slot number in the port coordinate system, current container slot status corresponding to the container slot number, and container type size corresponding to the container slot number through the port TOS system, and construct a theoretical container slot table containing the theoretical number of layers. The specific steps for calculating the abnormal state include: generating an actual container location detection table based on the container identification method and multimodal inspection data, and outputting a geometric anomaly set based on the actual container location detection table and the theoretical container location table; Infrared temperature anomaly detection is performed using visible light image sequences and infrared thermal image sequences, and a set of temperature anomalies is output. Anomalies in concentration and wind speed are detected by combining gas concentration sequences and wind speed values, and an environmental anomaly set is output.

6. The port panoramic operation safety AI monitoring method based on big data according to claim 5, characterized in that, The specific steps for generating the actual container location detection table include: S301. Run the box detection model on the visible light image sequence in the multimodal inspection data and combine it with the theoretical box position table to output the center pixel coordinates of the top and bottom surfaces of each vertically stacked box. S302. Based on the visible light camera intrinsic parameter calibration data and visible light camera installation parameter calibration data obtained by joint calibration, combined with UAV pose data and gimbal attitude data, the top surface center pixel coordinates are projected to obtain the ray starting point and top surface ray direction in the port coordinate system, and the bottom surface center pixel coordinates are projected to obtain the bottom surface ray direction in the port coordinate system. S303. Based on the port ground plane equation, the intersection of the ray starting point and the bottom ray direction, the coordinates of the bottom center of the stacked box in the port coordinate system are obtained. S304. Construct the vertical line parametric equation using the center coordinates of the bottom of the stacked container, construct the top ray parametric equation based on the ray starting point and the ray direction of the top surface, solve for the shortest distance between the vertical line parametric equation and the top ray parametric equation, obtain the measured top height, and output the center coordinates of the top of the stacked container in the port coordinate system. S305. Select the box type and size of each layer in the theoretical box position table according to the center coordinates of the bottom of the stacked box, and calculate the number of measured layers in combination with the measured top height. S306. Integrate the measured number of layers corresponding to each vertically stacked box, the center coordinates of the top of the stacked box, the center coordinates of the bottom of the stacked box, and the timestamps of the visible light image sequence to obtain the actual box position detection table.

7. The port panoramic operation safety AI monitoring method based on big data according to claim 5, characterized in that, The specific steps for outputting a geometric anomaly set based on the actual container location detection table and the theoretical container location table include: if the actual number of measured layers in the actual container location detection table is less than the theoretical number of layers, it is determined that the goods are lost, and geometric anomaly data containing the anomaly number corresponding to the lost goods, the corresponding container location number, and the severity level of the consequences corresponding to the anomaly number is output. If the measured number of layers in the actual container location detection table is greater than the theoretical number of layers, it is judged as an abnormal stacking height, and the output includes the abnormality number corresponding to the abnormal stacking height, the corresponding container location number, and the geometric anomaly data corresponding to the severity level of the consequences of the abnormality number. If the difference between the center coordinates of the bottom of the stacked boxes and the coordinates of the corresponding box position is greater than the preset maximum tolerance threshold, it is judged as a positional anomaly, and geometric anomaly data containing the anomaly number of the corresponding positional anomaly, the corresponding box position number, and the severity level of the consequences corresponding to the anomaly number is output. All geometric anomaly data are integrated to obtain a geometric anomaly set.

8. A port panoramic operation safety AI monitoring method based on big data according to claim 6, characterized in that, The specific steps for detecting infrared temperature anomalies and outputting a temperature anomaly set using visible light image sequences and infrared thermal image sequences include: S311. Perform timestamp alignment processing on the visible light image sequence and the infrared thermal image sequence; S312. Generate an original temperature matrix based on the infrared images of the infrared thermogram sequence; S313. Detect high reflectance regions in visible light images at the same timestamp and generate a reflectance binary mask characterizing unreliable infrared regions. S314. Adjust the original temperature matrix according to the reflection binary mask to generate an effective temperature matrix. Calculate the median according to the effective temperature matrix. Calculate the robust standard deviation according to the median. Calculate and generate candidate hotspots according to the median and robust standard deviation to obtain a candidate hotspot set. If there are no candidate hotspots, skip the infrared temperature anomaly detection. S315. Adjust the effective temperature matrix according to the candidate hotspot set to generate a background temperature matrix. Calculate the background mean value according to the background temperature matrix. Judge and confirm each candidate hotspot according to the background mean value and a preset temperature difference threshold. If confirmed, it is regarded as a hotspot pixel. Integrate the hotspot pixels to obtain a hotspot pixel set. S316. Perform connected component analysis on the hot spot pixel set to obtain a hot spot region set. Perform multi-frame consistency verification on each hot spot region in the hot spot region set. If the verification fails, delete the corresponding hot spot region. The remaining hot spot regions are recorded as temperature abnormal regions and judged as temperature abnormal. S317. Identify the abnormal center point of each temperature abnormal region through visible light images, generate an abnormal ray equation based on the abnormal center point and calculate the abnormal coordinates, integrate the abnormal coordinates and the corresponding temperature abnormal region to obtain a temperature abnormal region set, and output a temperature abnormal set containing the abnormal number of the corresponding temperature abnormality, the corresponding temperature abnormal region set and the severity level of the consequences of the corresponding abnormal number.

9. A port panoramic operation safety AI monitoring system based on big data, characterized in that, The method for monitoring port panoramic operation safety based on big data, as described in any one of claims 1-8, includes: a drone, environmental sensors, and a server; The drone is used to fly along a preset route and collect multimodal inspection data to the server; The environmental sensor is fixedly installed within the port and is used to collect environmental sensor data and transmit it to the server. The server includes: The rule base construction module is used to build a port operation safety rule base with severity levels of consequences based on standard operating procedures and preset port operation procedures. The data acquisition module is used to acquire the multimodal inspection data collected by the UAV and the environmental sensor data collected by the environmental sensor. An anomaly calculation module is used to perform anomaly calculations on the multimodal inspection data and the environmental sensor data based on the port operation safety rule base, and output an anomaly set containing anomaly numbers and the severity level of the consequences of the corresponding anomaly numbers. The early warning module is used to trigger corresponding early warning actions based on the severity level of the consequences.

Images