Double-layer 360-degree look-around monitoring and early warning system based on structure of super-large mining electric shovel

Abstract

This invention discloses a dual-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure. The system adopts a physically layered architecture: the upper monitoring subsystem (installation height 8-12 meters) monitors a dynamic annular area of ​​5-15 meters around the rotation center (monitoring moving targets); the lower monitoring subsystem (installation height 0.8-1.5 meters) monitors a static near-ground area of ​​0-3 meters in front of the tracks (monitoring fixed obstacles); the central processing unit processes the two video streams independently, calculates the relative distance of the moving target and the size of the obstacle, and dynamically adjusts the fusion parameters according to the dust concentration; the early warning logic is linked with the electric shovel PLC signal to achieve adaptive mode switching, the data is processed locally throughout the process, and the early warning record includes the hash value of the video segment used for integrity verification; this invention completely solves the industry problem of single-layer systems being unclear at high altitudes and lacking coverage at low altitudes, providing a safe solution with clear structural parameters, clear logical decoupling, strong environmental adaptability, and compliance with data compliance requirements.

Description

Technical Field

[0001] This invention relates to the field of intelligent safety technology for engineering machinery, and more specifically, to a dual-layer 360-degree surround view monitoring and early warning system based on the structure of an ultra-large mining electric shovel. In particular, it relates to a dual-layer 360-degree surround view monitoring and early warning system and method that achieves precise prevention and control of dual risks of rotational collision and walking obstacle through physical layer architecture and multi-modal algorithm fusion. Background Technology

[0002] Extra-large electric mining shovels (such as the WK-20, WK-35, and P&H 4100 series) are core equipment for stripping and excavation in open-pit mines, and their operational safety is directly related to the safety of personnel and equipment assets. Taking the WK-35 as an example, the machine is over 15 meters high and has a turning radius of over 20 meters. Although the operator is located in the high cab, the massive structure of the machine (such as the counterweight box, canopy, and boom) creates multiple blind spots. Blind spots during slewing: directly behind the counterweight box, the right corner of the machine hangar, and the area under the boom. These areas are prone to collisions with personnel, pickup trucks, and auxiliary equipment behind the machine during slewing.

[0003] Blind spots: The area directly in front of and inside the tracks where the driver cannot see obstacles on the ground (such as boulders with a diameter greater than 0.5m or deep pits), which can easily lead to track plate breakage and drive wheel damage, with a single repair cost exceeding 500,000 yuan.

[0004] The existing technical solutions have significant drawbacks: Single-layer surround view system: The cameras are centrally installed on the upper structure. Although they can cover long-distance areas, the resolution for recognizing obstacles on the ground is insufficient (measured <5cm / pixel) due to the installation height >10 meters, which cannot meet the requirements for walking safety. If the installation height is reduced, the turning radius coverage is insufficient, resulting in a dilemma of covering one area while neglecting another.

[0005] Auxiliary radar solution: Although it can penetrate dust, it cannot identify the type and size of obstacles, and is prone to misjudging haystacks and small stones as dangerous obstacles, with a false alarm rate of over 30%, which can cause alarm fatigue for drivers.

[0006] General AI vision solution: The algorithm logic is not designed for the dual-modal operation (rotation / walking) characteristics of electric shovels, dynamic targets and static obstacles are treated in the same way, the warning logic is simple and cannot achieve scene adaptation.

[0007] The industry urgently needs a dedicated system that can simultaneously handle high-altitude, wide-area dynamic monitoring and near-ground, precise static identification, while also meeting the stringent environmental requirements of mines with high dust levels, strong vibrations, and a wide temperature range (-40℃ to +70℃). This invention addresses this critical technological challenge. Summary of the Invention

[0008] This invention overcomes the shortcomings of single-layer surround view systems, such as the inability to simultaneously ensure turning safety and walking safety, coupled early warning logic, poor environmental adaptability, and data compliance risks. It provides a two-layer monitoring system with clear physical structure parameters, clear decoupling of processing logic, strong environmental adaptability, and compliance with data security regulations.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a double-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure, characterized in that it includes: The upper monitoring subsystem has cameras installed at the high point of the edge of the electric shovel's slewing platform, at a height of 8–12 meters, configured to collect video streams of a dynamic annular area with a radius of 5–15 meters centered on the shovel's slewing center. The lower monitoring subsystem has a camera mounted on the frame of the lower walking mechanism of the electric shovel with the lens tilted downwards. The installation height is 0.8–1.5 meters, and it is configured to collect video streams of the near-ground static area 0–3 meters in front of the track in the direction of travel. The central processing unit is communicatively connected to both the upper-level and lower-level monitoring subsystems. It is configured to receive a first video stream output from the upper-level subsystem and a second video stream output from the lower-level subsystem, and to process the first and second video streams independently: performing distortion correction and bird's-eye view projection on the first video stream to calculate the relative distance between the moving target and the center of rotation. Depth estimation is performed on the second video stream to calculate the actual size of the obstacle. ; The human-machine interface is connected to the central processing unit and is configured to dynamically switch the displayed content according to the operating status of the electric shovel. The early warning execution device is connected to the central processing unit and configured to... The comparison result with the preset distance threshold triggers a tiered warning, or according to... The comparison result with the preset size threshold triggers an obstacle warning; The video processing flow of the upper-level monitoring subsystem and the lower-level monitoring subsystem is logically decoupled in the central processing unit, and the early warning decisions are independent of each other, jointly covering the omnidirectional safety blind spots of the electric shovel's slewing and walking operations.

[0010] Furthermore, the upper monitoring subsystem includes at least four wide-angle cameras with a field of view ≥150°, which are respectively fixedly installed on the front edge of the top of the cab, the high-level bracket on the right side wall of the cab, the top platform of the counterweight box at the rear of the cab, and the support structure of the boom on the left side of the cab.

[0011] Furthermore, the lower-level monitoring subsystem includes at least five ruggedized cameras, with lens covers integrating high-pressure air curtain nozzles. The lens tilt angle is adjustable from 30° to 60° and is respectively installed on the bracket above the left track front guide wheel, the bracket above the right track front guide wheel, the bracket above the left track rear drive wheel, the bracket above the right track rear drive wheel, and the front and rear ends of the chassis area between the two tracks.

[0012] Furthermore, the central processing unit includes: The image correction module is configured to perform distortion correction on fisheye images using an isometric projection model and through a homography matrix. The corrected image is projected onto a unified bird's-eye view coordinate system, where , This is the camera intrinsic parameter matrix. For rotation matrix The first two columns are concatenated with the translation vector t to form a 3×3 matrix; The image stitching module is configured to perform distance-weighted fusion on overlapping regions of multi-view images; The AI ​​analysis module is configured to perform upper-layer dynamic target detection and lower-layer static obstacle analysis respectively. The early warning decision module is configured to generate tiered early warning instructions.

[0013] Furthermore, in the AI ​​analysis module: Upper-layer dynamic target detection calculates relative distance ;when When a Level 1 warning is triggered, A level-two warning is triggered at this time, among which ,and Positively correlated with the turning radius of the electric shovel model; Analysis and calculation of actual dimensions of lower-level static obstacles ,when And when it is located within a preset region of interest (ROI) in the direction of track movement, an obstacle warning is triggered, among which The configurable size threshold ranges from 0.3 to 1.0 meters.

[0014] Furthermore, the central processing unit also integrates a millimeter-wave radar data interface, and the early warning decision module is configured to fuse visual detection results with radar point cloud data using a Kalman filter algorithm; the observation noise covariance of the Kalman filter... Based on the ambient dust concentration value output by the dust sensor (Unit: μg / m³) Dynamically adjusted to satisfy the relationship. ,in , These are calibration coefficients.

[0015] Furthermore, the human-machine interface is configured such that: when the electric shovel is in the slewing operation state, the main display area presents the upper layer synthesized panoramic bird's-eye view; after the electric shovel receives the walking command, it automatically switches the key road surface images of the lower layer monitoring subsystem to the main display area.

[0016] Furthermore, the early warning execution device includes a graded sound and light alarm unit: a first-level early warning triggers a high-frequency buzzer and a red flashing light; a second-level early warning triggers a medium-frequency prompt sound and a yellow constant-on light; and an obstacle early warning triggers a voice broadcast and a yellow pulse light.

[0017] Furthermore, all camera module housings meet IP69K protection standards, the lens surfaces are coated with a hydrophobic and oleophobic coating, and an automatic high-pressure air curtain cleaning device is provided.

[0018] Furthermore, the central processing unit is deployed in an industrial-grade embedded industrial control computer in the driver's cab, equipped with a hardware acceleration module, and the end-to-end latency of video processing is ≤300ms.

[0019] This invention discloses an early warning method for a dual-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure. S1. Video streams are collected from a 5-15 meter annular area around the shovel's rotation center using an upper-level camera installed at a height of 8-12 meters; S2. Video streams are collected from the area 0–3 meters in front of the track in the direction of travel by installing a lower-level camera at a height of 0.8–1.5 meters; S3. Perform distortion correction and homography transformation on the two video streams respectively, and project them onto a unified bird's-eye view coordinate system; S4. Based on the corrected image: Calculate the relative distance between the moving target and the center of rotation in the upper flow. ; Calculate the actual size of obstacles in the lower laminar flow ; S5. According to and preset distance threshold , The relationship triggers a tiered warning, or based on With preset size threshold Relationship triggers obstacle warning; S6. Dynamically switch the content displayed on the human-machine interface according to the PLC signal of the electric shovel, and drive the audible and visual alarm device to perform the corresponding warning action.

[0020] Further, in step S4, the... The turning radius is positively correlated with the electric shovel model. It can be configured within the range of 0.3–1.0 meters; in step S5, when the ambient dust concentration... As the noise level increases, the observation noise covariance of the Kalman filter is dynamically increased. To improve the robustness of integration.

[0021] Furthermore, in step S6, when the electric shovel travel command is detected, the road surface image of the lower monitoring subsystem is displayed on the main screen area first, and the location and size information of the obstacle are highlighted.

[0022] Furthermore, all video data processing and early warning decisions are completed on the local industrial control computer of the electric shovel. The original video stream is not stored or transmitted externally. The early warning event record only retains the target type, coordinates, timestamp, and video segment hash value used for data integrity verification.

[0023] Furthermore, the connection between the camera bracket of the lower monitoring subsystem and the frame of the electric shovel walking mechanism is equipped with a vibration-damping rubber pad to isolate walking vibrations.

[0024] Furthermore, the aforementioned ,in It is a proportionality constant with a value range of 0.2–0.3. The turning radius specified for the electric shovel model.

[0025] This system consists of five core modules: 1. Upper-level monitoring subsystem: Cameras are installed at high points on the edge of the electric shovel's slewing platform (installation height 8–12 meters), capturing video streams of a dynamic ring-shaped area with a radius of 5–15 meters centered on the slewing center. The "dynamic" attribute of this area means that the monitored objects are moving targets such as personnel and vehicles, while the area boundaries remain fixed. A wide-angle lens with a field of view ≥150° is used, along with a built-in lightweight target detection model.

[0026] 2. Lower-level monitoring subsystem: The camera is mounted on the lower walking mechanism frame of the electric shovel (installation height 0.8–1.5 meters), with the lens tilted downwards at 30°–60°, capturing video streams of the near-ground static area 0–3 meters in front of the track's direction of travel. The "static" attribute of this area means that the primary monitoring object is fixed ground obstacles (rocks, potholes), and the area's location updates as the electric shovel moves. It employs an IP69K ruggedized binocular module, providing dust and shock resistance.

[0027] 3. Central Processing Unit: Communicatively connected to both the upper and lower layer subsystems, configured to receive the first video stream output from the upper-layer monitoring subsystem and the second video stream output from the lower-layer monitoring subsystem, and to process the first and second video streams independently respectively. Perform distortion correction and bird's-eye view projection on the first video stream, and calculate the relative distance between the moving target and the center of rotation. Perform binocular matching and depth estimation on the second video stream to calculate the actual size of the obstacles. ; The term "separate independent processing" means that the processing flow of the two video streams is logically completely decoupled, with no cross-references of data or nested conditions. Specific implementation methods include multi-threaded scheduling, hardware-accelerated pipelines, or efficient single-threaded time-sharing processing, as long as the end-to-end latency is ≤300ms. It integrates a millimeter-wave radar interface and dynamically adjusts the Kalman filter parameters based on dust concentration.

[0028] 4. Human-machine interface: Connected to the central processing unit, the display content is dynamically switched according to the signal of the electric shovel PLC (programmable logic controller): when rotating, the main display shows the upper bird's-eye view, and when the walking command is triggered, it automatically switches to the lower road surface image.

[0029] 5. Early warning execution device: based on and preset distance threshold ( , )or With preset size threshold ( The comparison results trigger a graded audible and visual alarm.

[0030] The beneficial effects of this invention compared to existing technologies are as follows: This invention pioneers a physically layered and logically decoupled architecture, with the upper layer focusing on dynamic risks and the lower layer focusing on static risks, increasing blind spot coverage to 98.7% (actual test data), completely solving the industry problem of single-layer systems being unclear at high levels and lacking comprehensive coverage at low levels. The algorithm used in this invention is precise, through... and Dual-formula quantification to mitigate risk and avoid subjective misjudgment; Kalman filter dynamic adjustment mechanism ( This invention improves the accuracy of early warning in dusty environments from 78.3% with monocular vision to 95.2%; obstacle size recognition error is ≤8% (verified by on-site calibration); the invention also further reduces camera vibration acceleration by using a vibration-damping rubber pad design, improving image clarity, reducing warning delay, and fully meeting the safety response requirement of electric shovel rotation speed ≤1.5° / s; while the use of automatic air curtain cleaning further reduces the failure time caused by lens dirt. The video data collected by this invention is processed locally throughout the process, and the early warning record only retains de-identified information and the hash value of the video segment used for integrity verification, which complies with the relevant requirements for personal information protection.

[0031] The actual test results are compared below:

[0032] Note: The data comes from 90 consecutive days of field testing of the WK-35 electric shovel, covering three working conditions: sunny, moderate dust (PM10=300–500μg / m³), and heavy dust (PM10>500μg / m³). Attached Figure Description

[0033] The present invention will now be further described with reference to the accompanying drawings.

[0034] Figure 1 This is a schematic diagram of the framework structure of the present invention. Detailed Implementation

[0035] Key terms are clearly defined in this invention: Circular dynamic zone: refers to a fixed geometric area with a radius of 5-15 meters centered on the rotation center of the electric shovel. "Dynamic" indicates that the monitored object is a moving target. Near-ground static area: refers to a fixed rectangular area 0–3 meters in front of the track travel direction. "Static" indicates that the monitoring object is a fixed obstacle on the ground. The first and second video streams are processed independently: This means that the processing flow of the two video streams is completely decoupled logically, with no cross-references of data or nested conditions. Specific implementation methods include multi-threaded scheduling, hardware-accelerated pipelines or efficient single-threaded time-sharing processing, as long as the end-to-end latency is ≤300ms. The early warning logics are independent of each other: the upper-level early warning decision is based solely on... Compared to distance thresholds, lower-level early warning decisions are based solely on... Compare with size thresholds; PLC signals: Operation status signals output by the electric shovel's programmable logic controller via the CAN bus, including slewing angle encoding signal, travel command signal, and slewing speed signal; Hardware acceleration module: refers to a dedicated computing unit integrated into the motherboard of an industrial control computer, including one or more of GPU, FPGA or NPU, which is dedicated to accelerating convolution operations, matrix transformations and Kalman filter iterations; Video clip hash value: A 256-bit digest value generated using the SHA-256 algorithm, specifically used to verify the integrity and tamper-proof nature of video data for early warning events.

[0036] The present invention will be further described below with reference to specific embodiments.

[0037] like Figure 1 As shown, this embodiment uses the WK-35 electric shovel as an example, and the hardware deployment is as follows: 1. Upper-level monitoring subsystem: Camera: 4 industrial-grade global shutter camera modules (resolution ≥ 8 million pixels, field of view ≥ 150°, frame rate ≥ 30fps@4K, operating temperature -40℃~+85℃); Installation location: The cab roof front edge support (horizontal distance from the center of rotation 6.2m, height 11.5m); High-level support frame on the right side wall of the machine shed (9.8m high); The top platform of the counterweight box at the rear of the machine shed (10.3m high, avoiding obstruction by the counterweight block). Left side boom support structure of the machine shed (8.7m high); Protection: The lens surface is coated with a nano-hydrophobic and oleophobic coating, and the outer cover integrates a high-pressure air curtain nozzle (working air pressure 0.6MPa), which automatically cleans for 3 seconds every 15 minutes or when the dust sensor is triggered.

[0038] 2. Lower-level monitoring subsystem: Camera: 5 IP69K certified ruggedized binocular camera modules (baseline distance 15cm, global shutter, lens tilt angle adjustable 45°); Installation location: Support above the left track front guide wheel (1.2m above the ground); Support above the right track front guide wheel (1.2m above the ground); Support above the left track rear drive wheel (1.0m above the ground); Support above the right track rear drive wheel (1.0m above the ground). The front and rear ends of the chassis area between the two tracks (0.8m above the ground). In this embodiment, a vibration-damping rubber pad (Shore hardness 70A) is configured at the connection between the camera bracket and the electric shovel walking mechanism frame. Actual measurements show that the walking vibration acceleration can be suppressed from 18.3g to ≤5g, significantly improving image clarity. Protection: The lens cover has a built-in miniature vibration motor (50Hz frequency) to help remove adhering dust.

[0039] 3. Central Processing Unit: Hardware: Industrial-grade embedded industrial control computer (CPU frequency ≥ 2.5GHz, memory ≥ 16GB, equipped with hardware acceleration module), with interfaces including 8-channel PoE+ video input, CAN bus (connecting to electric shovel PLC), and RS485 (connecting to millimeter-wave radar and dust sensor). Environmental adaptability: Wide temperature range (-40℃~+70℃), tri-proof coated circuit board, thermal grease combined with metal fin passive heat dissipation (fanless design). Processing capacity: The actual measured end-to-end video processing latency is ≤218ms (meets the requirement of ≤300ms).

[0040] 4. Human-computer interaction and early warning device: Display: 10.1-inch IPS industrial touchscreen (1280×800, brightness 1000cd / m², anti-glare glass). Alarm system: Red / yellow LED warning light (installed on the cab ceiling), buzzer (volume ≥ 95dB), voice module (supports Chinese and English broadcast); Linkage logic: Listen to PLC signals via CAN bus - when the walking command = 1, automatically switch the lower layer screen to the main screen; when the rotation speed > 0.3° / s, activate the upper layer warning logic.

[0041] The specific algorithm implementation process in this embodiment is as follows: 1. Camera calibration: Intrinsic parameter calibration employed the Zhang Zhengyou checkerboard method (Zhang Z. A Flexible New Technique for Camera Calibration. IEEE TPAMI 2000), using an 11×8 checkerboard (25 mm grid length) to acquire 20 sets of images from different viewpoints. The intrinsic parameter matrix K and distortion coefficients were calculated using the OpenCV calibrateCamera function, with reprojection error controlled within 0.3 pixels. Extrinsic parameter calibration was performed on a 10m×10m calibration cloth (including ArUco markers) laid on the electric shovel's parking area. The rotation matrix R and translation vector t were calculated to ensure that the origin of the bird's-eye view coordinate system coincided with the shovel's rotation center, with a calibration error of less than 2 cm. The calibration process is a well-known technique in the field of computer vision and can be directly implemented by those skilled in the art.

[0042] 2. Core Image Processing Algorithms: Distortion correction (equidistant projection model): This system uses an equidistant projection model to correct distortion in fisheye lenses, with the target image being a standard perspective projection image. The correction principle is based on optical geometry: the radial distance between pixels in a fisheye image is linearly related to the angle of incidence. In perspective projection, the radial distance and the angle of incidence satisfy a tangent relationship. The correction process first calculates the incident angle θ from the original coordinates, and then maps it to perspective projection coordinates. The specific formula is as follows: , ; , ; Where x and y represent the coordinates of the pixel to be corrected in the original fisheye image (unit: pixels), with the origin located at the optical center of the image; f represents the equivalent focal length of the camera (unit: pixels), which is accurately obtained through the calibration process and reflects the optical characteristics of the lens. θ represents the angle of incidence of the light ray (unit: radians), which is calculated from the ratio of the original radial distance to the focal length; Represents the radial distance corresponding to this pixel after correction (in the perspective projection image), (unit: pixels); and This represents the coordinates of the corresponding pixel in the corrected image (unit: pixels).

[0043] When the original radial distance is zero (i.e., the pixel is located at the optical center), the corrected coordinates are set to zero to avoid division by zero errors. This method effectively eliminates the barrel distortion of fisheye lenses, providing geometrically accurate input for subsequent bird's-eye projection. Actual measurements show that the image reprojection error after correction is less than 0.5 pixels.

[0044] Bird's-eye view projection (homophonic transformation): The distortion-corrected image is projected onto a bird's-eye view coordinate system (world coordinate system Zw=0 plane) with the shovel's rotation center as the origin through homography transformation, thus achieving top-down perspective synthesis. The transformation formula is: , ; in: and This represents the coordinates of a point on the ground in the bird's-eye view coordinate system (unit: meters), with the origin set as the center of rotation of the electric shovel. and Represents the coordinates of a pixel in the image after distortion correction (unit: pixels); H represents the homography matrix used for the inverse mapping, a 3×3 invertible matrix; K represents the camera intrinsic parameter matrix, a 3×3 constant matrix, fixed after calibration, and its form is: ; in, This represents the equivalent focal length in the x-direction (unit: pixels), obtained through calibration. Represents the equivalent focal length in the y-direction (unit: pixels); It represents the coordinates of the principal point of the image (the intersection of the optical axis and the image plane) in the x-direction (unit: pixels), and the calibration value is usually close to half the width of the image; This represents the coordinates of the principal point of the image in the y-direction (unit: pixels), and the calibration value is usually close to half the image height. The 1 in the lower right corner of the matrix is ​​a homogeneous coordinate normalization term, which is dimensionless.

[0045] The calibration result of the WK-35 upper-layer camera in this embodiment is as follows: =2850.3 pixels, =2848.7 pixels, =1920.5 pixels, =1080.2 pixels.

[0046] M represents a 3×3 matrix derived from the camera's extrinsic parameters, constructed by taking the first two columns of the rotation matrix R. Concatenate with the translation vector t, i.e. ,in These are the elements (dimensionless) of the rotation matrix R. , , Let t be the component of the translation vector t (unit: meters).

[0047] Multi-perspective fusion: To eliminate brightness and viewing angle differences when stitching multiple cameras together, distance-weighted fusion is performed on the overlapping areas: , ; in, This represents the pixel value (RGB or grayscale) at coordinates (u,v) of the merged image. and These represent the pixel values ​​at (u,v) of the corrected images from the first and second paths, respectively. and This is a weighting coefficient; the larger the value, the higher the credibility of that perspective at that location. This represents the Euclidean distance (in meters) from the world coordinates of the current pixel (u,v) calculated by the bird's-eye view projection to the optical center of the i-th camera in the bird's-eye view coordinate system. The closer the distance, the greater the weight. To prevent division by zero constant, the value is taken as follows: (Dimensionless) to avoid numerical errors with a denominator of zero. This strategy significantly improves the visual continuity of the stitched area, with measured brightness jumps at the fusion boundary reduced by 76%.

[0048] 3. AI Analysis and Early Warning Decision Making: Upper-level dynamic early warning: ; in, This indicates the relative distance between the center point of the detected moving target and the rotation center of the electric shovel (unit: meters). and This represents the coordinates of the center point of the moving target in the bird's-eye view coordinate system after target detection and coordinate transformation (unit: meters). and This represents the fixed coordinates of the electric shovel's rotation center in the bird's-eye view coordinate system (unit: meters), set to (0,0) during calibration. This distance is the core input of the upper-level early warning logic.

[0049] Lower level obstacle recognition: , ; Where Z represents the depth from the surface of the obstacle to the midpoint of the binocular camera baseline (unit: meters); f represents the camera's focal length (unit: pixels), which is consistent with the meaning of f in distortion correction; B represents the distance between the optical centers of the left and right lenses of the binocular camera (baseline distance), (unit: meters), which is 0.15 meters in this embodiment; d represents parallax (unit: pixels), defined as the difference in horizontal coordinates of the same obstacle point in the left and right corrected images; δ represents the dust compensation coefficient, which is dimensionless and has a value of 0.8. It is used to suppress parallax calculation errors in high-dust environments. Indicates the actual physical size (equivalent diameter) of the obstacle's projection onto the ground (unit: meter); Represents the equivalent diameter of the segmented region of the obstacle in the image (unit: pixels); This represents the ground calibration scale factor, which is dimensionless and has a value of 0.98. It is obtained by placing a calibration object of known size (such as a square with a side length of 1 meter) on-site for calibration and is used to correct dimensional errors caused by unevenness between the installation tilt angle and the ground.

[0050] Multi-source fusion (dynamic adjustment of Kalman filter): ; in, This represents the observation noise covariance of the Kalman filter, with the same dimensions as the state vector (affecting the variance of the position estimation). A larger value indicates lower observation reliability. This represents the calibration coefficient, which is dimensionless and has a value of 1.2. It is calibrated through experiments with different dust concentration gradients. This indicates the real-time ambient dust concentration value output by the dust sensor, in micrograms per cubic meter (μg / m³). This represents the normalization calibration coefficient, expressed in micrograms per cubic meter (μg / m³), with a value of 100. This ensures the logarithmic input is dimensionless and remains within the typical dust concentration range (100–1000 μg / m³). Smooth, monotonic growth (measured from 0.5 to 5.0). This mechanism improves the number of consecutive frames for target tracking in dusty environments by 187%.

[0051] 4. Model Training and Deployment: Dataset: Collected 6 months of operation videos, labeled 12,850 frames of images (including people, vehicles, rocks, potholes, etc.); Model: Improved YOLOv5s (with added CBAM attention module), mAP@0.5 reaches 89.7%; Deployment: TensorRT FP16 quantization, achieving inference speeds of up to 47fps on the hardware acceleration module.

[0052] The workflow of this invention is as follows: 1. Initialization: The system powers on and loads calibration parameters, including the camera intrinsic parameter matrix K, rotation matrix R, translation vector t, and ground calibration scale coefficient. And establish CAN communication with the electric shovel PLC; 2. Video Acquisition: Video streams are acquired synchronously by cameras on both upper and lower layers, with a timestamp alignment error of <10ms; 3. Process them separately and independently: First video stream (upper layer): Distortion correction → Bird's-eye view projection → Target detection → Calculation

[0053] Second video stream (lower layer): Binocular matching → Depth calculation → Obstacle segmentation → Calculation

[0054] In this invention, the central processing unit employs a logically decoupled architecture for processing the first and second video streams: the first video stream processing link focuses on target detection and distance calculation in the turning area, while the second video stream processing link focuses on obstacle recognition and size quantization along the travel path. Both share hardware resources but their decision-making logic is completely independent. Even if the second video stream processing is temporarily interrupted due to extreme vibration, the first video stream warning function can still operate normally, ensuring the engineering reliability of the logically decoupled design.

[0055] 4. Fusion and Decision-Making: If the radar data is valid, perform Kalman filtering to update the target status; generate early warning commands based on threshold comparisons; 5. Human-computer interaction: Dynamically switches the display content based on PLC signals, highlighting warning targets / obstacles; 6. Data Management: The early warning event only records the de-identified information (target type, coordinates, timestamp) and the hash value of the video clip used for data integrity verification. The original video stream is overwritten in real time and is not stored or transmitted.

[0056] In this embodiment, the WK-35 electric shovel was subjected to a 90-day field test.

[0057] Turning warning test: Simulated personnel / vehicles entering the turning area 127 times, the system correctly warned 121 times and missed 6 times (all due to extreme backlighting), with an accuracy rate of 95.3% and an average response time of 218ms; Walking obstacle test: 86 times with pre-placed stones ≥50cm, the system recognized the stones 82 times, with 3 false alarms (misjudgment of shadows), a recognition rate of 95.3%, and an average size measurement error of 6.8%; Environmental adaptability: In environments with PM10 concentrations of 300–800 μg / m³, the fusion algorithm increases the number of consecutive warning frames from 8.2 frames to 23.6 frames; User feedback: Driver confidence has significantly improved, and the number of emergency braking incidents due to blind spots has decreased by 76%; maintenance personnel report that the automatic lens cleaning function has reduced daily maintenance time by 85%.

[0058] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A dual-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure, characterized in that, include: The upper monitoring subsystem has cameras installed at the high point of the edge of the electric shovel's slewing platform, at a height of 8–12 meters, configured to collect video streams of a dynamic annular area with a radius of 5–15 meters centered on the shovel's slewing center. The lower monitoring subsystem has a camera mounted on the frame of the lower walking mechanism of the electric shovel with the lens tilted downwards. The installation height is 0.8–1.5 meters, and it is configured to collect video streams of the near-ground static area 0–3 meters in front of the track in the direction of travel. The central processing unit is communicatively connected to both the upper-level and lower-level monitoring subsystems. It is configured to receive a first video stream output from the upper-level subsystem and a second video stream output from the lower-level subsystem, and to process the first and second video streams independently: performing distortion correction and bird's-eye view projection on the first video stream to calculate the relative distance between the moving target and the center of rotation. Depth estimation is performed on the second video stream to calculate the actual size of the obstacle. ; The human-machine interface is connected to the central processing unit and is configured to dynamically switch the displayed content according to the operating status of the electric shovel. The early warning execution device is connected to the central processing unit and configured to... The comparison result with the preset distance threshold triggers a tiered warning, or according to... The comparison result with the preset size threshold triggers an obstacle warning; The video processing flow of the upper-level monitoring subsystem and the lower-level monitoring subsystem is logically decoupled in the central processing unit, and the early warning decisions are independent of each other, jointly covering the omnidirectional safety blind spots of the electric shovel's slewing and walking operations.

2. The double-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure according to claim 1, characterized in that, The upper monitoring subsystem includes at least four wide-angle cameras with a field of view ≥150°, which are respectively fixedly installed on the front edge of the top of the cab, the high-level bracket on the right side wall of the cab, the top platform of the counterweight box at the rear of the cab, and the support structure of the boom on the left side of the cab.

3. The double-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure according to claim 1, characterized in that, The lower-level monitoring subsystem includes at least five ruggedized cameras. The lens cover integrates a high-pressure air curtain nozzle. The lens tilt angle is adjustable from 30° to 60°. They are respectively installed on the bracket above the left track front guide wheel, the bracket above the right track front guide wheel, the bracket above the left track rear drive wheel, the bracket above the right track rear drive wheel, and the front and rear ends of the chassis area between the two tracks.

4. The double-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure according to claim 1, characterized in that, The central processing unit includes: The image correction module is configured to perform distortion correction on fisheye images using an isometric projection model and through a homography matrix. The corrected image is projected onto a unified bird's-eye view coordinate system, where , This is the camera intrinsic parameter matrix. For rotation matrix The first two columns are concatenated with the translation vector t to form a 3×3 matrix; The image stitching module is configured to perform distance-weighted fusion on overlapping regions of multi-view images; The AI ​​analysis module is configured to perform upper-layer dynamic target detection and lower-layer static obstacle analysis respectively. The early warning decision module is configured to generate tiered early warning instructions.

5. A double-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure according to claim 4, characterized in that, In the AI ​​analysis module: Upper-layer dynamic target detection calculates relative distance ;when When a Level 1 warning is triggered, A level-two warning is triggered at this time, among which ,and Positively correlated with the turning radius of the electric shovel model; Analysis and calculation of actual dimensions of lower-level static obstacles ,when And when it is located within a preset region of interest (ROI) in the direction of track movement, an obstacle warning is triggered, among which The configurable size threshold ranges from 0.3 to 1.0 meters.

6. The double-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure according to claim 4, characterized in that, The central processing unit also integrates a millimeter-wave radar data interface, and the early warning decision module is configured to fuse visual detection results with radar point cloud data using a Kalman filter algorithm; the observation noise covariance of the Kalman filter... Based on the ambient dust concentration value output by the dust sensor (Unit: μg / m³) Dynamically adjusted to satisfy the relationship. ,in , These are calibration coefficients.

7. A double-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure according to claim 1, characterized in that, The human-machine interface is configured such that when the electric shovel is in the slewing operation state, the main display area presents the upper layer synthesized panoramic bird's-eye view; after the electric shovel receives the walking command, it automatically switches the key road surface images of the lower layer monitoring subsystem to the main display area.

8. A double-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure according to claim 1, characterized in that, The early warning execution device includes a graded sound and light alarm unit: a first-level early warning triggers a high-frequency buzzer and a red flashing light; a second-level early warning triggers a medium-frequency prompt sound and a yellow constant-on light; and an obstacle early warning triggers a voice broadcast and a yellow pulse light.

9. A double-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure according to claim 1, characterized in that, All camera module housings meet IP69K protection standards, the lens surfaces are coated with a hydrophobic and oleophobic coating, and are equipped with an automatic high-pressure air curtain cleaning device.

10. A method for early warning of a double-layer 360-degree surround-view monitoring and early warning system based on an ultra-large mining electric shovel structure according to any one of claims 1 to 9, characterized in that, S1. Video streams are collected from a 5-15 meter annular area around the shovel's rotation center using an upper-level camera installed at a height of 8-12 meters; S2. Video streams are collected from the area 0–3 meters in front of the track in the direction of travel by installing a lower-level camera at a height of 0.8–1.5 meters; S3. Perform distortion correction and homography transformation on the two video streams respectively, and project them onto a unified bird's-eye view coordinate system; S4. Based on the corrected image: Calculate the relative distance between the moving target and the center of rotation in the upper flow. ; Calculate the actual size of obstacles in the lower laminar flow ; S5. According to and preset distance threshold , The relationship triggers a tiered warning, or based on With preset size threshold Relationship triggers obstacle warning; S6. Dynamically switch the content displayed on the human-machine interface according to the PLC signal of the electric shovel, and drive the audible and visual alarm device to perform the corresponding warning action.

Images