An industrial vehicle intelligent safety management system and method based on multi-modal perception

Abstract

The application provides a kind of industrial vehicle intelligent safety management system and method based on multi-modal perception, it is related to industrial vehicle safety technical field.The system includes synchronous control box, multi-camera array, vehicle-mounted processing unit and vehicle control unit.The vehicle-mounted processing unit synchronously collects the images of driver's face and vehicle panorama through the multi-camera array.The vehicle-mounted processing unit integrates environmental perception function, adopts the target detection algorithm of fusion improved HOG and deep learning features, and conducts hierarchical risk assessment on pedestrians in combination with dynamic safety area model.The system has identity authentication and fatigue analysis function, which verifies the identity of driver and detects the fatigue state based on driver image.The electronic fence function compares real-time position information with electronic fence model to determine whether a border-crossing event occurs.The system integrates identity verification, environmental perception, fatigue monitoring and electronic fence management in a unified platform, which can improve the safety and management intelligence level of forklift operation.

Description

Technical Field

[0001] This invention relates to the field of industrial vehicle safety technology, specifically to an intelligent safety management system and method for industrial vehicles based on multimodal perception. Background Technology

[0002] Although existing industrial vehicle safety management technologies have been developed for many years, they still exhibit a "fragmented" characteristic and have failed to form a systematic proactive protection that covers the entire life cycle of operations.

[0003] Regarding identity verification and access control, the card / key solution represented by the patent with authorization announcement number CN110949319B relies on easily lost and copyable physical media, achieving only one-time authorization. It cannot continuously bind the operator's identity during operation, leading to issues of unauthorized access and difficulties in management and traceability. While the fingerprint recognition solution with authorization announcement number CN120526488A achieves biometric activation, its functionality is limited to vehicle control; the operating trajectory, boundary crossing events, and alarm data are not deeply integrated with the cloud management platform.

[0004] There are also shortcomings in monitoring the driver's condition during operation. Although most environmental perception systems (such as CN118306384B) mention the driver's condition, they lack effective real-time monitoring methods; a few fatigue detection solutions use a general threshold and do not consider individual physiological differences of drivers (such as eye shape and mouth habits), resulting in a high false alarm rate in industrial environments with vibration and changes in lighting, thus limiting their practicality.

[0005] In terms of environmental perception, solutions based on ultrasound or monocular cameras have blind spots; while the solution with application publication number CN120902634A uses advanced perception technology, the system is complex and costly. More importantly, the safety decision-making models of the above solutions are mostly based on fixed distance thresholds and are not dynamically coupled with the real-time status of the vehicle (speed, lifting height, load), thus failing to reflect real-time changes in risk levels.

[0006] In summary, most existing solutions target a single safety aspect, with each functional module operating independently. Data and decision-making are not integrated, and a closed-loop management system encompassing "perception-early warning-control-recording-analysis" has not yet been formed. Therefore, a unified platform is needed that integrates highly reliable identity access, personalized driver status monitoring, multi-dimensional environmental perception, and cloud management. Summary of the Invention

[0007] The purpose of this invention is to overcome at least one technical problem existing in the prior art and to provide an intelligent safety management system and method for industrial vehicles based on multimodal perception.

[0008] On one hand, embodiments of the present invention provide an intelligent safety management system for industrial vehicles based on multimodal perception. The system includes: a synchronization control box for generating a hardware synchronization clock signal; a multi-camera array, including a driver monitoring camera and at least four environmental monitoring cameras, wherein the multi-camera array synchronously acquires images according to the hardware synchronization clock signal; and an on-board processing unit for processing the images synchronously acquired by the multi-camera array to achieve driver status monitoring, vehicle surrounding environment perception, and risk assessment, and generating corresponding graded early warning commands based on the assessment results and sending them to the vehicle control unit. This unit includes: an image processing submodule for real-time stitching of multiple images acquired by the environmental monitoring cameras using a YUV component separation and stitching algorithm to generate a panoramic view; and a driver monitoring submodule for verifying driver identity based on the images acquired by the driver monitoring camera. The system includes: a driver authentication and fatigue detection submodule; a driver monitoring and early warning submodule, used to generate corresponding graded early warning instructions based on driver authentication results and fatigue detection results; a pedestrian detection submodule, used to detect pedestrians in the panoramic view and calculate their three-dimensional spatial coordinates; a pedestrian detection and early warning submodule, used to compare the spatial relationship between the three-dimensional coordinates and the pre-built three-dimensional dynamic safety area model, and output graded early warning instructions; a positioning submodule, used to acquire the real-time location information of the industrial vehicle and receive an electronic fence model from the cloud management platform; compare the real-time location information with the electronic fence model to determine whether a boundary crossing event has occurred; when a boundary crossing event occurs, trigger a predetermined graded early warning instruction and upload the boundary crossing event data to the cloud management platform; and a vehicle control unit, used to control the vehicle to generate corresponding display and control information based on the graded early warning instructions.

[0009] Furthermore, the driver monitoring submodule includes a face recognition submodule, which comprises: an image preprocessing module for performing illumination adaptive enhancement and size normalization on images captured by the driver monitoring camera; a face alignment module for detecting facial key points and correcting the face to a standard pose through geometric transformation; a lightweight multi-task recognition module containing a jointly trained convolutional neural network for simultaneously extracting features for authentication and facial state features for assisting in liveness detection from the aligned image; and a security comparison and decision module for comparing the extracted features with a local encrypted feature library and outputting the authentication result.

[0010] Furthermore, the driver monitoring submodule includes a fatigue state detection submodule, which comprises: an individual baseline acquisition submodule, used to retrieve or establish the current driver's personalized physiological baseline, which includes at least an eye feature baseline and a mouth feature baseline; an individualized relative index calculation submodule, used to acquire the driver's current eye and mouth feature values ​​in real time, calculate their deviation from the corresponding personalized physiological baseline, and obtain individualized relative indices for the eyes and mouth; a comprehensive fatigue index generation submodule, used to weight and fuse the individualized relative indices for the eyes and mouth according to preset weights to generate a unified comprehensive fatigue index; and a state determination submodule, used to input the comprehensive fatigue index into a preset state determination model and output the driver's fatigue level.

[0011] Furthermore, the specific method for calculating the individualized relative index is as follows: For the eyes, the calculation formula is: Individualized relative index of the eyes = 1 - (Current eye feature value / Baseline value of eye feature); For the mouth, the calculation formula is: Individualized relative index of the mouth = (Current mouth feature value / Baseline value of mouth feature) - 1, and when the result is less than 0, it is counted as 0.

[0012] Furthermore, the current eye feature value is the eye aspect ratio, and the current mouth feature value is the mouth aspect ratio; the formula for generating the comprehensive fatigue index is: Overall fatigue index = (individualized relative index of the eyes × W1) + (individualized relative index of the mouth × W2); Where W1 is the weight of the eye and W2 is the weight of the mouth, and W1 > W2.

[0013] Furthermore, the state determination model is a state machine, and its state transition conditions are based on the magnitude and duration of the comprehensive fatigue index. The conditions include: when the comprehensive fatigue index continuously exceeds the first threshold for a first time length, it is determined to be mild fatigue; when in a mild fatigue state, if the comprehensive fatigue index further increases and continuously exceeds the second threshold for a second time length, it is determined to be moderate fatigue; when in a moderate fatigue state and the system has issued an alarm, if the comprehensive fatigue index does not decrease to below the third threshold within a third time length, it is determined to be severe fatigue; the second threshold is greater than the first threshold, and the second time length is less than the first time length.

[0014] Furthermore, the pedestrian detection submodule is used to perform pedestrian detection on the panoramic view using a multi-scale detection model that integrates histogram of oriented gradients (HGP) features and deep learning features, and to calculate the three-dimensional spatial coordinates of the pedestrian relative to the vehicle based on multi-view geometry principles. This includes: using an improved YOLOv5s model and an improved HOG feature descriptor, dynamically weighting and fusing the extracted deep features and traditional features on the feature layer to detect pedestrians; wherein the weighting coefficients are dynamically adjusted according to the estimated distance between the detected target and the vehicle; and calculating the three-dimensional spatial coordinates of the detected pedestrian using a stereo vision algorithm based on images from at least two different perspectives from at least four environmental monitoring cameras.

[0015] Furthermore, the three-dimensional dynamic safety zone model includes a warning zone, a alert zone, and a danger zone centered on the vehicle and with dynamically adjustable range. The adjustment parameters for the zone range include the vehicle's real-time speed, fork lifting height, and load information. By calculating the distance between the center point of the human silhouette and the boundary of the forklift safety zone, it is determined whether the human has entered the warning zone. This includes: when the distance between the center point of the human silhouette and the boundary of the forklift safety zone is less than a preset first distance threshold, it is determined that the human has entered the danger zone, and a first warning instruction is generated; when the distance between the center point of the human silhouette and the boundary of the forklift safety zone is greater than or equal to the preset first distance threshold and less than a preset second distance threshold, it is determined that the human has entered the warning zone, and a second warning instruction is generated; when the distance between the center point of the human silhouette and the boundary of the forklift safety zone is greater than or equal to the preset second distance threshold and less than a preset third distance threshold, it is determined that the human has entered the warning zone, and a third warning instruction is generated.

[0016] Furthermore, the three-dimensional dynamic safety zone model dynamically determines the three-dimensional safety zone boundary based on the current operating state of the industrial vehicle, including: acquiring at least one of the vehicle's real-time speed, lifting height, and load weight; and scaling or extending the preset basic safety zone model in the corresponding direction based on the state parameters, wherein the higher the driving speed, the longer the forward safety zone extends; and the higher the lifting height, the larger the upper safety zone expands.

[0017] Secondly, embodiments of the present invention provide an intelligent safety management method for industrial vehicles based on multimodal perception. This method is applied to the aforementioned intelligent safety management system for industrial vehicles based on multimodal perception. The method includes: Step S1, sending a hardware synchronization trigger signal to a multi-camera array via a synchronization control box to control at least four environmental monitoring cameras distributed around the vehicle and a driver monitoring camera facing the driver's cabin to simultaneously acquire images; Step S2, using an image processing submodule to stitch the multiple images acquired by the environmental monitoring cameras in real-time in the YUV color space to generate a 360-degree panoramic view; Step S3, using a driver monitoring submodule to perform driver authentication and fatigue detection based on the images acquired by the driver monitoring camera; Step S4, using a driver monitoring and early warning submodule to generate corresponding graded early warning instructions based on the driver authentication results and fatigue detection results; Step S5, using a pedestrian detection submodule to employ a multi-scale detection model that fuses directional gradient histogram features and deep learning features. Step S6: The pedestrian detection and warning submodule performs pedestrian detection on the panoramic view and calculates the three-dimensional spatial coordinates of the pedestrian relative to the vehicle based on the multi-view geometry principle; Step S7: The pedestrian detection and warning submodule uses a spatial relationship judgment and warning triggering algorithm to generate corresponding graded warning instructions in real time based on the spatial relationship between the three-dimensional spatial coordinates and the pre-constructed three-dimensional dynamic safety area model; Step S8: The positioning submodule obtains the real-time location information of the industrial vehicle and receives the electronic fence model from the cloud management platform; The real-time location information is compared with the electronic fence model to determine whether a boundary crossing event has occurred; When a boundary crossing event occurs, a predetermined graded warning instruction is triggered, and the boundary crossing event data is uploaded to the cloud management platform; The electronic fence model defines the geographical boundaries of the allowed or prohibited passage areas; Step S9: The vehicle control unit controls the vehicle to generate corresponding display information and control information based on the corresponding graded warning instructions sent by the pedestrian detection and warning submodule, the driver monitoring and warning submodule, and the positioning submodule.

[0018] The advantages of this invention compared to the prior art are: (1) It integrates functions such as AI face recognition, fatigue driving detection, 360 panoramic imaging and GPS electronic fence, covering a variety of safety hazards in the use of forklifts and providing comprehensive safety protection.

[0019] (2) Real-time monitoring and abnormal alarm functions effectively reduce the possibility of accidents. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] Figure 1This is a structural diagram of an intelligent safety management system for industrial vehicles based on multimodal perception, provided in Embodiment 1 of the present invention.

[0022] Figure 2 This is a schematic diagram of the workflow of a pedestrian detection submodule provided in Embodiment 1 of the present invention.

[0023] Figure 3 This is a schematic diagram of the workflow of a fatigue state detection submodule provided in Embodiment 1 of the present invention.

[0024] Figure 4 and 5 This is a schematic diagram of an iterative update mechanism for a face recognition feature library provided in Embodiment 1 of the present invention.

[0025] Figure 6 This is a flowchart of an intelligent safety management method for industrial vehicles based on multimodal perception, provided in Embodiment 2 of the present invention. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0027] Example 1 To facilitate understanding, the working principle of this system will be generally described before the detailed description of the embodiments of the present invention: Before the vehicle starts, the driver monitoring camera captures the driver's facial image, and the on-board processing unit performs face comparison to complete identity authentication; the driver monitoring camera continuously captures the driver's facial image, and the on-board processing unit analyzes the features such as the eyes and mouth, and compares them with the driver's personalized physiological baseline to determine the fatigue level; four environmental monitoring cameras simultaneously capture images of the vehicle's surroundings, and the on-board processing unit stitches them into a 360-degree panoramic view, identifies pedestrians through a pedestrian detection algorithm and calculates their three-dimensional spatial coordinates, and performs risk assessment in combination with a dynamic safety area model; the positioning module obtains the vehicle's real-time location and compares it with the electronic fence model issued by the cloud to determine whether it has crossed the boundary; according to different types and levels of risk, corresponding graded warning instructions are generated to drive the vehicle control unit to execute audible and visual alarms, display prompts, or vehicle control.

[0028] The specific implementation method is as follows: like Figure 1 The diagram shown is a structural diagram of an intelligent safety management system for industrial vehicles based on multimodal perception provided by the present invention.

[0029] The following graded warning commands appearing in this embodiment include: First warning command: Pedestrian enters the danger zone, triggering the highest level alarm (continuous rapid audible and visual alarm, which can be linked to vehicle deceleration); Second warning command: Pedestrian enters the warning zone, triggering the medium level alarm (audible and visual alarm); Third warning command: Pedestrian enters the reminder zone, triggering the low level alarm (voice prompt); Fourth warning command: Identity verification fails, triggering the unauthorized driving alarm; Fifth warning command: Mild fatigue, triggering the voice reminder; Sixth warning command: Moderate fatigue, triggering the audible and visual alarm; Seventh warning command: Severe fatigue, triggering the highest level alarm and can be linked to vehicle control.

[0030] Subsequently, each early warning submodule triggers the corresponding early warning instruction based on its own judgment logic.

[0031] As an example, the system includes: a synchronization control box 1 for generating a hardware synchronization clock signal; a multi-camera array 2, including at least one driver monitoring camera facing the interior of the cockpit and at least four environmental monitoring cameras distributed around the vehicle, the multi-camera array being connected to the synchronization control box 1 and performing synchronized image acquisition based on the received hardware synchronization clock signal; and an on-board processing unit 3, installed in the cockpit of the industrial vehicle, for processing the images synchronously acquired by the multi-camera array 2 to achieve driver status monitoring, vehicle surrounding environment perception, and risk assessment, and generating corresponding graded warning commands based on the assessment results and sending them to the vehicle control unit, including: an image processing submodule 30 for real-time stitching of multiple images acquired by the environmental monitoring cameras in the YUV color space using a YUV component separation and stitching algorithm to generate a 360-degree panoramic view; a driver monitoring submodule 31 for driver authentication and fatigue detection based on the images acquired by the driver monitoring cameras; and a driver monitoring and warning submodule 32 for generating corresponding graded warning commands based on the driver authentication results and fatigue detection results. The pedestrian detection submodule 33 is used to perform pedestrian detection on the panoramic view using a multi-scale detection model that integrates directional gradient histogram features and deep learning features, and calculates the three-dimensional spatial coordinates of the pedestrian relative to the vehicle based on the multi-view geometric principle. The pedestrian detection early warning submodule 34 is used to generate corresponding hierarchical early warning instructions in real time based on the spatial relationship between the three-dimensional spatial coordinates and the pre-constructed three-dimensional dynamic safety area model using a spatial relationship judgment and early warning triggering algorithm. The positioning submodule 35 is used to obtain the real-time location information of the industrial vehicle and receive the electronic fence model from the cloud management platform; compare the real-time location information with the electronic fence model to determine whether a boundary crossing event has occurred; when a boundary crossing event occurs, trigger the predetermined hierarchical early warning instruction and upload the boundary crossing event data to the cloud management platform; wherein, the electronic fence model defines the geographical boundaries of the allowed or prohibited passage areas. The vehicle control unit 4 is used to control the vehicle to generate corresponding display information and control information based on the hierarchical early warning instructions sent by the pedestrian detection early warning submodule 34, the driver monitoring early warning submodule 32 and the positioning submodule 35. In some feasible implementations, the multi-camera array includes one driver monitoring camera in the cockpit and four vehicle perimeter monitoring cameras. A synchronization clock signal is output by a synchronization control box to ensure that all cameras acquire data synchronously, with the acquisition time difference controlled within ±1ms.

[0032] In some feasible implementations, the system integrates a self-healing function for camera link failures. When a persistent anomaly is detected in the preview data of a certain camera (anti-shake threshold of 3 seconds), software reconnection and GPIO hardware reset are executed sequentially: software reconnection is achieved by releasing current device resources, rescanning USB devices, and restoring the preview stream; if three consecutive software reconnection attempts fail, the USB hub is powered off for 1-2 seconds via GPIO pins and then powered back on, triggering device re-enumeration. After the reset is completed, the system automatically restores the timing synchronization of the four cameras. The software reconnection takes 18-47ms, and the hardware reset adds an additional 1-2 seconds of power-off time; neither affects the continuity of panoramic stitching during normal vehicle operation.

[0033] In some feasible implementations, the image processing submodule 30 is used to perform real-time stitching of multiple images captured by the environmental monitoring camera in the YUV color space using a YUV component separation and stitching algorithm to generate a 360-degree panoramic view.

[0034] Preferably, the image processing submodule 30 is used to: acquire NV21 format image data from four cameras respectively, each image data containing a Y component and a UV component, wherein the Y component and UV component are stored continuously in NV21 format; separate the Y component and UV component from each image data to obtain four Y component data and four UV component data; stitch the four Y component data in a 2x2 layout, wherein the Y component data of the first and second channels are interleaved and written into the upper half of the panoramic image, and the Y component data of the third and fourth channels are interleaved and written into the lower half of the panoramic image; stitch the four UV component data in a 2x2 layout, wherein the UV component data of the first and second channels are interleaved and written into the upper half of the UV component of the panoramic image, and the UV component data of the third and fourth channels are interleaved and written into the lower half of the UV component of the panoramic image, wherein the stitching position of the UV component corresponds to the stitching position of the Y component, ensuring the spatial correspondence between chroma and luminance.

[0035] Preferably, the specific steps of writing the Y component in a row-by-row interleaved manner include: for the upper part, for each row i (i from 0 to 479), the i-th row of the Y component of the first camera is written into the (2×i)-th row of the panoramic Y component, and the i-th row of the Y component of the second camera is written into the (2×i+1)-th row of the panoramic Y component; for the lower part, for each row i (i from 0 to 479), the i-th row of the Y component of the third camera is written into the (960+2×i)-th row of the panoramic Y component, and the i-th row of the Y component of the fourth camera is written into the (960+2×i+1)-th row of the panoramic Y component.

[0036] Preferably, the specific steps of writing the UV components in a row-by-row staggered manner include: for the upper part, for each row i (i from 0 to 239), the i-th row of the UV components of the first camera is written into the (2×i)-th row of the panoramic image UV components, and the i-th row of the UV components of the second camera is written into the (2×i+1)-th row of the panoramic image UV components; for the lower part, for each row i (i from 0 to 239), the i-th row of the UV components of the third camera is written into the (480+2×i)-th row of the panoramic image UV components, and the i-th row of the UV components of the fourth camera is written into the (480+2×i+1)-th row of the panoramic image UV components.

[0037] Tests have shown that direct YUV domain stitching reduces computation by approximately 60% and memory usage by 50% compared to traditional RGB domain conversion stitching. End-to-end latency is controlled at 50-80ms, with a stable frame rate of 15-30fps and a color distortion rate of less than 1% (compared to 11-17% in traditional solutions), meeting the real-time requirements of forklift panoramic monitoring.

[0038] The aforementioned YUV domain stitching method enables real-time stitching and panoramic output of images from four cameras on an ARM embedded platform.

[0039] In some feasible implementations, the driver monitoring submodule 31 includes a face recognition submodule 310, which includes: an image preprocessing module 3101, used to perform illumination adaptive enhancement and size normalization on the image acquired by the driver monitoring camera; a face alignment module 3102, used to detect facial key points and correct the face to a standard pose through geometric transformation; a lightweight multi-task recognition module 3103, including a jointly trained convolutional neural network, used to simultaneously extract features for identity verification and facial state features for assisting in liveness detection from the aligned image; and a security comparison and decision module 3104, used to compare the extracted features with a local encrypted feature library and output the identity verification result.

[0040] Preferably, the liveness detection is implemented based on at least one of the following methods: (1) distinguishing live faces from static photos or screen captures by analyzing facial micro-movements (such as blinking and slight lip movements) between consecutive frames; (2) identifying texture periodicity or screen reflection features unique to photos / videos by using texture information in the facial state features output by the multi-task branch. When the liveness detection fails, the system still outputs a 'failed identity verification' result even if the identity feature comparison is successful. The liveness detection is an auxiliary security measure, and the system can choose to skip this detection according to the configuration in low-security scenarios. It should be noted that the liveness detection can be implemented using any liveness detection method known in the art, and the present invention does not limit this.

[0041] Preferably, the face recognition process of the driver monitoring submodule 31 is as follows: The acquired driver's facial image is first enhanced by illumination adaptation (based on Retinex theory to compensate for illumination components) and normalized in size. Then, the face region is located through the SCRFD lightweight detection network. After detecting more than 5 key points, it is corrected to a standard frontal pose through affine transformation. The corrected face image is fed into the improved EfficientNet-B0 backbone network to extract 128-dimensional features. This network embeds channel and spatial attention mechanisms, is trained using the ArcFace loss function, and undergoes 8-bit quantization and channel pruning to adapt to embedded deployment. The extracted features are compared with the local encrypted feature library (cosine distance). If the similarity exceeds the threshold, the verification is successful; otherwise, an unauthorized alarm is triggered. For partial occlusion situations such as wearing a helmet or mask, the system automatically switches to the unoccluded eye area for local feature comparison, expanding the effective recognition angle from ±15° to ±45°.

[0042] It should be noted that the improved EfficientNet-B0 backbone network used above is only one specific implementation example of the lightweight multi-task recognition module. In actual deployment, those skilled in the art can choose other lightweight convolutional neural network architectures such as MobileNetV3 and ShuffleNetV2 as alternatives, depending on the computing power constraints and accuracy requirements of the edge device. As long as the network has the multi-task capability of simultaneously extracting authentication features and facial state features from the aligned image, it falls within the technical scope of the "lightweight multi-task recognition module" described in this embodiment.

[0043] In addition, the face recognition submodule 310 is also equipped with an iterative update mechanism for the face recognition feature library, which is used to continuously optimize the recognition accuracy of the local encrypted feature library during actual operation, so that the system can adapt to the slow drift of facial features caused by factors such as driver aging, changes in makeup, and changes in clothing (such as helmets, goggles, and masks), and avoid the recognition accuracy from decreasing with the increase of usage time.

[0044] Combination Figure 4 As shown, the specific workflow of this iterative update mechanism is as follows: During driver identity verification, the system calculates the recognition confidence score for each captured facial image. The initially collected sample data is divided into three categories according to data type and stored in a primary cache pool (i.e., the initial sample library): employee ID, facial feature HASH value, and terminal device ID; The system continuously validates samples in the primary cache pool, with no fewer than n validations (n≥10). Based on the validation results, the following hierarchical migration logic is executed: when the recognition accuracy of a sample in continuous validation is consistently higher than a preset high-confidence threshold (preset to 99% in this embodiment), the confidence level of the sample is progressively increased (e.g., from level 02.10 to level 02.11, then to level 02.1n, where n is the number of consecutive validations passed), and it is eventually included in the high-confidence sample pool (i.e., ...). Figure 4 The data in this high-confidence sample pool (shown as the 01 agent pool) is considered high-quality positive samples and can be used for incremental training of the subsequent feature library and model fine-tuning. When the accuracy of a sample in continuous validation remains below a preset low confidence threshold (1% in this embodiment), the confidence level of the sample decreases progressively and it is eventually relegated to the discard pool. Data in the discard pool is no longer used for feature library updates and model training. like Figure 5 As shown, the high-confidence sample pool adopts a hierarchical management structure, divided into multiple sub-pools (Agent 01, Agent 02, Agent 03, ..., Agent n) according to confidence level. The hierarchical relationship between the sub-pools reflects the progressive gradient of sample data quality: the higher the confidence level (the larger the number), the higher the level of the corresponding sub-pool, and the better the performance of the samples stored in it in terms of feature stability and representativeness. Data in higher-level sub-pools enjoy the highest priority weight when updating the feature library and are preferentially used for incremental training and model fine-tuning.

[0045] Data in the high-confidence sample pool is processed in batches periodically and then used to incrementally update the local encrypted feature library. The updated feature library is synchronized to the local encrypted feature library of the onboard processing unit via a secure channel, for use by the secure comparison and decision module 3104 in subsequent identity verification processes. This forms a continuous iterative closed loop of "collection → evaluation → screening → storage → update → deployment," ensuring that the face recognition model always maintains a high recognition accuracy.

[0046] It should be noted that the above iterative update mechanism can be executed during system idle periods or when vehicles are out of service for maintenance, without affecting the real-time authentication performance during normal vehicle operation.

[0047] In some feasible implementations, the driver monitoring submodule 31 further includes a fatigue state detection submodule 311, which includes: an individual baseline acquisition submodule 3111, used to retrieve or establish the current driver's personalized physiological baseline, which includes at least an eye feature baseline and a mouth feature baseline; an individualized relative index calculation submodule 3112, used to acquire the current feature values ​​of the driver's eyes and mouth in real time, calculate their deviation from the corresponding personalized physiological baseline, and obtain individualized relative indices for the eyes and mouth; a comprehensive fatigue index generation submodule 3113, used to weight and fuse the individualized relative indices for the eyes and mouth according to preset weights to generate a unified comprehensive fatigue index; and a state determination submodule 3114, used to input the comprehensive fatigue index into a preset state determination model and output the driver's fatigue level.

[0048] Preferably, the specific method for calculating individualized relative indicators is as follows: For the eye, the calculation formula is: Individualized relative index of the eye = 1 - (current eye feature value / baseline value of eye feature); For the mouth, the calculation formula is: Mouth Individualized Relative Index = (Current Mouth Feature Value / Mouth Feature Baseline Value) - 1, and when the result is less than 0, it is counted as 0.

[0049] Preferably, the current eye feature value is the eye aspect ratio, and the current mouth feature value is the mouth aspect ratio; the formula for generating the comprehensive fatigue index is: Overall fatigue index = (individualized relative index of the eyes × W1) + (individualized relative index of the mouth × W2); Where W1 is the weight of the eye and W2 is the weight of the mouth, and W1 > W2.

[0050] Preferably, the individualized relative index calculation submodule 3112 is also used to acquire the current feature values ​​of the driver's eyes, mouth and head in real time, calculate their deviation from the corresponding personalized physiological baseline, and obtain the individualized relative index of the eyes, the individualized relative index of the mouth and the individualized relative index of the head.

[0051] Preferably, the state determination model is a state machine, and its state transition conditions are based on the magnitude and duration of the comprehensive fatigue index. The conditions include: when the comprehensive fatigue index continuously exceeds a first threshold for a first time length, it is determined to be mild fatigue; when in a mild fatigue state, if the comprehensive fatigue index further increases and continuously exceeds a second threshold for a second time length, it is determined to be moderate fatigue; when in a moderate fatigue state and the system has issued an alarm, if the comprehensive fatigue index does not decrease to below the third threshold within a third time length, it is determined to be severe fatigue; the second threshold is greater than the first threshold, and the second time length is less than the first time length.

[0052] As an example, in actual deployment, the first threshold can be preset to 0.35, the second threshold can be preset to 0.65, the first time length can be preset to 5 seconds, the second time length can be preset to 2 seconds, the third time length can be preset to 8 seconds, and the third threshold can be preset to 0.30. These values ​​can be calibrated and adjusted according to the actual safety requirements of different industrial vehicle types (such as counterbalance forklifts and reach trucks) and operating scenarios (such as indoor warehousing and outdoor storage yards). This embodiment does not impose a unique limitation on these values.

[0053] Specifically, in combination Figure 3 As shown, fatigue detection is based on relative index calculation using individualized baselines. The system pre-establishes baselines for the current driver's eye aspect ratio and mouth aspect ratio. During real-time detection, the individualized relative index for the eyes = 1 - (current eye aspect ratio / eye baseline value), and the individualized relative index for the mouth = (current mouth aspect ratio / mouth baseline value) - 1 (negative values ​​are 0). The comprehensive fatigue index = (eye index × W1) + (mouth index × W2), where W1 > W2. A state machine is used for state determination: if the comprehensive fatigue index exceeds the first threshold for a first duration, it indicates mild fatigue; if the index continues to rise beyond the second threshold for a second duration after mild fatigue (second threshold > first threshold, second duration < first duration), it indicates moderate fatigue; if the system issues an alarm after moderate fatigue but the index does not drop below the third threshold within a third duration, it indicates severe fatigue. The three levels correspond to voice reminders, audible and visual alarms, and vehicle control linkage, respectively.

[0054] In some feasible implementations, the pedestrian detection submodule 33 is used to: use an improved YOLOv5s model and an improved HOG (Histogram of Oriented Gradients) feature descriptor to dynamically weight and fuse the extracted depth features and traditional features on the feature layer to detect pedestrians; wherein the weighting coefficients are dynamically adjusted according to the estimated distance between the detected target and the vehicle; and calculate the three-dimensional spatial coordinates of the detected pedestrians using a stereo vision algorithm based on images from at least two different perspectives from at least four environmental monitoring cameras.

[0055] Preferably, the three-dimensional dynamic safety zone model includes a warning zone, a alert zone, and a danger zone centered on the vehicle and with dynamically adjustable range. The adjustment parameters for the zone range include the vehicle's real-time speed, fork lifting height, and load information. The system determines whether a person has entered the warning zone by calculating the distance between the center point of the human silhouette and the boundary of the forklift safety zone. This includes: determining if the person has entered the danger zone and generating a first warning instruction when the distance is less than a preset first distance threshold; determining if the person has entered the warning zone and generating a second warning instruction when the distance is greater than or equal to the preset first distance threshold and less than a preset second distance threshold; and determining if the person has entered the warning zone and generating a third warning instruction when the distance is greater than or equal to the preset second distance threshold and less than a preset third distance threshold.

[0056] Preferably, the three-dimensional dynamic safety zone model dynamically determines the three-dimensional safety zone boundary based on the current operating state of the industrial vehicle, including: acquiring at least one of the vehicle's real-time speed, lifting height, and load weight; and scaling or extending the preset basic safety zone model in the corresponding direction based on the state parameters, wherein the higher the driving speed, the longer the forward safety zone extends; and the higher the lifting height, the larger the upper safety zone expands.

[0057] Specifically, in combination Figure 2 As shown, the pedestrian detection submodule 33 is used to stitch four environmental images into a 360-degree panoramic image (during stitching, SIFT feature point matching and RANSAC purification are used to eliminate overlapping area misalignment, and illumination compensation is performed). A multi-scale image pyramid is constructed on the panoramic image, and improved HOG features (gradient direction bins increased to 12) and YOLOv5s depth features are extracted for each scale. At the feature map level, dynamic weighted fusion is performed according to the target scale (depth weight is greater for near objects, and HOG weight is greater for distant objects). After a pedestrian is detected, the three-dimensional spatial coordinates of the pedestrian are calculated by triangulation using images from at least two different camera perspectives.

[0058] The 3D dynamic safety zone model is centered on the vehicle and divided into a warning zone, a alert zone, and a danger zone. Its boundaries are adjusted in real time based on vehicle speed, lifting height, and load (the higher the speed, the longer the forward extension; the higher the lifting height, the greater the upward extension). The system compares the pedestrian's 3D coordinates with the dynamic zone boundaries: if the coordinates are below the first threshold, the highest level alarm is triggered and deceleration is activated; if the coordinates are between the first and second thresholds, an audible and visual alarm is triggered; if the coordinates are between the second and third thresholds, a voice prompt is triggered.

[0059] In terms of environmental adaptability, digital image stabilization algorithms (such as those based on optical flow) are used to compensate for vibrations during forklift operation. For motion-blurred images, deblurring algorithms (such as Wiener filtering) are used to repair them, ensuring image quality. Simultaneously, the contrast threshold and feature extraction parameters of the detection algorithm are automatically adjusted based on the overall brightness distribution of the image to adapt to strong light, backlight, and dim lighting environments. Regarding false alarm suppression, the system establishes a static background model of the current work scene (such as fixed shelves and columns), using foreground extraction technology to distinguish moving pedestrians from the static background, reducing false alarms caused by background interference. It also integrates the DeepSORT multi-target tracking algorithm, assigning a unique ID to each detected pedestrian and performing cross-frame tracking. Combining historical location data, it predicts the movement trajectory for the next 1-2 seconds, providing early warnings for pedestrians who may enter dangerous areas.

[0060] In some feasible implementations, the positioning submodule 35 uses a multi-mode GNSS receiver to acquire the real-time position of the vehicle, and uses a combination algorithm of Kalman filtering and sliding window dynamic weighted averaging to filter and optimize the original positioning data: Kalman filtering predicts the current position based on the vehicle motion model and merges it with the measured GNSS values ​​to smooth the trajectory; sliding window weighted averaging assigns weights to the filtered positioning points in the recent period according to their deviation from the predicted trajectory, removes obvious drift points, and calculates the weighted center as the final output, which can stably control the positioning error within 3-5 meters (2-3 meters in open areas) and eliminate false boundary crossing alarms caused by signal jitter.

[0061] Electronic fences support both circular and polygonal shapes. Circular fences are defined by their center point and radius, while polygonal fences are defined by an ordered sequence of vertices. Fence attributes are divided into static fences (fixed work areas or restricted areas) and dynamic fences (which can be created temporarily or bound to a work order system). The effectiveness and size of the fence can also be linked to vehicle status; for example, when the forks are raised above a set height, the activity radius can be automatically reduced to lower the risk of tipping over.

[0062] In boundary violation detection, circular fences are determined by calculating the distance between the vehicle's position and the center of the fence's great circle and comparing it with the radius. Polygonal fences employ an improved ray method, radiating a ray horizontally from the vehicle's position and counting the number of intersections with each side of the polygon, with odd numbers inside and even numbers outside. For boundary points, the shortest distance to each side is calculated; if it is less than a set threshold (e.g., 0.5 meters), it is considered a boundary condition. Upon boundary violation detection, the system triggers a voice prompt or audible and visual alarm locally, and simultaneously reports the alarm information (time, location, fence name, vehicle number, driver ID) to the cloud management platform via 4G / 5G network, displaying the alarm vehicle's location and trajectory on the monitoring interface. Alarm rules support delayed alarms and time-based configuration.

[0063] In some feasible implementations, to adapt to the limited computing power of embedded edge devices, the system adopts a channel pruning and layer fusion strategy for the convolutional neural network: by analyzing the channel importance of each convolutional layer of the model (calculating the L1 norm of each channel and removing redundant channels with norms below the threshold), the model parameter size is compressed by more than 40% while ensuring that the model accuracy loss is ≤3%; at the same time, adjacent convolutional layers are fused with BN (batch normalization) layers to reduce the amount of computation during model inference (such as merging convolution operations and normalization operations into a single operation) and improve the inference speed of edge devices.

[0064] In traditional solutions, face recognition and fatigue detection require separate extraction of facial features, leading to redundant computation. Therefore, this embodiment utilizes the improved EfficientNet-B0 backbone network, which shares 128-dimensional basic features between face recognition and fatigue detection. These features are then mapped to the identity recognition and fatigue detection task spaces via different fully connected layers, avoiding redundant computation and reducing feature extraction computation by approximately 60% and processing latency to 60ms. In pedestrian detection, HOG features and deep learning features are weighted and fused at the feature map level instead of vector concatenation. The weights are dynamically adjusted based on scene complexity, reducing memory usage from 8MB to 3MB while maintaining detection accuracy. The system also dynamically adjusts image resolution (1080P / 720P / 480P levels) based on scene target density, motion state, and lighting conditions, reducing average computation by approximately 35% and stabilizing processing latency for each detection task to within 100ms.

[0065] In some feasible implementations, the system dynamically adjusts the image processing resolution based on scene complexity to optimize edge computing power allocation. Scene complexity is comprehensively evaluated from four dimensions: target density (the ratio of the number of targets to their area in the scene), target movement speed (calculated based on optical flow, combined with the forklift's stationary / moving / lifting state), lighting conditions (brightness distribution and contrast analysis), and environmental interference factors (occlusions, reflective surfaces, dynamic backgrounds, etc.). The evaluation results are used to drive the switching between three resolution levels (1080P, 720P, 480P).

[0066] Preferably, different resolution strategies are adopted for different visual tasks: fatigue detection maintains 1080P to ensure clear eye details, face recognition is no less than 720P, and pedestrian detection switches between 720P and 480P depending on the target distance. A smooth transition mechanism is implemented for resolution switching, with the switching time controlled within 100ms to avoid fluctuations in detection accuracy. The system also monitors the processing latency and detection accuracy of each task in real time; when the latency exceeds 100ms, the resolution is automatically reduced; when the accuracy decreases, the resolution is automatically increased.

[0067] Actual testing showed that dynamic resolution adjustment can reduce the average computational load by about 30%, the peak computational load by about 50%, and the memory usage from about 2.5GB in the fixed mode to an average of 1.8GB. The processing latency of each detection task is stabilized within 100ms.

[0068] In some feasible implementations, when the system detects an alarm or other abnormal situation, it immediately triggers a hardware-controlled vehicle power-off mechanism. This mechanism is independent of the vehicle's conventional control system, employing physically isolated control circuitry and an independent power module to ensure reliable operation even if the main system fails or is interfered with. Specifically, upon receiving a warning command that has undergone multiple verifications, the vehicle control unit, while generating corresponding display and control information, can also quickly cut off the power supply to the vehicle's drive motor, high-voltage battery output circuit, and power steering system, while maintaining basic power supply to the braking system to ensure vehicle braking function. This allows the vehicle to stop smoothly within a safe distance, effectively preventing loss of vehicle control due to abnormal situations and ensuring driving safety.

[0069] Example 2 Please see Figure 6 This embodiment provides a flowchart of an intelligent safety management method for industrial vehicles based on multimodal perception.

[0070] As an example, the method is applied to the intelligent safety management system for industrial vehicles based on multimodal perception described in Embodiment 1, and the method includes: Step S1: Send a hardware synchronization trigger signal to the multi-camera array through the synchronization control box to control at least four environmental monitoring cameras distributed around the vehicle and driver monitoring cameras facing the cockpit to synchronously acquire images.

[0071] Step S2: The image processing submodule stitches the multiple images captured by the environmental monitoring camera in real time in the YUV color space to generate a 360-degree panoramic view.

[0072] Step S3: The driver monitoring submodule performs driver authentication and fatigue detection based on the images captured by the driver monitoring camera.

[0073] Step S4: The driver monitoring and early warning submodule generates a corresponding graded early warning instruction based on the driver's identity verification result and fatigue status detection result.

[0074] Step S5: The pedestrian detection submodule uses a multi-scale detection model that integrates directional gradient histogram features and deep learning features to detect pedestrians in the panoramic view, and calculates the three-dimensional spatial coordinates of the pedestrians relative to the vehicles based on the multi-view geometric principle.

[0075] Step S6: The pedestrian detection and early warning submodule uses a spatial relationship judgment and early warning triggering algorithm to generate corresponding hierarchical early warning instructions in real time based on the spatial relationship between the three-dimensional spatial coordinates and the pre-built three-dimensional dynamic safety area model.

[0076] Step S7: Obtain the real-time location information of the industrial vehicle through the positioning submodule and receive the electronic fence model from the cloud management platform; compare the real-time location information with the electronic fence model to determine whether a boundary crossing event has occurred; when a boundary crossing event occurs, trigger a predetermined graded early warning instruction and upload the boundary crossing event data to the cloud management platform; wherein, the electronic fence model defines the geographical boundaries of the areas where passage is permitted or prohibited.

[0077] Step S8: The vehicle control unit controls the vehicle to generate corresponding display and control information based on the corresponding graded warning commands sent by the pedestrian detection warning submodule, driver monitoring warning submodule and positioning submodule.

[0078] It is not difficult to see that this embodiment is a method embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0079] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. An intelligent safety management system for industrial vehicles based on multimodal perception, characterized in that, The system includes: Synchronization control box, used to generate hardware synchronization clock signals; A multi-camera array, including a driver monitoring camera and at least four environmental monitoring cameras, wherein the multi-camera array synchronously acquires images according to the hardware synchronization clock signal; The onboard processing unit processes the images synchronously acquired by the multi-camera array to achieve driver status monitoring, vehicle surrounding environment perception, and risk assessment. Based on the assessment results, it generates corresponding tiered warning commands and sends them to the vehicle control unit. This includes: The image processing submodule is used to stitch together multiple images captured by the environmental monitoring camera in real time using a YUV component separation and stitching algorithm to generate a panoramic view. The driver monitoring submodule is used to perform driver authentication and fatigue detection based on images captured by the driver monitoring camera. The driver monitoring and early warning submodule is used to generate corresponding graded early warning instructions based on the driver's identity verification results and fatigue status detection results. The pedestrian detection submodule is used to detect pedestrians in the panoramic view and calculate their three-dimensional spatial coordinates; the pedestrian detection early warning submodule is used to compare the spatial position relationship between the three-dimensional coordinates and the pre-built three-dimensional dynamic safety area model, and output graded early warning instructions. The positioning submodule is used to acquire the real-time location information of the industrial vehicle, receive the electronic fence model from the cloud management platform, compare the real-time location information with the electronic fence model to determine whether a boundary crossing event has occurred, and when a boundary crossing event occurs, trigger a predetermined graded early warning command and upload the boundary crossing event data to the cloud management platform. The vehicle control unit is used to control the vehicle to generate corresponding display and control information based on the graded warning instructions.

2. The intelligent safety management system for industrial vehicles based on multimodal perception according to claim 1, characterized in that, The driver monitoring submodule includes a face recognition submodule, which includes: The image preprocessing module is used to perform illumination adaptive enhancement and size normalization on the images captured by the driver monitoring camera; The face alignment module is used to detect key facial points and correct the face to a standard pose through geometric transformation; The lightweight multi-task recognition module includes a jointly trained convolutional neural network for simultaneously extracting features for identity verification and facial state features for assisting in liveness detection from the aligned image. The security comparison and decision module is used to compare the extracted features with the local encrypted feature library and output the authentication result.

3. The intelligent safety management system for industrial vehicles based on multimodal perception according to claim 1, characterized in that, The driver monitoring submodule includes a fatigue state detection submodule, which includes: The individual baseline acquisition submodule is used to retrieve or establish the current driver's personalized physiological baseline, which includes at least the eye feature baseline and the mouth feature baseline. The individualized relative index calculation submodule is used to obtain the current feature values ​​of the driver's eyes and mouth in real time, calculate their deviation from the corresponding personalized physiological baseline, and obtain the individualized relative index of the eyes and the individualized relative index of the mouth. The comprehensive fatigue index generation submodule is used to weight and fuse the individualized relative indicators of the eyes and mouth according to preset weights to generate a unified comprehensive fatigue index. The state determination submodule is used to input the comprehensive fatigue index into a preset state determination model and output the driver's fatigue level.

4. The intelligent safety management system for industrial vehicles based on multimodal perception according to claim 3, characterized in that, The specific method for calculating individualized relative indicators is as follows: For the eye, the calculation formula is: Individualized relative index of the eye = 1 - (current eye feature value / baseline value of eye feature); For the mouth, the calculation formula is: Mouth Individualized Relative Index = (Current Mouth Feature Value / Mouth Feature Baseline Value) - 1, and when the result is less than 0, it is counted as 0.

5. The intelligent safety management system for industrial vehicles based on multimodal perception according to claim 4, characterized in that, The current eye feature value is the eye aspect ratio, and the current mouth feature value is the mouth aspect ratio. The formula for generating the comprehensive fatigue index is: Overall fatigue index = (individualized relative index of the eyes × W1) + (individualized relative index of the mouth × W2); Where W1 is the weight of the eye and W2 is the weight of the mouth, and W1 > W2.

6. The intelligent safety management system for industrial vehicles based on multimodal perception according to claim 5, characterized in that, The state determination model is a state machine, and its state transition conditions are based on the magnitude and duration of the comprehensive fatigue index. These conditions include: When the comprehensive fatigue index continues to exceed the first threshold for a first time period, it is judged as mild fatigue; When a person is in a state of mild fatigue, if the comprehensive fatigue index rises further and continues to exceed the second threshold for a second duration, it is judged as moderate fatigue. When a person is in a state of moderate fatigue and the system has issued an alarm, if the comprehensive fatigue index does not drop below the third threshold within the third time period, it is determined to be severe fatigue; the second threshold is greater than the first threshold, and the second time period is less than the first time period.

7. The intelligent safety management system for industrial vehicles based on multimodal perception according to claim 1, characterized in that, The pedestrian detection submodule is used to perform pedestrian detection on the panoramic view using a multi-scale detection model that fuses histogram of oriented gradients (HBP) features and deep learning features, and to calculate the three-dimensional spatial coordinates of the pedestrians relative to the vehicles based on multi-view geometry principles, including: Using an improved YOLOv5s model and an improved HOG feature descriptor, dynamic weighted fusion of extracted deep features and traditional features is performed on the feature layer to detect pedestrians; where the weighting coefficients are dynamically adjusted according to the estimated distance between the detected target and the vehicle. Based on images from at least two different perspectives from at least four environmental monitoring cameras, the three-dimensional spatial coordinates of the detected pedestrian are calculated using a stereo vision algorithm.

8. The intelligent safety management system for industrial vehicles based on multimodal perception according to claim 1, characterized in that, The three-dimensional dynamic safety zone model includes a reminder zone, a warning zone, and a danger zone centered on the vehicle and whose range can be dynamically adjusted; wherein, the adjustment parameters of the zone range include the vehicle's real-time speed, fork lifting height, and load information; The system calculates the distance between the center point of the human body's outline and the boundary of the forklift's safe zone to determine whether the human body has entered the warning zone. This includes: when the distance between the center point of the human body's outline and the boundary of the forklift's safe zone is less than a preset first distance threshold, it is determined that the human body has entered the danger zone, and a first warning instruction is generated; when the distance between the center point of the human body's outline and the boundary of the forklift's safe zone is greater than or equal to the preset first distance threshold and less than a preset second distance threshold, it is determined that the human body has entered the warning zone, and a second warning instruction is generated; when the distance between the center point of the human body's outline and the boundary of the forklift's safe zone is greater than or equal to the preset second distance threshold and less than a preset third distance threshold, it is determined that the human body has entered the alert zone, and a third warning instruction is generated.

9. The intelligent safety management system for industrial vehicles based on multimodal perception according to claim 8, characterized in that, The three-dimensional dynamic safety zone model dynamically determines the three-dimensional safety zone boundary based on the current operating state of the industrial vehicle, including: acquiring at least one of the vehicle's real-time speed, lifting height, and load weight; and scaling or extending the preset basic safety zone model in the corresponding direction based on the state parameters, wherein the higher the driving speed, the longer the forward safety zone extends; and the higher the lifting height, the larger the upper safety zone expands.

10. A method for intelligent safety management of industrial vehicles based on multimodal perception, the method being applied to the intelligent safety management system for industrial vehicles based on multimodal perception as described in any one of claims 1-9, characterized in that, The method includes: Step S1: Send a hardware synchronization trigger signal to the multi-camera array through the synchronization control box to control at least four environmental monitoring cameras distributed around the vehicle and driver monitoring cameras facing the cockpit to synchronously acquire images. Step S2: The image processing submodule stitches together the multiple images captured by the environmental monitoring camera in the YUV color space in real time to generate a 360-degree panoramic view. Step S3: The driver monitoring submodule performs driver authentication and fatigue detection based on the images captured by the driver monitoring camera; Step S4: The driver monitoring and early warning submodule generates a corresponding graded early warning instruction based on the driver's identity verification result and fatigue status detection result; Step S5: The pedestrian detection submodule uses a multi-scale detection model that integrates histogram of oriented gradients and deep learning features to detect pedestrians in the panoramic view, and calculates the three-dimensional spatial coordinates of the pedestrians relative to the vehicles based on the multi-view geometric principle. Step S6: The pedestrian detection and early warning submodule uses a spatial relationship judgment and early warning triggering algorithm to generate corresponding hierarchical early warning instructions in real time based on the spatial relationship between the three-dimensional spatial coordinates and the pre-built three-dimensional dynamic safety area model. Step S7: Obtain the real-time location information of the industrial vehicle through the positioning submodule, and receive the electronic fence model from the cloud management platform; compare the real-time location information with the electronic fence model to determine whether a boundary crossing event has occurred; when a boundary crossing event occurs, trigger a predetermined graded early warning instruction and upload the boundary crossing event data to the cloud management platform; wherein, the electronic fence model defines the geographical boundaries of the areas where passage is permitted or prohibited. Step S8: The vehicle control unit controls the vehicle to generate corresponding display and control information based on the corresponding graded warning commands sent by the pedestrian detection warning submodule, driver monitoring warning submodule and positioning submodule.

Images