Intelligent training and examination system for carry-scraper truck based on multi-modal perception

Abstract

The invention discloses an intelligent forklift training and checking system based on multi-modal perception. The system comprises a multi-source image acquisition module, a data preprocessing module, a multi-modal perception module, an area and state judgment module, an intelligent decision engine module and a visualization and communication module. Through the multi-mode perception and intelligent decision technology, the objectivity of evaluation is improved, the artificial subjective deviation is eliminated, the full-scene coverage capability is enhanced, the real-time performance of the system meets the practical operation requirement, multi-thread parallel processing supports synchronous analysis of data of three paths of cameras, the target detection accuracy is high, when a single camera fails, the system is automatically switched to a degradation mode, and the system is convenient to operate. The assessment continuation rate is 100%, the efficiency is improved by 5 times compared with a manual mode, the safety accident risk is reduced, a quantitative assessment report is generated, and the skill improvement period of trainees is shortened.

Description

Technical Field

[0001] This invention relates to the field of practical assessment technology for special operations of construction machinery, and in particular to an intelligent training and assessment system for loader trucks based on multimodal perception. Background Technology

[0002] Current practical assessments for loader operators suffer from several problems, including high subjectivity, limited coverage, low efficiency, safety risks, and lack of data traceability. Specifically: 1. Assessment results are influenced by the examiner's experience and fatigue level, potentially leading to inconsistent scores for the same operation, making fairness difficult to guarantee. 2. It is difficult for humans to simultaneously monitor multiple dimensions of information, including driver operation, vehicle status, and the surrounding environment, making it prone to omissions. 3. Each assessment requires 1-2 examiners throughout, and the daily assessment volume is typically limited to no more than 10 people, making it difficult to meet the needs of large-scale training. 4. Examiners need to observe the operation process closely, facing potential dangers such as vehicle misoperation and environmental hazards. 5. The lack of quantitative records in the assessment process prevents trainees from conducting targeted review and improvement, resulting in slow improvement in training effectiveness.

[0003] While existing intelligent assessment technologies are in use, they have significant shortcomings: 1. They mostly rely on single cameras to collect data, making it difficult to cover all scenarios inside and outside the vehicle. 2. They depend on single-target detection models, resulting in insufficient accuracy in recognizing complex operations. 3. They lack dynamic process judgment mechanisms, making it impossible to differentiate assessment standards for pre-, during, and post-operation stages. 4. They have poor real-time performance, with inference latency often exceeding 500ms, failing to meet the real-time evaluation needs during practical operations.

[0004] Therefore, there is an urgent need for a loader assessment system that integrates multi-source sensing, intelligent decision-making, and real-time processing to achieve full-process, objective, and high-precision automated evaluation. Summary of the Invention

[0005] To address the problems mentioned in the background art, the present invention provides an intelligent training and assessment system for loader trucks based on multimodal perception.

[0006] The specific implementation of the present invention is as follows: an intelligent training and assessment system for loader trucks based on multimodal perception, including a multi-source image acquisition module, a data preprocessing module, a multimodal perception module, a region and state judgment module, an intelligent decision engine module, and a visualization and communication module.

[0007] Furthermore, the multi-source image acquisition module includes three sets of industrial-grade cameras: a camera on the left side of the vehicle exterior to capture the bucket's movement, a camera on the rear side of the vehicle exterior to capture the vehicle's position and surrounding environment, and a camera inside the vehicle to capture the driver's operation and equipment status. The camera synchronous control is achieved using an SDK interface, supporting a resolution of 1920×1080@30fps, and adapting to complex lighting conditions such as strong light and shadows through automatic exposure and white balance adjustment.

[0008] Furthermore, the data preprocessing module performs denoising (Gaussian filtering), distortion correction (based on camera intrinsic parameter matrix), and scale normalization (uniform resizing to 640×640) on the acquired images. It aligns multi-source data through timestamps and uses a thread-safe queue to decouple the acquisition thread from the processing thread, avoiding data blocking.

[0009] Furthermore, the multimodal perception module uses the YOLOv8 model to detect external targets such as loaders and pedestrians; it extracts the bucket mask using the YOLOv8-seg segmentation model and calculates the rotation angle of the minimum bounding rectangle to estimate the bucket attitude; it extracts human key points based on the attitude estimation model YOLOv8-pose to determine head turning and hand raising behaviors for driver behavior recognition; and it uses a classification model to identify the status of dashboard indicator lights and vehicle start button, thereby achieving equipment status recognition.

[0010] Furthermore, the intelligent decision engine module includes the construction of a rule base and the preset of assessment indicators for each stage, such as the requirement to complete "walking around the vehicle" (judgment criteria: human trajectory covers 4 key areas around the vehicle) and "fire extinguisher check" (staying time ≥ 3 seconds) before the operation; a real-time scoring mechanism is adopted, and a deduction system is used to record and calculate the total score for incomplete items (such as failure to avoid pedestrians) in real time; in the case of abnormal camera network disconnection, that is, the camera has no data for 3 consecutive frames, triggering the degradation mode, and only retaining the available data source for evaluation.

[0011] Furthermore, the visualization and communication module includes visualization output, i.e., labeling the detection results in the image and displaying them in real time through a multi-threaded window; and data interaction, establishing client-server communication based on the Socket protocol and sending the assessment status in real time.

[0012] Furthermore, the YOLOv8 model can be replaced by the YOLOv8-nano model, which improves inference speed by 50% and is suitable for edge devices. When calculating the bucket posture, for unobstructed buckets, Hough transform can be used to extract edge lines to calculate angles, reducing model dependence. This system also supports 4G modules to realize remote data transmission, thus adapting to scenarios without wireless networks.

[0013] This invention, through multimodal perception and intelligent decision-making technology, firstly significantly improves the objectivity of the assessment, eliminating subjective bias and achieving 100% consistency in judgments of key operations. Quantitative indicators cover 100% of the assessment items, reducing the false negative rate by over 85% compared to manual scoring. Secondly, it enhances full-scene coverage, simultaneously monitoring 12 types of key targets inside and outside the vehicle (such as the bucket, driver, and pedestrians), expanding the coverage area by three times compared to single-camera solutions. It supports automatic switching between three operational stages, with an accuracy recognition rate ≥95%. Thirdly, the system's real-time performance meets practical operational requirements, with end-to-end processing latency ≤80ms, an improvement over existing technologies (average 300ms). With a 73% success rate, multi-threaded parallel processing supports simultaneous analysis of data from three cameras, maintaining a stable frame rate of over 25fps. Furthermore, it exhibits robust adaptability to complex environments, maintaining target detection accuracy above 90% even under varying lighting conditions (0-10000 lux) and dust interference. In the event of a single camera failure, the system automatically switches to a degraded mode, ensuring a 100% assessment continuation rate. Finally, in terms of economic and social benefits, a single system can assess an average of 50 people per day, increasing efficiency fivefold compared to manual methods, reducing examiner exposure time, lowering the risk of safety incidents by 90%, generating quantitative evaluation reports, and shortening the skill improvement cycle for trainees by 40%. Attached Figure Description

[0014] Figure 1 This is a flowchart of the overall system of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0016] Figure 1 The above is a flowchart of the overall system of the present invention. The system includes a multi-source image acquisition module, a data preprocessing module, a multimodal perception module, a region and state judgment module, an intelligent decision engine module, and a visualization and communication module.

[0017] The multi-source image acquisition module includes three sets of industrial-grade cameras: a camera on the left side of the vehicle exterior captures the bucket's movement, a camera on the rear side captures the vehicle's position and surrounding environment, and an in-vehicle camera captures driver operations and equipment status. An SDK interface is used to achieve synchronous camera control, supporting a resolution of 1920×1080@30fps. Automatic exposure and white balance adjustment adapt to complex lighting conditions such as strong light and shadows. The data preprocessing module performs noise reduction (Gaussian filtering), distortion correction (based on the camera intrinsic parameter matrix), and scale normalization (uniform resizing to 640×640) on the acquired images. Multi-source data is aligned using timestamps, and a thread-safe queue (Queue) is used to decouple the acquisition thread from the processing thread, avoiding data blocking. The multimodal perception module uses the YOLOv8 model to detect external targets such as loaders and pedestrians; it extracts the bucket mask using the YOLOv8-seg segmentation model and calculates the rotation angle of the minimum bounding rectangle to estimate the bucket attitude; it extracts human key points based on the YOLOv8-pose attitude estimation model to determine head turning and hand raising behaviors for driver behavior recognition; and it uses a classification model to identify the status of dashboard indicator lights and vehicle start button, thereby achieving equipment status recognition. The intelligent decision engine module includes the construction of a rule base and the pre-setting of assessment indicators for each stage, such as completing "walking around the vehicle" (judgment criteria: human trajectory covers 4 key areas around the vehicle) and "fire extinguisher check" (dwelling time ≥ 3 seconds) before operation; it adopts a real-time scoring mechanism, using a deduction system to record and calculate the total score for incomplete items (such as failure to avoid pedestrians); in the event of a camera network outage, i.e., the camera has no data for 3 consecutive frames, a degradation mode is triggered, retaining only the available data source for evaluation. The visualization and communication module includes visualization output, which involves annotating detection results in images and displaying them in real time through a multi-threaded window; and data interaction, which establishes client-server communication based on the Socket protocol to send assessment status in real time. The YOLOv8 model can be replaced by the YOLOv8-nano model, improving inference speed by 50% and making it suitable for edge devices. When calculating bucket posture, for unobstructed buckets, Hough transform can be used to extract edge lines and calculate angles, reducing model dependency. This system also supports a 4G module for remote data transmission to adapt to scenarios without wireless networks.

[0018] Specifically, the system is deployed in an indoor simulated testing area, equipped with three 2-megapixel industrial cameras, installed on the left side of the vehicle 2m above the ground, the rear side of the vehicle 3m above the ground, and the rearview mirror position inside the driver's cab. The edge computing terminal uses an NVIDIA Jetson AGX Xavier (16GB of video memory), and the client is a monitoring host running a Windows system. The system software is developed using Python 3.8 and relies on libraries such as OpenCV (image processing), Ultralytics (YOLO model), Loguru (logging), and Socket (communication). Each module achieves low coupling through modular design.

[0019] Specific implementation steps: S1. Device initialization and parameter configuration: Load the global configuration file (YAML format), including camera IP, model path (e.g., . / models / yolov8x.pt), region coordinates (e.g., area A Polygon: [(100,200), (300,200),...]), and threshold parameters (e.g., stillness detection threshold of 5 pixels); Initialize the camera: Establish a connection through CameraSDK, set the resolution to 1920×1080, frame rate to 30fps, and enable automatic exposure; Start auxiliary threads: Set the display thread (display_frames) and Socket server thread (run_server) as daemon threads.

[0020] S2. Multi-source image acquisition and preprocessing: Three acquisition threads are started to read images from the camera and store them in frame_queue_get (maximum length 10) using the read_frames_deque function. The preprocessing thread retrieves images from the queue and performs the following: 1. Distortion correction: Correcting fisheye effect using the camera intrinsic parameter matrix (pre-acquired using Zhang Zhengyou calibration method); 2. Noise reduction and enhancement: Using bilateral filtering to remove salt-and-pepper noise and histogram equalization to improve contrast; 3. Scale adjustment: Resize to 640×640 pixels and store in the processing queue.

[0021] S3. Multimodal model inference: The external detection thread loads the YOLOv8x model, infers the image from the rear camera, outputs the loader's bounding box (x1, y1, x2, y2), calculates the center point (cx, cy), and records the timestamp of a pedestrian entering the work region when detecting pedestrians; the bucket recognition thread loads YOLOv8-seg... The model processes the image from the left-side camera, extracts the bucket mask, and calculates the minimum bounding rectangle angle θ (range -90°~90°). If the change in θ is less than 5° for 10 consecutive frames, it is considered stationary. The retraction / tilting state is determined by comparing θ with a threshold (e.g., 30° for area A). The in-vehicle recognition thread loads the YOLOv8-pose model, analyzes the image inside the driver's cab, extracts the head key points (point 0) and shoulder key points (points 5 and 6), and calculates the head deflection angle α=arctan[(y5-y6) / (x5-x6)]. When α>30° and lasts for 2 frames, it is determined to be a head turning action. When the y-coordinate of the wrist point (points 9 and 10) is less than the y-coordinate of the shoulder -50 pixels, it is determined to be a raised hand.

[0022] S4. Region and Status Judgment: Vehicle Position Judgment: Extract the coordinates (x, y) of the right front wheel of the loader from the rear camera detection results, and determine whether the point is located within the Polygon of area A (using the contains method of the shapely library). If so, set InA=True. Motion Status Judgment: Store the coordinates of the right front wheel for 20 consecutive frames into the _deque queue, calculate the average distance between frames, and if it is <5 pixels and lasts for 1 second, it is judged as stationary (forklift_stop=True). Work Stage Switching: The initial state is "before work" (work_status="before"). When the indicator light inside the vehicle is detected to be on and the vehicle leaves area A (InA=False), switch to "in work". When the vehicle returns to area A (InA=True) and stops, switch to "after work".

[0023] S5. Assessment and scoring implementation: The rule base is loaded, and assessment items are read from the configuration file, such as "checking the fire extinguisher" (weight 10 points) and "walking around the vehicle" (15 points) before the operation. Real-time scoring: Before the operation, if the driver enters the fire extinguisher area (polygon) and stays for ≥3 seconds, 10 points are awarded; otherwise, 10 points are deducted. During the operation, if a pedestrian is detected approaching and the vehicle does not stop (driving=True), 20 points are deducted. After the operation, if the vehicle stops in area A and the bucket is placed on the ground, 15 points are awarded. Anomaly handling: If camera 1 loses network connection, the system log records a warning, stops bucket-related scoring, and retains other item evaluations.

[0024] S6. Visualization and Communication: Visualization output: Mark pedestrians with red rectangles and area A with green polygons on the original image, and display the current status with superimposed text (e.g., "Homework in progress | Score: 80"). Display the image through cv2.imshow, with the frame rate controlled at 25fps. Data communication: The Socket server is bound to port 8888, receives the client's "Start Exam" command (

[201] ), and sends a status_send array (e.g., [200,1,99,1,...]) to the client every 500ms, containing the scores for each item.

[0025] S7. Optimization measures: Inference acceleration: The model is converted to TensorRT format, reducing YOLOv8 inference time from 80ms to 25ms. FP16 precision is enabled, and memory usage is reduced by 50%. Multi-threading optimization: Image acquisition, inference, and decision-making run on independent threads. Data is passed through a queue, and a lock mechanism is used to protect shared variables (such as status_send) to avoid race conditions.

[0026] In addition, a lightweight model can be used to replace YOLOv8 with YOLOv8-nano on terminals with limited computing power (such as Jetson Nano), improving the inference speed to 15ms and reducing the accuracy by ≤3%. Alternatively, traditional algorithms can be used to assist in the process. For scenes with stable lighting, Canny edge detection + Hough line extraction can be used to extract the bucket edge, replacing the segmentation model and reducing the amount of computation.

[0027] The above method enables fully automated evaluation of the practical assessment of loaders. In actual operation, the assessment accuracy rate reaches 98.7%, with an average delay of 72ms, which fully meets the needs of practical applications.

[0028] The above description is illustrative of the present invention and its embodiments, and is not restrictive. The accompanying drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs shall fall within the protection scope of the present invention.

Claims

1. A multimodal perception-based intelligent training and assessment system for loader trucks, characterized in that: It includes a multi-source image acquisition module, a data preprocessing module, a multimodal perception module, a region and state judgment module, an intelligent decision engine module, and a visualization and communication module.

2. The intelligent training and assessment system for loader trucks based on multimodal perception according to claim 1, characterized in that: The multi-source image acquisition module includes three sets of industrial-grade cameras, which respectively acquire image data from the left side of the vehicle exterior (bucket movement), the rear side of the vehicle exterior (vehicle position and surrounding environment), and the interior of the vehicle (driver operation and equipment status).

3. The intelligent training and assessment system for loader trucks based on multimodal perception according to claim 1, characterized in that: The data preprocessing module performs noise reduction, distortion correction, and scale normalization on the acquired images, and aligns multi-source data using timestamps.

4. The intelligent training and assessment system for loader trucks based on multimodal perception according to claim 1, characterized in that: The multimodal perception module uses the YOLOv8 model to detect external targets such as loaders and pedestrians; it extracts the bucket mask through the YOLOv8-seg segmentation model and calculates the rotation angle of the minimum bounding rectangle to estimate the bucket attitude. Based on the pose estimation model YOLOv8-pose, key points of the human body are extracted to determine head turning and hand raising behaviors, and driver behavior recognition is performed. A classification model is used to identify the status of dashboard indicator lights and vehicle start button, thereby enabling equipment status recognition.

5. The intelligent training and assessment system for loader trucks based on multimodal perception according to claim 1, characterized in that: The intelligent decision engine module includes the construction of a rule base and the preset of assessment indicators for each stage; it adopts a real-time scoring mechanism, uses a deduction system to record incomplete items in real time and calculate the total score; in case of camera network outage, it triggers a degradation mode, retaining only available data sources for evaluation.

6. The intelligent training and assessment system for loader trucks based on multimodal perception according to claim 5, characterized in that: The abnormal situation is that the camera has no data for 3 consecutive frames.

7. The intelligent training and assessment system for loader trucks based on multimodal perception according to claim 1, characterized in that: The visualization and communication module includes visualization output, which involves annotating the detection results in the image and displaying them in real time through a multi-threaded window; and data interaction, which establishes client-server communication based on the Socket protocol and sends the assessment status in real time.

8. The intelligent training and assessment system for loader trucks based on multimodal perception according to claim 4, characterized in that: The YOLOv8 model can be replaced by the YOLOv8-nano model. When calculating the bucket attitude, for unobstructed buckets, Hough transform can be used to extract edge lines to calculate angles, reducing model dependence.

Images