Heavy truck battery compartment guiding method and system based on visual perception

Abstract

The invention provides a heavy truck battery compartment guiding method and system based on visual perception. The method and system are used for automatic battery replacement operation in a complex industrial environment. According to the system, a multi-camera fusion visual perception platform is constructed, a plurality of industrial cameras arranged on the ground or ceiling of a battery swap station are used for collecting local images of different visual angles of a battery compartment, and a complete visual field image is generated through feature matching and image splicing. A battery compartment is coarsely positioned by adopting a YOLO series model, a bounding box region is extracted, pixel-level contour segmentation is realized by introducing SAM, and the complex background and multi-interference environment recognition capability is enhanced. And after segmentation, calculating a minimum enclosing rectangle of the battery compartment, obtaining a center coordinate and a deviation angle, and transmitting a pose parameter to an upper computer control system. According to the method, the defects of the laser radar are avoided, image processing, the deep neural network and multi-view information are fused, the recognition precision and stability are improved, the battery replacement efficiency and the unmanned level of the electric heavy truck can be remarkably improved, and reliable support is provided for green traffic.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and industrial automation control technology, and in particular to a heavy truck battery compartment guidance method and system based on visual perception. Background Art

[0002] With the accelerated progress towards global carbon peak and carbon neutrality goals, the penetration of new energy vehicles in the transportation sector continues to rise. Electric heavy-duty trucks are particularly important in high-intensity operations such as long-distance logistics, port loading and unloading, and mining transport. Compared to passenger cars and light commercial vehicles, heavy-duty trucks have higher requirements for range and energy supply efficiency. Traditional charging methods, with their long charging times, heavy grid loads, and low dispatch efficiency, are no longer able to meet the high-frequency operation demands of heavy-duty trucks.

[0003] Battery swapping technology has emerged as a typical solution for decoupling charging and power consumption. By quickly replacing standardized battery packs on the bottom or side of the vehicle, recharging time can be shortened to 3-5 minutes, significantly improving vehicle availability. Since 2023, many OEMs and energy companies have deployed heavy-duty truck battery swapping stations across the country. The level of automation at these stations has become a key factor restricting their large-scale application.

[0004] In the battery swap process, accurate positioning and posture estimation of the battery compartment are the basis for ensuring the accuracy of the battery swap mechanism. The current mainstream technical routes are mainly divided into the following categories: 1. LiDAR + Point Cloud Matching Solution: This solution uses 3D LiDAR to obtain a point cloud of the battery compartment's shape and performs ICP registration with a pre-set model to determine its pose. This solution offers excellent 3D perception capabilities, but its high cost, sensitivity to environmental changes (such as metal reflections and interference from rain and fog), and complex deployment have limited its adoption at large sites.

[0005] 2. Manual guidance + simple image recognition solution: Some sites use operator handheld terminals or rely on fixed-angle industrial cameras, using traditional vision algorithms such as edge detection or template matching for auxiliary guidance. However, this method has poor adaptability to environmental interference such as lighting, reflections, and stains, requires frequent manual intervention, and has a low level of automation.

[0006] 3. Monocular Vision + Deep Learning Solution: In recent years, thanks to the popularity of object detection and segmentation models such as YOLO and Mask R-CNN, some systems have attempted to use a single camera to complete battery compartment recognition tasks. Although the detection results are significantly better than traditional methods, due to limited viewing angles, frequent occlusions, and the inability to accurately infer spatial posture, the overall accuracy and stability are still insufficient to support fully automatic battery swapping tasks.

[0007] With the decreasing cost of industrial cameras and the increasing computing power of edge AI, multi-camera fusion and deep visual perception have become key technical directions for the next generation of battery swap guidance systems. Deploying multiple cameras to build multi-view image input effectively overcomes single-view occlusion and blind spot issues. Combining Transformer-based segmentation models (such as SAM) with lightweight detection networks (such as YOLOv8) enables rapid coarse target positioning and fine boundary extraction. Combined with low-latency communication protocols and edge inference optimization, the entire system offers the following advantages: High precision: Segmentation-level edge perception and geometric modeling can achieve millimeter-level positioning accuracy; Low cost: Avoid expensive radar equipment, and the overall hardware investment of the system can be controlled within the 10,000 yuan level; High versatility: Adaptable to different vehicle models and site structures, with good engineering scalability; High robustness: Adapts to lighting changes, reflective interference, rainy and foggy environments to ensure stable operation.

[0008] Therefore, there is an urgent need for a comprehensive solution that integrates multi-camera vision acquisition, deep learning perception, geometric modeling and control linkage to achieve the precision, intelligence and engineering of the battery swapping system at the perception layer, and provide reliable support for the large-scale deployment of automatic battery swapping. Summary of the Invention

[0009] In response to the problems of existing technologies such as insufficient battery compartment positioning accuracy, poor environmental adaptability and high cost, the present invention proposes a heavy-duty truck battery compartment guidance method and system based on visual perception, which integrates multi-perspective image acquisition, deep learning recognition, geometric calculation and control system visualization operation to construct a closed-loop guidance process from perception to execution, with the characteristics of high precision, low cost and high robustness.

[0010] In order to solve the above problems, the technical solution adopted by the present invention is: A heavy truck battery compartment guidance method based on visual perception includes the following steps: S1. Multiple industrial cameras deployed around the battery swap station collect local images of the battery compartment at different angles. By deploying two to four high-resolution industrial cameras at battery swap stations, they simultaneously capture images of the battery compartment from multiple perspectives, addressing the blind spots and angle blind spots of a single camera. Image registration and stitching technologies are used to generate a global view, improving the integrity and accuracy of target detection and adapting to complex vehicle structures and scene occlusion conditions.

[0011] S2. Register and stitch the acquired multiple images to obtain a view image covering the entire battery compartment area; The image stitching and fusion process uses a matching algorithm based on ORB, SIFT or SURF feature points, or combines it with a deep learning feature pyramid matching network. After the image stitching is completed, the resulting image is subjected to image distortion correction, non-local mean denoising and contrast enhancement to optimize image quality.

[0012] S3. Use the YOLO series deep learning object detection model to infer the stitched image and obtain the rough positioning bounding box of the battery compartment; S4. Based on the above bounding box, use the Segment Anything Model to perform refined pixel-level segmentation and extract the outer contour of the battery compartment; A two-stage visual recognition process of "coarse positioning + fine segmentation" is proposed, which integrates the complementary functions of the YOLO and SAM deep learning models to effectively balance detection speed and segmentation accuracy, significantly improving positioning capabilities in complex industrial scenarios. This architecture has the following technical advantages and implementation details: YOLO is used for rapid and coarse object localization: The YOLO (You Only Look Once) model series is an efficient, one-stage object detection framework with advantages such as end-to-end modeling, fast inference speed, and a small number of parameters. This paper uses the YOLOv8s model as the first-stage detector, deployed on the RK3588 edge computing platform. YOLO extracts multi-scale semantic information from the image using a feature pyramid structure and outputs a bounding box (BoundingBox) approximate to the battery compartment. Detection confidence levels greater than 0.85 are considered valid. This stage achieves real-time inference response times of less than 50ms, ensuring the overall perception efficiency of the system.

[0013] SAM achieves pixel-level fine segmentation: The candidate regions obtained in the detection phase are fed into the second-stage SAM (SegmentAnythingModel) model for image segmentation. SAM is a general-purpose large-scale model capable of "zero-shot prompt segmentation." It utilizes the ViT-H backbone network and combines point prompt and box prompt mechanisms to extract high-quality boundary masks without the need for detailed annotation. In this system, the bounding box output by YOLO is used as the prompt region input to SAM, and an automatic point sampling strategy is enabled to assist in enhancing segmentation edges. The output mask is a single-channel binary image with high contour accuracy, suitable for interference suppression in complex backgrounds.

[0014] A two-stage, coarse-to-fine approach: This dual-stage architecture utilizes a "coarse screening + fine extraction" process. In the first stage, YOLO rapidly locates targets, narrowing the search range and avoiding the resource waste associated with full-image segmentation. In the second stage, SAM accurately identifies contours within candidate regions, addressing YOLO's shortcomings in handling boundary details and occlusions. These two modules collaborate efficiently through inter-module intermediate result caching and asynchronous thread scheduling, ensuring smooth processing and module independence.

[0015] Enhanced robustness to occlusion and complex backgrounds: Traditional detection models are prone to false positives or missed detections under interference conditions common in heavy truck battery compartments, such as oil contamination, bolt occlusion, and uneven lighting. This two-stage architecture mitigates this problem through a collaborative approach: YOLO eliminates false positives due to large background interference, while SAM enhances local edge extraction through a multi-layer visual attention mechanism. Experiments demonstrate that even with the addition of occluders and reflectors, the IoU of the segmentation mask remains above 0.85, significantly outperforming solutions using only a single model.

[0016] Model lightweighting and edge-optimized deployment: Both the YOLOv8s and SAM models are exported in the ONNX format and inference is accelerated by the NPU on the RK3588 platform. The YOLO model undergoes quantization optimization, reducing weight precision to FP16. The SAM model uses a layered loading mechanism, performing local inference only on candidate regions. The overall dual-model joint inference time is kept within 120ms, meeting the real-time requirements of industrial applications.

[0017] Scalable design: This architecture supports modular replacement and custom model combination. Users can replace YOLO with advanced detectors such as RT-DETR and YOLO-World according to actual application scenarios, or replace SAM with multi-task segmentation models such as Mask2Former, thereby expanding the application boundaries to higher-dimensional semantic recognition, key point localization and other tasks.

[0018] S5. Based on the segmented outline, a geometric calculation method is used to fit the minimum circumscribed rectangle, and then the center coordinates and relative angle offset values ​​of the rectangle are extracted; Minimum bounding rectangle calculation includes: The center point of the fitted rectangle is used as the two-dimensional position coordinate of the battery compartment in the battery swapping work plane; The angle between the long side of the rectangle and the horizontal axis of the image coordinate system is used as the battery compartment attitude offset angle for subsequent motion compensation and trajectory planning.

[0019] By fitting the minimum circumscribed rectangle of the segmentation result contour point set, key parameters such as the rectangle center coordinates, length, width, and rotation angle are extracted. A coordinate transformation algorithm is used to convert the posture information in the image coordinate system into the battery swapping device control coordinate system, facilitating the mechanical system's motion execution and feedback control.

[0020] S6. The positioning parameters obtained above are transmitted to the upper computer control system in real time through the control protocol to realize the visualization and action execution of the battery replacement guidance process.

[0021] Visual guidance includes the following functional modules: Render the deviation information between the current position of the battery compartment and the target position in real time on the user control interface; Dynamically draw direction arrows to show movement trends, and overlay offset value prompts on the interface (such as ); The positioning data is synchronized to the execution control end through TCP / IP, ROS2 or CAN bus communication protocols to achieve real-time collaboration between multiple modules.

[0022] The present invention also provides a heavy truck battery compartment positioning system based on visual perception, which includes the following modules: a multi-camera image acquisition module, an image processing module, and a control end visualization module.

[0023] The multi-camera image acquisition module is used to be arranged in the working area of ​​the battery swap station to form a full-coverage monitoring angle of the battery compartment and acquire image data in real time; The image processing module includes stitching, target detection, segmentation and geometric calculation submodules, which are used to implement the image analysis and positioning functions of the method; The control-end visualization module integrates display, interaction and communication functions, supports linkage with execution equipment such as robotic arms and lifting platforms, and realizes real-time indication of battery compartment positioning information and guidance of battery replacement tasks.

[0024] Real-time visual guidance and multi-protocol communication: The control software interface integrates functional modules such as position deviation visualization, direction indication, and fault-tolerant feedback. It supports data transmission based on protocols such as ROS2, TCP / IP, and CAN bus, enabling communication and linkage with AGV chassis, robotic arms, lifting platforms, and other equipment. It offers excellent system scalability and industrial interface compatibility.

[0025] The system leverages the Rockchip RK3588 platform to implement localized deployment and inference optimization for YOLO and SAM models, leveraging the NPU to accelerate model inference, with single-frame processing latency below 150ms. This eliminates the need for high-performance GPU servers, reducing deployment costs and adapting to edge deployment and outdoor operation requirements.

[0026] Complex Environment Adaptation: To address adverse operating conditions such as rain, fog, strong reflections, and dust, the system introduces enhanced features such as HDR fusion imaging, adaptive filtering, and polarization filtering. The system also supports battery-swap vehicle identification and dynamic loading of model parameters, improving adaptability to different vehicle models and site structures, enhancing versatility and deployment efficiency.

[0027] The beneficial effects of the present invention are: 1. The present invention integrates multi-perspective image acquisition, deep learning recognition, geometric calculation and control system visualization operation to construct a closed-loop guidance process from perception to execution, with the characteristics of high precision, low cost and high robustness; it proposes systematic technical solutions from multiple levels such as visual acquisition, intelligent recognition, data fusion, parameter analysis, and interactive control, effectively solving the shortcomings of existing battery swap positioning methods in terms of accuracy, speed, cost and environmental adaptability, and has broad engineering application prospects and promotion value.

[0028] 2. Compared to traditional methods that rely on lidar or monocular vision, this invention offers significant advantages: it avoids the drawbacks of lidar, such as its high cost and sensitivity to harsh environments; and it improves recognition accuracy and stability by integrating image processing, deep neural networks, and multi-view information. This method is suitable for automated battery swapping in complex industrial environments, significantly improving the efficiency and unmanned operation of electric heavy-duty trucks, and providing reliable support for green transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is the overall flow chart of the system of the present invention. DETAILED DESCRIPTION

[0030] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0031] Reference Figure 1 A heavy truck battery compartment guidance method based on visual perception includes the following steps and modules: 1. Multi-camera image acquisition: Four high-resolution industrial cameras are deployed at the battery swap station to capture images of the battery compartment area under heavy trucks from multiple perspectives, ensuring there are no blind spots or obstructions. The images are then sent to downstream processing modules through a synchronous acquisition mechanism.

[0032] 2. Image stitching and preprocessing: The collected multi-view images are stitched and fused using the ORB feature point matching algorithm to generate a panoramic image of the complete battery compartment field of view. After stitching, image enhancement processes such as radial distortion correction, non-local mean denoising, and histogram equalization are performed to provide high-quality input for subsequent perception modules.

[0033] 3. Battery Compartment Detection (YOLO Module): The stitched image is input into the YOLOv8s model for rapid object detection, obtaining a rough bounding box for the battery compartment. This stage primarily locates the candidate area and preliminarily eliminates background interference.

[0034] 4. Battery Compartment Segmentation (SAM Module): The bounding box region output by YOLO is fed into the SAM model for pixel-level segmentation. Based on the ViT architecture, the SAM model combines the hint box to generate a precise mask outline, enabling precise extraction of the battery compartment boundary.

[0035] 5. Parameter extraction and coordinate conversion: Based on the segmentation results, OpenCV is used to fit the minimum bounding rectangle and extract the center point coordinates and rotation angle. Simultaneously, the perspective transformation matrix obtained through camera calibration is used to map the image coordinates to the battery swapping device's operating coordinate system, achieving spatial unification of positioning parameters.

[0036] 6. Visual guidance and control interaction: The calculated positioning parameters are sent to the control terminal via ROS2 or TCP / IP protocol. In the control terminal interface, the deviation between the battery compartment and the target position is displayed in real time ( ), with arrows indicating the direction of deviation. The interface supports color-coded feedback, data logging, and historical trajectory query, assisting in stable system operation.

[0037] 7. System linkage and control response: The positioning results are used to guide the displacement adjustment operations of the battery swapping mechanical platform, such as the battery lifting mechanism and the vehicle slide rail moving platform, to form a closed-loop control process and achieve precise docking between the battery compartment and the actuator during the battery swapping process.

[0038] Figure 1 It embodies the data flow, control flow and information interaction mode between the perception-processing-guidance-control modules of the present invention, and has the characteristics of clear structure, closed path, and strong engineering feasibility. It is the core logical skeleton of the entire battery swap guidance system.

[0039] Combine Figure 1 The following describes the specific implementation of the present invention in detail, covering the system architecture, algorithm flow, software and hardware deployment, and edge inference optimization strategy: 1. System hardware deployment: Camera layout: Four industrial-grade high-resolution cameras (40 megapixel resolution, 30fps) are installed above or on the left and right sides of the battery swap station. A fisheye lens + wide-angle lens combination layout strategy is used to achieve visual coverage of key areas in front, behind, left, and right of the battery compartment, avoiding obstructions and blind spots.

[0040] Edge Computing Platform: The Rockchip RK3588 embedded edge computing board is used as the core processing unit. The platform features an 8-core 64-bit ARM CPU (4 x Cortex-A76 + 4 x Cortex-A55), 6TOPS NPU computing power, supports GPU / NPU heterogeneous parallel computing, and includes 8GB of DDR memory and 64GB of eMMC. It runs Ubuntu 20.04 and comes pre-installed with OpenCV, PyTorch, ONNX Runtime, and other dependencies.

[0041] Transmission and power supply: The industrial camera is connected to the RK3588 via the Gigabit Ethernet port. The system power supply is centrally managed by a 24V industrial power supply with overvoltage and undervoltage protection mechanisms.

[0042] 2. Image processing and model inference process: Image stitching module: Partial images from different cameras are fed into the module. First, an affine transformation matrix is ​​obtained using ORB feature point detection, brute force matching, and RANSAC filtering. The images are then geometrically registered and stitched together. The stitched image resolution must be at least 8192×4096 to ensure the complete battery compartment structure is captured.

[0043] Image preprocessing: To improve the subsequent model inference effect, a series of preprocessing operations are performed on the stitched image, including: Distortion correction: Perform radial distortion correction based on the intrinsic parameter matrix and distortion coefficient; Adaptive Histogram Equalization (CLAHE): Enhances local contrast of images; Gaussian filtering and non-local means denoising (NLM, h=3): suppress high-frequency noise and improve edge stability.

[0044] Object detection: The YOLOv8s model is called to perform inference on the preprocessed image, detecting the bounding box of the battery compartment and outputting information including confidence, coordinates, and dimensions. The model uses an anchor-free mechanism with a confidence threshold of 0.85 and an IoU threshold of 0.5 to suppress redundant boxes.

[0045] Fine segmentation: Based on the YOLO detection results, candidate regions are cropped and fed into the SAM (SegmentAnythingModel) model for segmentation. This model uses the ViT-H backbone network, has an input size of 1024×1024, and supports segmentation guided by both prompt boxes and point prompts. The output is a binary mask image that reflects the battery compartment outline.

[0046] 3. Geometric calculation and coordinate transformation: Minimum bounding rectangle: Call the OpenCV function minAreaRect() on the mask contour point set to fit a rotated rectangle, and record parameters such as the center point (xc, yc), the direction of the long side θ, and the side length (w, h) as the two-dimensional plane positioning information of the battery compartment.

[0047] Coordinate system conversion: The perspective matrix H obtained by Zhang Zhengyou calibration method is used to convert the image coordinate system to the workstation coordinate system, realizing the mapping from pixels to actual physical units (mm). The conversion formula is as follows:

[0048] The experimental calibration error is controlled within 0.1mm.

[0049] 3D pose estimation: When multi-view images or additional markers such as AprilTag are available, the EPnP algorithm can be used to further estimate the target's 6DOF pose and obtain the complete spatial pose (x, y, z, α, β, γ).

[0050] 4. Control terminal and guidance visualization: Interface layout: The control end builds a graphical interface based on Qt or ROS2RViz, which displays the deviation of the battery compartment position from the expected position in real time, including horizontal (∆x), vertical (∆y) and rotation angle deviation (∆θ), and draws guide arrows and prompt text.

[0051] Communication module: Guide parameters are sent to the battery swap platform execution controller via DDS middleware or CAN bus protocol. Communication latency is less than 20ms, and heartbeat and data verification mechanisms are supported to ensure data real-time and security.

[0052] Status Indication and Fault Tolerance: The interface uses color coding to indicate status: green indicates alignment, yellow indicates moderate deviation, and red indicates correction is required. If the deviation exceeds the threshold for three consecutive frames, the system enters re-identification mode, prompting manual confirmation or automatic retry.

[0053] 5. System optimization measures: Coping with harsh environments: Use HDR imaging in rainy and foggy weather: capture images of different exposures and synthesize high dynamic range images; Light reflection interference: The camera is equipped with a polarizing filter, combined with the reflective area discrimination model to block high-contrast interference; When using infrared fill light at night, the SAM model can adapt to IR images.

[0054] Dynamic model loading: During system startup, the ResNet18 network is first used to identify the truck model. Based on the model results, the corresponding YOLO+SAM weight combination is dynamically loaded to improve recognition accuracy and versatility.

[0055] Motion filter enhancement: For moving trucks or critical frame errors, the Extended Kalman Filter (EKF) is introduced for state estimation and observation fusion. The state variables include position, velocity, and angular velocity: X=[x,y,θ,vx,vy,ω]T The system updates predictions and observations every frame, significantly suppressing inter-frame jitter and improving guidance stability.

[0056] Performance indicators: The end-to-end processing delay of the entire process is less than 150ms; Detection accuracy (mAP@0.5) 0.92; Segmentation contour IoU 0.87; Positioning error ±5mm, angle error ±1° The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A heavy truck battery compartment guidance method based on visual perception, characterized in that: The following steps are involved: S1. Multiple industrial cameras deployed around the battery swap station collect local images of the battery compartment at different angles. S2. Register and stitch the acquired multiple images to obtain a view image covering the entire battery compartment area; S3. Use the YOLO series deep learning object detection model to infer the stitched image and obtain the rough positioning bounding box of the battery compartment; S4. Based on the above bounding box, use the Segment Anything Model to perform refined pixel-level segmentation and extract the outer contour of the battery compartment; S5. Based on the segmented outline, a geometric calculation method is used to fit the minimum circumscribed rectangle, and then the center coordinates and relative angle offset values ​​of the rectangle are extracted; S6. The positioning parameters obtained above are transmitted to the upper computer control system in real time through the control protocol to realize the visualization and action execution of the battery replacement guidance process.

2. A heavy truck battery compartment guidance method based on visual perception according to claim 1, characterized in that: The image stitching and fusion process in S2 adopts a matching algorithm based on ORB, SIFT or SURF feature points, or is combined with a deep learning feature pyramid matching network. After the image stitching is completed, image distortion correction, non-local mean denoising and contrast enhancement are performed on the result image to optimize the image quality.

3. The heavy truck battery compartment guidance method based on visual perception according to claim 1 is characterized in that: The minimum bounding rectangle calculation in S5 includes: The center point of the fitted rectangle is used as the two-dimensional position coordinate of the battery compartment in the battery swapping work plane; The angle between the long side of the rectangle and the horizontal axis of the image coordinate system is used as the battery compartment attitude offset angle for subsequent motion compensation and trajectory planning.

4. The heavy truck battery compartment guidance method based on visual perception according to claim 1 is characterized in that: The visual guidance in S6 includes the following functional modules: Render the deviation information between the current position of the battery compartment and the target position in real time on the user control interface; Dynamically draw direction arrows to show movement trends, and overlay offset value prompts on the interface; The positioning data is synchronized to the execution control end through TCP / IP, ROS2 or CAN bus communication protocols to achieve real-time collaboration between multiple modules.

5. A heavy truck battery compartment positioning system based on visual perception, characterized in that: It includes the following modules: multi-camera image acquisition module, image processing module, and control end visualization module.

6. A heavy truck battery compartment positioning system based on visual perception according to claim 5, characterized in that: The multi-camera image acquisition module is used to be arranged in the working area of ​​the battery swap station to form a full-coverage monitoring angle of the battery compartment and acquire image data in real time; The image processing module includes stitching, target detection, segmentation and geometric calculation submodules, which are used to implement the image analysis and positioning functions of the method according to any one of claims 1 to 4; The control-end visualization module integrates display, interaction and communication functions, supports linkage with execution equipment such as robotic arms and lifting platforms, and realizes real-time indication of battery compartment positioning information and guidance of battery replacement tasks.

Images