Multimodal data fusion battery swap guidance method, system, storage medium and battery swap station

Abstract

The present invention discloses a multimodal data fusion battery swap guidance method, system, storage medium, and battery swap station, belonging to the field of new energy vehicle battery swap technology, including: obtaining the battery pack's position information and the battery pack's height from the ground, calculating the battery swap guide frame pixel coordinates corresponding to the current battery pack height; identifying the battery pack to obtain the battery pack's detection rectangular frame pixel coordinates; then comparing and matching the battery swap guide frame pixel coordinates with the detection rectangular frame pixel coordinates. If the battery pack's detection rectangular frame is completely within the battery swap guide frame, then obtaining the battery pack coordinates, converting the battery pack coordinates to the battery pack coordinates based on the battery swap robot coordinate system to form a complete battery pack position information data packet; and controlling the battery swap robot to complete the battery swap operation based on the battery pack position information data packet. Through multimodal data fusion, millimeter-level precise positioning of the battery pack in three-dimensional space is achieved, significantly improving the battery swap robot's docking success rate.

Description

Technical Field

[0001] The present invention relates to the field of new energy vehicle battery replacement technology, and in particular to a multimodal data fusion battery replacement guidance method, system, storage medium and battery replacement station. Background Art

[0002] With the increasing popularity of new energy vehicles, battery swapping has attracted increasing attention as a fast and efficient way to replenish energy. This model offers numerous advantages, including rapid battery replacement (the entire process takes only 3-5 minutes), significantly increasing vehicle utilization, resolving charging difficulties, reducing vehicle purchase costs, facilitating centralized battery management, testing, and maintenance, extending battery life, and assisting in grid load balancing.

[0003] However, existing battery swapping technology still faces some challenges in vehicle guidance. For example, the battery swapping process requires precise vehicle positioning. If the vehicle's position is not accurate, it may lead to inaccurate docking between the battery swapping equipment and the battery pack, increasing the risk of mechanical wear and even causing the equipment to become stuck. Furthermore, the battery swapping guidance system lacks adaptability and stability in different scenarios, making it difficult to meet the complex and ever-changing battery swapping needs. Summary of the Invention

[0004] In order to solve the problems of the prior art, the present invention provides a multimodal data fusion battery swap guidance method, system, storage medium and battery swap station.

[0005] In a first aspect, the present invention provides a multimodal data fusion battery swap guidance method, the method comprising:

[0006] S1: Capture the image information of the vehicle entering the station for battery swapping and obtain the location information of the battery pack;

[0007] S2: Acquire continuous ranging data of the battery pack based on the position information to obtain the height of the battery pack above the ground;

[0008] S3: Based on the pre-calibrated coordinates of the battery swap guide frame at the reference battery pack height, the pixel coordinates of the battery swap guide frame corresponding to the current battery pack height are calculated by linear interpolation according to the proportional relationship between the current battery pack height from the ground and the reference battery pack height;

[0009] S4: Capturing a video image in the battery swap channel of the battery swap station, identifying the battery pack in the video image, obtaining pixel coordinates of a detection rectangle of the battery pack, and determining whether the vehicle is stationary based on the pixel coordinates of the detection rectangle and the continuous ranging data;

[0010] If the vehicle is stationary, the pixel coordinates of the battery swap guide frame are compared and matched with the pixel coordinates of the detection rectangular frame. If the detection rectangular frame of the battery pack is completely within the battery swap guide frame, the vehicle battery pack position meets the battery swap requirements.

[0011] S5: Obtain the (X, Y) coordinates of the battery pack, and convert the (X, Y) coordinates of the battery pack into the (X1, Y1) coordinates of the battery pack based on the coordinate system of the battery swap robot according to the installation position and coordinate system of the battery swap robot. A complete battery pack position information data packet is formed according to the height of the battery pack from the ground and the (X1, Y1) coordinates of the battery pack.

[0012] S6: The battery swap robot control device controls the battery swap robot to move to the battery pack position according to the battery pack position information data packet to complete the battery swap operation.

[0013] Furthermore, in S2, the height of the battery pack from the ground is obtained by laser ranging or binocular ranging;

[0014] If it is laser ranging, the laser rangefinder emits laser pulses at a set frequency and receives reflected light to obtain continuous ranging data on the surface of the vehicle battery pack in real time;

[0015] The continuous ranging data is processed according to the set sliding window size. For the continuous ranging data in each sliding window, the data interval is first screened to determine whether it is within the specified range. If it is not within the specified range, the sliding window is moved down by one data point and the screening is continued until a sliding window that meets the conditions is found.

[0016] Calculate the variance of the continuous ranging data in the sliding window that meets the conditions. If the variance is less than the preset limit, further check whether the continuous ranging data in the subsequent n consecutive sliding windows are also within the limited range and the variance is less than the preset limit. If so, take the average of all the continuous ranging data in the n + 1 sliding windows as the distance m1 between the current vehicle battery pack and the laser rangefinder. Then, based on the height m of the laser rangefinder above the ground, calculate the height of the battery pack above the ground as m - m1;

[0017] If binocular ranging is used, the binocular camera collects images in real time when the vehicle enters the station, and identifies the vehicle battery pack entering the field of view. Using the calibrated binocular camera parameters, the field of view difference calculation and triangulation principle are used to obtain the continuous ranging data and calculate the height of the battery pack from the ground.

[0018] Furthermore, in S4, determining whether the vehicle is in a stationary state based on the pixel coordinates of the detection rectangular frame and the continuous distance measurement data specifically includes:

[0019] If the pixel coordinate deviations of the detection rectangle of the battery pack in three consecutive video frames are all lower than a preset threshold, and the deviations of the continuous ranging data obtained in the same time period are also lower than the preset threshold, it is determined that the vehicle is stationary.

[0020] Furthermore, the S1 specifically includes:

[0021] S11: Record the video of battery swapping vehicles entering and leaving the station;

[0022] S12: Label the battery packs of the battery swapping vehicles in the entry and exit videos, and train them using an improved YOLOv8 model to obtain a first battery pack detection model;

[0023] S13: Based on the first battery pack detection model, complete the vehicle battery pack identification at the entrance of the battery swap station and obtain the location information of the battery pack.

[0024] Furthermore, the improved YOLOv8 model includes improvements to the feature extraction module, feature fusion module, and detection head module in the YOLOv8 model;

[0025] The improvement of the feature extraction module is to introduce depth-wise separable convolution on the basis of CSPDarknet, and decompose the standard convolution into depth-wise convolution and point-wise convolution;

[0026] The improvement of the feature fusion module is to replace the bidirectional path aggregation network with a bidirectional feature pyramid network and combine it with depthwise separable convolution;

[0027] The improvement of the detection head module is to replace the conventional convolution with depth-wise separable convolution.

[0028] In a second aspect, the present invention provides a multimodal data fusion battery swap guidance system for implementing the multimodal data fusion battery swap guidance method described in the first aspect, the system comprising:

[0029] The first battery pack identification device is used to capture image information of vehicles entering the station for battery replacement;

[0030] Battery height ranging device, used to obtain continuous ranging data of the battery pack;

[0031] A second battery pack identification device is used to capture video images of the battery swap channel in the battery swap station;

[0032] Battery pack coordinate measuring device, used to obtain the (X, Y) coordinates of the battery pack;

[0033] an industrial computer, configured to obtain position information of the vehicle battery pack based on the image information, and control the battery height ranging device to obtain continuous ranging data of the battery pack based on the position information to obtain the height of the battery pack from the ground;

[0034] The industrial computer is also used to calculate the pixel coordinates of the battery swap guide frame corresponding to the current battery pack height by linear interpolation based on the pre-calibrated battery swap guide frame coordinates at the reference battery pack height and the proportional relationship between the current battery pack height from the ground and the reference battery pack height;

[0035] The industrial computer is further used to identify the battery pack according to the video image, obtain the pixel coordinates of the detection rectangular frame of the battery pack, and determine whether the vehicle is in a stationary state through the pixel coordinates of the detection rectangular frame and the continuous ranging data. If the vehicle is in a stationary state, the pixel coordinates of the battery swap guide frame are compared and matched with the pixel coordinates of the detection rectangular frame. If the detection rectangular frame of the battery pack is completely within the battery swap guide frame, the position of the vehicle battery pack meets the battery swap requirements.

[0036] The industrial computer is also used to convert the battery pack (X, Y) coordinates into the battery pack (X1, Y1) coordinates based on the battery swap robot coordinate system according to the installation position and coordinate system of the battery swap robot, and form a complete battery pack position information data packet based on the battery pack height from the ground and the battery pack (X1, Y1) coordinates, and send the battery pack position information data packet to the battery swap robot control device. The battery swap robot control device controls the battery swap robot to move to the battery pack position to complete the battery swap operation.

[0037] Furthermore, the system also includes: a display device for displaying the video image in the battery swap channel of the battery swap station captured by the second battery pack identification device, and the battery swap guide frame and battery pack position status information corresponding to the current battery pack height above the ground sent by the industrial computer.

[0038] Furthermore, the first battery pack identification device is a monocular camera or a binocular camera

[0039] The battery height distance measuring device is a laser rangefinder or a binocular camera;

[0040] The second battery pack identification device is a monocular camera;

[0041] The battery pack coordinate measuring device is a slide rail type laser rangefinder.

[0042] In a third aspect, the present invention provides a computer-readable storage medium storing a battery replacement guidance program. When the battery replacement guidance program is executed by a processor, the multimodal data fusion battery replacement guidance method described in the first aspect is implemented.

[0043] In a fourth aspect, the present invention provides a battery swap station, comprising the multimodal data fusion battery swap guidance system described in the second aspect.

[0044] The beneficial effects of the technical solution provided by the embodiment of the present invention are: in the present invention, through multimodal data fusion (visual recognition + ranging + coordinate measurement), millimeter-level precise positioning of the battery pack in three-dimensional space (X / Y / Z) is achieved, significantly improving the docking success rate of the battery swap robot; and it can dynamically and adaptively guide, automatically matching the battery swap guide frame according to the height of the battery pack from the ground, adapting to the battery swap needs of different models, and enhancing versatility and flexibility.

[0045] Secondly, it has strong robustness: it uses technologies such as sliding window variance filtering and multi-sensor cross-validation to effectively reduce measurement errors caused by environmental interference (such as lighting changes, occlusions, etc.) and improve system stability.

[0046] Furthermore, fully automated operation: combining deep learning target detection (improved YOLOV8 model) with real-time posture judgment reduces manual intervention and improves battery replacement efficiency by more than 30%.

[0047] In addition, intuitive human-computer interaction helps drivers park quickly and accurately through a dynamic visual interface, optimizing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 This is a structural block diagram of a multimodal data fusion battery swap guidance system provided by the present invention;

[0050] Figure 2 This is a network architecture diagram of an improved YOLOv8 model provided by the present invention;

[0051] Figure 3 This is a flowchart of the depth-separable convolution of an improved YOLOv8 model provided by the present invention;

[0052] Figure 4 This is a network architecture diagram of an improved triple attention mechanism of the YOLOv8 model provided by the present invention;

[0053] Figure 5 This is a flow chart of a multimodal data fusion battery swap guidance method provided by the present invention;

[0054] Figure 6 This is a layout diagram of a multimodal data fusion battery swap guidance system provided by the present invention in a battery swap station.

[0055] Figure numerals: 1 - battery height measuring device; 2 - first battery pack identification device; 3 - second battery pack identification device; 4 - display device; 5 - industrial computer; 6 - battery pack coordinate measuring device. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0057] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0058] Example 1

[0059] See also Figure 1 and Figure 6 A multimodal data fusion battery replacement guidance system includes: a first battery pack identification device 2, a battery height ranging device 1, a second battery pack identification device 3, a battery pack coordinate measuring device 6, a display device 4 and an industrial computer 5.

[0060] The first battery pack recognition device 2 is a visual sensor (monocular or multi-camera) installed outside the station entrance. Its field of view covers the station entrance area downward, allowing it to capture images of incoming battery swapping vehicles. The first battery pack recognition device is calibrated to obtain its internal and external parameters to ensure the accuracy and reliability of the captured images. Furthermore, image acquisition parameters such as resolution and frame rate are set to meet the requirements of subsequent battery pack recognition.

[0061] The battery height ranging device 1 is a laser rangefinder or a binocular camera. If a laser rangefinder is selected, it is installed vertically in the middle of the battery exchange channel at the entrance of the battery exchange station, so that the laser beam is emitted vertically downward, and the height m of the laser rangefinder from the ground is measured.

[0062] If a binocular camera is selected, it is the same device as the binocular camera selected for the first battery pack identification device, and its installation position and angle are consistent. The binocular camera is calibrated as mentioned above to obtain parameters such as camera intrinsic parameters, extrinsic parameters, and field of view difference. Subsequently, the depth and distance information needs to be calculated and calibrated using the principle of triangulation.

[0063] The second battery pack identification device 3 is a monocular camera installed at the top of the intersection of the battery swap station's battery swap bay exit and the traffic lane. Its viewing angle is adjusted downward, and its installation angle is tilted 45° downward, enabling clear bird's-eye view of the battery swapping vehicles within the station's lanes, particularly the vehicle's battery pack area. This monocular camera is calibrated to determine its imaging parameters, and appropriate image acquisition parameters are set to ensure accurate battery pack identification and location in subsequent processes.

[0064] The battery pack coordinate measuring device 6 is a slide rail type laser rangefinder. The slide rail can be a ball screw slide rail or a synchronous belt slide rail. The laser rangefinder is installed on the synchronous belt or ball nut seat of the slide rail. The slide rail is installed at the lower part of the battery exit in the battery swap station and remains parallel to the direction of the vehicle entering the station to ensure that the laser rangefinder can slide stably on the slide rail and can accurately collect data. The laser rangefinder is calibrated and the parameters are set to ensure the accuracy of the measurement data.

[0065] Industrial computer 5 features a 20 TOPS GPU computing power (based on the NVIDIA Jetson Orin Nano, powered by the Ampere architecture GPU), a 6-core ARM Cortex-A78AE CPU, and 8GB of memory. It is suitable for lightweight AI inference scenarios and meets the requirements of machine vision battery pack identification. Installed inside the battery swap station's workstation, it is connected and configured with communication interfaces and network connections to the first battery pack identification device 2, the battery height ranging device 1, the second battery pack identification device 3, the battery pack coordinate measurement device 6, and the display device 4. The appropriate deep learning framework and model library are installed and deployed on industrial computer 5 to run the improved YOLOv8 model and other related algorithms. Simultaneously, control software for the battery swap guidance system is developed or integrated to enable control of each device, data processing, and execution of the battery swap guidance logic.

[0066] The display device 4 is a 43-inch LCD screen installed on the side of the battery exchange channel exit in the battery exchange station to ensure that it can clearly display the video image collected by the second battery pack identification device, as well as the battery exchange guide frame and battery pack position status information (whether it is inside / outside the guide frame) corresponding to the current battery pack height from the ground sent by the industrial computer.

[0067] The working process of the multimodal data fusion battery swap guidance system is as follows: when a vehicle enters the entrance of the battery swap station, the first battery pack recognition device 2 captures the image information of the vehicle and sends it to the industrial computer 5. The industrial computer 5 uses the first battery pack detection model to perform real-time analysis and processing on each frame of the image information. If the battery pack of the vehicle is detected, the first battery pack detection model will output the location information of the vehicle battery pack (a bounding box expressed in pixel coordinates). After obtaining the location information of the vehicle battery pack, the industrial computer 5 triggers the battery height ranging device 1 to obtain continuous ranging data of the battery pack to obtain the height of the battery pack from the ground.

[0068] It should be noted that when measuring the height of the battery pack from the ground, if a laser rangefinder is used, the pre-installed laser rangefinder will start and start measuring the distance immediately after receiving the trigger signal. The laser rangefinder emits laser pulses at the set frequency and receives the reflected light to obtain continuous ranging data of the vehicle battery pack surface in real time. After receiving these continuous ranging data, the industrial computer processes the continuous ranging data according to the set sliding window size (assuming the sliding window size is n). For the continuous data in each sliding window, the data interval is first screened to determine whether it is within the limited range [K1, K2]. If it is not within the range, the sliding window is moved down by one data point and the screening is continued until a sliding window that meets the conditions is found. For the sliding window that meets the conditions, the variance of the continuous ranging data under the sliding window is calculated. If the variance is less than the preset limit value K3, it is further checked whether the continuous ranging data in the subsequent n consecutive sliding windows are also within the limited range [K1, K2] and the variance is less than K3; if it meets the requirements, the (n + 1) The average value of all continuous distance measurement data in the sliding window is used as the distance m1 between the current vehicle battery pack and the laser rangefinder. Then, based on the pre-measured height m of the laser rangefinder from the ground, the height of the battery pack from the ground is calculated as m - m1.

[0069] If a binocular camera is used, it will capture images in real time as the vehicle enters the station and identify the vehicle battery pack that enters its field of view. Using the calibrated binocular camera parameters, continuous ranging data is obtained through field of view difference calculation and triangulation principles to calculate the height of the battery pack above the ground. Specifically, based on the position difference (parallax) of the battery pack in the images taken by the left and right cameras, combined with parameters such as the baseline distance and focal length of the binocular camera, the depth distance between the battery pack and the binocular camera is calculated, and the height of the battery pack above the ground is then determined.

[0070] After obtaining the current battery pack height from the ground, the industrial computer 5 dynamically adjusts the battery swap guide frame, and based on the pre-calibrated battery swap guide frame coordinates at the reference battery pack height, calculates the pixel coordinates of the battery swap guide frame corresponding to the current battery pack height by linear interpolation according to the proportional relationship between the current battery pack height from the ground and the reference battery pack height, and then draws the dynamically adjusted battery swap guide frame on the video image taken by the second battery pack recognition device 3, and records its pixel coordinate information for subsequent battery pack position judgment; the display device displays the battery swap guide frame information of the second battery pack recognition device 3 ... pixel coordinates of the battery swap guide frame. The video collected by device 3 is displayed in real time, and the battery replacement guide frame corresponding to the current battery pack height from the ground sent by the industrial computer 5 is displayed. The battery replacement guide frame line is blue and is used by the driver to park, showing whether the current battery pack is inside / outside the battery replacement guide frame; when the industrial computer 5 detects that the battery pack is not inside the battery replacement guide frame, the battery replacement guide frame line displayed on the display device 4 changes from blue to red, and the text "not in place" is displayed; when the industrial computer 5 detects that the battery pack is inside the battery replacement guide frame, the battery replacement guide frame line displayed on the display device 4 changes from blue to green, and the text "in place" is displayed.

[0071] The industrial computer 5 receives the video image in the battery swap channel of the battery swap station sent by the second battery pack identification device 3, and triggers the second battery pack detection model to analyze each frame of the video to obtain the pixel coordinates of the detection rectangular box of the battery pack, and judges whether the vehicle is in a stationary state by detecting the pixel coordinates of the rectangular box and the continuous ranging data. When judging, if the pixel coordinate deviations of the detection rectangular box of the battery pack in three consecutive frames of video images are all lower than the preset threshold (in this embodiment, the coordinate deviation threshold is set to 2 pixels), and the deviation of the continuous ranging data obtained in the same time period is also lower than the preset threshold (in this embodiment, the ranging data deviation threshold can be set to 1 mm), then it is judged that the vehicle is in a stationary state.

[0072] If the vehicle is stationary, the industrial computer 5 compares and matches the pixel coordinates of the battery swap guide frame with the pixel coordinates of the detection rectangular frame. If the detection rectangular frame of the battery pack is completely within the battery swap guide frame, the position of the vehicle battery pack meets the battery swap requirements. When the industrial computer 5 receives the information that the vehicle is stationary and the battery pack is in the corresponding battery swap guide frame, it triggers the battery pack coordinate measurement device to start working, and the slide-type laser rangefinder slides on the slide according to the preset path and speed, and continuously collects data on the battery pack surface during the sliding process. After the slide stops, the industrial computer 5 performs data preprocessing on the collected data, including noise removal, data filtering and other operations, and then, through geometric calculations and coordinate transformation The method converts the laser ranging data into the (X, Y) coordinates of the battery pack based on the slide rail coordinate system, and further converts the (X, Y) coordinates of the battery pack in the slide rail coordinate system into the (X1, Y1) coordinates of the battery pack based on the battery swap robot coordinate system according to the installation position and coordinate system of the battery swap robot. At the same time, the ground clearance information of the vehicle battery pack measured previously is packaged with the battery pack (X1, Y1) coordinates to form a complete battery pack position information data packet; the industrial computer 5 sends the packaged battery pack position information data packet to the battery swap robot control device through the communication connection established with the battery swap robot control device (in this embodiment, industrial Ethernet communication is used).

[0073] After the battery swap robot control device receives the battery pack location information data packet, it controls the battery swap robot's robotic arm and other actuators to move precisely to the battery pack position based on the battery pack (X1, Y1) coordinates and the battery pack's height from the ground to complete the subsequent battery swap operation.

[0074] It should be noted that the training of the first battery pack detection model is as follows: according to the determined installation position of the camera outside the entrance of the battery swap station, a large number of video materials of battery swap vehicles entering and exiting the station from this perspective are recorded, and the battery packs of the battery swap vehicles in these videos are labeled. The annotation content includes information such as the category and location (bounding box represented by pixel coordinates) of the battery pack. A training dataset containing enough samples and annotation information is constructed, and the improved YOLOv8 model is used to train the labeled dataset. During the training process, hyperparameters such as learning rate, number of iterations, and batch size are set according to actual conditions to ensure that the model can effectively learn the characteristics and detection patterns of the battery pack. After the training is completed, the obtained first battery pack detection model is saved, and deployed and tested on an industrial computer to verify its accuracy and efficiency in identifying the battery packs of vehicles entering the station.

[0075] It should also be noted that the specific method for calibrating the coordinates of the battery swap guide frame at the reference battery pack height is as follows. In this embodiment, a camera is installed on the top of the battery exit position in the battery swap station, and its viewing angle is adjusted to tilt downward 40° to ensure that the bird's-eye view of the vehicle entering and exiting the battery swap channel can be captured. A vehicle with a reference battery pack height is selected. In this embodiment, a battery swap traction heavy truck with a battery capacity of 282kwh (battery pack height 2051mm) is selected. A ground guide line is set in the battery swap channel and it is driven into the battery swap channel to ensure that the projections of the outer edges of the wheels on one side of the vehicle fall on this guide line. On the wire, park the car at the front and rear limit positions for battery swapping, take an image of the vehicle through the second battery pack recognition device, and use the second battery pack detection model (the training method is similar to that of the first battery pack detection model and will not be repeated here) to identify and locate the battery pack in the image, and obtain the pixel coordinates of the battery pack at the front and rear limit positions under the camera's perspective. According to the pixel coordinates of the front and rear limit positions of the reference battery pack, combined with the working range and battery swap operation requirements of the battery swap robot, draw the corresponding battery swap guide frame at the reference battery pack height on the image, and record geometric information such as pixel coordinates.

[0076] It should also be noted that the network architecture diagram of the improved YOLOv8 model is as follows Figure 2 As shown in the figure: It shows the structure of the entire target detection model, including the feature extraction module (Backbone), the feature fusion module (Neck) and the detection head module (Head).

[0077] The goal of the feature extraction module (Backbone) is to extract features of the input image at different scales from small to large. The improvement of the feature extraction module (Backbone) in this invention is to introduce depthwise separable convolution on the basis of CSPDarknet, decompose the standard convolution into depthwise convolution and pointwise convolution, and Figure 2As can be seen in the figure, this module consists of multiple CBS (Convolution-BatchNorm-SiLU) blocks. Each CBS block includes convolution, batch normalization, and SiLU activation functions to extract image features layer by layer. Convolution is used to extract spatial features. The convolution stride S=2 is used for downsampling, and the number of convolution kernels K=6 is the number of output channels. Batch normalization stabilizes the training process, and the SiLU activation function introduces nonlinearity. In addition, it also includes a DWMP (Depthwise Max Pooling) module, which combines depthwise separable convolution with multi-layer pooling to enhance the feature extraction capabilities of the C2f module. First, each bottleneck module is pooled to obtain the aggregated features of each bottleneck layer, and then the three features are directly concatenated. Furthermore, to better measure the importance of features in different bottleneck layers in C2f, a fully connected (FC) layer is introduced in each branch to enable feature weight learning. Finally, the MP-C2f (Multi-Pooling-C2f) module, based on multi-layer pooling and splicing, is generated. This multi-layer pooling aggregates multi-scale features, enhancing the ability to capture details such as battery pack edges and corners. Parameters such as C2f and n=3 indicate specific configurations. C2f is an improved CSP (Cross Stage Partial) bottleneck module designed to improve computational efficiency and feature extraction capabilities, while n=3 indicates the number of repetitions. The Split module then splits the input tensor into multiple sub-tensors along the channel dimension to achieve feature fusion.

[0078] As a core component of the feature extraction module, Bottleneck significantly reduces the number of model parameters and computational complexity through its combination of residual connections and depthwise separable convolutions. The Bottleneck module, combined with the DWMP (Depthwise Max Pooling) module, enhances multi-scale feature extraction. This optimization primarily involves performing pooling operations (maximum pooling) on each bottleneck's output features at different scales to extract both local and global features, enhancing the ability to capture details such as battery pack edges and corners. Furthermore, the pooled multi-scale features are directly concatenated, and the weights of each branch feature are dynamically learned through a fully connected layer to enhance the representation of important features.

[0079] The feature fusion module (Neck) aims to fuse features at different levels to generate richer, multi-scale feature representations. Figure 2 The feature fusion module (Neck) in the middle part shows operations such as upsampling (Upsample) and feature concatenation (Concat) in the feature fusion process. Upsampling is used to amplify deep high-semantic features to the same spatial resolution as shallow features for easy fusion; feature concatenation connects features at different levels in the channel dimension to integrate information at different scales; in addition, Triplet Attention (triplet attention mechanism) is added, which can enhance the correlation modeling between features and make the fused features more discriminative. BiFPN uses a fast normalization method) for weighted fusion and a bidirectional fusion idea to construct a top-down and bottom-up bidirectional channel. For information at different scales from the backbone network, the feature resolution scale is unified by upsampling and downsampling when fusing between different scales, and horizontal connections are added between features at the same scale to alleviate the loss of feature information caused by too many network layers.

[0080] The improvement of the feature fusion module (Neck) in this invention is to replace the bidirectional path aggregation network (PANet) with a weighted bidirectional feature pyramid network (BiFPN) and combine it with depthwise separable convolution.

[0081] Among them, the original bidirectional path aggregation network has redundant connections, and some intermediate nodes have limited contribution to the final detection, but it increases the amount of calculation, and the feature weights are fixed, that is, when fusing features at different levels, simple splicing or addition is used, without considering the importance differences of features at different scales. The core improvement of the weighted bidirectional feature pyramid network (BiFPN) replaced by this invention is to first streamline the nodes: remove the intermediate nodes with low contribution in PANet, and only retain the key cross-scale connections (such as Figure 2 The algorithm uses a bidirectional channel in the image processing engine to reduce computational redundancy. Secondly, it performs weighted feature fusion: it assigns learnable weights to each input feature (fast normalized fusion) and dynamically adjusts the importance of features at different scales. This approach achieves more efficient multi-scale feature fusion and enhances the representation of important features through weight learning, improving detection accuracy.

[0082] At the same time, in each convolution operation of BiFPN, the standard convolution is optimized to a depthwise separable convolution, which decomposes the standard convolution into two independent steps: channel-by-channel convolution (DepthwiseConv) and point-by-point convolution (PointwiseConv). Its main purpose is to reduce computational complexity and parameter count while maintaining model performance.

[0083] The detection head module (Head) performs target detection based on the fused features and outputs the detection results, including the target's location, category probability, etc. The improvement to the detection head module (Head) in this invention is to replace the conventional convolution with depthwise separable convolution. Figure 2 It also contains CBS blocks and DWMP modules for further processing features and outputting the final detection results. Multiple Detect nodes represent target detection at different scales to accommodate targets of different sizes.

[0084] The depth-wise separable convolution mentioned above is an optimized convolution operation. Figure 3 As can be seen in the figure, the input data first undergoes a pointwise convolution (1×1 convolution) operation to increase its dimensionality, generating more channel features, enriching cross-channel interactions, and enhancing feature expressiveness. It then undergoes a depthwise convolution, which performs convolution on each channel of the input, primarily to extract spatial features. Using a separate convolution kernel for each channel reduces computational effort while preserving the spatial information of each channel. The data then undergoes a pointwise convolution (1×1 convolution) to fuse information between channels. This operation combines information from different channels, increasing the model's expressiveness. After the convolution operation, the data passes through a batch normalization (BN) layer, which accelerates training and improves model stability by normalizing the distribution of the input data to reduce internal covariate shift. Finally, the data passes through the Sigmoid Linear Unit (SiLU) activation function, introducing nonlinearity into the model, enabling the network to learn more complex feature representations. The SiLU function combines the smoothness of the Sigmoid function and the simplicity of the linear function, providing good nonlinear activation effects while maintaining computational efficiency.

[0085] Through this structure, depth-wise separable convolution can effectively extract and fuse features while reducing computing resource usage, thereby improving the efficiency and performance of the model.

[0086] See also Figure 4 ,The triple attention mechanism is an attention module for capturing ,multidimensional correlations between features. It consists of three parallel branches, each ,branch processes the input features in different dimensions to capture ,different attention information.

[0087] Specifically, each branch first adjusts the dimensional order of the input tensor through a permute operation to process features from different dimensions. For example, the channel dimension can be swapped with other spatial dimensions, enabling analysis of feature relationships from different perspectives. Next, the data passes through a Z-Pool (a custom pooling operation), which may be used to aggregate information along a specific dimension, similar to how traditional pooling is used to reduce spatial resolution. The data then passes through a Conv (convolution) layer for feature transformation. Parameters such as the kernel size and number of channels are set according to the specific task to extract higher-level feature representations. After convolution, the data is again normalized through a BN (batch normalization) layer to ensure the stability of the data distribution. Finally, the data passes through a Sigmoid activation function to generate attention weights, which represent the importance of the features along the corresponding dimensions.

[0088] The outputs from each of the three branches are restored to their original dimensional order through a permutation operation and then combined to form the final output tensor. This multi-branch structure captures correlations between features from different dimensions, such as between channels and between spatial locations, thereby enhancing the model's ability to understand and utilize feature information, enabling it to more accurately focus on key information in the input data and improving performance in tasks such as object detection.

[0089] It is worth noting that the multimodal data fusion battery swap guidance system achieves millimeter-level precise positioning of the battery pack in three-dimensional space (X / Y / Z) through multimodal data fusion (visual recognition + ranging + coordinate measurement), significantly improving the docking success rate of the battery swap robot; and can dynamically and adaptively guide, automatically matching the battery swap guidance frame according to the height of the battery pack from the ground, adapting to the battery swap needs of different models, and enhancing the system's versatility and flexibility.

[0090] Secondly, it has strong robustness: it uses technologies such as sliding window variance filtering and multi-sensor cross-validation to effectively reduce measurement errors caused by environmental interference (such as lighting changes, occlusions, etc.) and improve system stability.

[0091] Furthermore, fully automated operation: combining deep learning target detection (improved YOLOV8 model) with real-time posture judgment reduces manual intervention and improves battery replacement efficiency by more than 30%.

[0092] In addition, the intuitive human-computer interaction helps drivers park quickly and accurately through a dynamic visual interface (blue / red / green guide box prompts), optimizing the user experience.

[0093] Example 2

[0094] A multimodal data fusion battery swap guidance method includes the following steps:

[0095] Step (1): Capture the image information of the vehicle entering the station for battery replacement and obtain the location information of the battery pack.

[0096] Specifically, it includes: recording videos of battery-swapping vehicles entering and exiting the station, marking the battery packs of the battery-swapping vehicles in the videos, and training them with an improved YOLOv8 model to obtain the first battery pack detection model; the first battery pack detection model performs real-time analysis and processing on each frame of the image information. If the vehicle's battery pack is detected, the first battery pack detection model will output the location information of the vehicle's battery pack (a bounding box expressed in pixel coordinates).

[0097] Step (2): Obtain the continuous ranging data of the battery pack based on the location information to obtain the height of the battery pack from the ground.

[0098] Specifically, if it is laser ranging, the laser rangefinder emits laser pulses at a set frequency and receives reflected light to obtain continuous ranging data on the surface of the vehicle battery pack in real time.

[0099] The continuous ranging data is processed according to the set sliding window size. For the continuous ranging data in each sliding window, the data interval is first screened to determine whether it is within the specified range. If it is not within the specified range, the sliding window is moved down by one data point and the screening is continued until a sliding window that meets the conditions is found.

[0100] Calculate the variance of the continuous ranging data in the sliding window that meets the conditions. If the variance is less than the preset limit, further check whether the continuous ranging data in the subsequent n consecutive sliding windows are also within the limit and the variance is less than the preset limit. If so, take the average of all the continuous ranging data in the n + 1 sliding windows as the distance m1 between the current vehicle battery pack and the laser rangefinder. Then, based on the height m of the laser rangefinder above the ground, calculate the height of the battery pack above the ground as m - m1;

[0101] If binocular ranging is used, the binocular camera collects images in real time when the vehicle enters the station, and identifies the vehicle battery pack entering the field of view. Using the calibrated binocular camera parameters, through field of view difference calculation and triangulation principle, continuous ranging data is obtained, and the height of the battery pack above the ground is calculated.

[0102] Step (3): Based on the coordinates of the battery swap guide frame at the pre-calibrated reference battery pack height, according to the proportional relationship between the current battery pack height from the ground and the reference battery pack height, the pixel coordinates of the battery swap guide frame corresponding to the current battery pack height are calculated by linear interpolation.

[0103] Step (4): Capture the video image in the battery swap channel of the battery swap station, identify the battery pack in the video image, obtain the pixel coordinates of the detection rectangle of the battery pack through the second battery pack detection model, and judge whether the vehicle is in a stationary state by detecting the pixel coordinates of the rectangular frame and the continuous ranging data. Specifically, if the pixel coordinate deviations of the detection rectangle of the battery pack in three consecutive frames of video images are all lower than a preset threshold (in this embodiment, the coordinate deviation threshold is set to 2 pixels), and the continuous ranging data deviation obtained in the same time period is also lower than the preset threshold (in this embodiment, the ranging data deviation threshold can be set to 1 mm), then it is judged that the vehicle is in a stationary state.

[0104] If the vehicle is stationary, the pixel coordinates of the battery swap guide frame are compared and matched with the pixel coordinates of the detection rectangular frame. If the detection rectangular frame of the battery pack is completely within the battery swap guide frame, the position of the vehicle battery pack meets the battery swap requirements.

[0105] Step (5): Obtain the battery pack (X, Y) coordinates, and convert the battery pack (X, Y) coordinates into the battery pack (X1, Y1) coordinates based on the battery swap robot coordinate system according to the installation position and coordinate system of the battery swap robot. Form a complete battery pack location information data packet based on the battery pack height from the ground and the battery pack (X1, Y1) coordinates.

[0106] Step (6): The battery swap robot control device controls the battery swap robot to move to the battery pack position according to the battery pack position information data packet to complete the battery swap operation.

[0107] It is worth noting that this multimodal data fusion battery swap guidance method achieves precise guidance of battery swap vehicles through multimodal data fusion, deep learning models, and precise measurement and control, improves battery swap efficiency and reliability, and provides strong support for the efficient operation of battery swap stations.

[0108] Example 3

[0109] A computer-readable storage medium stores a battery swap guidance program. When the battery swap guidance program is executed by a processor, the multimodal data fusion battery swap guidance method in Example 2 is implemented.

[0110] Example 4

[0111] A battery swap station includes the multimodal data fusion battery swap guidance system of embodiment one.

[0112] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0113] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0114] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal data fusion battery swap guidance method, characterized in that: The method comprises: S1: Capture image information of the battery swapping vehicle entering the station and obtain the location information of the battery pack, specifically including: S11: Record the video of the battery swapping vehicle entering and leaving the station; S12: Label the battery packs of the battery swapping vehicles in the entry and exit videos, and train them using an improved YOLOv8 model to obtain a first battery pack detection model; S13: Based on the first battery pack detection model, complete the identification of the vehicle battery pack at the entrance of the battery swap station and obtain the location information of the battery pack; S2: Acquire continuous ranging data of the battery pack based on the position information to obtain the height of the battery pack above the ground; S3: Based on the pre-calibrated coordinates of the battery swap guide frame at the reference battery pack height, the pixel coordinates of the battery swap guide frame corresponding to the current battery pack height are calculated by linear interpolation according to the proportional relationship between the current battery pack height from the ground and the reference battery pack height; S4: Capturing a video image in the battery swap channel of the battery swap station, identifying the battery pack in the video image, obtaining pixel coordinates of a detection rectangle of the battery pack, and determining whether the vehicle is stationary based on the pixel coordinates of the detection rectangle and the continuous ranging data; If the vehicle is stationary, the pixel coordinates of the battery swap guide frame are compared and matched with the pixel coordinates of the detection rectangular frame. If the detection rectangular frame of the battery pack is completely within the battery swap guide frame, the vehicle battery pack position meets the battery swap requirements. S5: Obtain the (X, Y) coordinates of the battery pack, and convert the (X, Y) coordinates of the battery pack into the (X1, Y1) coordinates of the battery pack based on the coordinate system of the battery swap robot according to the installation position and coordinate system of the battery swap robot. A complete battery pack position information data packet is formed according to the height of the battery pack from the ground and the (X1, Y1) coordinates of the battery pack. S6: The battery swap robot control device controls the battery swap robot to move to the battery pack position according to the battery pack position information data packet to complete the battery swap operation.

2. The multimodal data fusion battery swap guidance method according to claim 1, characterized in that: In S2, the height of the battery pack from the ground is obtained by laser ranging or binocular ranging; If it is laser ranging, the laser rangefinder emits laser pulses at a set frequency and receives reflected light to obtain continuous ranging data on the surface of the vehicle battery pack in real time; The continuous ranging data is processed according to the set sliding window size. For the continuous ranging data in each sliding window, the data interval is first screened to determine whether it is within the specified range. If it is not within the specified range, the sliding window is moved down by one data point and the screening is continued until a sliding window that meets the conditions is found. Calculate the variance of the continuous ranging data in the sliding window that meets the conditions. If the variance is less than the preset limit, further check whether the continuous ranging data in the subsequent n consecutive sliding windows are also within the limited range and the variance is less than the preset limit. If so, take the average of all the continuous ranging data in the n + 1 sliding windows as the distance m1 between the current vehicle battery pack and the laser rangefinder. Then, based on the height m of the laser rangefinder above the ground, calculate the height of the battery pack above the ground as m - m1; If binocular ranging is used, the binocular camera collects images in real time when the vehicle enters the station, and identifies the vehicle battery pack entering the field of view. Using the calibrated binocular camera parameters, the field of view difference calculation and triangulation principle are used to obtain the continuous ranging data and calculate the height of the battery pack from the ground.

3. The multimodal data fusion battery swap guidance method according to claim 2, characterized in that: In S4, determining whether the vehicle is stationary based on the pixel coordinates of the detection rectangle and the continuous distance measurement data specifically includes: If the pixel coordinate deviations of the detection rectangle of the battery pack in three consecutive video frames are all lower than a preset threshold, and the deviations of the continuous ranging data obtained in the same time period are also lower than the preset threshold, it is determined that the vehicle is stationary.

4. The multimodal data fusion battery swap guidance method according to claim 3, characterized in that: The improved YOLOv8 model includes improvements to the feature extraction module, feature fusion module, and detection head module in the YOLOv8 model; The improvement of the feature extraction module is to introduce depth-wise separable convolution on the basis of CSPDarknet, and decompose the standard convolution into depth-wise convolution and point-wise convolution; The improvement of the feature fusion module is to replace the bidirectional path aggregation network with a bidirectional feature pyramid network and combine it with depthwise separable convolution; The improvement of the detection head module is to replace the conventional convolution with depth-wise separable convolution.

5. A multimodal data fusion battery swap guidance system, characterized in that: For implementing the multimodal data fusion battery swapping guidance method according to any one of claims 1 to 4, the system comprises: The first battery pack identification device is used to capture image information of vehicles entering the station for battery replacement; Battery height ranging device, used to obtain continuous ranging data of the battery pack; A second battery pack identification device is used to capture video images of the battery swap channel in the battery swap station; Battery pack coordinate measuring device, used to obtain the (X, Y) coordinates of the battery pack; an industrial computer, configured to run an improved YOLOv8 model and utilize a first battery pack detection model to perform real-time analysis and processing on each frame of the image information. If a vehicle battery pack is detected, the first battery pack detection model outputs location information of the vehicle battery pack and controls the battery height ranging device based on the location information to obtain continuous ranging data of the battery pack to determine the height of the battery pack above the ground. The industrial computer is also used to calculate the pixel coordinates of the battery swap guide frame corresponding to the current battery pack height by linear interpolation based on the pre-calibrated battery swap guide frame coordinates at the reference battery pack height and the proportional relationship between the current battery pack height from the ground and the reference battery pack height; The industrial computer is further used to identify the battery pack according to the video image, obtain the pixel coordinates of the detection rectangular frame of the battery pack, and determine whether the vehicle is in a stationary state through the pixel coordinates of the detection rectangular frame and the continuous ranging data. If the vehicle is in a stationary state, the pixel coordinates of the battery swap guide frame are compared and matched with the pixel coordinates of the detection rectangular frame. If the detection rectangular frame of the battery pack is completely within the battery swap guide frame, the position of the vehicle battery pack meets the battery swap requirements. The industrial computer is also used to convert the battery pack (X, Y) coordinates into the battery pack (X1, Y1) coordinates based on the battery swap robot coordinate system according to the installation position and coordinate system of the battery swap robot, and form a complete battery pack position information data packet based on the battery pack height from the ground and the battery pack (X1, Y1) coordinates, and send the battery pack position information data packet to the battery swap robot control device. The battery swap robot control device controls the battery swap robot to move to the battery pack position to complete the battery swap operation.

6. The multimodal data fusion battery swap guidance system according to claim 5, characterized in that: The system also includes: a display device for displaying the video image in the battery swap channel of the battery swap station captured by the second battery pack identification device, and the battery swap guide frame and battery pack position status information corresponding to the current battery pack height above the ground sent by the industrial computer.

7. The multimodal data fusion battery swap guidance system according to claim 6, characterized in that: The first battery pack identification device is a monocular camera or a binocular camera The battery height distance measuring device is a laser rangefinder or a binocular camera; The second battery pack identification device is a monocular camera; The battery pack coordinate measuring device is a slide rail type laser rangefinder.

8. A computer-readable storage medium, characterized in that A battery replacement guidance program is stored, and when the battery replacement guidance program is executed by the processor, the multimodal data fusion battery replacement guidance method as described in any one of claims 1-4 is implemented.

9. A battery swap station, characterized in that: It includes a multimodal data fusion battery replacement guidance system as described in any one of claims 5-7.

Images