Multi-modal data fusion battery swap guiding method and system, storage medium and battery swap station

Abstract

The invention discloses a multi-modal data fusion battery replacement guiding method and system, a storage medium and a battery replacement station, and belongs to the technical field of new energy automobile battery replacement, and the method comprises the steps: obtaining the position information of a battery pack and the ground clearance of the battery pack, and calculating the pixel point coordinates of a battery replacement guiding frame corresponding to the current height of the battery pack; identifying the battery pack to obtain a detection rectangular frame pixel point coordinate of the battery pack; if the detection rectangular frame of the battery pack is completely located in the battery replacement guide frame, comparing and matching the pixel point coordinates of the battery replacement guide frame with the pixel point coordinates of the detection rectangular frame, and if the detection rectangular frame of the battery pack is completely located in the battery replacement guide frame, obtaining the coordinates of the battery pack, converting the coordinates of the battery pack into the coordinates of the battery pack based on a battery replacement robot coordinate system, and forming a complete battery pack position information data packet; and controlling the battery replacing robot to complete battery replacing operation according to the battery pack position information data packet. And through multi-modal data fusion, millimeter-level accurate positioning of a three-dimensional space of the battery pack is realized, and the docking success rate of the battery replacement robot is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of new energy vehicle battery swapping, and particularly to a multimodal data fusion battery swapping guiding method, system, storage medium and battery swapping station. Background Art

[0002] With the popularization of new energy vehicles, the battery swapping mode, as a fast and efficient energy replenishment method, has received increasing attention. The battery swapping mode has many advantages, such as fast battery swapping speed, with the whole process only taking 3 - 5 minutes, which can greatly improve the vehicle utilization rate, solve the problem of difficult charging, reduce the user's vehicle purchase cost, facilitate centralized battery management, detection and maintenance, extend the battery service life, and assist in balancing the power grid load.

[0003] However, there are still some problems in the existing battery swapping technology in terms of vehicle guidance. For example, during the battery swapping process, there is a high precision requirement for the vehicle position. If the vehicle position is inaccurate, it may lead to inaccurate docking between the battery swapping equipment and the battery pack, increasing the risk of mechanical wear and even causing the equipment to jam. In addition, the adaptability and stability of the battery swapping guidance system in different scenarios are also insufficient, making it difficult to meet the complex and changeable battery swapping requirements. Summary of the Invention

[0004] To solve the problems of the existing technology, the present invention provides a multimodal data fusion battery swapping guiding method, system, storage medium and battery swapping station.

[0005] In a first aspect, the present invention provides a multimodal data fusion battery swapping guiding method, which includes: S1: Capture the image information of the incoming battery swapping vehicle to obtain the position information of the battery pack; S2: Obtain the continuous ranging data of the battery pack based on the position information to get the height of the battery pack from the ground; S3: Based on the coordinates of the battery swapping guiding frame under the pre - calibrated reference battery pack height, calculate the pixel coordinates of the battery swapping guiding frame corresponding to the current battery pack height through linear interpolation according to the proportional relationship between the current battery pack height from the ground and the reference battery pack height; S4: Capture the video image in the battery swapping channel of the battery swapping station, identify the battery pack in the video image to obtain the pixel coordinates of the detection rectangular frame of the battery pack, and judge whether the vehicle is in a stationary state through the pixel coordinates of the detection rectangular frame and the continuous ranging data; If the vehicle is in a stationary state, compare and match the pixel coordinates of the battery swapping guiding frame with the pixel coordinates of the detection rectangular frame. If the detection rectangular frame of the battery pack is completely within the battery swapping guiding frame, the position of the vehicle battery pack meets the battery swapping requirements; S5: Obtain the coordinates of the battery pack (X, Y), and convert the coordinates of the battery pack (X, Y) into the coordinates of the battery pack (X1, Y1) based on the coordinate system of the battery swapping robot according to the installation position and coordinate system of the battery swapping robot. Form a complete battery pack position information data packet according to the height of the battery pack from the ground and the coordinates of the battery pack (X1, Y1); S6: The battery swapping robot control device controls the battery swapping robot to move to the battery pack position according to the battery pack position information data packet to complete the battery swapping operation.

[0006] Further, in S2, the height of the battery pack from the ground is obtained by laser ranging or binocular ranging; If it is laser ranging, the laser rangefinder emits laser pulses at a set frequency, receives the reflected light, and obtains continuous ranging data on the surface of the vehicle battery pack in real time; Process the continuous ranging data according to the set sliding window size. For the continuous ranging data in each sliding window, first perform data interval screening to determine whether it is within the defined range. If it is not within the defined range, the sliding window moves down one data point position and continues screening until a sliding window that meets the conditions is found; Calculate the variance of the continuous ranging data under the sliding window that meets the conditions. If the variance is less than the preset limit value, further check whether the continuous ranging data in the subsequent n consecutive sliding windows are also within the defined range and the variances are all less than the preset limit value. If it is satisfied, take the average value of all the continuous ranging data in the n + 1 sliding windows as the distance m1 from the current vehicle battery pack to the laser rangefinder. Then, according to the height m of the laser rangefinder from the ground, calculate that the height of the battery pack from the ground is m - m1; If it is binocular ranging, the binocular camera captures images in real time during the process of the vehicle entering the station, identifies the vehicle battery pack that enters the field of view, and uses the calibrated binocular camera parameters to obtain the continuous ranging data through the calculation of the field of view difference and the principle of triangulation ranging, and calculates the height of the battery pack from the ground.

[0007] Further, in S4, determining whether the vehicle is in a stationary state by the pixel coordinates of the detection rectangular frame and the continuous ranging data specifically includes: If the deviation amount of the pixel coordinates of the detection rectangular frame of the battery pack in three consecutive video images is lower than the preset threshold, and the deviation amount of the continuous ranging data obtained in the same time period is also lower than the preset threshold, it is determined that the vehicle is in a stationary state.

[0008] Further, S1 specifically includes: S11: Record the video of the battery swapping vehicle entering and leaving the station; S12: Label the battery packs of the battery swapping vehicles in the inbound and outbound videos, and train them using the improved YOLOv8 model to obtain the first battery pack detection model; S13: Based on the first battery pack detection model, complete the identification of the vehicle battery packs at the inbound port of the battery swapping station to obtain the position information of the battery packs.

[0009] Furthermore, the improved YOLOv8 model includes improvements to the feature extraction module, feature fusion module, and detection head module in the YOLOv8 model; The improvement to the feature extraction module is to introduce depthwise separable convolution on the basis of CSPDarknet, and decompose the standard convolution into depth convolution and pointwise convolution; The improvement to the feature fusion module is to replace the bidirectional path aggregation network with a bidirectional feature pyramid network and combine depthwise separable convolution; The improvement to the detection head module is to replace the conventional convolution with depthwise separable convolution.

[0010] In a second aspect, the present invention provides a multi-modal data fusion battery swapping guidance system for implementing the multi-modal data fusion battery swapping guidance method described in the first aspect. The system includes: A first battery pack identification device for capturing image information of inbound battery swapping vehicles; A battery height ranging device for obtaining continuous ranging data of the battery pack; A second battery pack identification device for capturing video images in the battery swapping channel of the battery swapping station; A battery pack coordinate measurement device for obtaining the (X, Y) coordinates of the battery pack; An industrial control computer for obtaining the position information of the vehicle battery pack based on the image information, and controlling the battery height ranging device to obtain continuous ranging data of the battery pack according to the position information to obtain the height of the battery pack from the ground; The industrial control computer is further configured to calculate the pixel coordinates of the battery swapping guidance frame corresponding to the current battery pack height by linear interpolation based on the pre-calibrated coordinates of the battery swapping guidance frame under the reference battery pack height and according to the proportional relationship between the current battery pack height from the ground and the reference battery pack height; The industrial control computer is further configured to identify the battery pack based on the video image to obtain the pixel coordinates of the detection rectangle frame of the battery pack, and determine whether the vehicle is in a stationary state by comparing the pixel coordinates of the detection rectangle frame with the continuous ranging data. If the vehicle is in a stationary state, the pixel coordinates of the battery swapping guidance frame are compared and matched with the pixel coordinates of the detection rectangle frame of the battery pack. If the detection rectangle frame of the battery pack is completely within the battery swapping guidance frame, the position of the vehicle battery pack meets the battery swapping requirements; The industrial control computer is also used to convert the coordinates of the battery pack (X, Y) into the coordinates of the battery pack (X1, Y1) based on the coordinate system of the battery swapping robot according to the installation position and coordinate system of the battery swapping robot, and form a complete battery pack position information data packet according to the height of the battery pack from the ground and the coordinates of the battery pack (X1, Y1), and send the battery pack position information data packet to the battery swapping robot control device, and the battery swapping robot control device controls the battery swapping robot to move to the battery pack position to complete the battery swapping operation.

[0011] Further, the system further includes: a display device for displaying the video image in the battery swapping channel of the battery swapping station captured by the second battery pack identification device, and the battery swapping guide frame and the battery pack position status information corresponding to the current height of the battery pack from the ground sent by the industrial control computer.

[0012] Further, the first battery pack identification device is a monocular camera or a binocular camera The battery height ranging device is a laser rangefinder or a binocular camera; The second battery pack identification device is a monocular camera; The battery pack coordinate measuring device is a slide rail type laser rangefinder.

[0013] In a third aspect, the present invention provides a computer-readable storage medium storing a battery swapping guide program, and when the battery swapping guide program is executed by a processor, it implements the multi-modal data fusion battery swapping guide method described in the first aspect.

[0014] In a fourth aspect, the present invention provides a battery swapping station including the multi-modal data fusion battery swapping guide system described in the second aspect.

[0015] The beneficial effects brought by the technical solution provided by the embodiment of the present invention are: in the present invention, through multi-modal data fusion (visual recognition + ranging + coordinate measurement), millimeter-level precise positioning of the battery pack in three-dimensional space (X / Y / Z) is realized, significantly improving the docking success rate of the battery swapping robot; and dynamic adaptive guidance can be performed, automatically matching the battery swapping guide frame through the height of the battery pack from the ground to adapt to the battery swapping requirements of different vehicle models, enhancing versatility and flexibility.

[0016] Secondly, it has strong robustness: technologies such as sliding window variance filtering and multi-sensor cross-validation are adopted to effectively reduce the measurement errors caused by environmental interferences (such as light changes, occlusions, etc.) and improve the system stability.

[0017] Furthermore, it has fully automated operation: combining deep learning object detection (improved YOLOV8 model) with real-time pose judgment reduces manual intervention and improves the battery swapping efficiency by more than 30%.

[0018] In addition, intuitive human-machine interaction, through a dynamic visualization interface, helps the driver park the vehicle quickly and accurately, optimizing the user experience. Description of the Drawings

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0020] Figure 1 is the structural block diagram of a multi-modal data fusion battery swapping guidance system provided by the present invention; Figure 2 is the network architecture diagram of an improved YOLOv8 model provided by the present invention; Figure 3 is the depthwise separable convolution flow chart of an improved YOLOv8 model provided by the present invention; Figure 4 is the network architecture diagram of the triplet attention mechanism of an improved YOLOv8 model provided by the present invention; Figure 5 is the flow chart of a multi-modal data fusion battery swapping guidance method provided by the present invention; Figure 6 is the layout diagram of a multi-modal data fusion battery swapping guidance system in a battery swapping station.

[0021] Reference numerals: 1 - Battery height ranging device; 2 - First battery pack identification device; 3 - Second battery pack identification device; 4 - Display device; 5 - Industrial control computer; 6 - Battery pack coordinate measuring device. Detailed Embodiments

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0023] To make the purpose, technical solutions and advantages of the present invention clearer, the following will further describe the embodiments of the present invention in detail with reference to the drawings.

[0024] Embodiment 1

[0025] See Figure 1 and Figure 6, a multi-modal data fusion battery swapping guidance system, comprising: a first battery pack identification device 2, a battery height ranging device 1, a second battery pack identification device 3, a battery pack coordinate measurement device 6, a display device 4, and an industrial control computer 5.

[0026] The first battery pack identification device 2 is a vision sensor (such as a monocular camera or a multi-camera), installed outside the inbound channel of the battery swapping station, ensuring that its viewing angle can cover the inbound area of the battery swapping station downward, so as to capture the image information of the inbound battery swapping vehicle. Calibrate the first battery pack identification device to obtain its internal and external parameters to ensure the accuracy and reliability of the collected images. At the same time, set parameters such as the resolution and frame rate of image acquisition to meet the requirements of subsequent battery pack identification.

[0027] The battery height ranging device 1 is a laser rangefinder or a binocular camera. If a laser rangefinder is selected, it is vertically installed in the middle of the battery swapping channel at the inbound port of the battery swapping station, so that the laser beam is emitted vertically downward, and the height m of the laser rangefinder from the ground is measured.

[0028] If a binocular camera is selected, it is the same device as the binocular camera selected for the first battery pack identification device, and its installation position and angle are kept consistent. Calibrate the binocular camera to obtain parameters such as the internal and external parameters of the camera and the field of view difference. Subsequently, it is necessary to calculate and calibrate the depth and distance information using the principle of triangulation ranging.

[0029] The second battery pack identification device 3 is a monocular camera, installed at the top position of the intersection of the battery swapping bin exit and the driving channel of the battery swapping station, adjusting its viewing angle downward, and its installation angle is inclined downward by 45°, so that it can clearly capture the top-view video image of the battery swapping vehicle in the battery swapping station channel, especially the vehicle battery pack area. Calibrate the monocular camera to determine its imaging parameters and set appropriate image acquisition parameters to ensure accurate identification and positioning of the battery pack in the subsequent process.

[0030] The battery pack coordinate measurement device 6 is a slide-type laser rangefinder. The slide can be a ball screw slide or a synchronous belt slide. The laser rangefinder is installed on the synchronous belt or the ball nut seat of the slide. The slide is installed under the battery out-loading port in the battery swapping station, parallel to the vehicle inbound driving direction, ensuring that the laser rangefinder can slide stably on the slide and can accurately collect data. Calibrate and set parameters for the laser rangefinder to ensure the accuracy of the measurement data.

[0031] The industrial control computer 5 has a GPU computing power of 20 TOPS (developed based on NVIDIA Jetson Orin Nano, equipped with an Ampere architecture GPU), a 6-core ARM Cortex-A78AE CPU, 8GB of memory, and is suitable for lightweight AI inference scenarios, meeting the battery pack recognition in machine vision; it is installed inside the working bin of the battery swapping station and the communication interfaces and network connections with the first battery pack recognition device 2, the battery height ranging device 1, the second battery pack recognition device 3, the battery pack coordinate measurement device 6, and the display device 4 are connected and configured. The corresponding deep learning framework and model library are installed and deployed on the industrial control computer 5 for running the improved YOLOv8 model and other related algorithms. At the same time, the control software of the battery swapping guidance system is developed or integrated to implement the control of each device, data processing, and the execution of the battery swapping guidance logic.

[0032] The display device 4 is a 43-inch liquid crystal display screen, installed on one side of the exit of the battery swapping channel in the battery swapping station, ensuring that it can clearly display the video images collected by the second battery pack recognition device, as well as the battery swapping guidance frame corresponding to the current battery pack's height from the ground and the battery pack position status information (whether inside / outside the guidance frame) sent by the industrial control computer.

[0033] The working process of this multi-modal data fusion battery swapping guidance system is as follows: when a vehicle drives into the entrance of the battery swapping station, the first battery pack recognition device 2 captures the image information of the vehicle and sends it to the industrial control computer 5. The industrial control computer 5 uses the first battery pack detection model to perform real-time analysis and processing on each frame of the image information in the image. If the battery pack of the vehicle is detected, the first battery pack detection model will output the position information of the vehicle's battery pack (the bounding box represented by pixel coordinates). After the industrial control computer 5 obtains the position information of the vehicle's battery pack, it triggers the battery height ranging device 1 to obtain continuous ranging data of the battery pack, and gets the height of the battery pack from the ground.

[0034] It should be noted that when measuring the height of the battery pack from the ground, if a laser rangefinder is used, the pre-installed laser rangefinder will be immediately turned on and start ranging after receiving a trigger signal. The laser rangefinder emits laser pulses at a set frequency, receives the reflected light, and obtains continuous ranging data on the surface of the vehicle battery pack in real time. After the industrial control computer receives this continuous ranging data, it processes the continuous ranging data according to the set sliding window size (assuming the sliding window size is n). For the continuous data in each sliding window, first, data interval screening is performed to determine whether it is within the limited range [K1, K2]. If it is not within this range, the sliding window moves down one data point position and continues screening until a sliding window that meets the conditions is found. For the sliding window that meets the conditions, the variance of the continuous ranging data under this sliding window is calculated. If the variance is less than the preset limit value K3, it is further checked whether the continuous ranging data in the subsequent n consecutive sliding windows are also within the limited range [K1, K2] and the variances are all less than K3. If it is satisfied, the average value of all the continuous ranging data in these (n + 1) sliding windows is taken as the distance m1 from the current vehicle battery pack to the laser rangefinder. Then, based on the pre-measured height m of the laser rangefinder from the ground, the height of the battery pack from the ground is calculated as m - m1.

[0035] If a binocular camera is used, the binocular camera collects images in real time during the vehicle's approach to the station, identifies the vehicle battery pack that enters the field of view, and obtains continuous ranging data and calculates the height of the battery pack from the ground by using the calibrated parameters of the binocular camera and through the calculation of the field of view difference and the principle of triangulation ranging. Specifically, according to the position difference (parallax) of the battery pack in the images taken by the left and right cameras, combined with parameters such as the baseline distance and focal length of the binocular camera, the depth distance between the battery pack and the binocular camera is calculated, and then the height of the battery pack from the ground is obtained.

[0036] After obtaining the current ground clearance of the battery pack, the industrial control computer 5 dynamically adjusts the battery swapping guiding frame. Based on the coordinates of the battery swapping guiding frame at the pre-calibrated reference battery pack height, and according to the proportional relationship between the current ground clearance of the battery pack and the reference battery pack height, the pixel coordinates of the battery swapping guiding frame corresponding to the current battery pack height are calculated by the linear interpolation method. Then, the dynamically adjusted battery swapping guiding frame is drawn on the video image captured by the second battery pack recognition device 3, and its pixel coordinate information is recorded for subsequent battery pack position judgment. The display device displays the video collected by the second battery pack recognition device 3 in real time, and at the same time displays the battery swapping guiding frame corresponding to the current ground clearance of the battery pack sent by the industrial control computer 5. The line of the battery swapping guiding frame is blue for the driver to park the vehicle, and it shows whether the current battery pack is inside / outside the battery swapping guiding frame. When the industrial control computer 5 detects that the battery pack is not inside the battery swapping guiding frame, the line of the battery swapping guiding frame displayed on the display device 4 changes from blue to red, and the text "not in place" is displayed. When the industrial control computer 5 detects that the battery pack is inside the battery swapping guiding frame, the line of the battery swapping guiding frame displayed on the display device 4 changes from blue to green, and the text "in place" is displayed.

[0037] The industrial control computer 5 receives the video image in the battery swapping channel of the battery swapping station sent by the second battery pack recognition device 3, and triggers the second battery pack detection model to analyze each frame of the video to obtain the pixel coordinates of the detection rectangle frame of the battery pack. Whether the vehicle is in a stationary state is judged by the pixel coordinates of the detection rectangle frame of the battery pack and the continuous ranging data. When judging, if the deviation amount of the pixel coordinates of the detection rectangle frame of the battery pack in three consecutive frames of video images is lower than the preset threshold (in this embodiment, the coordinate deviation amount threshold is set to 2 pixels), and the deviation amount of the continuous ranging data obtained in the same time period is also lower than the preset threshold (in this embodiment, the ranging data deviation amount threshold can be set to 1 mm), it is judged that the vehicle is in a stationary state.

[0038] If the vehicle is in a stationary state, the industrial control computer 5 compares and matches the pixel coordinates of the battery swapping guide frame with the pixel coordinates of the detection rectangle frame. If the detection rectangle frame of the battery pack is completely within the battery swapping guide frame, the position of the vehicle's battery pack meets the battery swapping requirements; when the industrial control computer 5 receives the information that the vehicle is stationary and the battery pack is in the corresponding battery swapping guide frame, it triggers the battery pack coordinate measuring device to start working. The rail-mounted laser rangefinder slides on the rail according to the preset path and speed, and continuously collects data on the surface of the battery pack during the sliding process. After the rail stops, the industrial control computer 5 performs data preprocessing on the collected data, including operations such as noise removal and data filtering. Then, through geometric calculation and coordinate transformation methods, the laser ranging data is converted into the (X, Y) coordinates of the battery pack based on the rail coordinate system. According to the installation position and coordinate system of the battery swapping robot, the (X, Y) coordinates of the battery pack in the rail coordinate system are further converted into the (X1, Y1) coordinates of the battery pack based on the battery swapping robot coordinate system. At the same time, the previously measured ground clearance information of the vehicle's battery pack is data-packed with the (X1, Y1) coordinates of the battery pack to form a complete battery pack position information data packet; the industrial control computer 5 sends the packed battery pack position information data packet to the battery swapping robot control device through the established communication connection (in this embodiment, industrial Ethernet communication) with the battery swapping robot control device.

[0039] After receiving the battery pack position information data packet, the battery swapping robot control device controls the actuators such as the robotic arm of the battery swapping robot to accurately move to the battery pack position according to the (X1, Y1) coordinates of the battery pack and the ground clearance information of the battery pack, and completes the subsequent battery swapping operation.

[0040] It should be noted that the first battery pack detection model is trained as follows: According to the determined installation position of the camera outside the entrance of the battery swapping station, a large number of video materials of battery swapping vehicles entering and leaving the station from this perspective are recorded, and the battery packs of the battery swapping vehicles in these videos are labeled. The labeling content includes information such as the category and position (bounding box represented by pixel coordinates) of the battery pack, constructing a training data set containing sufficient diverse samples and labeling information, using the improved YOLOv8 model to train the labeled data set. During the training process, hyperparameters such as the learning rate, number of iterations, and batch size are set according to the actual situation to ensure that the model can effectively learn the characteristics and detection patterns of the battery pack. After the training is completed, the obtained first battery pack detection model is saved, and it is deployed and tested on the industrial control computer to verify its recognition accuracy and efficiency for the battery packs of incoming vehicles.

[0041] It should also be noted that the specific method for calibrating the coordinates of the battery swapping guide frame at the height of the reference battery pack is as follows. In this embodiment, a camera is installed at the top of the battery exit position in the battery swapping station, and its viewing angle is adjusted to tilt downward by 40°, ensuring that the top-down view of the vehicle entering and exiting the battery swapping channel can be captured. A vehicle with the height of the reference battery pack is selected. In this embodiment, a battery swapping heavy-duty tractor with a battery capacity of 282 kWh (battery pack height 2051 mm) is selected. A ground guide line is set in the battery swapping channel, and the vehicle is driven into the battery swapping channel, ensuring that the projection of the outer edge of one side of the vehicle's wheels all falls on this guide line. The vehicle is parked at the front and rear extreme positions where battery swapping is possible. The image of the vehicle is captured by the second battery pack recognition device, and the battery pack in the image is recognized and located using the second battery pack detection model (the training method is similar to that of the first battery pack detection model and will not be elaborated here). The pixel coordinates at the front and rear extreme positions of the battery pack in the viewing angle of this camera are obtained. According to the pixel coordinates at the front and rear extreme positions of this reference battery pack, combined with the working range of the battery swapping robot and the requirements of the battery swapping operation, a corresponding battery swapping guide frame at the height of the reference battery pack is drawn on the image, and geometric information such as pixel coordinates is recorded.

[0042] It should also be noted that the network architecture diagram of the improved YOLOv8 model is as Figure 2 shown: It shows the structure of the entire object detection model, including a feature extraction module (Backbone), a feature fusion module (Neck), and a detection head module (Head).

[0043] The goal of the feature extraction module (Backbone) is to extract the features of the input image at different scales from small to large. In the present invention, the improvement to the feature extraction module (Backbone) is to introduce depthwise separable convolution on the basis of CSPDarknet, decomposing the standard convolution into depth convolution and pointwise convolution, from Figure 2As can be seen, this module consists of multiple CBS (Convolution-BatchNorm-SiLU) blocks. Each CBS block contains convolution, batch normalization, and SiLU activation functions, which are used to extract image features layer by layer. The convolution operation is used to extract spatial features. The convolution stride S = 2 for downsampling, and the number of convolution kernels K = 6, which is the number of output channels. Batch normalization stabilizes the training process, and the SiLU activation function introduces non-linearity. In addition, it also includes a DWMP (Depthwise Max Pooling) module, which combines depthwise separable convolution with multi-layer pooling to enhance the feature extraction ability of the C2f module. First, pooling is performed on each Bottleneck module to obtain the aggregated features of each Bottleneck layer, and then the three parts of features are directly connected. Further, to better measure the importance of features in different Bottleneck layers in C2f, a fully connected (full Concatenate, FC) layer is introduced for each branch to achieve the learning of feature weights. Finally, an MP-C2f (Multi-Pooling-C2f) module based on multi-layer pooling concatenation is generated. By aggregating multi-scale features through multi-layer pooling, it enhances the ability to capture details such as the edges and corners of the battery pack. Parameters such as C2f and n = 3 represent specific configurations. C2f is an improved CSP (Cross Stage Partial) bottleneck module designed to improve computational efficiency and feature extraction ability, and n = 3 represents the number of repetitions. Then, the Split module is used to split the input tensor into multiple sub-tensors along the channel dimension to achieve feature fusion.

[0044] As one of the core components of the feature extraction module, the Bottleneck plays a major role in the following aspects: Through the combined design of residual connection and depthwise separable convolution, the Bottleneck module significantly reduces the number of model parameters and computational volume. The Bottleneck module combines with the DWMP (Depthwise Max Pooling) module to enhance the multi-scale feature extraction ability. It optimizes feature extraction mainly in the following ways: performing pooling operations (max pooling) on the output features of each Bottleneck at different scales to extract local and global features, enhancing the ability to capture details such as the edges and corners of the battery pack. At the same time, the multi-scale features after pooling are directly concatenated, and the weights of the features of each branch are dynamically learned through a fully connected layer to strengthen the expression of important features.

[0045] The purpose of the feature fusion module (Neck) is to fuse features at different levels to generate a richer and multi-scale feature representation.Figure 2 The feature fusion module (Neck) part shows operations such as upsampling (Upsample) and feature concatenation (Concat) in the feature fusion process. Upsampling is used to enlarge the deep high-semantic features to the same spatial resolution as the shallow features for easy fusion. Feature concatenation connects features at different levels in the channel dimension to integrate information at different scales. In addition, Triplet Attention (triplet attention mechanism) is added, which can enhance the correlation modeling between features and make the fused features more discriminative. BiFPN uses a fast normalization method for weighted fusion and the idea of bidirectional fusion to construct top-down and bottom-up bidirectional channels. When fusing information from different scales of the backbone network, the feature resolution scale is unified through upsampling and downsampling during fusion between different scales, and horizontal connections are added between features at the same scale to alleviate the loss of feature information caused by too many network levels.

[0046] In the present invention, the improvement to the feature fusion module (Neck) is to replace the bidirectional path aggregation network (PathAggregation Network, PANet) with a weighted bidirectional feature pyramid network (bidirectional feature pyramid network, BiFPN) and combine it with depthwise separable convolution.

[0047] Among them, the original bidirectional path aggregation network has redundant connections. Some intermediate nodes contribute limitedly to the final detection but increase the computational amount, and the feature weights are fixed, that is, simple concatenation or addition is used when fusing features at different levels without considering the importance differences of features at different scales. The core improvement of the replaced weighted bidirectional feature pyramid network (BiFPN) in the present invention is to first streamline the nodes: remove the intermediate nodes with low contribution in PANet and only retain the key cross-scale connections (such as Figure 2 the bidirectional channels in), reducing computational redundancy. Secondly, weighted feature fusion is carried out: assign learnable weights (fast normalization fusion) to each input feature to dynamically adjust the importance of features at different scales. Its advantage is to achieve more efficient multi-scale feature fusion and enhance the expression of important features through weight learning, improving the detection accuracy.

[0048] At the same time, in each convolution operation of BiFPN, the standard convolution is optimized into depthwise separable convolution (Depthwise Separable Convolution), which decomposes the standard convolution into two independent steps: depthwise convolution (Depthwise Conv) and pointwise convolution (Pointwise Conv). Its main purpose is to reduce the computational complexity and the number of parameters while maintaining the model performance.

[0049] The detection head module (Head) performs object detection based on the fused features and outputs the detection results, including the location of the object, class probability, etc. In the present invention, the improvement to the detection head module (Head) is to replace the conventional convolution with depthwise separable convolution. Figure 2 It also includes CBS blocks and DWMP modules for further processing the features and outputting the final detection results. Multiple Detect nodes represent object detection at different scales to adapt to objects of different sizes.

[0050] The depthwise separable convolution mentioned above is an optimized convolution operation. As Figure 3 can be seen, the input data first undergoes a pointwise convolution (Pointwise Conv, a 1×1 convolution) operation to increase the dimension, generating more channel features, enriching cross-channel interactions, and enhancing the feature expression ability. Then, it undergoes a depthwise convolution (Depthwise Conv). This operation performs convolution on each channel of the input separately, mainly for extracting spatial features. Each channel uses a separate convolution kernel, which can reduce the computational amount while retaining the spatial information of each channel. Next, the data passes through a pointwise convolution (Pointwise Conv), that is, a 1×1 convolution operation, for information fusion between channels. This operation can combine information from different channels and increase the expression ability of the model. After the convolution operation, the data passes through a batch normalization (BN, BatchNormalization) layer, which can accelerate the training process and improve the stability of the model. It reduces the internal covariate shift by normalizing the distribution of the input data. Finally, the data passes through a SiLU (Sigmoid Linear Unit) activation function, which introduces non-linearity to the model, enabling the network to learn more complex feature representations. The SiLU function combines the smoothness of the Sigmoid function and the simplicity of the linear function, providing a good non-linear activation effect while maintaining computational efficiency.

[0051] Through this structure, the depthwise separable convolution can effectively extract and fuse features while reducing the occupation of computing resources, improving the efficiency and performance of the model.

[0052] See Figure 4 , the triplet attention mechanism is an attention module for capturing multi-dimensional correlations between features. It contains three parallel branches, and each branch processes the input features in different dimensions to capture different attention information.

[0053] Specifically: For each branch, first, the Permute operation is used to adjust the dimension order of the input tensor so that features can be processed from different dimensions. For example, the channel dimension is swapped with other spatial dimensions, enabling the analysis of the relationships between features from different perspectives. Next, the data passes through the Z-Pool (a custom pooling operation), which may be used to aggregate information in a specific dimension, similar to how traditional pooling operations are used to reduce spatial resolution. Then, the data goes through the Conv (convolution) layer for feature transformation. Parameters such as the convolution kernel size and the number of channels are set according to the specific task and are used to extract higher-level feature representations. After convolution, it is again normalized through the BN (batch normalization) layer to ensure the stability of the data distribution. Finally, the data passes through the Sigmoid activation function to generate attention weights, which represent the degree of importance of features in the corresponding dimensions.

[0054] The outputs after processing by the three branches are respectively restored to the original dimension order through the Permute operation, and then they are integrated together to form the final output tensor. This multi-branch structure can capture the correlations between features from different dimensions, such as between channels and between spatial positions, thereby enhancing the model's ability to understand and utilize feature information, enabling the model to more accurately focus on the key information in the input data, and improving the performance of tasks such as object detection.

[0055] It is worth noting that this multi-modal data fusion battery swapping guidance system realizes millimeter-level precise positioning of the battery pack in the three-dimensional space (X / Y / Z) through multi-modal data fusion (visual recognition + ranging + coordinate measurement), significantly improving the docking success rate of the battery swapping robot; and it can provide dynamic and adaptive guidance. By automatically matching the battery swapping guidance frame according to the height of the battery pack from the ground, it adapts to the battery swapping requirements of different vehicle models, enhancing the versatility and flexibility of the system.

[0056] Secondly, it has strong robustness: By adopting technologies such as sliding window variance filtering and multi-sensor cross-validation, it effectively reduces the measurement errors caused by environmental interferences (such as light changes, occlusions, etc.), improving the system stability.

[0057] Furthermore, it operates fully automatically: By combining deep learning object detection (improving the YOLOV8 model) with real-time pose judgment, it reduces manual intervention and increases the battery swapping efficiency by more than 30%.

[0058] In addition, it has an intuitive human-computer interaction. Through a dynamic visualization interface (blue / red / green guidance box prompts), it helps the driver park quickly and accurately, optimizing the user experience.

[0059] Embodiment 2 A multi-modal data fusion battery swapping guidance method, comprising the following steps: Step (1): Capture the image information of the incoming vehicle for battery swapping to obtain the position information of the battery pack.

[0060] Specifically, it includes: recording the videos of the vehicle entering and leaving the station for battery swapping, annotating the battery packs of the vehicles in the videos of entering and leaving the station, and training with an improved YOLOv8 model to obtain the first battery pack detection model; the first battery pack detection model performs real-time analysis and processing on each frame of the image information. If the battery pack of the vehicle is detected, the first battery pack detection model will output the position information of the vehicle's battery pack (the bounding box represented by pixel coordinates).

[0061] Step (2): Obtain the continuous ranging data of the battery pack based on the position information to obtain the height of the battery pack from the ground.

[0062] Specifically, it includes: if it is laser ranging, the laser rangefinder emits laser pulses at a set frequency and receives the reflected light to obtain the continuous ranging data on the surface of the vehicle's battery pack in real time.

[0063] Process the continuous ranging data according to the set sliding window size. For the continuous ranging data in each sliding window, first perform data interval screening to determine whether it is within the limited range. If it is not within the limited range, the sliding window moves down one data point position and continues the screening until a sliding window that meets the conditions is found; Calculate the variance of the continuous ranging data under the sliding window that meets the conditions. If the variance is less than the preset limit value, further check whether the continuous ranging data in the subsequent n consecutive sliding windows are also within the limited range and the variances are all less than the preset limit value. If it is satisfied, take the average value of all the continuous ranging data in the n + 1 sliding windows as the distance m1 from the current vehicle's battery pack to the laser rangefinder. Then, according to the height m of the laser rangefinder from the ground, calculate the height of the battery pack from the ground as m - m1; If it is binocular ranging, the binocular camera captures images in real time during the vehicle's entry into the station and identifies the vehicle's battery pack that enters the field of view. Using the calibrated parameters of the binocular camera, through the calculation of the field of view difference and the principle of triangulation ranging, obtain the continuous ranging data and calculate the height of the battery pack from the ground.

[0064] Step (3): Based on the coordinates of the battery swapping guiding frame under the pre-calibrated reference battery pack height, calculate the pixel coordinates of the battery swapping guiding frame corresponding to the current battery pack height through linear interpolation according to the proportional relationship between the current battery pack height and the reference battery pack height.

[0065] Step (4): Capture the video image in the battery swapping channel of the battery swapping station, identify the battery pack in the video image, obtain the pixel coordinates of the detection rectangle of the battery pack through the second battery pack detection model, and determine whether the vehicle is in a stationary state based on the pixel coordinates of the detection rectangle and the continuous ranging data. Specifically, it includes: if the deviation of the pixel coordinates of the detection rectangle of the battery pack in three consecutive video images is lower than the preset threshold (in this embodiment, the coordinate deviation threshold is set to 2 pixels), and the deviation of the continuous ranging data obtained within the same time period is also lower than the preset threshold (in this embodiment, the ranging data deviation threshold can be set to 1 mm), then it is determined that the vehicle is in a stationary state.

[0066] If the vehicle is in a stationary state, compare and match the pixel coordinates of the battery swapping guide frame with the pixel coordinates of the detection rectangle. If the detection rectangle of the battery pack is completely within the battery swapping guide frame, the position of the vehicle's battery pack meets the battery swapping requirements.

[0067] Step (5): Obtain the (X, Y) coordinates of the battery pack, and convert the (X, Y) coordinates of the battery pack into the (X1, Y1) coordinates of the battery pack based on the coordinate system of the battery swapping robot according to the installation position and coordinate system of the battery swapping robot. Form a complete battery pack position information data packet based on the height of the battery pack from the ground and the (X1, Y1) coordinates of the battery pack.

[0068] Step (6): The battery swapping robot control device controls the battery swapping robot to move to the battery pack position according to the battery pack position information data packet to complete the battery swapping operation.

[0069] It should be noted that this multi-modal data fusion battery swapping guidance method realizes the precise guidance of battery swapping vehicles through multi-modal data fusion, deep learning models, and precise measurement and control, improves the battery swapping efficiency and reliability, and provides strong support for the efficient operation of the battery swapping station.

[0070] Embodiment III

[0071] A computer-readable storage medium stores a battery swapping guidance program. When the battery swapping guidance program is executed by a processor, it realizes the multi-modal data fusion battery swapping guidance method in Embodiment II.

[0072] Embodiment IV

[0073] A battery swapping station includes the multi-modal data fusion battery swapping guidance system in Embodiment I.

[0074] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device.

[0075] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection part (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, a computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or otherwise processing it as appropriate, and then storing it in a computer memory.

[0076] The serial numbers of the embodiments of the present invention described above are only for description and do not represent the superiority or inferiority of the embodiments.

[0077] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multimodal data fusion method for guiding battery swapping, characterized in that, The method includes: S1: Capture the image information of the incoming vehicle for battery swapping to obtain the position information of the battery pack; S2: Obtain the continuous ranging data of the battery pack based on the position information to get the height of the battery pack from the ground; S3: Based on the coordinates of the battery swapping guiding frame at the pre-calibrated reference battery pack height, according to the proportional relationship between the current height of the battery pack from the ground and the reference battery pack height, calculate the pixel coordinates of the battery swapping guiding frame corresponding to the current battery pack height through linear interpolation; S4: Capture the video image in the battery swapping channel of the battery swapping station, identify the battery pack in the video image to obtain the pixel coordinates of the detection rectangular frame of the battery pack, and determine whether the vehicle is in a stationary state based on the pixel coordinates of the detection rectangular frame and the continuous ranging data; If the vehicle is in a stationary state, compare and match the pixel coordinates of the battery swapping guiding frame with the pixel coordinates of the detection rectangular frame of the battery pack. If the detection rectangular frame of the battery pack is completely within the battery swapping guiding frame, the position of the vehicle's battery pack meets the requirements for battery swapping; S6: Obtain the (X, Y) coordinates of the battery pack, and based on the installation position and coordinate system of the battery swapping robot, convert the (X, Y) coordinates of the battery pack into the (X1, Y1) coordinates of the battery pack based on the coordinate system of the battery swapping robot, and form a complete data packet of the battery pack position information according to the height of the battery pack from the ground and the (X1, Y1) coordinates of the battery pack; S7: The control device of the battery swapping robot controls the battery swapping robot to move to the position of the battery pack according to the data packet of the battery pack position information to complete the battery swapping operation.

2. The multi-modal data fusion battery swapping guidance method according to claim 1, wherein In the above S2, the height of the battery pack from the ground is obtained by laser ranging or binocular ranging; If it is laser ranging, the laser rangefinder emits laser pulses at a set frequency and receives the reflected light to obtain the continuous ranging data on the surface of the vehicle's battery pack in real time; Process the continuous ranging data according to the set sliding window size. For the continuous ranging data in each sliding window, first perform data interval screening to determine whether it is within the defined range. If it is not within the defined range, move the sliding window down by the position of one data point and continue the screening until a sliding window that meets the conditions is found; Calculate the variance of the continuous ranging data under the sliding window that meets the conditions. If the variance is less than the preset defined value, further check whether the continuous ranging data in the subsequent n consecutive sliding windows are also within the defined range and the variances are all less than the preset defined value. If it is satisfied, take the average value of all the continuous ranging data in the n + 1 sliding windows as the distance m1 from the current vehicle's battery pack to the laser rangefinder. Then, according to the height m of the laser rangefinder from the ground, calculate the height of the battery pack from the ground as m - m1; If it is binocular ranging, the binocular camera captures images in real time during the process of the vehicle entering the station, identifies the vehicle's battery pack that enters the field of view, and uses the calibrated parameters of the binocular camera to obtain the continuous ranging data through the calculation of the field of view difference and the principle of triangulation ranging, and calculates the height of the battery pack from the ground.

3. The multimodal data fusion power replacement guiding method according to claim 2, wherein In S4, determining whether the vehicle is in a stationary state by using the pixel coordinates of the detected rectangular box and the continuous ranging data specifically includes: If the deviation of the pixel coordinates of the detected rectangular box of the battery pack in three consecutive video images is lower than a preset threshold, and the deviation of the continuous ranging data obtained within the same time period is also lower than the preset threshold, it is determined that the vehicle is in a stationary state.

4. The multimodal data fusion power replacement guiding method according to claim 3, wherein, S1 specifically includes: S11: Recording the video of the battery swapping vehicle entering and leaving the station; S12: Annotating the battery pack of the battery swapping vehicle in the entering and leaving station video and training it with an improved YOLOv8 model to obtain the first battery pack detection model; S13: Based on the first battery pack detection model, completing the identification of the vehicle battery pack at the entrance of the battery swapping station to obtain the position information of the battery pack.

5. The multimodal data fusion battery swapping guidance method according to claim 4, wherein The improved YOLOv8 model includes improvements to the feature extraction module, feature fusion module, and detection head module in the YOLOv8 model; The improvement of the feature extraction module is to introduce depthwise separable convolution on the basis of CSPDarknet, decomposing the standard convolution into depth convolution and pointwise convolution; The improvement of the feature fusion module is to replace the bidirectional path aggregation network with a bidirectional feature pyramid network and combine depthwise separable convolution; The improvement of the detection head module is to replace the conventional convolution with depthwise separable convolution.

6. A multimodal data fusion battery swapping guidance system, characterized in that, For implementing the multi-modal data fusion battery swapping guidance method according to any one of claims 1-5, the system includes: A first battery pack identification device for capturing the image information of the incoming battery swapping vehicle; A battery height ranging device for obtaining the continuous ranging data of the battery pack; A second battery pack identification device for capturing the video image in the battery swapping channel of the battery swapping station; A battery pack coordinate measuring device for obtaining the (X, Y) coordinates of the battery pack; An industrial control computer for obtaining the position information of the vehicle battery pack according to the image information, and controlling the battery height ranging device to obtain the continuous ranging data of the battery pack according to the position information to obtain the height of the battery pack from the ground; The industrial control computer is further configured to, based on the coordinates of the battery swapping guidance frame at the pre-calibrated reference battery pack height, calculate the pixel coordinates of the battery swapping guidance frame corresponding to the current battery pack height through linear interpolation according to the proportional relationship between the current battery pack height from the ground and the reference battery pack height; The industrial control computer is further configured to identify the battery pack according to the video image to obtain the pixel coordinates of the detected rectangular box of the battery pack, and determine whether the vehicle is in a stationary state by using the pixel coordinates of the detected rectangular box and the continuous ranging data. If the vehicle is in a stationary state, the pixel coordinates of the battery swapping guidance frame are compared and matched with the pixel coordinates of the detected rectangular box. If the detected rectangular box of the battery pack is completely located within the battery swapping guidance frame, the position of the vehicle battery pack meets the battery swapping requirements. The industrial control computer is also used to convert the coordinates of the battery pack (X, Y) into the coordinates of the battery pack (X1, Y1) based on the coordinate system of the battery swapping robot according to the installation position and coordinate system of the battery swapping robot, form a complete battery pack position information data packet according to the height of the battery pack from the ground and the coordinates of the battery pack (X1, Y1), and send the battery pack position information data packet to the battery swapping robot control device, and the battery swapping robot control device controls the battery swapping robot to move to the battery pack position to complete the battery swapping operation.

7. The multimodal data fusion power replacement guidance system according to claim 6, wherein The system further includes: a display device for displaying the video image in the battery swapping channel of the battery swapping station captured by the second battery pack identification device, and the battery swapping guide frame and battery pack position status information corresponding to the current height of the battery pack from the ground sent by the industrial control computer.

8. The multi-modal data fusion battery swapping guidance system according to claim 7, wherein The first battery pack identification device is a monocular camera or a binocular camera The battery height ranging device is a laser rangefinder or a binocular camera; The second battery pack identification device is a monocular camera; The battery pack coordinate measuring device is a slide rail type laser rangefinder.

9. A computer-readable storage medium, characterized in that, A battery swapping guidance program is stored, and when the battery swapping guidance program is executed by a processor, the multi-modal data fusion battery swapping guidance method according to any one of claims 1-5 is implemented.

10. A battery swapping station, characterized in that, It includes the multi-modal data fusion battery swapping guidance system according to any one of claims 6-8.

Images