A charging port recognition and positioning method and device based on multi-modal perception

Abstract

The application provides a charging port recognition and positioning method and device based on multi-modal perception, which comprises the following steps: acquiring binocular visible light images and infrared depth maps, synchronizing and pixel-level registering based on calibration external parameters to obtain RGBD data; inputting the visible light images into a detection model embedded with coordinate attention to obtain a candidate detection box; combining the geometric parameters of the detection box and local depth features to output two-dimensional correction offsets from a correction network, and correcting at least three two-dimensional feature points; establishing a two-dimensional-three-dimensional correspondence according to three-dimensional template points and outputting rotation parameters and translation parameters by differentiable PnP solving; weighting the reprojection error by using the variance output by the uncertainty branch; when the reprojection error meets the trigger condition, writing the sample into the difficult sample buffer and performing incremental updating on at least part of the parameters of the correction network and / or the uncertainty branch. The technical scheme of the application realizes stable output of the charging port pose, and improves the success rate of automatic charging alignment and plugging.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a charging port identification and positioning method and device based on multimodal perception. Background Technology

[0002] With the increasing popularity of new energy vehicles, the requirements for charging efficiency and user experience are constantly improving. Accurate identification and positioning of charging ports has become a crucial aspect of automatic charging systems (such as automatic alignment by a charging robotic arm). Current mainstream solutions typically include manual positioning, sensor-assisted positioning, and visual positioning.

[0003] Manual positioning relies on the driver's visual operation, which is inefficient and prone to errors. Sensor-assisted positioning often uses ultrasonic radar or near-field communication technology (such as NFC / RFID). Ultrasonic solutions have insufficient detection accuracy, while near-field communication solutions often require implanting special tags in the charging port and are subject to signal attenuation due to interference from the metal body, increasing deployment costs and application limitations.

[0004] In terms of visual positioning technology, traditional image processing methods (such as edge detection combined with template matching) are significantly constrained by lighting conditions and frequently fail in low light or rainy / foggy environments. Although deep learning-based detection models (such as YOLO, SSD, or pre-trained CNN) improve detection accuracy, they usually rely on large-scale, vehicle-specific data for training and have limited generalization ability for new or modified vehicles.

[0005] Existing technical solutions generally suffer from insufficient environmental adaptability (positioning accuracy decreases in rain, fog, strong light, or obstruction by foreign objects), increased costs due to reliance on additional hardware or vehicle modifications, insufficient cross-vehicle compatibility, difficulty in balancing real-time performance and accuracy, and deficiencies in functional dimensions such as charging port type and open / closed status recognition. These issues restrict the reliability and large-scale deployment of automatic charging systems.

[0006] Therefore, there is an urgent need for a new energy vehicle charging port identification and positioning method to improve the identification and positioning accuracy in complex environments while also ensuring real-time performance. Summary of the Invention

[0007] This invention provides a charging port identification and localization method based on multimodal perception. The method constructs a multimodal perception input including binocular visible light images and infrared depth maps, and introduces a coordinate attention mechanism during target detection to obtain candidate regions for the charging port. It combines the geometric information and local depth information of the candidate regions to generate a two-dimensional correction offset, and outputs the charging port pose parameters based on the correspondence between the corrected two-dimensional feature points and three-dimensional template points. This enables stable identification and localization of the charging port when a new energy vehicle needs to be automatically charged.

[0008] In a first aspect, the present invention provides a charging port identification and positioning method based on multimodal sensing, the method comprising: Acquire binocular visible light images and depth maps of the target vehicle's charging area. Based on pre-calibrated extrinsic parameters, synchronize and pixel-level register the depth map and the binocular visible light images to obtain multimodal fused RGBD data with color image and depth information alignment. The visible light image in the RGBD data is input into the target detection model with an embedded coordinate attention mechanism, and the candidate detection box of the charging port is output. Local depth features corresponding to the candidate detection box are extracted from the RGBD data. The geometric parameters of the candidate detection box and the local depth features are input into the correction network, and a two-dimensional correction offset is output. Based on the candidate detection box, at least three two-dimensional feature points of the charging port are determined, and the at least three two-dimensional feature points are corrected based on the two-dimensional correction offset. A three-dimensional template point set matching the charging port structure is called to establish the correspondence between the corrected two-dimensional feature points and the three-dimensional template points. According to the correspondence, the rotation matrix and translation vector of the charging port relative to the camera coordinate system are solved by the PnP algorithm, and the rotation matrix and translation vector are used as the pose output of the charging port. The PnP algorithm is iteratively solved based on the variance-weighted reprojection error, and the variance is output by the uncertainty estimation branch.

[0009] Using the above scheme, the charging port identification and localization method based on multimodal perception of the present invention achieves stable perception input of the charging area under complex lighting and occlusion conditions by performing synchronous and pixel-level registration of binocular visible light images and infrared depth maps to form aligned RGBD data; by embedding a target detection model with a coordinate attention mechanism into the visible light image input to output candidate detection boxes for the charging port, robust detection of small targets and background interference of the charging port is achieved; by extracting the local depth features corresponding to the candidate detection boxes and combining them with the geometric parameters of the detection boxes to generate a two-dimensional correction offset, adaptive correction of the localization deviation of the two-dimensional feature points is achieved. Furthermore, by establishing a two-dimensional-three-dimensional correspondence between the corrected two-dimensional feature points and the three-dimensional template points, and inputting the rotation and translation parameters into a differentiable PnP solver, end-to-end calculation of the charging port pose is achieved. Even further, by weighting the reprojection error with the variance output from the uncertainty estimation branch, and incrementally updating at least some parameters of the correction network and / or the uncertainty estimation branch based on a hard sample buffer when the error meets the trigger condition, suppression of noise points and abnormal samples and continuous adaptation across vehicle models and environments are achieved, significantly improving the accuracy, stability, and real-time reliability of charging port identification and positioning. The technical solution of this invention achieves highly robust output of the charging port pose, improving the success rate of automatic charging alignment and insertion.

[0010] In some embodiments of the present invention, the binocular visible light image and the depth map are obtained by the binocular visible light camera and the infrared depth sensor, respectively, and the two are rigidly fixed together. The extrinsic parameter relationship between the two is obtained through joint calibration. Distortion correction and stereo correction are performed on the binocular visible light image, and filtering and completion processing are performed on the depth map to generate the aligned RGBD data.

[0011] In some embodiments of the present invention, the synchronization and pixel-level registration includes: performing frame-level alignment between the binocular visible light image and the depth map based on the acquisition timestamp, and reprojecting the depth map onto the pixel coordinate system of the binocular visible light image based on the extrinsic parameter relationship to obtain a depth value corresponding to each pixel of the binocular visible light image.

[0012] In some embodiments of the present invention, the target detection model with embedded coordinate attention mechanism is a lightweight YOLO detection network, wherein the coordinate attention mechanism is inserted after the backbone feature extraction layer and / or before the regression branch of the lightweight YOLO detection network.

[0013] In some embodiments of the present invention, the correction network is a multilayer perceptron (MLP), the geometric parameters of the candidate detection box include at least normalized center coordinates, width and height, the local depth features include depth statistics sampled within the corresponding region of the candidate detection box, and the two-dimensional correction offset output by the correction network.

[0014] In some embodiments of the present invention, the at least three two-dimensional feature points include: the center point of the candidate detection box and at least two boundary points determined based on the preset boundary position of the candidate detection box; the two-dimensional correction offset is superimposed on the pixel coordinates of the center point and the boundary points respectively to obtain the corrected two-dimensional feature points.

[0015] In some embodiments of the present invention, the three-dimensional template point set includes template points that correspond one-to-one with the corrected two-dimensional feature points, and the correspondence between the corrected two-dimensional feature points and the three-dimensional template points is established based on a preset point mapping rule.

[0016] In some embodiments of the present invention, when the PnP algorithm iteratively solves the rotation matrix and the translation vector, it introduces the variance parameter output by the uncertainty estimation branch to each corresponding point pair, and weights the reprojection error based on the variance parameter; wherein, the variance parameter includes variance scalars corresponding to different corresponding point pairs.

[0017] In some embodiments of the present invention, the method further includes: calculating the reprojection error based on the rotation matrix and the translation vector; when the reprojection error meets a preset triggering condition, writing the corresponding sample into a hard sample buffer; and performing incremental updates on at least some parameters of the correction network and / or the uncertainty estimation branch based on the hard sample buffer; the preset triggering condition includes: the mean of the reprojection error is greater than a preset threshold within a consecutive preset number of frames; the hard sample buffer is a circular buffer; and the incremental update includes: extracting samples from the hard sample buffer to form a small dataset, and performing incremental updates on at least some parameters of the correction network and / or the uncertainty estimation branch.

[0018] Compared with existing technologies, the beneficial effects of this invention are as follows: By rigidly fixing a binocular visible light camera and an infrared depth sensor and obtaining extrinsic parameter relationships through joint calibration, while simultaneously performing distortion correction and stereo correction on the visible light image and filtering and completion processing on the depth map, stable input of RGBD data in terms of geometric consistency and depth integrity is achieved; by performing frame-level alignment of multi-source data based on timestamps and reprojecting the depth map onto the visible light pixel coordinate system, pixel-by-pixel correspondence between depth information and image information is achieved, reducing the impact of cross-sensor latency and parallax errors on positioning; by employing a lightweight YOLO detection network and inserting a coordinate attention mechanism after the backbone feature extraction layer and / or before the regression branch, rapid detection of charging port candidate regions and enhancement of position-sensitive features are achieved; furthermore, by inputting the normalized center coordinates, width, height, and other geometric parameters of the candidate detection box and the depth statistics sampled within the candidate region into the MLP correction network and outputting a two-dimensional offset (Δx, Δy), unified correction of two-dimensional positioning deviations within the candidate region is achieved; and further still, through… By determining at least three two-dimensional feature points based on candidate detection boxes, the two-dimensional feature points include the center point of the candidate detection box and at least two boundary feature points located on the boundary of the candidate detection box, and superimposing the two-dimensional offset for correction, and simultaneously corresponding the corrected two-dimensional points with three-dimensional template points one-to-one according to a preset point mapping rule, a stable two-dimensional-three-dimensional correspondence is established. Furthermore, by introducing a variance scalar configured by point pairs from the output of the uncertainty estimation branch in the differentiable PnP iterative solution, and weighting the reprojection error accordingly, the suppression of low-confidence corresponding point pairs and stable convergence of pose calculation are achieved. Furthermore, by setting a trigger condition for the mean reprojection error to exceed a threshold within a consecutive preset number of frames and managing difficult samples in a circular buffer manner, and then extracting samples from the difficult sample buffer to form a small dataset to perform incremental updates only on at least some parameters of the correction network and / or uncertainty estimation branch, continuous self-correction under abnormal working conditions and gradual adaptation to cross-vehicle and cross-scene disturbances are achieved, significantly improving the overall solution's positioning consistency, long-term stability, and engineering deployability in complex environments. The technical solution of this invention achieves stable and reliable output of the charging port position and improves the success rate of automatic charging alignment and insertion.

[0019] A second aspect of the present invention provides a charging port identification and positioning system based on multimodal sensing, comprising: The data acquisition and registration module is used to acquire a binocular visible light image of the target vehicle charging area and a depth map acquired by an infrared depth sensor; based on a pre-calibrated extrinsic parameter relationship, the depth map and the binocular visible light image are synchronized and registered at the pixel level to obtain multimodal fused RGBD data with color image and depth information alignment; The target detection module is used to input the visible light image in the RGBD data into the target detection model with an embedded coordinate attention mechanism, and output the candidate detection box of the charging port; The correction calculation module is used to extract local depth features corresponding to the candidate detection box from the RGBD data, and input the geometric parameters of the candidate detection box and the local depth features into the correction network to output a two-dimensional correction offset. The pose solving module is used to determine at least three two-dimensional feature points of the charging port based on the candidate detection box, and to correct the two-dimensional feature points based on the two-dimensional correction offset; it calls a three-dimensional template point set that matches the charging port structure, establishes a two-dimensional-three-dimensional correspondence between the corrected two-dimensional feature points and the three-dimensional template points, and inputs the two-dimensional-three-dimensional correspondence into a differentiable PnP solver to output the rotation and translation parameters of the charging port relative to the camera coordinate system; wherein, the differentiable PnP solver performs iterative solving based on variance-weighted reprojection error, and the variance is output by the uncertainty estimation branch; The optimization and update module is used to calculate the reprojection error based on the rotation parameters and translation parameters; when the reprojection error meets the preset triggering conditions, the corresponding sample is written into the hard sample buffer, and incremental updates are performed on at least some parameters of the correction network and / or the uncertainty estimation branch based on the hard sample buffer.

[0020] A third aspect of the present invention provides a charging port identification and positioning device based on multimodal perception, characterized in that the device includes a computer device, the computer device includes a processor and a memory, the processor stores computer instructions, and when the computer instructions are executed, the device implements the charging port identification and positioning method based on multimodal perception.

[0021] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention will become apparent from the description and the accompanying drawings.

[0022] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] In the attached diagram: Figure 1 This is a flowchart illustrating a charging port identification and positioning method based on multimodal sensing, provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of a charging port identification and positioning system based on multimodal sensing, provided as an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0027] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0028] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0029] Figure 1 This is a flowchart illustrating a charging port identification and positioning method based on multimodal sensing, provided in an embodiment of the present invention.

[0030] Example 1, such as Figure 1 As shown, this invention provides a charging port identification and positioning method based on multimodal sensing, the method comprising the following steps: S1. Acquire a binocular visible light image and a depth map of the target vehicle's charging area. Based on a pre-calibrated external parameter relationship, synchronize and pixel-level register the depth map and the binocular visible light image to obtain multimodal fused RGBD data with color image and depth information alignment. S2. Input the visible light image in the RGBD data into the target detection model with an embedded coordinate attention mechanism, and output the candidate detection box of the charging port; S3. Extract the local depth features corresponding to the candidate detection box from the RGBD data, input the geometric parameters of the candidate detection box and the local depth features into the correction network, and output the two-dimensional correction offset. S4. Based on the candidate detection box, determine at least three two-dimensional feature points of the charging port, and correct the at least three two-dimensional feature points based on the two-dimensional correction offset; call the three-dimensional template point set that matches the charging port structure, establish the correspondence between the corrected two-dimensional feature points and the three-dimensional template points, and solve the rotation matrix and translation vector of the charging port relative to the camera coordinate system using the PnP algorithm according to the correspondence, and output the rotation matrix and translation vector as the pose of the charging port; wherein, the PnP algorithm performs iterative solution based on variance-weighted reprojection error, and the variance is output by the uncertainty estimation branch; S5. Calculate the reprojection error based on the rotation matrix and translation vector. When the reprojection error meets the preset triggering condition, write the corresponding sample into the hard sample buffer. Perform incremental updates on at least some parameters of the correction network and / or the uncertainty estimation branch based on the hard sample buffer.

[0031] Using the above scheme, the charging port identification and localization method based on multimodal perception of the present invention achieves stable perception input of the charging area under complex lighting and occlusion conditions by performing synchronous and pixel-level registration of binocular visible light images and infrared depth maps to form aligned RGBD data; by embedding a target detection model with a coordinate attention mechanism into the visible light image input to output candidate detection boxes for the charging port, robust detection of small targets and background interference of the charging port is achieved; by extracting the local depth features corresponding to the candidate detection boxes and combining them with the geometric parameters of the detection boxes to generate a two-dimensional correction offset, adaptive correction of the localization deviation of the two-dimensional feature points is achieved. Furthermore, by establishing a two-dimensional-three-dimensional correspondence between the corrected two-dimensional feature points and the three-dimensional template points, and inputting the rotation and translation parameters into a differentiable PnP solver, end-to-end calculation of the charging port pose is achieved. Even further, by weighting the reprojection error with the variance output from the uncertainty estimation branch, and incrementally updating at least some parameters of the correction network and / or the uncertainty estimation branch based on a hard sample buffer when the error meets the trigger condition, suppression of noise points and abnormal samples and continuous adaptation across vehicle models and environments are achieved, significantly improving the accuracy, stability, and real-time reliability of charging port identification and positioning. The technical solution of this invention achieves highly robust output of the charging port pose, improving the success rate of automatic charging alignment and insertion.

[0032] In some embodiments of the present invention, the binocular visible light image and depth map distribution are obtained by the binocular visible light camera and the infrared depth sensor, and the binocular visible light camera and the infrared depth sensor are rigidly fixed together, and the external parameter relationship between them is obtained through joint calibration; distortion correction and stereo correction are performed on the binocular visible light image, and filtering and completion processing are performed on the depth map to generate the aligned RGBD data.

[0033] In some embodiments of the present invention, the synchronization and pixel-level registration includes: performing frame-level alignment between the binocular visible light image and the depth map based on the acquisition timestamp, and reprojecting the depth map onto the pixel coordinate system of the binocular visible light image based on the extrinsic parameter relationship to obtain a depth value corresponding to each pixel of the binocular visible light image.

[0034] In embodiments of this invention, a solid-state LiDAR integrating RGB and infrared sensing is used, or a system consisting of two industrial cameras and an infrared depth sensor rigidly fixed in place. The cameras are mounted at the end of a charging robot arm or behind the front bumper of a vehicle, with the center of their field of view aligned with the area where the expected charging port will appear. The baseline distance between the binocular RGB cameras is preferably 10-15 cm to ensure sufficient parallax perception accuracy. All sensors are hardware synchronized via trigger lines to ensure consistent data acquisition timestamps. The NVIDIA Jetson AGX Orin embedded AI platform is used as the core computing unit, with its built-in GPU used to run deep learning models.

[0035] The intrinsic parameters and distortion coefficients of the stereo RGB camera were calibrated using Zhang Zhengyou's calibration method, and the rotation matrix and translation vector of the right camera relative to the left camera were obtained through stereo calibration. The RGB camera and the ToF camera were jointly calibrated using a calibration board provided by the manufacturer to obtain their extrinsic parameters, achieving pixel-level spatial coordinate alignment.

[0036] The hardware system was used to collect charging port image data under various environments, including sunny, rainy, dusky, low light, and partially occluded conditions, for network training and validation.

[0037] In some embodiments of the present invention, the target detection model with embedded coordinate attention mechanism is a lightweight YOLO detection network, wherein the coordinate attention mechanism is inserted after the backbone feature extraction layer and / or before the regression branch of the lightweight YOLO detection network.

[0038] In this embodiment of the invention, YOLOv8n (nano version) is used as the baseline model for improvement. Specifically, coordinate attention mechanisms are inserted after the C2f module of its backbone and before the regression branch of its head. The model is initialized using weights pre-trained on the COCO dataset, and then fine-tuned on a self-built dataset of new energy vehicle charging ports. During training, the input image size is set to 640x640, the batch size is 32, the SGD optimizer is used, the initial learning rate is 0.01, and the training lasts for 100 epochs.

[0039] In some embodiments of the present invention, the correction network is a multilayer perceptron (MLP), the geometric parameters of the candidate detection box include at least normalized center coordinates, width and height, the local depth features include depth statistics sampled within the corresponding region of the candidate detection box, and the two-dimensional correction offset output by the correction network.

[0040] This module is a simple multilayer perceptron (MLP), and its structure is as follows: Input layer: Normalized center coordinates of 7 bounding boxes ( x i , y i ),width w i ,high h i (Total 28 dimensions), and the average depth value obtained from bilinear interpolation sampling of the corresponding region. d i (7 dimensions), totaling 35 dimensions.

[0041] Hidden layers: two fully connected layers with 64 and 32 neurons respectively, each followed by a LayerNorm layer and a ReLU activation function.

[0042] Output layer: A 14-dimensional output layer (corresponding to an offset Δx of 7 points) i , Δy i The Tanh activation function is used to limit the output to the range of (-1, 1), corresponding to a correction of ±1 pixel.

[0043] This module is trained jointly with the YOLO network, and its loss is driven by backpropagation of the subsequent total loss function.

[0044] In some embodiments of the present invention, the at least three two-dimensional feature points include: the center point of the candidate detection box and at least two boundary points determined based on the preset boundary position of the candidate detection box; the two-dimensional correction offset is superimposed on the pixel coordinates of the center point and the boundary points respectively to obtain the corrected two-dimensional feature points.

[0045] In some embodiments of the present invention, the three-dimensional template point set includes template points that correspond one-to-one with the corrected two-dimensional feature points, and the correspondence between the corrected two-dimensional feature points and the three-dimensional template points is established based on a preset point mapping rule.

[0046] In some embodiments of the present invention, when the PnP algorithm iteratively solves the rotation matrix and the translation vector, it introduces the variance parameter output by the uncertainty estimation branch to each corresponding point pair, and weights the reprojection error based on the variance parameter; wherein, the variance parameter includes variance scalars corresponding to different corresponding point pairs.

[0047] The `solve_pnp_differentiable` function from the Kornia library is used as the differentiable PnP solver. Its inputs are a corrected 2D point set (7 points) and a predefined 3D template point set (unit: meters). The 3D template points are precisely measured according to the national standard charging port dimensions. The solver outputs a rotation matrix. R Translation vector t Simultaneously, a parallel uncertainty estimation branch (consisting of three fully connected layers) receives features from corresponding 2D-3D points and outputs a variance scalar σ. i 2 , used to characterize the uncertainty of each corresponding point.

[0048] In some embodiments of the present invention, the preset triggering conditions include: the mean of the reprojection error is greater than a preset threshold within a consecutive preset number of frames; the hard sample buffer is a circular buffer; the incremental update includes: extracting samples from the hard sample buffer to form a small dataset, and performing incremental updates on at least some parameters of the correction network and / or the uncertainty estimation branch.

[0049] To improve noise resistance, an uncertainty estimation branch can be added to the differentiable PnP iterative solution to predict the uncertainty variance σ² at each point, and a weighted loss function can be used to further enhance noise resistance.

[0050] Specifically, in one embodiment, an uncertainty estimation branch is set in the model used for charging port identification and localization. This uncertainty estimation branch shares a feature extraction layer with the target detection model and / or correction network, and outputs variance parameters σ² corresponding to the two-dimensional feature points of the charging port based on the shared features. The variance parameter is used to characterize the confidence level of the localization result of the corresponding two-dimensional feature point, wherein the larger the variance value, the higher the possibility that the feature point is affected by factors such as noise, reflection, occlusion, or depth loss.

[0051] When performing the incremental update, a dynamic total loss function is constructed, which includes a reprojection loss term, a geometric constraint loss term, and an uncertainty regularization term. The loss term is weighted based on dynamic weights to update at least some parameters of the correction network and / or the uncertainty estimation branch.

[0052] Reprojection loss measures the error between the predicted pose (projecting 3D template points onto the image plane) and the corrected detection center. To give greater weight to samples that are difficult to regress, a Focal modulation factor is introduced. The reprojection loss can be expressed as:

[0053] in, L reproj The reprojection loss term is represented by IoU, which is the intersection-union ratio. The error term includes the intersection, center distance, and width-to-height difference. and To adjust the parameters.

[0054] Geometric constraint loss:

[0055] Among them, L geo For geometric constraint loss; To predict the Euclidean distance between any two points ij in the template; Any two points in the real target template ij The Euclidean distance between them.

[0056] The uncertainty regularization term is the logarithmic parameter of the variance output by the uncertainty estimation branch. The optimization is achieved through adaptive weighting using the following formula to reduce the impact of noise points on the optimization. The specific formula for the uncertainty regularization term is expressed as follows:

[0057] Finally, the total loss function is the weighted sum of the parts: L total = w 1 L reproj + w 2 L geo + w 3 L var In the formula, L total The dynamic total loss function is derived from the reprojection loss. L reproj Geometric constraint loss Lgeo Uncertainty regularization term L var We get the result by weighted summation; w 1 ，w 2 ，w3 represents the weight parameters, which are dynamically calculated and adaptively adjusted in each training phase based on the current gradient magnitude of each loss term. For example, when the reprojection error... L reproj When the gradient is large, its weight w The value will automatically increase by 1, forcing the network to allocate more resources to the task.

[0058] After deployment using the technical solution of this invention, it operates according to the following logic: 1) Normal operation: The sensors collect data, the model performs reasoning, and outputs the pose result to the robotic arm control system.

[0059] 2) Monitoring and triggering: If the average reprojection error of the current frame is greater than the set threshold (2.5 pixels according to the experiment), then the RGB image, depth map, detection result and calculated reprojection error of the frame are stored in a circular buffer (FIFO, capacity of 100 frames).

[0060] 3) Fine-tuning learning: When the buffer is full, the system automatically pauses the charging task (if safety permits) and retrieves data from the buffer to form a small dataset. Using a small learning rate (e.g., 0.0001) and the Adam optimizer, fine-tuning is performed only on the uncertainty branch of the post-processing correction module and the pose estimation network for 50 iterations to avoid catastrophic forgetting.

[0061] 4) Model Update: The original parameters are overwritten with the fine-tuned model parameters, the buffer is cleared, and the system resumes normal operation. This process achieves adaptive learning in response to environmental changes.

[0062] Compared with the prior art, the beneficial effects of the present invention are as follows: This application constructs a multimodal perception system composed of binocular vision and infrared sensors, and performs initial detection using a lightweight YOLO network incorporating a coordinate attention mechanism. A learnable post-processing correction module is designed to perform sub-pixel-level accuracy offset correction on the detection box center. An end-to-end pose solving network based on differentiable PnP and uncertainty estimation is proposed to replace the traditional Homography decomposition algorithm, directly regressing rotation and translation vectors and avoiding error accumulation during the decomposition process. A dynamically weighted multi-task loss function is employed to adaptively adjust the weights of reprojection error, geometric constraints, and uncertainty regularization terms according to the training state. An online calibration module based on real-time feedback is integrated to continuously optimize model parameters in practical applications, forming a closed-loop optimization system. The technical solution of this invention can achieve high-precision and robust identification and localization of charging ports for multiple vehicle models in complex scenarios such as changing lighting and partial occlusion, significantly improving the success rate and safety of autonomous driving charging processes.

[0063] Figure 2 This is a flowchart illustrating a charging port identification and positioning system based on multimodal sensing, provided in an embodiment of the present invention.

[0064] Example 2, as Figure 2 As shown, the present invention also provides a charging port identification and positioning system based on multimodal perception, including: a data acquisition and registration module S11, a target detection module S12, a correction calculation module S13, and a pose solving module S14.

[0065] The data acquisition and registration module is used to acquire binocular visible light images and depth maps of the target vehicle charging area. Based on the pre-calibrated extrinsic parameter relationship, the depth map and the binocular visible light image are synchronized and registered at the pixel level to obtain multimodal fused RGBD data with color image and depth information alignment. The target detection module is used to input the visible light image in the RGBD data into the target detection model with an embedded coordinate attention mechanism, and output the candidate detection box of the charging port; The correction calculation module is used to extract local depth features corresponding to the candidate detection box from the RGBD data, input the geometric parameters of the candidate detection box and the local depth features into the correction network, and output a two-dimensional correction offset. The pose solving module is used to determine at least three two-dimensional feature points of the charging port based on the candidate detection box, and to correct the two-dimensional feature points based on the two-dimensional correction offset; it calls a three-dimensional template point set that matches the charging port structure, establishes a two-dimensional-three-dimensional correspondence between the corrected two-dimensional feature points and the three-dimensional template points, and inputs the two-dimensional-three-dimensional correspondence into a differentiable PnP solver to output the rotation and translation parameters of the charging port relative to the camera coordinate system; wherein, the differentiable PnP solver performs iterative solving based on variance-weighted reprojection error, and the variance is output by the uncertainty estimation branch; Furthermore, the system also includes an optimization and update module, which is used to calculate the reprojection error based on the rotation parameters and translation parameters; when the reprojection error meets the preset triggering conditions, the corresponding sample is written into the hard sample buffer, and incremental updates are performed on at least some parameters of the correction network and / or the uncertainty estimation branch based on the hard sample buffer.

[0066] The data acquisition and registration module uses a rigidly connected and jointly calibrated binocular RGB camera and infrared depth sensor to synchronously acquire data. Distortion and stereo corrections are performed on the RGB images, and the infrared depth data is filtered and completed, ultimately providing four-channel information (R, G, B, D) for each pixel, laying the foundation for subsequent fusion processing.

[0067] The correction calculation module includes a lightweight detection network based on an attention mechanism and a learnable post-processing correction module with fused depth. The lightweight detection network uses a cropped and lightweight YOLOv8 as its basic detection framework, embedding a coordinate attention mechanism at key locations in the backbone and head. This mechanism can simultaneously capture inter-channel relationships and long-range positional dependencies, allowing the network to focus more on the circular edge and center features of the charging port's circular hole, significantly improving recall and precision under conditions of uneven lighting and cluttered backgrounds. The correction module is a lightweight neural network that takes the center coordinates, width, height, and corresponding depth values ​​of the initial detection box as input, passes through two fully connected layers and a LayerNorm layer, and outputs a sub-pixel level correction offset (Δx, Δy). This correction module, through a data-driven approach, can predict and compensate for systematic biases in the detection system caused by lighting, occlusion, etc.

[0068] The pose estimation module employs a differentiable PnP solver as the core of the pose estimation network. This module takes the corrected 2D-3D point pair correspondence as input and directly regresses the rotation matrix R and translation vector t using an iteratively optimized PnP algorithm. The entire process is fully differentiable, allowing the gradient to propagate directly back from the pose loss to the preceding correction module and feature extraction network, achieving true end-to-end optimization.

[0069] Meanwhile, an additional branch is added to the network to predict the uncertainty variance σ² for each point, using a weighted loss function to improve the system's noise resistance.

[0070] In one embodiment, an uncertainty estimation branch is set in the model used for charging port identification and localization. This uncertainty estimation branch shares a portion of the feature extraction layer with the target detection model and / or correction network, and outputs variance parameters σ² corresponding to the two-dimensional feature points of the charging port based on the shared features. The variance parameter is used to characterize the confidence level of the localization result of the corresponding two-dimensional feature point, wherein the larger the variance value, the higher the possibility that the feature point is affected by factors such as noise, reflection, occlusion, or depth loss.

[0071] In the pose calculation process, the variance parameter σ² of the uncertainty estimation branch output is introduced to weight the reprojection error: a weight coefficient is configured for each pair of two-dimensional-three-dimensional corresponding points, and the weight coefficient is determined according to the corresponding variance parameter, so that the contribution of corresponding point pairs with larger variance to the objective function in the iterative solution is reduced, and the contribution of corresponding point pairs with smaller variance to the objective function is increased; thereby completing the iterative solution of the rotation matrix and translation vector.

[0072] When performing the incremental update, the optimization update module constructs a dynamic total loss function, which includes a reprojection loss term, a geometric constraint loss term, and an uncertainty regularization term. The loss term is weighted based on dynamic weights to update at least some parameters of the correction network and / or the uncertainty estimation branch.

[0073] The dynamic total loss function consists of three parts: reprojection loss, geometric constraint loss, and uncertainty regularization term. The reprojection loss measures the error between the predicted pose (projecting the 3D template points onto the image plane) and the corrected detection center. To give greater weight to samples that are difficult to regress, a Focal modulation factor is introduced. The reprojection loss can be expressed as:

[0074] in, Error terms include intersection, center distance, and width-to-height difference. and To adjust the parameters.

[0075] Geometric constraint loss:

[0076] in, L geo For geometric constraint loss; For any two points in the prediction template ij The Euclidean distance between them; For any two real points ij The Euclidean distance between them.

[0077] The uncertainty regularization term is the logarithmic parameter of the variance output by the uncertainty estimation branch. The optimization is achieved through adaptive weighting using the following formula to reduce the impact of noise points on the optimization. The specific formula for the uncertainty regularization term is expressed as follows:

[0078] Ultimately, the total loss function is the weighted sum of the individual parts: L total = w 1 L reproj + w 2 L geo + w 3 L var In the formula, L total The dynamic total loss function is derived from the reprojection loss.L reproj Geometric constraint loss L geo Uncertainty regularization term L var We get the result by weighted summation; w 1 ，w 2 ，w 3 represents the weight parameters, which are dynamically calculated and adaptively adjusted in each training phase based on the current gradient magnitude of each loss term. For example, when the reprojection error... L reproj When the gradient is large, its weight w The value will automatically increase by 1, forcing the network to allocate more resources to the task.

[0079] During runtime, the system continuously monitors the reprojection error of pose estimation. If the error of a certain frame is found to be consistently higher than a threshold, it is marked as a "hard sample" and stored in a buffer. When the number of samples in the buffer reaches a certain amount, the system automatically initiates a lightweight fine-tuning process, using new data to incrementally learn the model. This enables the system to adapt to new environments and new vehicle models, forming a continuously self-improving closed-loop system.

[0080] Example 3: The present invention also provides a charging port identification and positioning device based on multimodal perception. The device includes a computer device, which includes a processor and a memory. The processor stores computer instructions. When the computer instructions are executed, the device implements the charging port identification and positioning method based on multimodal perception.

[0081] Example 4, as Figure 3 As shown, the present invention also provides an electronic device 100 for implementing a charging port identification and positioning method based on multimodal sensing.

[0082] The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104.

[0083] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the charging port identification and positioning method based on multimodal perception described in the first aspect of the present invention by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0084] The memory 101 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0085] At least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0086] The memory 101 in the electronic device 100 stores multiple instructions to implement a charging port identification and positioning method based on multimodal sensing, and the processor 102 can execute multiple instructions to achieve the following: The system acquires a binocular visible light image and a depth map of the target vehicle's charging area. Based on a pre-calibrated extrinsic parameter relationship, it synchronizes and performs pixel-level registration of the depth map and the binocular visible light image to obtain aligned RGBD data. The visible light image in the RGBD data is input into the target detection model with an embedded coordinate attention mechanism, and the candidate detection box of the charging port is output. Local depth features corresponding to the candidate detection box are extracted from the RGBD data. The geometric parameters of the candidate detection box and the local depth features are input into the correction network, and a two-dimensional correction offset is output. Based on the candidate detection box, at least three two-dimensional feature points of the charging port are determined, and the at least three two-dimensional feature points are corrected based on the two-dimensional correction offset. A three-dimensional template point set matching the charging port structure is called to establish the correspondence between the corrected two-dimensional feature points and the three-dimensional template points. According to the correspondence, the rotation matrix and translation vector of the charging port relative to the camera coordinate system are solved by the PnP algorithm, and the rotation matrix and translation vector are output as the pose of the charging port. The PnP algorithm is iteratively solved based on variance-weighted reprojection error, and the variance is output by the uncertainty estimation branch. The reprojection error is calculated based on the rotation matrix and translation vector. When the reprojection error meets the preset triggering condition, the corresponding sample is written into the hard sample buffer. Based on the hard sample buffer, at least some parameters of the correction network and / or the uncertainty estimation branch are incrementally updated.

[0087] Example 5: If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A charging port identification and positioning method based on multimodal sensing, characterized in that, The method includes: Acquire binocular visible light images and depth maps of the target vehicle's charging area. Based on pre-calibrated extrinsic parameters, synchronize and pixel-level register the depth map and the binocular visible light images to obtain multimodal fusion data with color image and depth information alignment. The visible light image from the multimodal fusion data is input into a target detection model with an embedded coordinate attention mechanism, and candidate detection boxes for the charging port are output. The geometric parameters of the candidate detection box and the local depth features corresponding to the candidate detection box are input into the correction network, and the two-dimensional correction offset is output. Based on the candidate detection box, at least three two-dimensional feature points of the charging port are determined, and the at least three two-dimensional feature points are corrected based on the two-dimensional correction offset. A three-dimensional template point set matching the charging port structure is called to establish the correspondence between the corrected two-dimensional feature points and the three-dimensional template points. Based on the correspondence, the rotation matrix and translation vector of the charging port relative to the camera coordinate system are solved by a pose solving algorithm. The rotation matrix and translation vector are used as the pose output result of the charging port.

2. The charging port identification and positioning method based on multimodal sensing according to claim 1, characterized in that, The binocular visible light image is obtained through a binocular visible light camera, and the depth map is obtained through an infrared depth sensor. The binocular visible light camera and the infrared depth sensor are rigidly fixed together. The extrinsic parameter relationship between the binocular visible light camera and the infrared depth sensor is obtained through joint calibration. Distortion correction and stereo correction are performed on the binocular visible light image, and filtering and completion processing are performed on the depth map to generate the aligned multimodal fusion data.

3. The charging port identification and positioning method based on multimodal sensing according to claim 1, characterized in that, The synchronization and pixel-level registration include: performing frame-level alignment between the binocular visible light image and the depth map based on the acquisition timestamp, and reprojecting the depth map onto the pixel coordinate system of the binocular visible light image based on the extrinsic parameter relationship to obtain the depth value corresponding to each pixel of the binocular visible light image.

4. The charging port identification and positioning method based on multimodal sensing according to claim 1, characterized in that, The target detection model with embedded coordinate attention mechanism is a lightweight YOLO detection network, and the coordinate attention mechanism is inserted after the backbone feature extraction layer and / or before the regression branch of the lightweight YOLO detection network.

5. The charging port identification and positioning method based on multimodal sensing according to claim 1, characterized in that, The correction network is a multilayer perceptron (MLP), and the geometric parameters of the candidate detection box include at least normalized center coordinates, width, and height. The local depth features include depth statistics sampled within the corresponding region of the candidate detection box. The two-dimensional correction offset is output by the correction network.

6. The charging port identification and positioning method based on multimodal sensing according to claim 1, characterized in that, The at least three two-dimensional feature points include: the center point of the candidate detection box and at least two boundary points determined based on the preset boundary position of the candidate detection box; the two-dimensional correction offset is superimposed on the pixel coordinates of the center point and the boundary points respectively to obtain the corrected two-dimensional feature points.

7. The charging port identification and positioning method based on multimodal sensing according to claim 1, characterized in that, The three-dimensional template point set includes template points that correspond one-to-one with the corrected two-dimensional feature points, and the correspondence between the corrected two-dimensional feature points and the three-dimensional template points is established based on a preset point mapping rule.

8. The charging port identification and positioning method based on multimodal sensing according to claim 1, characterized in that, When iteratively solving the rotation matrix and the translation vector, the pose solving algorithm introduces the variance parameter output by the uncertainty estimation branch for each corresponding point pair, and weights the reprojection error based on the variance parameter; wherein, the variance parameter includes variance scalars corresponding to different corresponding point pairs.

9. The charging port identification and positioning method based on multimodal sensing according to claim 1, characterized in that, Also includes: The reprojection error is calculated based on the rotation matrix and translation vector. When the reprojection error meets the preset triggering condition, the corresponding sample is written into the hard sample buffer. Based on the hard sample buffer, at least some parameters of the correction network are incrementally updated. The preset triggering conditions include: the mean value of the reprojection error is greater than a preset threshold within a preset number of consecutive frames; and the hard sample buffer is a circular buffer. The incremental update includes: drawing samples from the hard sample buffer to form a small dataset, and performing incremental updates on at least some parameters of the correction network and / or the uncertainty estimation branch.

10. A charging port identification and positioning device based on multimodal sensing, characterized in that, The device includes a computer device, which includes a processor and a memory. The processor stores computer instructions. When the computer instructions are executed, the device implements the charging port identification and positioning method based on multimodal perception as described in any one of claims 1 to 9.

Images