Charging robot automobile charging port pose measuring method, charging method and charging system

Abstract

The invention discloses a charging robot automobile charging port pose vision measurement method comprising the following steps: 1, controlling a mechanical arm to drive a monocular camera to collect a charging port image, and constructing a charging port image data set; 2, extracting a charging hole contour in each image by adopting an improved Mask R-CNN instance segmentation model; step 3, performing robust ellipse fitting based on each charging hole contour to obtain center coordinates of each charging hole in the two-dimensional image; step 4, establishing a world coordinate system based on charging hole space distribution defined by the automobile charging port standard model, and determining three-dimensional coordinates of each charging hole; and 5, the multi-view two-dimensional center coordinates and the corresponding three-dimensional coordinates are input into the PnP graph optimization model, and the pose of the charging port in the mechanical arm base coordinate system is solved by fusing the re-projection error constraint, the structure prior constraint and the kinematics chain constraint. The invention further discloses a charging robot automobile charging port pose vision measurement charging method and a charging system.

Description

Technical Field

[0001] The present invention belongs to the technical field of new energy vehicles, and specifically relates to a method for measuring the posture of a charging port of a charging robot, a charging method, and a charging system. Background Art

[0002] With the rapid development of new energy vehicles, automated charging is a critical requirement. Precisely connecting charging piles to vehicle charging ports is a key technology for achieving automated charging and intelligent management. 3D position measurement of charging ports is crucial for ensuring precise operation and safe charging of charging robots, directly impacting charging efficiency and user experience.

[0003] Currently, the posture measurement of charging ports mostly relies on multi-camera systems or laser scanning equipment to obtain multi-view or three-dimensional point cloud data. Although these methods can achieve high measurement accuracy, they usually have disadvantages such as high equipment cost, high system complexity, and strict requirements on the installation environment, which limits their widespread application in actual industrial scenarios.

[0004] In contrast, monocular camera systems have attracted much attention due to their low cost and simple structure. However, monocular cameras provide limited two-dimensional image information, and pose estimation based on single-view acquisition is easily affected by factors such as occlusion, illumination variations, and image noise. This results in insufficient measurement accuracy and stability, making it difficult to meet the requirements of high-precision pose measurement. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a charging robot car charging port posture measurement method, charging method and charging system, which integrates multi-view image segmentation and PnP (Perspective-n-Point) graph optimization to achieve high-precision and high-robustness measurement of the charging port posture to meet the needs of intelligent applications such as automatic charging of new energy vehicles.

[0006] In order to achieve the above object, the present invention provides the following technical solutions: A method for visually measuring the posture of a charging robot and a vehicle charging port includes the following steps: Step 1: Build a charging port image dataset Control the robotic arm to drive the monocular camera to collect charging port images under different viewing angles, lighting conditions, and background interference, and construct a charging port image dataset; Step 2: Extract the charging port outline An improved Mask R-CNN instance segmentation model is used to extract the charging hole outline in each image. The improved Mask R-CNN instance segmentation model includes an edge prediction branch on the mask header and weights the feature map through a spatial attention mechanism. Step 3: Extract the center of the charging port Performing robust ellipse fitting based on the contour of each charging hole to obtain the center coordinates of each charging hole in the two-dimensional image; Step 4: Establish the world coordinate system Establish a world coordinate system based on the spatial distribution of charging ports defined by the standard model of automotive charging ports to determine the three-dimensional coordinates of each charging port; Step 5: Solve the position of the car charging port in the coordinate system of the robot arm end effector The two-dimensional center coordinates of multiple perspectives and the corresponding three-dimensional coordinates are input into the PnP graph optimization model, and the posture of the charging port in the robot base coordinate system is solved by integrating the reprojection error constraint, structural prior constraint and kinematic chain constraint. The posture is determined by the rotation matrix and translation vectors express.

[0007] Furthermore, in step 1, before collecting the charging port image, camera calibration and hand-eye calibration are performed, including: Obtaining the camera intrinsic parameter matrix through checkerboard calibration and distortion coefficients : in: and Indicates that the camera is Axis and The focal length in the axial direction; and Indicates the horizontal and vertical coordinates of the camera's optical center; Represent the first-order, second-order and third-order radial distortion respectively; Respectively Direction and Tangential distortion of the direction; Solve the hand-eye transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system through the ArUco code calibration plate : in: is the rotation matrix; is the translation vector; Then the transformation relationship between the camera coordinate system and the robot arm end effector coordinate system is: in: Represents the pose transformation of the robot end in the world coordinate system; Represents the transformation of the camera's observation object from one frame to the next.

[0008] Furthermore, in step 2, the following image enhancement strategy is adopted during training of the improved Mask R-CNN instance segmentation model: Geometric transformation enhancement: Perform geometric transformation operations including random rotation, scaling, translation, and mirroring on the original charging port image to simulate different shooting angles and relative postures; Color enhancement: Performs color enhancement operations on the original charging port image, including brightness adjustment, contrast adjustment, and color perturbation, to simulate the diversity of natural lighting and industrial scene light sources. Noise perturbation: Add noise perturbations including Gaussian noise, motion blur, and compression artifacts to the original charging port image to enhance the model's robustness to low-quality images; Occlusion simulation: Occlusion mapping is performed on local areas of the original charging port image to improve the model's segmentation ability under incomplete visual information.

[0009] Furthermore, in step 2, the method of weighting the feature map by the spatial attention mechanism is: The spatial attention module generates a single-channel attention map through convolution operations : in: Feature map output by the backbone network; is the number of channels, and is the feature map height and width; and 3×3 and 1×1 convolutional layers respectively; Is the Sigmoid function, used to normalize the attention value to , indicating the importance of each spatial position; Attention map With the original feature map Multiply element by element to generate a weighted feature map .

[0010] Furthermore, in step 2, the method of adding an edge prediction branch to the mask branch is as follows: a parallel edge prediction branch is added to the mask branch of Mask R-CNN, sharing features with the mask prediction branch: in: Generate an edge prediction map through the convolution layer, which indicates the probability that each pixel belongs to the edge; The edge branch receives the same RoI features as the mask branch; is a sequence of convolutional layers; is the Sigmoid function; and Indicates the height and width of the RoI feature; Define binary cross entropy loss: in: is the binary edge map extracted using the true segmentation mask; Indicates that the true edge map is at position The value of , 1 represents edge, 0 represents non-edge; Indicates the predicted edge map at position The probability value of .

[0011] Furthermore, the improved Mask R-CNN instance segmentation model implements object detection and pixel-level segmentation from the input image through a feature extraction backbone network, a region proposal network, and classification, bounding box regression, and mask prediction branches. Its total loss function is: in: is the classification loss, is the bounding box regression loss, is the mask loss.

[0012] Furthermore, in step three, the RANSAC algorithm, the least squares algorithm, or the image moment algorithm is used to perform robust fitting on the contour points to extract the center of the charging hole; wherein the method steps for performing robust fitting on the contour points and extracting the center of the charging hole using the RANSAC algorithm are as follows: 31) Single binary mask map based on segmentation output , combined with opening and closing operations to remove noise points and pseudo contours; 32) Use the Canny edge detection algorithm to extract the contour boundary and obtain the boundary point set : in: Represents the first Pixel coordinates; 33) Based on the general expression of the quadratic curve of the ellipse, fit the boundary point set C into an elliptic curve: The RANSAC algorithm is introduced to perform robust fitting on the contour points to construct the optimal ellipse model, including: 331) Randomly from the boundary point set Select the minimum sample subset and fit the ellipse model based on the selected points , calculate the algebraic distance residuals from all points to the fitted ellipse: 332) After each fitting is completed, determine whether each point in the minimum sample subset belongs to the interior point of the ellipse model, and calculate the proportion of interior points to all data points and update iterations : in: represents the expected probability of success; If the proportion of inliers in the current iteration If the ratio of the inner points exceeds the current optimal one, the current ellipse model is updated to the optimal ellipse model; 333) Loop through steps 331)-332) until the number of iterations reaches , get the optimal ellipse model; 34) Based on the optimal ellipse model, obtain the coordinates of the center point of the ellipse image: in: and Indicates the coordinate value of the center point of the image; 35) Execute steps 31) to 34) for each mask to realize the parallel extraction of multiple charging hole center points; for any frame image , the extracted two-dimensional charging hole center point set is: in: For images Middle The coordinates of the center point of each charging port.

[0013] Furthermore, in step 5, the method for solving the position and posture of the vehicle charging port in the coordinate system of the robot arm end effector includes the following steps: 51) Two-dimensional feature points extracted for each frame of image , and the three-dimensional feature points established in step 4 Establish a corresponding relationship: Among them: function To include camera intrinsic parameters and distortion parameters Pinhole projection model; 52) For each frame of image , use the PnP algorithm to quickly solve the initial pose of the target relative to the camera coordinate system: in: For images The rotation matrix of For images The translation vector of 53) Using the posture of the end effector of the robotic arm and hand-eye calibration results , transform the initial pose to the base coordinate system: in: For the The pose transformation matrix of the charging port target relative to the robot arm base coordinate system obtained by the PnP algorithm in the frame image; 54) Introduce graph optimization methods to jointly optimize the target global pose and construct the optimization objective function: in: is the number of observed images; For the The number of feature points in the frame; It is a robust kernel function that suppresses the influence of external points; is the covariance matrix, which represents the weight distribution strategy; and is an adaptive weight factor, which is dynamically adjusted according to the optimization stage; is the reprojection error constraint; is a structural prior constraint; is a kinematic chain constraint; and: The reprojection error constraint is: in: Indicates the Frame image Image coordinates of the center point of each charging hole; Indicates the actual charging port The three-dimensional feature point coordinates of the center of the charging port; Indicates the The pose transformation matrix of the charging port target relative to the camera obtained by transforming in the globally consistent pose in the frame image; and: The structural prior constraints are: in: Indicates the charging port number defined based on the national standard. The three-dimensional feature point coordinates of the center of the charging port; The kinematic chain constraints are: in: Represents a globally consistent target pose; 55) The Levenberg-Marquardt algorithm is used for iterative optimization to obtain the precise position of the target in the manipulator base coordinate system.

[0014] The present invention also proposes a charging method for measuring the posture of a charging robot and a charging port of a vehicle, comprising the following steps: S1: Use the above-mentioned visual measurement method of the charging robot's car charging port to obtain the precise position and pose of the target in the robot arm's base coordinate system; S2: Execute automatic charging tasks, including: S21: Based on the target pose, the inverse kinematics solver is used to calculate the required joint angle solution of the manipulator, and the interpolation trajectory is generated in combination with the trajectory planning algorithm to ensure that the manipulator reaches the target position without interference and vibration; S22: After confirming that the charging gun is successfully plugged into the charging port and the physical connection is stable, the charging process is started; S23: Record the current charging task execution data, and after charging is completed, control the charging gun to detach from the charging port through the robotic arm; S24: The robotic arm returns to its initial standby position to prepare for the next charging task.

[0015] The present invention also proposes a charging robot and a vehicle charging port posture measurement charging system, which is used to implement the above-mentioned charging robot and a vehicle charging port posture visual measurement method, and is characterized by comprising: A robotic arm equipped with an end effector and a monocular camera; An image processing module configured to perform improved Mask R-CNN segmentation and ellipse center extraction; A pose optimization module configured to construct and solve a PnP graph optimization model; The control module generates the motion trajectory of the robotic arm based on the optimized posture, starts the charging process after the physical connection between the charging gun and the charging port is stable, and controls the charging gun to detach from the charging port through the robotic arm after charging is completed, and controls the robotic arm to reset to the initial standby position.

[0016] The beneficial effects of the present invention are: The present invention provides a method for visually measuring the posture of a charging port of a charging robot car. It adopts a structure driven by a monocular camera and a robotic arm, combined with deep learning image segmentation and a multi-view PnP graph optimization algorithm. Compared with existing measurement methods, it significantly reduces the system cost and complexity, while improving the accuracy and robustness of posture measurement. The method includes: using a structure in which a monocular camera is fixed at the end of a robotic arm, and using the controllable movement of the robotic arm to collect images of the charging port at multiple angles, replacing traditional multi-camera or laser measurement systems, reducing system hardware costs and deployment difficulty; introducing a deep learning image segmentation algorithm to accurately extract the boundary information of multiple charging holes, and obtain their two-dimensional image coordinates through robust ellipse fitting; using the prior three-dimensional position information of the charging hole in the world coordinate system to construct 2D-3D point correspondences under multiple perspectives; performing posture estimation and optimization fusion based on multi-view observations, significantly improving the posture estimation accuracy and robustness of the monocular camera system; realizing high-precision, stable, and low-cost three-dimensional posture measurement of the charging port of a new energy vehicle in a complex environment, and providing technical support for robotic arm visual guidance and automatic charging.

[0017] In summary, the method of the present invention combines hardware simplicity, algorithm accuracy and engineering practicality, can adapt to a variety of application scenarios, and achieve high-precision and high-robustness measurement of the charging port posture. It has good promotion and application prospects, promotes the intelligence and automation of the charging process of new energy vehicles, and is especially suitable for the demand for accurate acquisition of the charging interface posture in the field of automatic charging of new energy vehicles, and has broad promotion and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention provides the following drawings for illustration: Figure 1 This is a flow chart of the visual measurement method for the charging port of a charging robot of the present invention; Figure 2 is the charging hole geometry prior; Figure 3 A flowchart for optimizing the position of the car charging port in the robot arm end effector coordinate system based on the PnP graph; Figure 4 This is a flow chart of the charging method for visually measuring the posture of a charging robot and a car charging port of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0020] like Figure 1 As shown, the visual measurement method of the charging robot vehicle charging port posture in this embodiment includes the following steps.

[0021] Step 1: Build a charging port image dataset The robotic arm is controlled to drive the monocular camera to collect charging port images under different viewing angles, lighting conditions, and background interference, and a charging port image dataset is constructed.

[0022] (1) Camera calibration and hand-eye calibration Before capturing images of the charging port, perform camera and hand-eye calibration. Ensure that the camera and end effector (such as a charging gun) are fixed to the end of the robotic arm and maintain their relative positions.

[0023] Prepare a standard plane chessboard as a calibration board, place the chessboard in multiple positions, and ensure that the camera takes multiple images containing the chessboard from different angles and distances. To ensure the diversity of the calibration data, 15 to 20 images should be taken, and the relative position of the camera and the chessboard should be different each time. Through the Zhang Zhengyou calibration method, the camera's intrinsic parameter matrix K and distortion coefficient D are calculated using the corner point data of multiple images and the known geometric dimensions of the chessboard. That is, in this embodiment, the camera's intrinsic parameter matrix is ​​obtained by chessboard calibration. and distortion coefficients : in: and Indicates that the camera is Axis and The focal length in the axial direction; and Indicates the horizontal and vertical coordinates of the camera's optical center; Represent the first-order, second-order and third-order radial distortion respectively; Respectively Direction and Tangential distortion in the direction.

[0024] This embodiment uses an eye-on-hand installation method. After camera calibration, hand-eye calibration is performed to solve the transformation relationship between the camera coordinate system and the robot arm end-effector coordinate system (End-Effector Frame, EE coordinate system), thereby accurately linking the image captured by the camera with the movement of the robot arm. Specifically, the transformation relationship between the camera coordinate system and the robot arm end-effector coordinate system is: in: Represents the pose transformation of the robot end in the world coordinate system; Represents the transformation of the camera's observation object from one frame to the next. The hand-eye transformation matrix from the camera coordinate system to the robot end effector coordinate system. The goal of hand-eye calibration is to solve the unknown transformation matrix .

[0025] In traditional hand-eye calibration, a checkerboard is usually used as a calibration board. However, in this embodiment, in order to improve the robustness and automation of corner detection, ArUco code (i.e., a binary matrix QR code with a unique identifier) ​​is used as a calibration auxiliary tool. The ArUco code calibration board printed with known size and posture is placed in a fixed position so that it can be fully recognized by the camera field of view fixed at the end of the robotic arm. The robotic arm is controlled to move to multiple postures (no less than 10 groups), and the camera takes the ArUco code image. In each group of postures: the posture of the ArUco code is extracted using the camera image as ; At the same time, obtain the current position transformation of the end of the manipulator (from the robot controller) for construction .

[0026] Using multiple pose pairs provided by ArUco code , use the common Tsai-Lenz algorithm to solve. The Tsai-Lenz method decouples the rotation and translation parts and first solves the rotation matrix , and then substitute the translation vector Finally, the complete hand-eye transformation matrix is ​​obtained. That is, this embodiment uses the ArUco code calibration plate to solve the hand-eye transformation matrix from the camera coordinate system to the robot arm end effector coordinate system. : in: is the rotation matrix; is the translation vector.

[0027] Once calibration is complete, the spatial information captured by the camera in the image can be converted to the coordinate system of the robotic arm's end effector using the hand-eye transformation matrix, enabling coordinated control of the camera and robotic arm's motion. This process provides the coordinate foundation and geometric guarantee for subsequent image-guided precise positioning and control.

[0028] Step 2: Extract the charging port outline An improved Mask R-CNN instance segmentation model is used to extract the charging hole contours in each image. The improved Mask R-CNN instance segmentation model includes adding an edge prediction branch to the mask header and weighting the feature map through a spatial attention mechanism.

[0029] (1) Charging port dataset construction and image enhancement To improve the accuracy and robustness of the image segmentation model for recognizing charging ports of new energy vehicles, this embodiment first constructs an image dataset dedicated to charging port recognition and performs multiple enhancement processes on the original images to ensure that the model still has good generalization capabilities in complex environments.

[0030] A robotic arm controls a monocular camera to capture image data of new energy vehicle charging ports from multiple angles, under varying lighting conditions, and with varying background interference, ensuring a diverse image of the charging ports' orientation, proportions, and degree of occlusion. During the acquisition process, the robotic arm maintains a viewing angle of at least 60°.

[0031] Use open-source annotation tools (such as LabelMe) to accurately annotate the charging port outlines in the captured images with polygons, generating standard segmentation mask labels. To improve annotation consistency, develop unified charging port boundary determination rules and introduce a manual review process to ensure label quality.

[0032] In response to illumination changes, background interference, image blur, and other situations in actual applications, the improved Mask R-CNN instance segmentation model of this embodiment uses the following image enhancement strategies during training to expand the dataset size and improve the robustness of the model: Geometric transformation enhancement: Perform geometric transformation operations including random rotation, scaling, translation, and mirroring on the original charging port image to simulate different shooting angles and relative postures; Color enhancement: Performs color enhancement operations on the original charging port image, including brightness adjustment, contrast adjustment, and color perturbation, to simulate the diversity of natural lighting and industrial scene light sources. Noise perturbation: Add noise perturbations including Gaussian noise, motion blur, and compression artifacts to the original charging port image to enhance the model's robustness to low-quality images; Occlusion simulation: Occlusion mapping is performed on local areas of the original charging port image to improve the model's segmentation ability under incomplete visual information.

[0033] After the construction is completed, the dataset is divided into training set, validation set and test set in a ratio of 8:1:1 to ensure that the performance evaluation during the model training process is representative and stable.

[0034] (2) Extracting the charging hole contour based on the improved Mask R-CNN algorithm Mask R-CNN is a deep learning instance segmentation model that implements object detection and pixel-level segmentation from input images through a feature extraction backbone network (ResNet+FPN), a region proposal network (RPN), and classification, bounding box regression, and mask prediction branches. In this embodiment, Mask R-CNN is used to extract the contours of the charging hole from images captured by a monocular camera. However, the standard Mask R-CNN may have insufficient boundary extraction accuracy in complex industrial scenes due to background noise, occlusion, and illumination changes. This embodiment introduces a spatial attention mechanism and an edge supervision module based on the traditional Mask R-CNN instance segmentation framework, and improves the loss function to achieve a more refined and stable segmentation of the charging hole contour.

[0035] a. Weighted feature maps via spatial attention mechanism In order to enhance the model's attention to the charging hole features, the spatial attention mechanism is introduced to improve the representation ability of key areas by generating weighted features of the attention map. Assume that the feature map output by the backbone network is ,in is the number of channels, and is the height and width of the feature map. The spatial attention module generates a single-channel attention map through convolution operation. : in: Feature map output by the backbone network; is the number of channels, and is the feature map height and width; and 3×3 and 1×1 convolutional layers respectively; Is the Sigmoid function, used to normalize the attention value to , indicating the importance of each spatial position.

[0036] Attention map With the original feature map Multiply element by element to generate a weighted feature map , weighted The feature expression of the charging hole area is enhanced, while background noise is suppressed, improving the accuracy of region proposal and mask prediction.

[0037] b. Add an edge prediction branch at the pier To enhance the model's ability to perceive the boundaries of charging holes, an edge supervision module is introduced. This module provides additional supervision of boundaries during training, generating smoother and more accurate contours. In the Mask Head of Mask R-CNN, a parallel edge prediction branch is added, sharing features with the mask prediction branch. The edge branch receives the same RoI features as the mask branch. , generate edge prediction map through convolution layer : in: Generate an edge prediction map through the convolution layer, which indicates the probability that each pixel belongs to the edge; The edge branch receives the same RoI features as the mask branch; is a sequence of convolutional layers; is the Sigmoid function; and Indicates the height and width of the RoI feature.

[0038] Binary edge map extracted using the true segmentation mask , define the binary cross entropy loss: in: is the binary edge map extracted using the true segmentation mask; Indicates that the true edge map is at position The value of , 1 represents edge, 0 represents non-edge; Indicates the predicted edge map at position The probability value of .

[0039] Furthermore, the improved Mask R-CNN instance segmentation model implements object detection and pixel-level segmentation from the input image through the feature extraction backbone network, region proposal network, and classification, bounding box regression, and mask prediction branches. Its total loss function is: in: is the classification loss, is the bounding box regression loss, is the mask loss.

[0040] The enhanced new energy vehicle charging port image dataset is fed into the improved Mask R-CNN algorithm and trained until convergence. In the actual pose estimation process, the converged model is used to segment and obtain several charging port mask images.

[0041] Step 3: Extract the center of the charging port Based on the contour of each charging hole, robust ellipse fitting is performed to obtain the center coordinates of each charging hole in the two-dimensional image. Specifically, a RANSAC algorithm, a least squares algorithm, or an image moment algorithm can be used to robustly fit the contour points to extract the charging hole center. In this embodiment, the method steps for robustly fitting the contour points and extracting the charging hole center using the RANSAC algorithm are as follows.

[0042] 31) Single binary mask map based on segmentation output , combined with opening and closing operations to remove noise points and pseudo contours.

[0043] 32) Use the Canny edge detection algorithm to extract the contour boundary and obtain the boundary point set : in: Represents the first Pixel coordinates.

[0044] 33) Based on the general expression of the quadratic curve of the ellipse, fit the boundary point set C into an elliptic curve: In order to prevent pseudo boundary points in the segmentation mask from interfering with the fitting accuracy, this embodiment introduces the RANSAC algorithm to perform robust fitting on the contour points to construct an optimal ellipse model, including steps 331)-333).

[0045] 331) Randomly from the boundary point set Select the minimum sample subset (5 points) and fit the ellipse model based on the selected points , calculate the algebraic distance residuals from all points to the fitted ellipse: 332) After each fitting is completed, determine whether each point in the minimum sample subset belongs to the interior point of the ellipse model by giving a threshold ,when When , the point is considered to be an interior point of the fitted ellipse model; when When , the point is considered to be an external point that does not conform to the fitted ellipse model.

[0046] Calculate the proportion of inliers to all data points and update iterations : in: Indicates the expected success probability, which is 99% in this embodiment.

[0047] Each time 5 points are randomly selected for ellipse fitting, if the ratio of the inner points in this iteration is If the inlier ratio exceeds the current optimal model, the optimal model is updated.

[0048] 333) Loop through steps 331)-332) until the number of iterations reaches , and obtain the optimal ellipse model.

[0049] 34) Based on the optimal ellipse model, obtain the coordinates of the center point of the ellipse image: in: and Indicates the coordinate value of the center point of the ellipse image.

[0050] 35) Execute steps 31) to 34) for each mask to realize the parallel extraction of multiple charging hole center points; for any frame image , the extracted two-dimensional charging hole center point set.

[0051] Specifically, when there are multiple charging hole instances in the image, each mask is extracted separately through the multi-target detection frame index in the segmentation stage and the above ellipse fitting operation is performed separately to achieve the parallel extraction of multiple charging hole center points. , and obtain the extracted two-dimensional charging hole center point set: in: For images Middle The coordinates of the center point of each charging port.

[0052] Step 4: Establish the world coordinate system A world coordinate system is established based on the spatial distribution of charging ports defined by the standard model of automobile charging ports to determine the three-dimensional coordinates of each charging port.

[0053] To accurately locate the charging port in three-dimensional space and effectively solve the PnP algorithm, the spatial position of each charging port must be clearly defined in the world coordinate system. The physical structure of new energy vehicle charging ports is generally standardized, with highly consistent distribution, size, and relative arrangement of the ports. Therefore, a world coordinate model can be constructed based on the following geometric prior information: ① The three-dimensional center position of each charging port is arranged according to a known layout on the charging port panel; ② The relative spacing between the holes is fixed, and the arrangement relationship between the holes in the x and y directions is known; ③The charging port panel is a local plane and can be approximately modeled as the same z-plane.

[0054] Select the center of the charging port PE hole as the origin of the world coordinate system , the spatial position of this point is recorded as .like Figure 2 As shown, this embodiment takes the jack distribution defined in GB / T 20234.2-2015 as an example. By referring to the jack distribution defined in GB / T 20234.2-2015, the panel plane is selected as plane, with the normal direction set to The positive direction of the axis, establish a right-hand coordinate system. Then we can get the spatial point set of the charging hole. : Step 5: Solve the position of the car charging port in the coordinate system of the robot arm end effector The two-dimensional center coordinates of multiple perspectives and the corresponding three-dimensional coordinates are input into the PnP graph optimization model, and the posture of the charging port in the robot base coordinate system is solved by integrating the reprojection error constraint, structural prior constraint and kinematic chain constraint. The posture is determined by the rotation matrix and translation vectors express.

[0055] To achieve accurate 3D pose estimation of the target charging port, we use image observation data from multiple perspectives to improve the accuracy and robustness of pose estimation. We use a robotic arm to control the camera to capture images of the target charging port from different perspectives, and adopt a surround-type small-angle tilt shooting strategy (tilted 10° to 30° from the front of the target, up and down, left and right), to ensure that all feature points are fully included in each frame of the image, while enhancing geometric redundancy. Specifically, Figure 3 As shown, the method for solving the position and posture of the car charging port in the coordinate system of the robot end effector includes the following steps.

[0056] 51) Two-dimensional feature points extracted for each frame of image , and the three-dimensional feature points established in step 4 Establish a corresponding relationship: Among them: function To include camera intrinsic parameters and distortion parameters Pinhole projection model.

[0057] 52) For each frame of image , use the PnP algorithm to quickly solve the initial pose of the target relative to the camera coordinate system: in: For images The rotation matrix of For images The translation vector of .

[0058] 53) Using the posture of the end effector of the robotic arm (obtained by forward kinematics) and hand-eye calibration results , transform the initial pose to the base coordinate system: in: For the The pose transformation matrix of the charging port target relative to the robot arm base coordinate system obtained by the PnP algorithm in the frame image.

[0059] 54) To further improve the overall accuracy of pose estimation, a graph optimization method is introduced to jointly optimize the target global pose. In this optimization framework, a graph structure is constructed with the target global pose as the core node and the target high-precision three-dimensional feature points as general nodes. The reprojection error of multiple frame images, the charging port structure prior and the robot arm kinematic chain are used as constraint edges to perform global optimization solution.

[0060] Assume there is a globally consistent target pose , initialize it to the weighted result of the pose conversion of multiple frames PnP, which is The expression in the camera coordinate system is: The reprojection error constraint (observation constraint) is: in: Indicates the Frame image Image coordinates of the center point of each charging hole; Indicates the actual charging port The three-dimensional feature point coordinates of the center of the charging port; Indicates the The pose transformation matrix of the charging port target relative to the camera obtained by transforming in a globally consistent pose in the frame image.

[0061] Considering the manufacturing error of the charging port and the aging caused by long-term use, the structural prior constraints are constructed. The prior values ​​are derived from the national standard model: in: Indicates the charging port number defined based on the national standard. The three-dimensional feature point coordinates of the center of the charging hole.

[0062] Considering the consistency between the global pose X and the pose estimated from a single frame, a kinematic chain constraint is constructed: in: represents a globally consistent target pose.

[0063] The final optimization objective function is: in: is the number of observed images; For the The number of feature points in the frame; It is a robust kernel function that suppresses the influence of external points; is the covariance matrix, which represents the weight distribution strategy; and is an adaptive weight factor, which is dynamically adjusted according to the optimization stage; is the reprojection error constraint; is a structural prior constraint; is a kinematic chain constraint.

[0064] 55) Use the Levenberg-Marquardt algorithm for iterative optimization to obtain the precise position of the target in the manipulator base coordinate system: This method effectively overcomes the pathological problem of single-frame PnP (especially when feature points are coplanar) by integrating multi-view visual observations, kinematic chain transfer and structural prior constraints, and significantly improves the accuracy and robustness of pose estimation.

[0065] like Figure 4 As shown, this embodiment also proposes a charging method for measuring the posture of a charging robot and a car charging port, which includes the following steps.

[0066] S1: Use the visual measurement method for the position and posture of the charging port of the charging robot as described above in this embodiment to obtain the precise position and posture of the target in the robot arm base coordinate system.

[0067] S2: After completing the three-dimensional pose optimization solution of the charging port, the final pose result is used as the control input to drive the robotic arm to perform the automatic charging task, including.

[0068] S21: Based on the target pose, the inverse kinematics solver is used to calculate the required joint angle solution of the robot arm, and combined with the trajectory planning algorithm to generate a smooth and safe interpolation trajectory to ensure that the robot arm reaches the target position without interference and vibration.

[0069] S22: After confirming that the charging gun is successfully plugged into the charging port and detecting that the physical connection is stable, the charging process is started.

[0070] S23: Record the current charging task execution data, and control the charging gun to detach from the charging port through the robotic arm after charging is completed.

[0071] S24: The robotic arm returns to its initial standby position to prepare for the next charging task.

[0072] This embodiment also proposes a charging robot vehicle charging port posture measurement charging system, which is used to implement the charging robot vehicle charging port posture visual measurement method described above in this embodiment, including: A robotic arm equipped with an end effector and a monocular camera; An image processing module configured to perform improved Mask R-CNN segmentation and ellipse center extraction; A pose optimization module configured to construct and solve a PnP graph optimization model; The control module generates the motion trajectory of the robotic arm based on the optimized posture, starts the charging process after the physical connection between the charging gun and the charging port is stable, and controls the charging gun to detach from the charging port through the robotic arm after charging is completed, and controls the robotic arm to reset to the initial standby position.

[0073] The above embodiments are merely preferred embodiments for the purpose of fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A visual measurement method for the position and posture of a charging robot and a car charging port, characterized by: The steps include: Step 1: Build a charging port image dataset Control the robotic arm to drive the monocular camera to collect charging port images under different viewing angles, lighting conditions, and background interference, and construct a charging port image dataset; Step 2: Extract the charging port outline The charging hole outlines in each image are extracted using an improved Mask R-CNN instance segmentation model. The improved Mask R-CNN instance segmentation model includes an edge prediction branch on the mask header and weights the feature map through a spatial attention mechanism. Step 3: Extract the center of the charging port Performing robust ellipse fitting based on the contour of each charging hole to obtain the center coordinates of each charging hole in the two-dimensional image; Step 4: Establish the world coordinate system Establish a world coordinate system based on the spatial distribution of charging ports defined by the standard model of automotive charging ports to determine the three-dimensional coordinates of each charging port; Step 5: Solve the position of the car charging port in the coordinate system of the robot arm end effector The two-dimensional center coordinates of multiple perspectives and the corresponding three-dimensional coordinates are input into the PnP graph optimization model, and the posture of the charging port in the robot base coordinate system is solved by integrating the reprojection error constraint, structural prior constraint and kinematic chain constraint. The posture is determined by the rotation matrix and translation vectors express.

2. The method for visually measuring the posture of a charging robot and a vehicle charging port according to claim 1 is characterized in that: In step 1, before collecting the charging port image, camera calibration and hand-eye calibration are performed, including: Obtaining the camera intrinsic parameter matrix through checkerboard calibration and distortion coefficients : in: and Indicates that the camera is Axis and The focal length in the axial direction; and Indicates the horizontal and vertical coordinates of the camera's optical center; Represent the first-order, second-order and third-order radial distortion respectively; Respectively Direction and Tangential distortion of the direction; Solve the hand-eye transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system through the ArUco code calibration plate : in: is the rotation matrix; is the translation vector; Then the transformation relationship between the camera coordinate system and the robot arm end effector coordinate system is: in: Represents the pose transformation of the robot end in the world coordinate system; Represents the transformation of the camera's observation object from one frame to the next.

3. The method for visually measuring the posture of a charging robot and a vehicle charging port according to claim 1 is characterized in that: In step 2, the following image enhancement strategy is adopted during training of the improved Mask R-CNN instance segmentation model: Geometric transformation enhancement: Perform geometric transformation operations including random rotation, scaling, translation, and mirroring on the original charging port image to simulate different shooting angles and relative postures; Color enhancement: Performs color enhancement operations on the original charging port image, including brightness adjustment, contrast adjustment, and color perturbation, to simulate the diversity of natural lighting and industrial scene light sources. Noise perturbation: Add noise perturbations including Gaussian noise, motion blur, and compression artifacts to the original charging port image to enhance the model's robustness to low-quality images; Occlusion simulation: Occlusion mapping is performed on local areas of the original charging port image to improve the model's segmentation ability under incomplete visual information.

4. The method for visually measuring the position and posture of a charging robot and a vehicle charging port according to claim 1 is characterized in that: In step 2, the method of weighting the feature map through the spatial attention mechanism is: The spatial attention module generates a single-channel attention map through convolution operations : in: Feature map output by the backbone network; is the number of channels, and is the feature map height and width; and 3×3 and 1×1 convolutional layers respectively; Is the Sigmoid function, used to normalize the attention value to , indicating the importance of each spatial position; Attention map With the original feature map Multiply element by element to generate a weighted feature map .

5. The method for visually measuring the position and posture of a charging robot and a vehicle charging port according to claim 1 is characterized in that: In step 2, the method of adding an edge prediction branch to the mask branch is to add a parallel edge prediction branch to the mask branch of Mask R-CNN and share features with the mask prediction branch: in: Generate an edge prediction map through the convolution layer, which indicates the probability that each pixel belongs to the edge; The edge branch receives the same RoI features as the mask branch; is a sequence of convolutional layers; is the Sigmoid function; and Indicates the height and width of the RoI feature; Define binary cross entropy loss: in: is the binary edge map extracted using the true segmentation mask; Indicates that the true edge map is at position The value of , 1 represents edge, 0 represents non-edge; Indicates the predicted edge map at position The probability value of .

6. The method for visually measuring the position and posture of a charging robot and a vehicle charging port according to claim 5 is characterized in that: The improved Mask R-CNN instance segmentation model implements object detection and pixel-level segmentation from the input image through a feature extraction backbone network, a region proposal network, and classification, bounding box regression, and mask prediction branches. Its total loss function is: in: is the classification loss, is the bounding box regression loss, is the mask loss.

7. The method for visually measuring the position and posture of a charging robot and a vehicle charging port according to claim 1 is characterized in that: In step three, the RANSAC algorithm, the least squares algorithm, or the image moment algorithm is used to perform robust fitting on the contour points to extract the center of the charging hole. The steps of the method for robust fitting on the contour points and extracting the center of the charging hole using the RANSAC algorithm are as follows: 31) Single binary mask map based on segmentation output , combined with opening and closing operations to remove noise points and pseudo contours; 32) Use the Canny edge detection algorithm to extract the contour boundary and obtain the boundary point set : in: Represents the first Pixel coordinates; 33) Based on the general expression of the quadratic curve of the ellipse, fit the boundary point set C into an elliptic curve: The RANSAC algorithm is introduced to perform robust fitting on the contour points to construct the optimal ellipse model, including: 331) Randomly from the boundary point set Select the minimum sample subset and fit the ellipse model based on the selected points , calculate the algebraic distance residuals from all points to the fitted ellipse: 332) After each fitting is completed, determine whether each point in the minimum sample subset belongs to the interior point of the ellipse model, and calculate the proportion of interior points to all data points and update iterations : in: represents the expected probability of success; If the proportion of inliers in the current iteration If the ratio of the inner points exceeds the current optimal one, the current ellipse model is updated to the optimal ellipse model; 333) Loop through steps 331)-332) until the number of iterations reaches , get the optimal ellipse model; 34) Based on the optimal ellipse model, obtain the coordinates of the center point of the ellipse image: in: and Indicates the coordinate value of the center point of the image; 35) Execute steps 31) to 34) for each mask to realize the parallel extraction of multiple charging hole center points; for any frame of image , the extracted two-dimensional charging hole center point set is: in: For images Middle The coordinates of the center point of each charging port.

8. The method for visually measuring the position and posture of a charging robot and a vehicle charging port according to claim 1 is characterized in that: In step 5, the method for solving the position and posture of the vehicle charging port in the coordinate system of the robot arm end effector includes the following steps: 51) Two-dimensional feature points extracted for each frame of image , and the three-dimensional feature points established in step 4 Establish a corresponding relationship: Among them: function To include camera intrinsic parameters and distortion parameters Pinhole projection model; 52) For each frame of image , use the PnP algorithm to quickly solve the initial pose of the target relative to the camera coordinate system: in: For images The rotation matrix of For images The translation vector of 53) Using the posture of the end effector of the robotic arm and hand-eye calibration results , transform the initial pose to the base coordinate system: in: For the The pose transformation matrix of the charging port target relative to the robot arm base coordinate system obtained by the PnP algorithm in the frame image; 54) Introduce graph optimization methods to jointly optimize the target global pose and construct the optimization objective function: in: is the number of observed images; For the The number of feature points in the frame; It is a robust kernel function that suppresses the influence of external points; is the covariance matrix, which represents the weight distribution strategy; and is an adaptive weight factor, which is dynamically adjusted according to the optimization stage; is the reprojection error constraint; is a structural prior constraint; is a kinematic chain constraint; and: The reprojection error constraint is: in: Indicates the Frame image Image coordinates of the center point of each charging hole; Indicates the actual charging port The three-dimensional feature point coordinates of the center of the charging port; Indicates the The pose transformation matrix of the charging port target relative to the camera obtained by transforming in the globally consistent pose in the frame image; and: The structural prior constraints are: in: Indicates the charging port number defined based on the national standard. The three-dimensional feature point coordinates of the center of the charging port; The kinematic chain constraints are: in: Represents a globally consistent target pose; 55) The Levenberg-Marquardt algorithm is used for iterative optimization to obtain the precise position of the target in the manipulator base coordinate system.

9. A charging method for measuring the posture of a charging robot and a car charging port, characterized by: The steps include: S1: Using the visual measurement method for the position and posture of the charging port of the charging robot as described in any one of claims 1 to 8, the precise position and posture of the target in the robot arm base coordinate system is obtained; S2: Execute automatic charging tasks, including: S21: Based on the target pose, the inverse kinematics solver is used to calculate the required joint angle solution of the manipulator, and the interpolation trajectory is generated in combination with the trajectory planning algorithm to ensure that the manipulator reaches the target position without interference and vibration; S22: After confirming that the charging gun is successfully plugged into the charging port and the physical connection is stable, the charging process is started; S23: Record the current charging task execution data, and after charging is completed, control the charging gun to detach from the charging port through the robotic arm; S24: The robotic arm returns to its initial standby position to prepare for the next charging task.

10. A charging robot and a vehicle charging port posture measurement charging system, used to implement the charging robot and a vehicle charging port posture visual measurement method according to any one of claims 1 to 8, characterized in that: include: A robotic arm equipped with an end effector and a monocular camera; An image processing module configured to perform improved Mask R-CNN segmentation and ellipse center extraction; A pose optimization module configured to construct and solve a PnP graph optimization model; The control module generates the motion trajectory of the robotic arm based on the optimized posture, starts the charging process after the physical connection between the charging gun and the charging port is stable, and controls the charging gun to detach from the charging port through the robotic arm after charging is completed, and controls the robotic arm to reset to the initial standby position.

Images