A charging port recognition method and device based on radar and vision fusion

Abstract

The application discloses a charging port recognition method and system based on radar and vision fusion, comprising the following steps: time synchronization is performed on point cloud data collected by a fuselage radar and image data collected by a multi-view visual perception component to obtain a perception data set at the same recognition time; a vehicle lateral plane, a vehicle boundary and a real-time distance between the robot and the vehicle are extracted from the point cloud data, and a safety observation point is generated according to the real-time distance, a chassis motion envelope of the robot, a reachable range of the mechanical arm and a safety distance threshold. Beneficial effects: improve recognition robustness: through deep fusion of radar and vision, the safety observation area and the vehicle lateral geometric constraint are determined by using the radar, and the fine structure of the charging port is recognized in combination with the vision, so that stable recognition can be realized even in complex environments such as light change and shielding.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology for new energy charging, and in particular to a charging port recognition method and device based on radar and vision fusion. It is mainly applied in mobile charging scenarios, where a general-purpose unibody wheeled robot (hereinafter referred to as SRT) automatically identifies the vehicle charging port to achieve unmanned charging operation in collaboration with mobile charging equipment (hereinafter referred to as CUBE). Background Technology

[0002] With the increasing popularity of new energy vehicles, mobile charging robots are gradually becoming an important supplement to energy replenishment solutions. In existing technologies, most automatic charging robots use visual sensors to identify charging ports. However, in complex outdoor lighting conditions, vehicle reflections, and obstructions, a single visual sensor is prone to false detections or missed detections. Furthermore, the lack of dynamic assessment of safe distances and workable areas when the robot approaches a vehicle increases the risk of collisions. Some solutions utilize only LiDAR for obstacle detection, but LiDAR cannot accurately identify the center and normal of the charging port, resulting in insufficient accuracy of the charging gun.

[0003] Furthermore, existing methods typically separate environmental perception from target recognition, lacking deep temporal and spatial fusion of radar and vision. This fails to form a complete closed loop from selecting safe observation points, multi-view visual recognition, 3D localization, to calculating the gun insertion pose and assessing recognition confidence. Once recognition fails, the robot lacks an effective re-observation strategy, often resulting in direct shutdown or requesting manual intervention, making it difficult to meet the needs of fully unmanned automated charging. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to solve the technical problems in the prior art and provide a charging port identification method and device based on radar and vision fusion.

[0005] Technical Solution: In a first aspect, this application proposes a charging port recognition method based on radar and vision fusion, applied to the automatic recognition of vehicle charging ports by a general-purpose unibody wheeled robot in mobile charging scenarios. The robot is equipped with a body radar and a multi-view visual perception component. The method is characterized by the following steps:

[0006] Point cloud data collected by the fuselage radar and the image data acquired by the multi-view visual perception component Perform time synchronization to obtain the perception dataset at the same recognition time;

[0007] From the point cloud data Extracting the vehicle's lateral plane Vehicle boundaries and the real-time distance between the robot and the vehicle. And based on the real-time distance The robot's chassis motion envelope, the reachable range of the robotic arm, and the safe distance threshold. Generate security observation points ;

[0008] Control the robot to move to the safe observation point. The multi-view visual perception component is used to determine the candidate charging port area based on radar data. Visual recognition is performed to obtain two-dimensional candidate boxes for the charging port. ;

[0009] Using the two-dimensional candidate box of the charging port With respect to the vehicle's lateral plane The fusion process is performed to calculate the three-dimensional center position of the charging port. and socket normal ;

[0010] Based on the three-dimensional center position and socket normal Combined with the hand-eye calibration relationship of the robotic arm Based on the geometric parameters of the charging gun, calculate the pre-insertion pose of the robotic arm. and insertion target pose ;

[0011] According to the two-dimensional candidate box Image recognition confidence Geometric consistency of 3D fusion Time stability Combined confidence level for robotic arm reachability calculation If the overall confidence level The preset threshold is reached and the insertion target pose is If the target is within the reachable space of the robotic arm, then the insertion target pose will be... Output the identification result; otherwise, trigger a re-observation action based on the identification anomaly type. The re-observation action includes adjusting the robot's observation position, observation angle, or supplementary lighting.

[0012] Preferably, the time synchronization specifically includes:

[0013] Establish a unified time reference for the point cloud data. The image data The robot's body pose data and robotic arm joint status data Distribute timestamps separately , , , ;

[0014] Based on the timestamp, radar frames, visual frames, pose frames, and joint state frames with a time difference less than a preset maximum time difference threshold are selected as the perception dataset for the same recognition time.

[0015] Motion compensation is performed on the time difference between the selected data frames to map the data to the same fusion time. ;

[0016] If the time difference of any sensor data exceeds the threshold or a key frame is missing, the synchronization is deemed invalid, reducing the subsequent overall confidence level. This may trigger data re-collection.

[0017] Preferably, generate safety observation points. The steps include;

[0018] From the point cloud data The vehicle's lateral point cloud is separated and fitted to obtain the vehicle's lateral plane. Vehicle external boundary frame ;

[0019] Calculate the reference point on the robot to the lateral plane of the vehicle. The minimum distance is taken as the real-time distance. ;

[0020] According to the vehicle lateral plane The inner boundary, parking space boundary, or obstacle boundary is used as the outer boundary. A lateral passable zone is generated by combining the robot's chassis width, motion envelope, robotic arm sweep space, and cable dragging channel width.

[0021] Multiple candidate observation points are generated within the passable zone. And for each candidate observation point Based on the real-time distance Safe distance threshold Safety constraints are imposed based on visual field coverage, robotic arm accessibility, and cable dragging continuity; points that pass these constraints are selected as the safety observation points. ;

[0022] If no candidate observation points meet the conditions after the screening process... If the observation point fails, an observation point failure signal will be output, and a request will be made to adjust the service docking point of the mobile storage and charging equipment or the station position of the robot.

[0023] Preferably, the two-dimensional candidate frame for the charging port The method to obtain it is as follows:

[0024] The robot reaches the safe observation point. Then, the multi-view visual perception component in the charging port candidate area The device performs close-range recognition, and the objects to be recognized include at least the charging port cover boundary, the socket outline, the center of the socket, the direction of the socket, the reflective area and the obstructed area.

[0025] The robot is based on a vehicle model database. Or historical charging port location record The location, relative height, and shape information of the charging port are read and used as priors for visual search to constrain the candidate areas for the charging port. The search window;

[0026] When the multi-view visual perception component includes a binocular camera or a depth camera, it utilizes the acquired depth point cloud. Alternatively, binocular parallax data can be used to correct the depth value corresponding to the candidate box obtained by visual recognition, in order to compensate for the depth deviation caused by local concavity and convexity of the vehicle body's side plane;

[0027] When light reflection causes the recognition confidence of the charging port in any viewpoint image to fall below a preset image confidence threshold, the multi-view visual perception component performs weighted fusion based on the temporal stability and geometric consistency of the multi-view results, and selects the candidate box with the highest weighted score as the two-dimensional candidate box for the charging port. Output.

[0028] Preferably, the two-dimensional candidate box for the charging port is used. With respect to the vehicle's lateral plane The fusion process is performed to calculate the three-dimensional center position of the charging port. and socket normal ,include:

[0029] Obtain the two-dimensional candidate box of the charging port center pixel coordinates Using the camera intrinsics of the multi-view visual perception component Formation of space rays The formula is as follows;

[0030] ;

[0031] The space ray Transform to the map coordinate system or the robot arm base coordinate system, and calculate the spatial ray. With respect to the vehicle's lateral plane The intersection point is taken as the three-dimensional center position of the charging port. ;

[0032] Based on the visual recognition results of the socket outline and the normal of the vehicle's lateral plane... The normal of the socket was determined by the multi-view observation results. .

[0033] Preferably, the space ray Transform to the map coordinate system or the robot arm base coordinate system, and calculate the spatial ray. With respect to the vehicle's lateral plane The intersection point is taken as the three-dimensional center position of the charging port. ,include:

[0034] The spatial ray and the vehicle's lateral plane Solve for the intersection point:

[0035] ;

[0036] Where λ is a ray parameter, representing the distance from the camera's optical center along the ray direction to the vehicle's lateral plane, the ray parameter λ extends from the vehicle's lateral plane. The expression:

[0037] ;

[0038] Substituting into the ray equation, we get:

[0039] ;

[0040] in, The three-dimensional coordinate components of the optical center position O of the camera are given. The ray direction vector directional components, The vehicle's lateral plane The expression coefficients;

[0041] The intersection point The three-dimensional center position of the charging port is determined after confidence evaluation and screening. Output.

[0042] Preferably, if the multi-view visual perception component includes a binocular camera or a depth camera, then the depth point cloud acquired by the binocular camera or depth camera is also utilized. The intersection point is corrected.

[0043] Preferably, based on the three-dimensional center position and socket normal Combined with the hand-eye calibration relationship of the robotic arm Based on the geometric parameters of the charging gun, calculate the pre-insertion pose of the robotic arm. and insertion target pose ,include:

[0044] Based on the three-dimensional center position of the charging port and the socket normal Establish the transformation of the charging port coordinate system relative to the vision acquisition unit coordinate system. ;

[0045] According to the extrinsic parameters of the vision acquisition unit relative to the base coordinate system of the robotic arm Transform the charging port coordinate system to the robotic arm base coordinate system;

[0046] Based on the compensation transformation of the charging gun tip relative to the charging port insertion posture Calculate the insertion target pose of the robotic arm:

[0047] ;

[0048] According to the insertion target pose Along the normal direction of the socket Offset in the opposite direction by the preset pre-insertion distance The pre-insertion pose is obtained as follows:

[0049] .

[0050] Preferably, calculate the overall confidence level. The steps, and whether to insert the target pose. The steps for outputting the recognition results are as follows:

[0051] Obtain the two-dimensional candidate box of the charging port Corresponding visual recognition confidence level Shape consistency between the candidate area and the charging port geometry The three-dimensional center position of the charging port With respect to the vehicle's lateral plane Geometric consistency Temporal stability of multi-frame recognition results And penalties for degree of occlusion ;

[0052] According to the preset weighting The overall confidence level is obtained by weighted summation of the above items:

[0053] ;

[0054] The weights for each item are determined through calibration.

[0055] The overall confidence level Compared with the preset confidence threshold Compare the results and combine them with the safety distance judgment and the pose error judgment. If both conditions are met... Real-time distance Greater than or equal to the safe distance threshold Safety margin Greater than or equal to zero, and the insertion target pose pose estimation error If the error is less than the preset error threshold, the current recognition result is deemed valid, and the process of inserting the gun is allowed.

[0056] If the comprehensive confidence level Below the confidence threshold If there is a conflict in the number of candidates, the pose estimation error exceeds the limit, or the robotic arm cannot reach the target, the gun insertion stage will not be entered. A corresponding re-observation action will be generated based on the identified anomaly type. The re-observation action includes one or more of the following: lateral fine adjustment, re-observation from a different perspective, supplementary lighting, or re-shooting at a closer location.

[0057] Preferably, based on the output of the insertion target pose During the insertion of the gun, the insertion operation adopts a force / position hybrid control method, specifically including:

[0058] The axial insertion force is detected in real time by a force / torque sensor or joint current feedback at the end of the robotic arm. and lateral moment ;

[0059] The maximum allowable insertion force is preset based on the charging gun structure, socket specifications, and calibration results. Maximum permissible lateral moment and maximum insertion speed ;

[0060] During the pre-insertion phase, the device moves to the pre-insertion pose using position control. During the formal insertion phase, along the normal direction of the socket. Advance, and according to the axial insertion force Insertion force with target Adjust propulsion speed and attitude based on deviations;

[0061] When the axial insertion force exceeds the maximum permissible insertion force or the lateral moment exceeds the maximum permissible lateral moment, the advancement should be stopped immediately and the device should be retracted to the pre-insertion position. Or retract position This triggers re-identification;

[0062] Insertion is considered complete when the end displacement reaches the preset insertion depth and the gun lock signal or electrical handshake status confirms the insertion.

[0063] Secondly, a charging port identification system based on radar and vision fusion includes a time synchronization module for processing point cloud data collected by the fuselage radar. and the image data acquired by the multi-view visual perception component Perform time synchronization to obtain the perception dataset at the same recognition time;

[0064] The security observation point generation module is used to generate points from the point cloud data. Extracting the vehicle's lateral plane Vehicle boundaries and the real-time distance between the robot and the vehicle. And based on the real-time distance The robot's chassis motion envelope, the reachable range of the robotic arm, and the safe distance threshold. Generate security observation points ;

[0065] The visual recognition module is used to control the robot to move to the safe observation point. The multi-view visual perception component is used to determine the candidate charging port area based on radar data. Visual recognition is performed to obtain two-dimensional candidate boxes for the charging port. ;

[0066] The fusion positioning module is used to utilize the two-dimensional candidate frame of the charging port. With respect to the vehicle's lateral plane The fusion process is performed to calculate the three-dimensional center position of the charging port. and socket normal ;

[0067] The target pose calculation module is used to calculate the target pose based on the three-dimensional center position. and socket normal Combined with the hand-eye calibration relationship of the robotic arm Based on the geometric parameters of the charging gun, calculate the pre-insertion pose of the robotic arm. and insertion target pose ;

[0068] The confidence evaluation module is used to evaluate the confidence level based on the two-dimensional candidate boxes. Image recognition confidence Geometric consistency of 3D fusion Time stability Combined confidence level for robotic arm reachability calculation ;

[0069] The control output module is used to determine the overall confidence level. The preset threshold is reached and the insertion target pose is If the target is within the reachable space of the robotic arm, then the insertion target pose will be... Output the identification result; otherwise, trigger a re-observation action based on the identification anomaly type. The re-observation action includes adjusting the robot's observation position, observation angle, or supplementary lighting.

[0070] Thirdly, this application proposes an electronic device including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, the processor causes the processor to perform the method described in the above embodiments.

[0071] Fourthly, this application proposes a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the method described in the above embodiments.

[0072] Beneficial effects:

[0073] Improve recognition robustness: Through deep fusion of radar and vision, radar is used to determine the safe observation area and the vehicle's lateral geometric constraints (which has the advantage of resisting light interference), and combined with vision to recognize the fine structure of the charging port (which has the advantage of obtaining the center and normal of the socket), stable recognition can be achieved even in complex environments such as changes in lighting and occlusion.

[0074] Ensure operational safety: Through a multi-layered screening mechanism of safety observation points (distance, field of view, robotic arm accessibility, cable drag continuity), ensure that the robot will not collide with vehicles or obstacles during the identification and insertion of the gun, while avoiding cable entanglement or jamming.

[0075] Improve insertion accuracy: Through a multi-stage identification process (vehicle positioning → candidate area determination → charging port cover identification → socket precision positioning) and force / position hybrid insertion control, the positioning accuracy of the charging port meets the requirements of automatic insertion, reducing insertion failures caused by positioning deviations.

[0076] Supports anomaly recovery: Through reobservation strategy and anomaly handling mechanism, when the confidence level is insufficient or the charging port resistance is abnormal, reobservation or rollback is automatically triggered to avoid jamming or damage to the charging port and improve the fault tolerance of the system. Attached Figure Description

[0077] Figure 1 This is a schematic diagram of the method according to an embodiment of the present invention;

[0078] Figure 2 This is a schematic diagram of the system structure provided in an embodiment of the present invention.

[0079] Figure 3 This is a block diagram of an electronic device structure provided in one embodiment of this application. Detailed Implementation

[0080] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0081] Example 1:

[0082] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "including" and similar expressions used herein mean that the element or object preceding the term covers the element or object listed after the term and its equivalents, but do not exclude other elements or objects.

[0083] In response to the problems existing in the current technology, such as Figure 1-3 As shown, a charging port recognition method based on radar and vision fusion is proposed and applied to the automatic recognition of vehicle charging ports by a general-purpose unibody wheeled robot in mobile charging scenarios. The robot is equipped with a body radar and a multi-view visual perception component. The method is characterized by the following steps:

[0084] Step S1: Collect point cloud data from the fuselage radar. and the image data acquired by the multi-view visual perception component Perform time synchronization to obtain the perception dataset at the same recognition time;

[0085] The principle behind this step is to address the data asynchrony problem caused by differences in sampling frequency, triggering time, and transmission delay between different sensors. Specifically, radar point clouds and visual images are acquired at different frequencies (radar is typically 10-20Hz, and cameras are typically 30-60Hz), and there may be a fixed phase difference or random transmission delay between the acquisition times. Without time synchronization, the fusion algorithm will use data from different times for positioning, leading to deviations in the charging port location calculation. For example, if the SRT is moving laterally, fusing point cloud data from 20ms ago with the current image data will result in an error of several millimeters to several centimeters in the calculated charging port position, directly affecting the insertion accuracy. The effect of this step is to ensure that radar, vision, pose, and robotic arm status data correspond to the same moment, laying the foundation for subsequent accurate fusion positioning.

[0086] In some specific embodiments, the time synchronization specifically includes:

[0087] Establish a unified time reference for the point cloud data. The image data The robot's body pose data and robotic arm joint status data Distribute timestamps separately , , , ;

[0088] Based on the timestamp, radar frames, visual frames, pose frames, and joint state frames with a time difference less than a preset maximum time difference threshold are selected as the perception dataset for the same recognition time.

[0089] Motion compensation is performed on the time difference between the selected data frames to map the data to the same fusion time. ;

[0090] If the time difference of any sensor data exceeds the threshold or a key frame is missing, the synchronization is deemed invalid, reducing the subsequent overall confidence level. This may trigger data re-collection.

[0091] Specifically, the system uses the SRT master clock, robot control bus clock, ROS2 / DDS time source, or IEEE 1588PTP (Precision Time Protocol) timing module as a unified clock source. Each sensor's data is recorded with a corresponding timestamp at the moment of acquisition: point cloud data. Assign timestamps Each visual image Assign timestamps Body pose data Assign timestamps Robotic arm joint status data Assign timestamps .

[0092] Software timestamp matching. The fusion module uses the current identification time. Based on this, select the cache queue with the closest timestamps and a time difference less than the preset maximum time difference threshold. The radar frames, visual frames, pose frames, and joint state frames are used as the perception dataset at the same recognition time. The typical value is 10ms to 50ms, depending on the motion speed of the SRT. If the motion speed is high (e.g., 0.5m / s), then... A smaller value (such as 10ms) should be selected to ensure that the position error does not exceed 5mm within 50ms; if the movement speed is slow, the value can be appropriately relaxed.

[0093] Different sensors have different sampling frequencies, trigger times, and data transmission delays. Radar typically scans at a fixed period (e.g., 50ms per frame), while cameras expose at another fixed period (e.g., 33ms per frame), and their sampling times may not be aligned. Without timestamp matching, the fusion module might fuse radar data from 50ms ago with image data from the current moment, resulting in significant spatiotemporal misalignment in the fusion result. By recording timestamps and selecting thresholds, the system can ensure that the various types of data used for fusion are sufficiently close in time.

[0094] Ensure that radar, vision, pose, and robotic arm status data are aligned in time to avoid fusion positioning errors caused by time asynchrony. By setting a maximum time difference threshold, the system can automatically select the closest data combination when there is data latency or frame rate fluctuation, ensuring the input consistency of the fusion algorithm.

[0095] If the time difference of data from any sensor exceeds a threshold or a key frame is missing, the synchronization is deemed invalid, reducing the sensor's overall confidence level in subsequent tests. Individual weights in the calculation. Specifically, reducing the weighting weight corresponding to this sensor. to The corresponding item in.

[0096] For example, if the time synchronization of the vision sensor fails, the system will reduce the weight of the vision-related term in the overall confidence formula (e.g., weight The geometric consistency term will be reduced from the calibration value of 0.25 to 0.1 if radar time synchronization fails. weight If multiple sensors fail simultaneously, the system will trigger data reacquisition or enter a re-observation process.

[0097] Once the sensor is synchronized, the system gradually restores its weights to the calibrated values ​​to avoid continuous performance degradation caused by a single synchronization failure.

[0098] In some specific embodiments, the time synchronization further includes: when hardware synchronization conditions are available, the robot main control or synchronization trigger module sends a trigger signal to the body radar and the multi-view visual perception component, so that the radar scanning cycle and visual exposure time fall into a preset synchronization window; when hardware synchronization conditions are not available, each sensor records the acquisition completion time or exposure midpoint time at the acquisition end and uploads it with the data frame.

[0099] In embodiments with hardware synchronization capabilities, the SRT master control or synchronization trigger module sends trigger signals to the fuselage radar and multi-view visual perception components, ensuring that the radar scanning cycle and visual exposure time fall within a preset synchronization window. Specifically, the master control module simultaneously sends trigger pulses to all sensors at times t0, t1, t2, ... The radar begins scanning upon receiving the trigger signal, and the camera begins exposure upon receiving the trigger signal, thereby aligning the data acquisition times of each sensor to the same point in time. Hardware triggering can achieve nanosecond-level time alignment accuracy. In embodiments without hardware synchronization capabilities, each sensor records the acquisition completion time or exposure midpoint time at the acquisition end and uploads it with the data frame. For example, the camera records the exposure start and end times after image acquisition, taking the midpoint as the representative timestamp of the image; the radar records the scan end time after scanning; and the IMU records the measurement time at each measurement. These timestamps are transmitted to the fusion module along with the data frame.

[0100] Hardware triggering and timestamp recording are two synchronization schemes with different levels of precision. Hardware triggering is suitable for scenarios requiring nanosecond-level precision, but requires additional synchronization hardware support; timestamp recording is suitable for scenarios with slightly lower precision requirements (millisecond-level), requires no additional hardware, and can be implemented through software algorithms. This solution provides two implementation methods, which can be flexibly selected according to hardware configuration.

[0101] Time alignment of multi-sensor data can be achieved through hardware triggering or timestamp recording. Hardware triggering offers the highest synchronization accuracy and is suitable for high-speed motion scenarios; timestamp recording can achieve effective synchronization without increasing hardware costs and is suitable for cost-sensitive scenarios.

[0102] In some specific embodiments, the time synchronization further includes: if there is a time difference of less than a preset threshold between the radar point cloud, visual image, robot pose, or robotic arm joint state, interpolation or extrapolation compensation is performed on the radar point cloud, visual feature points, or coordinate transformation based on the robot chassis speed, angular velocity, robotic arm joint speed, or historical pose interpolation results, so that they correspond to the same fusion time.

[0103] Specifically, assuming the visual image is acquired at t=0 and the radar point cloud is acquired at t=20ms, with the fusion reference time being t=20ms. The system knows that the velocity of the SRT at t=0 is v=0.2m / s, meaning the SRT moves approximately 4mm within 20ms. To compensate the visual feature points to their positions at t=20ms, the system translates the positions of the visual feature points 4mm along the direction of SRT movement.

[0104] For compensation of the coordinate transformation matrix, the system uses the historical sequence of SRT poses for interpolation. For example, given the pose at time t=0 as... The pose at t=20ms is Then the pose at t=10ms can be obtained by linear interpolation or spherical interpolation (Slerp).

[0105] The principle is to compensate for the time difference between data frames using a motion model. Since the acquisition times of different sensors cannot be perfectly aligned, there will inevitably be a slight time difference. In static or low-speed scenarios, these time differences are negligible; however, in dynamic scenarios (such as SRT identifying the charging port during lateral movement), even a 20ms time difference can lead to positioning errors of several millimeters. Motion compensation estimates the amount of movement of the system within the time difference and corrects the data to the same moment, thereby eliminating this source of error.

[0106] Interpolation or extrapolation compensation is used to eliminate positioning errors caused by asynchronous sensor acquisition, thereby improving the accuracy of fusion positioning. In scenarios where the SRT moves at high speeds, the accuracy of motion compensation directly affects the positioning accuracy of the charging port and the success rate of plugging in the charging gun.

[0107] Step S2: From the point cloud data Extracting the vehicle's lateral plane Vehicle boundaries and the real-time distance between the robot and the vehicle. And based on the real-time distance The robot's chassis motion envelope, the reachable range of the robotic arm, and the safe distance threshold. Generate security observation points ;

[0108] Leveraging the spatial geometric perception capabilities of radar point clouds, an optimal pose point is determined to the side of the vehicle, allowing for safe parking and effective observation of the charging port. Unaffected by lighting conditions, radar accurately measures the vehicle's lateral position and orientation, as well as the real-time distance between the SRT and the vehicle. Multi-dimensional safety constraints are constructed using radar data to ensure the safety of the SRT during charging port identification and subsequent plug-in operations. If a suitable observation point does not exist, the system outputs an observation point failure signal, requesting adjustments to the mobile charging unit's (CUBE) service docking point or the SRT's position, or even entering a manual confirmation mode, thereby preventing the robot from operating under unsafe conditions.

[0109] In some specific embodiments, security watchpoints are generated. The steps include;

[0110] From the point cloud data The vehicle's lateral point cloud is separated and fitted to obtain the vehicle's lateral plane. Vehicle external boundary frame ;

[0111] vehicle lateral plane It is a geometric plane describing the reference plane of the vehicle body side, and its expression is:

[0112] ;

[0113] Where (a, b, c) is the normal vector of the plane. (x, y, z) represent three-dimensional spatial coordinates, and d is a plane constant. This plane is used to represent the vehicle's lateral reference plane and serves as the fundamental reference plane for all subsequent distance calculations and the construction of the safety zone. Vehicle outer bounding box. Used to define the front and rear boundaries, height range, and lateral protrusions of the vehicle body, such as the protrusions of rearview mirrors or wheel arches.

[0114] Calculate the reference point on the robot to the lateral plane of the vehicle. The minimum distance is taken as the real-time distance. Project the SRT body reference point (such as the chassis center or a preset safety detection point) onto the vehicle's lateral plane. or vehicle external boundary frame Calculate the distance from a point to the plane or the distance from a point to the bounding box. Specifically, if the coordinates of the SRT reference point are known, its distance to the plane... The distance can be calculated by substituting the coordinates into the plane equation. The system takes the minimum value among all reference point distances as the real-time distance from the SRT to the vehicle body. .

[0115] According to the vehicle lateral plane The inner boundary, parking space boundary, or obstacle boundary is used as the outer boundary. A lateral passable zone is generated by combining the robot's chassis width, motion envelope, robotic arm sweep space, and cable dragging channel width. This is based on the vehicle's lateral plane. The inner boundary is defined by the boundary of the parking space, the boundary of the adjacent vehicle, or the boundary of the obstacle, while the outer boundary is defined by the SRT chassis width, the wheel foot motion envelope in lateral movement mode, and the minimum safe distance. The robotic arm sweeps through the space and the cable drags, contracting or expanding the boundary to create a passable zone. This parameter can be preset in advance using robot parameters.

[0116] For example, if the vehicle's lateral plane is located at x=0, the parking space boundary is located at x=2.0m, and the chassis width is 0.8m, what is the minimum safe distance? If the width is 0.3m, then the inner boundary of the passable zone is x=0+0.3m=0.3m, and the outer boundary is x=2.0m−0.4m=1.6m (space needs to be reserved for the chassis width), and the width of the passable zone is 1.3m.

[0117] Multiple candidate observation points are generated within the passable zone. And for each candidate observation point Based on the real-time distance Safe distance threshold Safety constraints are imposed based on visual field coverage, robotic arm accessibility, and cable dragging continuity; points that pass these constraints are selected as the safety observation points. ;

[0118] The generation of safety observation points must simultaneously satisfy the following constraints:

[0119] (1) The real-time distance between the SRT and the vehicle shall not be less than the safe distance threshold to avoid collision;

[0120] (2) The SRT chassis can be stably parked at this location without exceeding the passable area;

[0121] (3) The robotic arm can reach the charging port from this position;

[0122] (4) The field of view of the visual acquisition unit can cover the candidate area of ​​the charging port;

[0123] (5) The cable dragging channel shall not interfere with vehicles or obstacles.

[0124] If no candidate observation points meet the conditions after the screening process... If the observation point fails, an observation point failure signal will be output, and a request will be made to adjust the service docking point of the mobile storage and charging equipment or the station position of the robot.

[0125] The multidimensional safety constraints are transformed into computable screening conditions. Through a series of screening steps, the final selected observation point is guaranteed to simultaneously satisfy safety, observability, and operability. The technical essence is a constrained multi-objective optimization problem: within the feasible domain of the traversable zone, find the optimal point that simultaneously satisfies constraints such as distance safety, field of view coverage, robotic arm accessibility, and cable unobstructed access.

[0126] Step S3: Control the robot to move to the safe observation point. The multi-view visual perception component is used to determine the candidate charging port area based on radar data. Visual recognition is performed to obtain two-dimensional candidate boxes for the charging port. ;

[0127] Leveraging the high resolution and high texture resolution of visual images, the two-dimensional location of the charging port is accurately identified within a localized area defined by radar. Radar data is used to limit the candidate region, effectively narrowing the scope of the visual search, reducing the time required for a full-vehicle search, and avoiding false detections that may occur in a global visual search. Visual recognition then uses deep learning object detection algorithms (such as YOLO and Faster R-CNN) or traditional template matching methods to locate the charging port cover or socket in the image. Combining the spatial constraint capabilities of radar with the fine-grained recognition capabilities of vision achieves fast and accurate two-dimensional localization of the charging port. This includes using a vehicle model database or historical charging port location records as prior constraints, using a depth camera to correct depth deviations, and employing multi-view weighted fusion under abnormal lighting conditions.

[0128] In some specific embodiments, the two-dimensional candidate box for the charging port The method to obtain it is as follows:

[0129] The robot reaches the safe observation point. Then, the multi-view visual perception component in the charging port candidate area The device performs close-range recognition, and the objects to be recognized include at least the charging port cover boundary, the socket outline, the center of the socket, the direction of the socket, the reflective area and the obstructed area.

[0130] The robot is based on a vehicle model database. Or historical charging port location record The location, relative height, and shape information of the charging port are read and used as priors for visual search to constrain the candidate areas for the charging port. The search window is narrowed down; for example, if the vehicle model database shows that the charging port of the model is located on the right rear side, about 0.7m to 0.9m above the ground, the system will limit the search window to this range, which will significantly reduce search time and reduce false detection rate.

[0131] When the multi-view visual perception component includes a binocular camera or a depth camera, it utilizes the acquired depth point cloud. Alternatively, binocular parallax data can be used to correct the depth values ​​corresponding to the candidate boxes obtained by visual recognition, in order to compensate for the depth deviation caused by local concavities and convexities on the side plane of the vehicle body. This is because there may be local concavities and convexities on the side plane of the vehicle body (such as a recessed charging port cover), which causes a deviation between the depth value calculated directly using the side plane model and the actual depth. Data from a depth camera can directly measure the true depth of the charging port area, thereby compensating for this deviation.

[0132] When light reflection causes the recognition confidence of the charging port in any viewpoint image to fall below a preset image confidence threshold, the multi-view visual perception component performs weighted fusion based on the temporal stability and geometric consistency of the multi-view results, and selects the candidate box with the highest weighted score as the two-dimensional candidate box for the charging port. Output. For example, if the left camera's recognition confidence is low due to reflections caused by direct sunlight, the recognition result of the right camera may not be affected, and the system will assign a higher weight to the right result.

[0133] Deep learning-based object detection algorithms are used to locate charging ports in images, and multi-view fusion and depth data correction are employed to improve localization accuracy. Deep learning methods can identify typical shape features of charging ports in complex backgrounds, such as the edges of rectangular covers and the outline of circular sockets. Multi-view fusion leverages the complementarity of different viewpoints; when one viewpoint fails due to lighting or occlusion, other viewpoints can still provide effective recognition results. Depth correction utilizes the direct depth measurement capabilities of binocular / depth cameras, compensating for errors caused by the curved surfaces of the vehicle body when estimating depth using a planar model with monocular vision.

[0134] The benefits of this step are: improving speed by narrowing the search range through prior constraints, enhancing 3D positioning accuracy through depth correction, and increasing the system's robustness to lighting variations and occlusion conditions through multi-view weighted fusion. Compared to single-view visual recognition, multi-view fusion recognition has a significantly higher success rate in complex environments.

[0135] Step S4: Utilize the two-dimensional candidate box for the charging port With respect to the vehicle's lateral plane The fusion process is performed to calculate the three-dimensional center position of the charging port. and socket normal ;

[0136] The charging port features in a 2D image are back-projected into 3D space using a camera projection model and hand-eye calibration, achieving a transformation from pixel coordinates to spatial coordinates. Specifically, the camera imaging process projects 3D spatial points onto a 2D image plane, while fusion localization is the reverse process—inferring the 3D spatial position from 2D pixel coordinates. However, since a single line of sight from a monocular camera corresponds to a ray in 3D space, depth cannot be directly determined. Therefore, the vehicle's lateral plane is used as a constraint surface, intersecting the ray with the plane to uniquely determine the 3D position. Accurately converting the 2D results of visual recognition into 3D spatial coordinates provides precise target pose for robotic arm motion planning. Compared to pure vision methods, radar-vision fusion localization utilizes the precise planar constraints provided by radar, avoiding the errors of monocular vision depth estimation; compared to pure radar methods, it leverages the fine recognition capabilities of vision, accurately locating the center point of the charging port rather than a rough approximation of the vehicle's overall position.

[0137] In some specific embodiments, the two-dimensional candidate box for the charging port is utilized. With respect to the vehicle's lateral plane The fusion process is performed to calculate the three-dimensional center position of the charging port. and socket normal ,include:

[0138] Obtain the two-dimensional candidate box of the charging port center pixel coordinates Using the camera intrinsics of the multi-view visual perception component Formation of space rays (The spatial ray direction vector in the camera coordinate system (the line of sight from the camera optical center, passing through pixel (u,v)), the formula is as follows;

[0139] ;

[0140] The space ray Transform to the map coordinate system or the robot arm base coordinate system, and calculate the spatial ray. With respect to the vehicle's lateral plane The intersection point is taken as the three-dimensional center position of the charging port. ; It is the inverse matrix of the camera intrinsic parameter matrix of the i-th visual acquisition unit; This is a homogeneous representation of pixel coordinates;

[0141] Based on the visual recognition results of the socket outline and the normal of the vehicle's lateral plane... The normal of the socket was determined by the multi-view observation results. .

[0142] The normal of the charging port determines the insertion direction of the charging gun, and its accuracy directly affects the success rate of insertion. When the outline of the socket can be clearly seen by visual recognition, the orientation information of the socket can be directly extracted from the image as the normal. When the normal cannot be accurately estimated by vision due to occlusion or lighting, the normal of the vehicle's side plane can be used as an approximation, because the normal of most vehicle charging ports is basically consistent with the normal of the vehicle's side plane (usually with a deviation within 5°). When there are multiple viewing angles, a more accurate estimate of the normal can be obtained through triangulation.

[0143] The imaging principle of a monocular camera dictates that a three-dimensional spatial position cannot be directly obtained from two-dimensional pixel coordinates, because any point on the entire ray from the camera's optical center to the pixel will project to the same pixel location. To determine the three-dimensional position of the charging port, additional constraints must be used. This solution uses the vehicle's lateral plane fitted by radar point cloud as a constraint surface, intersecting the ray with the plane to uniquely determine the three-dimensional position. The advantage of this method lies in utilizing the precise ranging and plane fitting capabilities of radar, making it more robust and accurate compared to purely vision-based methods.

[0144] This method converts two-dimensional pixel coordinates into rays in three-dimensional space, laying the foundation for subsequent ray-plane intersection localization. Compared to traditional methods that directly use depth maps or stereo matching to obtain depth values, this approach fully utilizes the precise planar constraints provided by radar point clouds through ray-plane intersection, improving the accuracy and robustness of depth estimation.

[0145] In some specific embodiments, the space ray Transform to the map coordinate system or the robot arm base coordinate system, and calculate the spatial ray. With respect to the vehicle's lateral plane The intersection point is taken as the three-dimensional center position of the charging port. ,include:

[0146] The spatial ray and the vehicle's lateral plane Solve for the intersection point:

[0147] ;

[0148] Where λ is a ray parameter, representing the distance from the camera's optical center along the ray direction to the vehicle's lateral plane, the ray parameter λ extends from the vehicle's lateral plane. The expression:

[0149] ;

[0150] Substituting into the ray equation, we get:

[0151] ;

[0152] in, The three-dimensional coordinate components of the optical center position O of the camera are given. The ray direction vector directional components, The vehicle's lateral plane The expression coefficients;

[0153] The intersection point The three-dimensional center position of the charging port is determined after confidence evaluation and screening (the specific scheme can be preset according to requirements). Output.

[0154] In three-dimensional space, the intersection point of a ray and a plane can be uniquely determined by simultaneously solving the ray equation and the plane equation. In this scheme, the ray equation is determined by the camera projection model, and the plane equation is determined by fitting the radar point cloud. Both are calculated independently in the visual and radar domains, respectively, and then spatial positioning is achieved through intersection. This fusion approach of "visual direction determination and radar depth determination" fully leverages the advantages of both sensors. The three-dimensional position of the charging port is accurately calculated using analytical geometry methods, avoiding the uncertainties introduced by using deep neural networks for depth estimation. Compared to learning-based depth estimation methods, analytical solutions offer advantages such as strong interpretability, high computational efficiency, and no accumulated error. Furthermore, because the precise plane fitted from the radar point cloud is utilized, the depth error primarily depends on the radar ranging accuracy and the plane fitting accuracy, typically at the millimeter level.

[0155] Step S5: Based on the three-dimensional center position and socket normal Combined with the hand-eye calibration relationship of the robotic arm Based on the geometric parameters of the charging gun, calculate the pre-insertion pose of the robotic arm. and insertion target pose ;

[0156] Robot kinematics is used to transform the position of the charging port in the camera coordinate system into the target pose in the tool coordinate system of the robotic arm's end effector. Specifically, the hand-eye calibration relationship describes the fixed transformation relationship between the camera coordinate system and the robotic arm's base coordinate system; multiplying these three values ​​yields the target pose that the robotic arm's end effector needs to reach. The pre-insertion pose is designed to allow the robotic arm to approach a certain distance in front of the charging port before the actual insertion, enabling more precise force / position hybrid control to complete the insertion during the actual insertion phase. Pre-insertion pose The typical value is 50mm~100mm, depending on the precision of the robotic arm and the structure of the charging gun.

[0157] In some specific embodiments, based on the three-dimensional center position and socket normal Combined with the hand-eye calibration relationship of the robotic arm The pre-insertion pose of the robotic arm is calculated using the extrinsic parameters of the visual acquisition unit relative to the base coordinate system of the robotic arm and the geometric parameters of the charging gun. and insertion target pose ,include:

[0158] Based on the three-dimensional center position of the charging port and the socket normal Establish the transformation of the charging port coordinate system relative to the vision acquisition unit coordinate system. ;

[0159] According to the extrinsic parameters of the vision acquisition unit relative to the base coordinate system of the robotic arm Transform the charging port coordinate system to the robotic arm base coordinate system;

[0160] Based on the compensation transformation of the charging gun tip relative to the charging port insertion posture (For example, if the grip structure of the charging gun needs to be offset by 5mm in the X direction and rotated by 2° in the Y direction when inserted, then...) (It should include corresponding translation and rotation components), calculate the insertion target pose of the robotic arm:

[0161] ;

[0162] According to the insertion target pose Along the normal direction of the socket Offset in the opposite direction by the preset pre-insertion distance (A pre-set safe insertion distance is used to ensure that the robotic arm end effector approaches a preset distance from the charging port before actual insertion), thus obtaining the pre-insertion pose:

[0163] .

[0164] in Indicates the direction along the socket normal Translate in the opposite direction by a preset distance The homogeneous transformation matrix;

[0165] Hand-eye calibration establishes a fixed transformation relationship between the camera coordinate system and the robotic arm's base coordinate system, allowing the charging port position obtained through visual recognition to be directly translated into the robotic arm's control space. Geometric parameter compensation for the charging gun considers the attitude deviation between the charging gun's end (insertion end) and the visually recognized charging port coordinate system. The pre-insertion pose design aims to allow the robotic arm to reach a certain distance in front of the charging port before actual insertion, enabling insertion at a lower speed and with more precise control during the actual insertion phase. It provides the robotic arm with a precise motion target, ensuring the charging gun is correctly aligned with the charging port. The pre-insertion pose setting provides a safety buffer for subsequent force-controlled insertion, reducing the risk of collisions during high-speed insertion. Simultaneously, by... As an independent parameter, the system can be adapted to charging guns of different models and structures, and has good versatility.

[0166] Step S6: Based on the two-dimensional candidate box Image recognition confidence Geometric consistency of 3D fusion Time stability Combined confidence level for robotic arm reachability calculation If the overall confidence level The preset threshold is reached and the insertion target pose is If the target is within the reachable space of the robotic arm, then the insertion target pose will be... Output the identification result; otherwise, trigger a re-observation action based on the identification anomaly type. The re-observation action includes adjusting the robot's observation position, observation angle, or supplementary lighting.

[0167] The reliability of the recognition results is comprehensively evaluated from multiple dimensions to avoid blindly performing the charging port insertion operation when the recognition quality is insufficient. Compared with a single confidence score evaluation, multi-dimensional evaluation can more comprehensively reflect the quality of the recognition results: image recognition confidence score reflects the reliability of visual detection; shape consistency reflects whether the candidate region conforms to the geometric features of the charging port; geometric consistency reflects whether the three-dimensional position reasonably falls on the side plane of the vehicle body; temporal stability reflects whether the recognition results are consistent between consecutive frames; and the occlusion penalty term reflects whether the charging port is partially occluded. Through weighted fusion, the comprehensive confidence score can more accurately reflect the overall reliability of the recognition results.

[0168] This step ensures that only reliable identification results are executed through multi-dimensional confidence evaluation and comprehensive validity judgment, avoiding insertion failures or equipment damage due to misidentification. If the overall confidence level is insufficient, the system automatically triggers a re-observation, re-executing the identification by adjusting observation conditions (position, angle, supplementary lighting, etc.), thus improving the system's robustness and fault tolerance.

[0169] In some specific embodiments, if the multi-view visual perception component includes a binocular camera or a depth camera, then the depth point cloud acquired by the binocular camera or depth camera is also utilized. The intersection point is corrected.

[0170] When the visual candidate bounding box has been identified, but there are local irregularities between the side plane of the vehicle body and the actual charging port cover (such as the charging port cover having a depression or bulge relative to the vehicle body surface), depth point cloud is used. The corresponding depth value in the depth point cloud replaces or weights the depth value obtained from ray-plane intersection. For example, if the actual depth of the charging port area shown in the depth point cloud is 5mm deeper than the depth predicted by the planar model, the system will use the value of the depth point cloud as the corrected depth.

[0171] The correction method can be a weighted average: α is the confidence weight of the planar model depth, which is dynamically adjusted according to the confidence of the planar fit. When the confidence of the planar fit is high, α takes a larger value (e.g., 0.8); when the confidence of the planar fit is low (e.g., the side of the vehicle is occluded or uneven), α takes a smaller value (e.g., 0.3), which relies more on the measured data of the depth camera.

[0172] The vehicle's lateral plane is the overall plane obtained by fitting the radar point cloud, representing the average position of the vehicle's side. However, the charging port cover usually has a slight indentation (e.g., 1mm~10mm) relative to the vehicle's surface, and this local unevenness cannot be reflected in the global plane fitting. A binocular / depth camera can directly measure the true depth value of the charging port area, thereby compensating for this local deviation.

[0173] In some specific embodiments, the overall confidence level is calculated. The steps, and whether to insert the target pose. The steps for outputting the recognition results are as follows:

[0174] Obtain the two-dimensional candidate box of the charging port Corresponding visual recognition confidence level Shape consistency between the candidate area and the charging port geometry The three-dimensional center position of the charging port With respect to the vehicle's lateral plane Geometric consistency Temporal stability of multi-frame recognition results And penalties for degree of occlusion ;

[0175] According to the preset weighting The overall confidence level is obtained by weighted summation of the above items:

[0176] ;

[0177] The weights for each item are determined through calibration.

[0178] The overall confidence level Compared with the preset confidence threshold Compare the results and combine them with the safety distance judgment and the pose error judgment. If both conditions are met... Real-time distance Greater than or equal to the safe distance threshold Safety margin Greater than or equal to zero, and the insertion target pose pose estimation error If the error is less than the preset error threshold, the current recognition result is deemed valid, and the process of inserting the gun is allowed.

[0179] If the comprehensive confidence level Below the confidence threshold If there is a conflict in the number of candidates, the pose estimation error exceeds the limit, or the robotic arm cannot reach the target, the gun insertion stage will not be entered. A corresponding re-observation action will be generated based on the identified anomaly type. The re-observation action includes one or more of the following: lateral fine adjustment, re-observation from a different perspective, supplementary lighting, or re-shooting at a closer location.

[0180] The principle of comprehensive confidence calculation is to comprehensively evaluate the credibility of the recognition result from multiple dimensions. A single confidence index (such as relying solely on visual detection confidence) cannot fully reflect the quality of the recognition result, because visual detection may give a high confidence score but the positioning is inaccurate, or it may give a low confidence score due to changes in lighting but the actual positioning is accurate. By introducing multiple dimensions such as shape consistency, geometric consistency, temporal stability, and occlusion penalty, the comprehensive confidence score can more accurately reflect whether the recognition result is truly reliable. The validity judgment further introduces hard constraints such as safety distance and pose error to ensure that the insertion is only performed when all conditions are met. The multi-dimensional and multi-level evaluation of the credibility of the recognition result avoids incorrect insertion caused by misjudgment due to a single index. When the comprehensive confidence score is insufficient, the system automatically triggers re-observation, and the recognition is re-executed by adjusting the observation conditions, which improves the recognition success rate and system robustness. At the same time, the introduction of hard constraints (safety distance, pose error) ensures the safety of the insertion operation.

[0181] In some specific embodiments, based on the output insertion target pose During the insertion of the gun, the insertion operation adopts a force / position hybrid control method, specifically including:

[0182] The axial insertion force is detected in real time by force / torque sensors (such as a six-dimensional force / torque sensor) at the end of the robotic arm or by joint current feedback. and lateral moment A six-dimensional force / torque sensor can directly measure force and torque in three directions, making it the most direct method of force control. Joint current feedback estimates the end effector force by detecting the drive current of each joint motor; it is less expensive but slightly less accurate.

[0183] The maximum allowable insertion force is preset based on the charging gun structure, socket specifications, and calibration results. Maximum permissible lateral moment and maximum insertion speed ;

[0184] During the pre-insertion phase, the device moves to the pre-insertion pose using position control. The main goal of this stage is to accurately position the device in front of the charging port. The speed can be slightly faster (e.g., 0.1 m / s). During the actual insertion stage, the device is positioned along the normal direction of the charging port. Advance, and according to the axial insertion force Insertion force with target The deviation is used to adjust the propulsion speed and attitude; for example, if < If so, then increase the propulsion speed appropriately; if near If so, reduce the speed to ensure a smooth insertion; if If abnormal fluctuations occur, proceeding will be suspended and the approach will be readjusted.

[0185] Anomaly Handling: When the axial insertion force exceeds the maximum permissible insertion force or the lateral moment exceeds the maximum permissible lateral moment, the advancement shall be stopped immediately and the position shall be retracted to the pre-insertion posture. Or retract position This triggers re-identification. This mechanism prevents damage to the charging port or charging gun due to excessive posture deviation.

[0186] Insertion is considered complete when the end displacement reaches the preset insertion depth and the gun lock signal or electrical handshake status is confirmed. The gun lock signal refers to the positioning signal issued by the mechanical locking mechanism inside the charging gun; the electrical handshake status refers to the establishment of a communication connection between the charging gun and the vehicle's charging port (e.g., conforming to ISO 15118 standard).

[0187] The principle of force / position hybrid control is to superimpose a force feedback closed loop on top of position control. Pure position control is suitable for rigid contact scenarios, but in flexible contact scenarios (such as plug-in guns), even a small deviation in the target position can lead to a huge contact force, making pure position control prone to damage. Force / position hybrid control introduces force feedback into the position control loop, allowing the robotic arm to monitor and adjust the contact force simultaneously as it approaches the target position. When the contact force reaches a preset target value, it stops advancing, achieving "smooth" insertion. Lateral torque monitoring detects posture deviations; when there is an angular deviation between the charging gun and the socket, a lateral torque is generated. Timely detection and correction can prevent the gun nozzle from tilting or scraping the socket. Force / position hybrid control achieves safe and smooth plug-in operation. The force feedback mechanism can adapt to small deviations in the charging port posture, avoiding damage caused by rigid advancement; the anomaly detection mechanism can promptly retract when the plug-in gun resistance is abnormal, protecting equipment and vehicles; the completion confirmation mechanism verifies through multiple signals to ensure successful plug-in, avoiding incomplete insertion due to sensor misjudgment.

[0188] In some specific embodiments, before extracting the vehicle lateral plane and vehicle boundary from the point cloud data, a preprocessing step is included: performing time stamp alignment, point cloud distortion correction, ground segmentation, outlier removal, and obstacle clustering on the point cloud data to obtain a vehicle point cloud set. Personnel / obstacles assembly and parking space boundary .

[0189] Timestamp alignment: Aligning the acquisition time of each point in the radar point cloud with the fusion reference time. Since a radar scan typically takes 50ms to 100ms, the acquisition times of each point within the frame are different. If the SRT moves during the scan, the robot pose corresponding to each point will also be different. Timestamp alignment uses IMU and odometry data to transform each point to the same coordinate system at the same time.

[0190] Point cloud distortion correction: Using IMU and odometry data, points collected at different times in a single radar point cloud frame are transformed to the same coordinate system at the same time, eliminating motion distortion. For example, if the SRT moves 5cm to the right during radar scanning, the distortion correction process subtracts this 5cm displacement from each point, ensuring that all points correspond to the robot's pose at the same time.

[0191] Ground segmentation: Using planar fitting or grid height thresholding methods, point clouds belonging to the ground are separated from obstacle point clouds. For example, in a parking lot scene, the ground is the largest continuous plane, and RANSAC planar fitting can effectively segment ground points, reducing the interference of ground points on subsequent vehicle detection.

[0192] Outlier removal: Statistical filtering or radius filtering methods are used to remove isolated noise points. For example, for each point, the number of points in its neighborhood is counted; if the number of neighborhood points is less than a threshold, the point is identified as an outlier and removed.

[0193] Obstacle clustering: Using Euclidean clustering or DBSCAN methods, adjacent obstacle points are aggregated into independent obstacle clusters, which facilitates the subsequent differentiation of vehicles, pedestrians and parking space boundaries.

[0194] After the above preprocessing, we obtain the vehicle point cloud set, the personnel / obstacle set, and the parking space boundary, which can be used in subsequent steps.

[0195] The principle of radar preprocessing is to extract useful information and filter out noise from the raw point cloud through a series of signal processing algorithms. The raw radar point cloud contains a large amount of invalid information (such as ground points), noise (such as outliers), and interference (such as motion distortion), which, if used directly, would severely affect the accuracy and efficiency of subsequent processing. The preprocessing steps remove these unfavorable factors step by step, retaining key information such as vehicle, obstacle, and parking space boundaries. This yields clean and accurate vehicle and obstacle point cloud data, providing high-quality input for subsequent vehicle lateral plane fitting, passable zone construction, and candidate observation point selection. The quality of preprocessing directly affects the accuracy and robustness of the entire recognition system.

[0196] In some specific embodiments, if no candidate observation points meet the criteria after screening, an observation point failure signal is returned to the collaborative control link between the mobile storage and charging device and the robot. The system requests adjustments to the service docking points of the mobile storage and charging equipment, the robot's position, or entry into manual confirmation mode. It provides a fallback path in case of failure, ensuring the system can safely stop or request assistance when it cannot automatically find a safe observation point, thus avoiding safety accidents caused by forced operation.

[0197] In some specific embodiments, the comprehensive confidence level The calculation also includes multi-view consistency factors, specifically: calculating the multi-view consistency confidence based on the degree of geometric consistency between visual recognition results from multiple perspectives. The multi-view consistency confidence is treated as an independent term or adjusted by the geometric consistency. The weighting method is incorporated into the overall confidence level. The calculation.

[0198] In some specific embodiments, the process of calculating the overall confidence level and triggering a re-observation action based on the identified anomaly type also includes at least one of the following judgments and processes:

[0199] If the real-time distance If the distance is less than the preset vehicle proximity safety threshold (e.g., 0.3m), the robot will stop approaching and retract laterally. This judgment is continuously performed during the recognition process. Even if the SRT has reached the safe observation point, if the vehicle or SRT pose changes and the distance becomes too close, the system can still stop in time.

[0200] If the vehicle's lateral plane fitting fails, the robot is controlled to switch its viewing angle or request repositioning. Vehicle lateral plane fitting may fail due to poor point cloud quality, surface reflection, or occlusion. In this case, the SRT adjusts its position and rescans, or requests repositioning from the upper-level system.

[0201] If the number of visual candidates is zero, then perform supplemental lighting, lateral fine-tuning, or close-up retake; when visual recognition cannot find any charging port features in the candidate area, the system first tries to turn on the supplemental light or adjust the exposure parameters, then slightly moves laterally to change the viewing angle, and finally tries to get closer to the vehicle to obtain a clearer image.

[0202] If multiple candidates exist and their fusion confidence levels are similar, the system combines vehicle-side information, model priors, historical records, and robotic arm accessibility to select a candidate. When the confidence levels of multiple candidate boxes are not significantly different, the system uses auxiliary information to make a decision. For example, if historical records show that the charging port of this model is usually located in the left half of the charging port candidate area, the candidate box located in the left half will be selected first.

[0203] If the three-dimensional center position of the charging port is mentioned in multiple consecutive frames before the gun is inserted. If the drift exceeds a preset drift threshold, plugging in the charging port will be prohibited and the identification process will be re-executed. The charging port position should theoretically remain stable. If the detected position drift exceeds 5mm in multiple consecutive frames, it indicates that the identification may be unstable, and the system will re-execute the identification process to avoid plugging in the charging port based on an unstable position.

[0204] In some specific embodiments, the loop control condition for the re-observation action is:

[0205] The loop starts when the robot has entered the identification phase and the cooperative link between the robot and the mobile storage and charging device allows for observation before the gun is inserted.

[0206] The loop ends when the overall confidence level reaches a preset threshold and the insertion target pose is within the reachable space of the robotic arm.

[0207] The number of re-observations reaches the preset maximum number of re-observations, or at least one of the following safety anomalies occurs: personnel intrusion, cable tension exceeding limits, or sensor malfunction.

[0208] The principle of reobservation loops is to allow the system to improve its success rate by making multiple attempts when the confidence level for identification is insufficient. Each reobservation may change one or more observation conditions (such as position, angle, supplementary lighting, etc.), thereby re-performing identification under different conditions. Loop control ensures that the number of attempts is limited, avoiding infinite loops; at the same time, monitoring of safety anomalies ensures that the system can stop in time under any circumstances, protecting the safety of personnel and equipment. Controlled multiple attempts improve the identification success rate, while safety monitoring ensures the system's security. Compared to solutions that abandon identification after a single failure, reobservation loops effectively improve the system's availability in complex environments.

[0209] In some specific embodiments, the use of multi-view visual perception components to visually identify the candidate charging port area employs a multi-stage identification process from far to near:

[0210] The first stage involves identifying the overall outline and orientation of the vehicle based on the radar point cloud. This stage is performed when the SRT is far from the vehicle (e.g., 3m to 5m). Its main purpose is to determine the vehicle's position and orientation in space, providing a navigation reference for subsequent approach.

[0211] In the second stage, the candidate area for the charging port is determined based on the vehicle model information, the side area of ​​the vehicle body, or user input. For example, if the vehicle model is an electric vehicle of a certain brand, the system reads from the database whether the charging port is located on the left front side or the right rear side of the vehicle, and then delineates the candidate area on the corresponding side of the vehicle body.

[0212] The third stage involves identifying the charging port cover boundary, socket outline, socket center, and socket direction within the candidate charging port area. This stage is performed when the SRT approaches approximately 1m to 1.5m, using high-resolution images to finely identify the structural features of the charging port.

[0213] In the fourth stage, the recognition results are converted into the target pose in the robotic arm's base coordinate system by combining the hand-eye calibration matrix. This stage converts the position of the charging port in the image obtained from visual recognition into a three-dimensional target that the robotic arm can execute.

[0214] In some specific embodiments, the multi-stage identification process from far to near includes:

[0215] If the recognition of the overall vehicle outline and body orientation fails in the first stage, the observation point is adjusted or the approach path is replanned; for example, if the radar point cloud quality is poor and the vehicle outline cannot be accurately recognized, the SRT moves laterally by about 0.3m to change the scanning angle and tries again.

[0216] If the candidate area for the charging port fails to be determined in the second stage, the viewpoint is switched or the search area is expanded; for example, if the charging port cannot be determined on which side under the current viewpoint, SRT slowly drives around the vehicle for about half a circle to observe from the other side.

[0217] If the third stage charging port cover recognition fails, the system will perform supplementary lighting, move closer, or request manual confirmation. For example, if the charging port cover cannot be recognized due to insufficient lighting, the system will turn on the supplementary light; if the image resolution is insufficient due to the distance, the system will move closer by 0.2m; if all the above measures fail, the system will request remote manual confirmation.

[0218] If the target pose after the coordinate transformation in the fourth stage is unattainable or lacks accuracy, the system reverts to the pre-inserted pose and re-executes the recognition. For example, if the hand-eye calibration parameters are biased, resulting in insufficient target pose accuracy, the system reverts and recalibrates or uses backup calibration parameters.

[0219] In some specific embodiments, the insertion operation is divided into five stages: coarse positioning, fine positioning, pre-insertion, force-controlled insertion, and locking confirmation.

[0220] In the coarse positioning stage, the robot chassis completes the adjustment of the vehicle's lateral distance and heading angle. In this stage, the SRT chassis is adjusted to a posture parallel to the vehicle's lateral plane using real-time radar ranging results, with a distance of approximately 0.5m to 0.8m from the target, providing a suitable chassis posture basis for the robotic arm operation.

[0221] In the precision positioning phase, the robotic arm aligns with the muzzle to the pre-insertion position; the robotic arm moves from the initial posture to the pre-insertion position, precisely positioning the end of the charging gun approximately 50mm in front of the charging port. This phase is primarily position-controlled at a moderate speed (approximately 0.1m / s to 0.2m / s).

[0222] During the pre-insertion phase, the robotic arm approaches the socket at a low speed; it slowly advances along the normal direction of the socket, bringing the tip of the charging gun into initial contact with the charging port socket. The speed decreases significantly during this phase, preparing for the force-controlled insertion that will begin.

[0223] During the force-controlled insertion phase, the robot advances along the normal direction of the insertion port and makes fine adjustments based on axial resistance and attitude deviation. The robot enters a force / position hybrid control mode, using the target insertion force as the control target, while monitoring the lateral torque to detect attitude deviation.

[0224] During the locking confirmation phase, successful insertion is confirmed by the gun lock signal, electrical handshake status, or end displacement change. Once the charging gun is fully inserted, the gun lock mechanism automatically locks, and electrical communication is established between the vehicle and the charging equipment. The system then makes a comprehensive judgment on whether the insertion was successful.

[0225] In some specific embodiments, the determination of the preset maximum allowable insertion force, maximum allowable lateral torque, and maximum insertion speed includes: setting the target insertion force, the maximum allowable insertion force, the maximum allowable lateral torque, the maximum insertion speed, and the maximum insertion depth based on the charging gun structure, socket specifications, robotic arm stiffness, muzzle attitude error, and test calibration results, and storing the above thresholds in the robotic arm controller parameter table.

[0226] In some specific embodiments, when the axial insertion force exceeds the maximum permissible insertion force or the lateral moment exceeds the maximum permissible lateral moment, the method further includes at least one of the following abnormal conditions detection and processing:

[0227] When it is detected that the insertion depth is insufficient but the resistance continues to increase, the end displacement is inconsistent with the commanded displacement, or the muzzle attitude error exceeds the preset attitude error threshold, the advancement is immediately stopped and the arm is withdrawn to the pre-insertion pose or withdrawal pose, and the charging port attitude is re-identified, the observation angle is adjusted, or the robotic arm insertion trajectory is replanned.

[0228] In some specific embodiments, determining that insertion is complete when the end displacement reaches a preset insertion depth and the gun lock signal or electrical handshake status confirms the insertion specifically includes:

[0229] The insertion is successfully determined by combining the feedback of the end displacement reaching the preset insertion depth, the decrease or stabilization of the insertion force, the gun lock signal, the electrical handshake status, and the charging interface status.

[0230] If the mechanical insertion is in place but the gun lock or electrical handshake is not confirmed, charging permission will not be released, and a slight retraction, realignment, or abnormal retraction will be performed.

[0231] In some other embodiments, combined Figure 2 Furthermore, a charging port identification system based on radar and vision fusion is proposed, with a time synchronization module 201 used for processing point cloud data collected by the fuselage radar. and the image data acquired by the multi-view visual perception component Perform time synchronization to obtain the perception dataset at the same recognition time;

[0232] The security observation point generation module 202 is used to generate points from the point cloud data. Extracting the vehicle's lateral plane Vehicle boundaries and the real-time distance between the robot and the vehicle. And based on the real-time distance The robot's chassis motion envelope, the reachable range of the robotic arm, and the safe distance threshold. Generate security observation points ;

[0233] The visual recognition module 203 is used to control the robot to move to the safe observation point. The multi-view visual perception component is used to determine the candidate charging port area based on radar data. Visual recognition is performed to obtain two-dimensional candidate boxes for the charging port. ;

[0234] The fusion positioning module 204 is used to utilize the two-dimensional candidate frame of the charging port. With respect to the vehicle's lateral plane The fusion process is performed to calculate the three-dimensional center position of the charging port. and socket normal ;

[0235] Target pose calculation module 205, used for calculating the target pose based on the three-dimensional center position and socket normal Combined with the hand-eye calibration relationship of the robotic arm Based on the geometric parameters of the charging gun, calculate the pre-insertion pose of the robotic arm. and insertion target pose ;

[0236] Confidence evaluation module 206 is used to evaluate the confidence level based on the two-dimensional candidate box. Image recognition confidence Geometric consistency of 3D fusion Time stability Combined confidence level for robotic arm reachability calculation ;

[0237] Control output module 207, used to determine the overall confidence level The preset threshold is reached and the insertion target pose is If the target is within the reachable space of the robotic arm, then the insertion target pose will be... Output the identification result; otherwise, trigger a re-observation action based on the identification anomaly type. The re-observation action includes adjusting the robot's observation position, observation angle, or supplementary lighting.

[0238] In other embodiments of the present invention, an electronic device 400 is disclosed, such as... Figure 3 As shown, the electronic device may include: one or more processors 401; a memory 402; a display 403; one or more application programs (not shown); and one or more computer programs 404. These devices can be connected via one or more communication buses 405. The one or more computer programs 404 are stored in the memory 402 and configured to be executed by the one or more processors 401. The one or more computer programs 404 include instructions that can be used to perform actions such as... Figure 1 And the steps in the corresponding embodiments.

[0239] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0240] In the various embodiments of this invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0241] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0242] The above description is merely a specific implementation of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of the present invention should be covered within the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention should be determined by the protection scope of the claims.

Claims

1. A charging port recognition method based on radar and vision fusion, applied to the automatic recognition of vehicle charging ports by a general-purpose unibody wheeled robot in mobile charging scenarios, wherein the robot is equipped with a body radar and a multi-view visual perception component, characterized in that... Includes the following steps: Point cloud data collected by the fuselage radar and the image data acquired by the multi-view visual perception component Perform time synchronization to obtain the perception dataset at the same recognition time; From the point cloud data Extracting the vehicle's lateral plane Vehicle boundaries and the real-time distance between the robot and the vehicle. And based on the real-time distance The robot's chassis motion envelope, the reachable range of the robotic arm, and the safe distance threshold. Generate security observation points ; Control the robot to move to the safe observation point. The multi-view visual perception component is used to determine the candidate charging port area based on radar data. Visual recognition is performed to obtain two-dimensional candidate boxes for the charging port. ; Using the two-dimensional candidate box of the charging port With respect to the vehicle's lateral plane The fusion process is performed to calculate the three-dimensional center position of the charging port. and socket normal ; Based on the three-dimensional center position and socket normal Combined with the hand-eye calibration relationship of the robotic arm Based on the geometric parameters of the charging gun, calculate the pre-insertion pose of the robotic arm. and insertion target pose ; According to the two-dimensional candidate box Image recognition confidence Geometric consistency of 3D fusion Time stability Combined confidence level for robotic arm reachability calculation If the overall confidence level The preset threshold is reached and the insertion target pose is If the target is within the reachable space of the robotic arm, then the insertion target pose will be... Output the identification result; otherwise, trigger a re-observation action based on the identification anomaly type. The re-observation action includes adjusting the robot's observation position, observation angle, or supplementary lighting.

2. The method according to claim 1, characterized in that, The time synchronization specifically includes: Establish a unified time reference for the point cloud data. The image data The robot's body pose data and robotic arm joint status data Distribute timestamps separately , , , ; Based on the timestamp, radar frames, visual frames, pose frames, and joint state frames with a time difference less than a preset maximum time difference threshold are selected as the perception dataset for the same recognition time. Motion compensation is performed on the time difference between the selected data frames to map the data to the same fusion time. ; If the time difference of any sensor data exceeds the threshold or a key frame is missing, the synchronization is deemed invalid, reducing the subsequent overall confidence level. This may trigger data re-collection.

3. The method according to claim 1, characterized in that, Generate security observation points The steps include: From the point cloud data The vehicle's lateral point cloud is separated and fitted to obtain the vehicle's lateral plane. Vehicle external boundary frame ; Calculate the reference point on the robot to the lateral plane of the vehicle. The minimum distance is taken as the real-time distance. ; According to the vehicle lateral plane The inner boundary, parking space boundary, or obstacle boundary is used as the outer boundary. A lateral passable zone is generated by combining the robot's chassis width, motion envelope, robotic arm sweep space, and cable dragging channel width. Multiple candidate observation points are generated within the passable zone. And for each candidate observation point Based on the real-time distance Safe distance threshold Safety constraints are imposed based on visual field coverage, robotic arm accessibility, and cable dragging continuity; points that pass these constraints are selected as the safety observation points. ; If no candidate observation points meet the conditions after the screening process... If the observation point fails, an observation point failure signal will be output, and a request will be made to adjust the service docking point of the mobile storage and charging equipment or the station position of the robot.

4. The method according to claim 1, characterized in that, Two-dimensional candidate box for charging port The method to obtain it is as follows: The robot reaches the safe observation point. Then, the multi-view visual perception component in the charging port candidate area The device performs close-range recognition, and the objects to be recognized include at least the charging port cover boundary, the socket outline, the center of the socket, the direction of the socket, the reflective area and the obstructed area. The robot is based on a vehicle model database. Or historical charging port location record The location, relative height, and shape information of the charging port are read and used as priors for visual search to constrain the candidate areas for the charging port. The search window retrieves one or more two-dimensional candidate boxes for charging ports. Output; When the multi-view visual perception component includes a binocular camera or a depth camera, it utilizes the acquired depth point cloud. Alternatively, binocular parallax data can be used to correct the depth value corresponding to the candidate box obtained by visual recognition, in order to compensate for the depth deviation caused by local concavity and convexity of the vehicle body's side plane; When light reflection causes the recognition confidence of the charging port in any viewpoint image to fall below a preset image confidence threshold, the multi-view visual perception component performs weighted fusion based on the temporal stability and geometric consistency of the multi-view results, and selects the candidate box with the highest weighted score as the two-dimensional candidate box for the charging port. Output.

5. The method according to claim 1, characterized in that, Using the two-dimensional candidate box of the charging port With respect to the vehicle's lateral plane The fusion process is performed to calculate the three-dimensional center position of the charging port. and socket normal ,include: Obtain the two-dimensional candidate box of the charging port center pixel coordinates Using the camera intrinsics of the multi-view visual perception component Formation of space rays The formula is as follows; ； The space ray Transform to the map coordinate system or the robot arm base coordinate system, and calculate the spatial ray. With respect to the vehicle's lateral plane The intersection point is taken as the three-dimensional center position of the charging port. ; Based on the visual recognition results of the socket outline and the normal of the vehicle's lateral plane... The normal of the socket was determined by the multi-view observation results. .

6. The method according to claim 5, characterized in that, The space ray Transform to the map coordinate system or the robot arm base coordinate system, and calculate the spatial ray. With respect to the vehicle's lateral plane The intersection point is taken as the three-dimensional center position of the charging port. ,include: The spatial ray and the vehicle's lateral plane Solve for the intersection point: ； Where λ is a ray parameter, representing the distance from the camera's optical center along the ray direction to the vehicle's lateral plane, the ray parameter λ extends from the vehicle's lateral plane. The expression: ； Substituting into the ray equation, we get: ； in, The three-dimensional coordinate components of the optical center position O of the camera are given. The ray direction vector directional components, The vehicle's lateral plane The expression coefficients; The intersection point The three-dimensional center position of the charging port is determined after confidence evaluation and screening. Output.

7. The method according to claim 1, characterized in that, Based on the three-dimensional center position and socket normal Combined with the hand-eye calibration relationship of the robotic arm Based on the geometric parameters of the charging gun, calculate the pre-insertion pose of the robotic arm. and insertion target pose ,include: Based on the three-dimensional center position of the charging port and the socket normal Establish the transformation of the charging port coordinate system relative to the vision acquisition unit coordinate system. ; According to the extrinsic parameters of the vision acquisition unit relative to the base coordinate system of the robotic arm Transform the charging port coordinate system to the robotic arm base coordinate system; Based on the compensation transformation of the charging gun tip relative to the charging port insertion posture Calculate the insertion target pose of the robotic arm: ； According to the insertion target pose Along the normal direction of the socket Offset in the opposite direction by the preset pre-insertion distance The pre-insertion pose is obtained as follows: 。 8. The method according to claim 1, characterized in that, Calculate the overall confidence level The steps, and whether to insert the target pose. The steps for outputting the recognition results are as follows: Obtain the two-dimensional candidate box of the charging port Corresponding visual recognition confidence level Shape consistency between the candidate area and the charging port geometry The three-dimensional center position of the charging port With respect to the vehicle's lateral plane Geometric consistency Temporal stability of multi-frame recognition results And penalties for degree of occlusion ; According to the preset weighting The overall confidence level is obtained by weighted summation of the above items: ； The weights for each item are determined through calibration. The overall confidence level Compared with the preset confidence threshold Compare the results and combine them with the safety distance judgment and the pose error judgment. If both conditions are met... Real-time distance Greater than or equal to the safe distance threshold Safety margin Greater than or equal to zero, and the insertion target pose pose estimation error If the error is less than the preset error threshold, the current recognition result is deemed valid, and the process of inserting the gun is allowed. If the comprehensive confidence level Below the confidence threshold If there is a conflict in the number of candidates, the pose estimation error exceeds the limit, or the robotic arm cannot reach the target, the gun insertion stage will not be entered. A corresponding re-observation action will be generated based on the identified anomaly type. The re-observation action includes one or more of the following: lateral fine adjustment, re-observation from a different perspective, supplementary lighting, or re-shooting at a closer location.

9. The method according to claim 1, characterized in that, Based on the output of the insertion target pose During the insertion of the gun, the insertion operation adopts a force / position hybrid control method, specifically including: The axial insertion force is detected in real time by a force / torque sensor or joint current feedback at the end of the robotic arm. and lateral moment ; The maximum allowable insertion force is preset based on the charging gun structure, socket specifications, and calibration results. Maximum permissible lateral moment and maximum insertion speed ; During the pre-insertion phase, the device moves to the pre-insertion pose using position control. During the formal insertion phase, along the normal direction of the socket. Advance, and according to the axial insertion force Insertion force with target Adjust propulsion speed and attitude based on deviations; When the axial insertion force exceeds the maximum permissible insertion force or the lateral moment exceeds the maximum permissible lateral moment, the advancement should be stopped immediately and the device should be retracted to the pre-insertion position. Or retract position This triggers re-identification; Insertion is considered complete when the end displacement reaches the preset insertion depth and the gun lock signal or electrical handshake status confirms the insertion.

10. A charging port identification system based on radar and vision fusion, characterized in that, Includes a time synchronization module for processing point cloud data acquired by the fuselage radar. and the image data acquired by the multi-view visual perception component Perform time synchronization to obtain the perception dataset at the same recognition time; The security observation point generation module is used to generate points from the point cloud data. Extracting the vehicle's lateral plane Vehicle boundaries and the real-time distance between the robot and the vehicle. And based on the real-time distance The robot's chassis motion envelope, the reachable range of the robotic arm, and the safe distance threshold. Generate security observation points ; The visual recognition module is used to control the robot to move to the safe observation point. The multi-view visual perception component is used to determine the candidate charging port area based on radar data. Visual recognition is performed to obtain two-dimensional candidate boxes for the charging port. ; The fusion positioning module is used to utilize the two-dimensional candidate frame of the charging port. With respect to the vehicle's lateral plane The fusion process is performed to calculate the three-dimensional center position of the charging port. and socket normal ; The target pose calculation module is used to calculate the target pose based on the three-dimensional center position. and socket normal Combined with the hand-eye calibration relationship of the robotic arm Based on the geometric parameters of the charging gun, calculate the pre-insertion pose of the robotic arm. and insertion target pose ; The confidence evaluation module is used to evaluate the confidence level based on the two-dimensional candidate boxes. Image recognition confidence Geometric consistency of 3D fusion Time stability Combined confidence level for robotic arm reachability calculation ; The control output module is used to determine the overall confidence level. The preset threshold is reached and the insertion target pose is If the target is within the reachable space of the robotic arm, then the insertion target pose will be... Output the identification result; otherwise, trigger a re-observation action based on the identification anomaly type. The re-observation action includes adjusting the robot's observation position, observation angle, or supplementary lighting.

Images