System and method for supporting the autonomous charging of an electric vehicle

Abstract

A system for supporting the autonomous connection of an EV charging connector into an EV inlet, the system comprising: - a controllable robot adapted to support the EV charging connector; - an imaging sensor in communication with the robot, the imaging sensor adapted to capture image data within a field of view; and - a controller including a processor and a non-transitory memory storing instructions executed by the processor to configure the controller, the controller configured to: - determining the presence of at least one target component within the field of view, the target component selected from at least one of the EV charging connector and a moveable component of the robot, - when the target component is found to be present, analysing an image received from the image sensor and determining a pose of the target component, utilizing at least the received image and a pose recognition algorithm, and - controlling the robot based on the determined pose.

Description

[0001] SYSTEM AND METHOD FOR SUPPORTING THE AUTONOMOUS CHARGING OF AN ELECTRIC VEHICLE

[0002] ABSTRACT

[0003] The invention relates to a system and method for supporting the autonomous charging of an electric vehicle. The system comprises a controllable robot adapted to support the EV charging connector, an imaging sensor and a controller configured to: determining the presence of a target component selected from at least one of the EV charging connector and a moveable component of the robot; when the target component is found to be present, analysing an image received from the image sensor and determining, based on at least the image and a pose recognition algorithm, a pose of the target component; and controlling the robot based on the determined pose.

[0004] TECHNICAL FIELD

[0005] The present invention relates to the field of autonomous charging systems for electric vehicles. Specifically, the invention focuses on robotic systems and methods designed to automate the charging process.

[0006] BACKGROUND OF THE INVENTION

[0007] The surge in electric vehicle (EV) adoption has propelled a substantial demand for autonomous charging solutions, often referred to as autonomous charging systems. Within the scope of this disclosure, these systems integrate robotic technologies designed to automate the charging process, addressing the needs of EV users, manufacturers, OEM's, and charging station operators, among others. Autonomous charging systems are generally equipped with actuated robotic systems, computer vision technologies, and neural networks, which facilitate recognizing the charging inlet to enable the mating and unmating of the charger into the vehicle inlet.

[0008] Devices and related methods for this purpose are known in the state of the art, for instance from the international patent applications PCT / NL2020 / 050266, PCT / NL2021 / 050115, PCT / NL2021 / 050410, PCT / NL2021 / 050495, PCT / NL2021 / 05061, PCT / EP2022 / 062233, and PCT / EP2022 / 088101, PCT / EP2023 / 063018, from the same applicant, all of which are herein incorporated by reference. Systems in accordance with the present disclosure are configured to support several connectors, including but not limited to Type 1, Type 2, CCS-1, CCS-2, CHAdeMO, Tesla, NACS, and MCS connectors, among others.

[0009] Recent progress in autonomous charging systems has brought about enhancements in the pose determination of the electric vehicle's charging inlet and the alignment of the charging connector with the inlet. However, several challenges are associated with such systems.

[0010] US 2021001736 Al describes an automated method for connecting a charging connector to a vehicle's charging connector socket. The process includes the initial optical capture of the charging connector socket or the vehicle as a first image using a movable image capturing unit attached to the charging connector. The position of the charging connector socket or the vehicle is determined based on this image. In addition to the charging plug, the end effector has a first image capturing unit in the form of a camera. This camera is stationary and located close to the charging plug, above it, with its detection area oriented in the direction in front of the charging plug. The viewing direction of the camera substantially corresponds to the joining direction or the joining axis of the charging plug, ensuring alignment during the connection process.

[0011] An inherent challenge in existing systems is their restricted ability to acquire comprehensive data related to the vehicle and robot components. These systems predominantly prioritize vision-based methods for detecting the pose of the EV inlet. Moreover, existing systems rely primarily on sensors integrated into their components, coupled with complex parametric models or black-box models like an artificial-intelligence-based model, to discern the behaviour, position, and responsiveness of these components.

[0012] It is therefore desired to propose a system and method of the aforementioned type that mitigates or eliminates the described drawbacks. Specifically, the invention seeks to enable an automated charging process by utilizing image data for determining the behaviour and position of essential components throughout the autonomous charging cycle.

[0013] SUMMARY OF THE INVENTION

[0014] It is an object of the present invention to overcome the above-described problems and to provide an autonomous charging system which overcomes or elevates at least one of the challenges above.

[0015] It is a further object of the present invention to provide such a system which is capable of assessing the motion, behaviour and positioning of essential components during the charging cycle utilizing image and computer vision mechanisms.

[0016] In a first embodiment, the present invention provides for a system for supporting the autonomous connection of an EV charging connector into an EV inlet, the system comprising: a controllable robot adapted to support the EV charging connector; an imaging sensor in communication with the robot, the imaging sensor adapted to capture image data within a field of view; and a controller including a processor and a non-transitory memory storing instructions executed by the processor to configure the controller, the controller configured to: determining the presence of a target component selected from at least one of the EV charging connector and a moveable component of the robot; when the target component is found to be present, analysing an image received from the image sensor and determining a pose of the target component based on at least the image and a pose recognition algorithm; and controlling the robot based on the determined pose

[0017] In a second embodiment, the present invention provides for a method for controlling a system for the autonomous charging of an electric vehicle, the method comprising determining the presence of a target component selected from at least one of the EV charging connector and a moveable component of the robot, when the target component is found to be present, analysing an image received from the image sensor and determining a pose of the target component based on at least the image and a pose recognition algorithm; and controlling the robot based on the determined pose.

[0018] In a third embodiment, the present invention provides a non-transitory computer program product, comprising program code which is stored on a machine-readable medium, and having computer-executable instructions for performing a method in accordance with the present disclosure.

[0019] Systems and methods in accordance with several embodiments enable realizing an autonomous mating and charging cycle by utilizing pose information from image feedback of one or more several components of the system.

[0020] Advantageously, the system and method in accordance with the invention utilize image data with a field of view which can include not only the EV inlet but the charging connector and other movable components of the robotic system, enhancing the responsiveness of the system to external factors and to effects not captured in parametric models. Thus, in several embodiments, a mating, charging and unmating cycle can be realized even where one or more of those components are not readily visible in the camera's field of view by obtaining pose information of other components which remain visible in the camera field of view. Moreover, the system's reliance on vision means allows assessing robot behaviour without necessarily relying solely on sensors and complex parametric models of robot components. A system and method in accordance with several embodiments utilizes imaging sensor means with a field of view which allows utilizing image data to gather comprehensive information associated to the robot behaviour.

[0021] Specific embodiments of the present disclosure are herein described.

[0022] It may be provided that controlling the robot based on the determined pose includes at least one of: determining a positioning motion and causing the robot to perform such motion, performing a mating of the charging connector into the EV inlet when the determined pose is within a predetermined position range for mating; and performing an unmating of the charging connector. It may be provided that the positioning motion includes at least one of a rotating, translating, retracting, locking, unlocking, wiggling, aligning, repositioning, vibrating, or a combination thereof. A positioning motion may include a motion to bring the target component to a predetermined position range for mating. Preferably, such corrective action is a motion of the robotic arm, the end effector or a charging connector manipulation.

[0023] It may be provided that the pose of the target component is visually determined by identifying recognizable features associated to such target component.

[0024] It may be provided that the controller is configured to visually determine if the connector is at a predetermined position range for mating.

[0025] It may be provided that the presence of a target component is determined by utilizing at least one of a received image, a signal from the EV, and the robot's internal knowledge.

[0026] It may be provided that if one or more recognizable features of the target component are not suitable or visible in the received image for pose determination, the pose of the target component is determined by utilizing a referential marker associated with such target component. In this context, "not suitable" denotes instances where the clarity, visibility, or distinctiveness of the recognizable features within the received image falls below a predefined threshold, impeding reliable pose determination based on these features. Alternatively or additionally, the pose of the target component may be determined by utilizing a previously acquired image of the target component, a referential marker associated to such target component, a known past pose of the target component, data associated with such target component (e.g., data from robot encoders, robot joint positions, and prior knowledge about such target component), a moving speed of the target component, or combinations thereof. By such implementation, the controller can achieve a reliable pose estimation even when not all recognizable features are visible in a received image.

[0027] It may be provided that a direct pose estimation of the target component can be realized. Pose determination may take place based on a single image or multiple images captured at different times. In some cases, a different image may be used to complete pose detection of a first target component, i.e., charging connector and a second image for a second target component, i.e., EV inlet. In certain embodiments, at least two or more images are received and analysed to determine a pose of the target component. Thereby, where the target component is the charging connector, the controller can obtain pose information of the controller by analysing one or more of the received images.

[0028] It may be provided that pose determination of a single target component, e.g., the EV inlet, may in several cases be sufficient to enable a successful charging cycle. In some cases, determination of poses of two or more target components is desirable, such as the pose of both the charging connector and EV inlet and to manipulate the endeffector so that both components are positioned at a predetermined pose relative to each other. It may be provided that, upon determining a misalignment between two target components, or any of such components not being at a desired position, the controller determines a suitable positioning motion. Such positioning motion may include translational adjustments, rotational realignments, or a combination of both, depending on the nature and extent of the misalignment.

[0029] It may be provided that the pose of the target component is determined by identifying recognizable features associated to such target component. Thereby, the pose recognition algorithm can be advantageously trained to achieve pose estimation based on such recognizable features.

[0030] It may be provided that the pose of the target component is determined by identifying recognizable features associated with, or in proximity to, such target component. In the event that one or more recognizable features of the target component are not suitable or visible in the received image for pose determination, the pose of the target component can determined by utilizing a pose of an auxiliary component. By such implementation, pose estimation can be achieved even where such target component is not readily visible, when it has to be recognized from a relatively large distance, or when the same is (partially) occluded by the EV charging connector or a movable component of the robot. In this scenario, the system has knowledge of the relative pose between the target and the auxiliary component. Alternatively, or additionally, the controller can determine a relative position between the pose of the auxiliary component and the target component by utilizing additional knowledge including at least one of historical data, machine learning algorithms, or predictive modelling. Thereby, pose determination reliability is ensured, for example, to facilitate charging connector positioning even in limited visual conditions.

[0031] It may be provided that the auxiliary component is a robot component selected from at least one of the end-effector, a compliance mechanism, a sensor, a locking mechanism, a connector attachment device, a robot arm actuator, and a joint assembly. As such, where the target component is the charging connector, and the charging connector is not readily visible, the controller may acquire pose information of an auxiliary component mechanically coupled to it, e.g., the end effector, and based on that information, estimating the current pose of the charging connector.

[0032] It may be provided that the target component is further selected from the EV inlet, and the controller determines its pose based on at least a received image and a pose recognition algorithm. Where the target component is further selected from the EV inlet, it may be provided that the auxiliary component is a vehicle component selected from at least one of an inlet cover, an inlet locking mechanism, an inlet light indicator, an inlet port frame, an inlet port housing.

[0033] It may be provided that the robot comprises an end effector configured to support, in an either releasable manner or not, a charging connector.

[0034] It may be provided that the controller performs one or more subsequent determinations of the pose of the target component and / or the auxiliary component after the robot has performed a positioning motion. It may be provided that the controller is configured to determine a pose of the EV inlet, positioning the charging connector towards the EV inlet, performing a new determination of the EV inlet pose through direct pose estimation or through indirect pose estimation, determining a pose of the charging connector, positioning the charging connector at a predetermined pose for mating, determining if the connector is at a predetermined position range for mating, and performing a mating of the connector into the inlet. Thereby, this iterative process allows for a continuous update on the pose of both the target and auxiliary components during manipulation of the robot. It may be provided that the controller is configured to determine if features of the auxiliary component of the EV inlet have moved compared to a previous image, indicating that the EV inlet has moved; and to reposition the charging connector to allow for a new pose determination of the charging connector.

[0035] It may be provided that the controller is configured to visually determine deviations of the pose of a robot moveable component compared to where a robot parametric model would expect that component to be based on sensor information.

[0036] It may be provided that at least one movable joint positioned between the charging connector and a component to which the imaging sensor is rigidly mounted. In this context, a "movable joint" is defined as a mechanical connection allowing relative motion between the EV charging connector and the imaging sensor. Thereby, the camera and its field of view may not be affected by any passive or active motion of the charging connector. It may be provided that at least two, three, four, five or six movable joint positioned between the charging connector and a component to which the imaging sensor is rigidly mounted.

[0037] It may be provided that at least one joint separating the EV charging connector and the imaging sensor is actively movable. In this arrangement, an "actively movable joint" is defined as a mechanical connection allowing controlled motion between the EV charging connector and the imaging sensor. It may be provided that at least one, two, three, four, five or six joints separating the EV charging connector and the imaging sensor are actively movable. It may be provided that at least one, two, three, four, five or six joints separating the EV charging connector and the imaging sensor are passively movable.

[0038] It may be provided that the controller can determine a compliant deflection as a result of an external force acting on a robot component by determining a pose of a target component and comparing it to an expected pose of such component. Herein, the target component can be the charging connector, the end effector, or any moveable component of the robot, and external forces acting can include forces caused by environmental conditions, physical obstructions, or variations in the charging interface. Thereby, the system can assess e.g., whether the connector is being subject to an external force by determining the pose of the charging connector and comparing it to an expected pose. As used herein, an expected pose denotes the predetermined or anticipated spatial configuration or position that the charging connector is expected to exhibit under normal or specified conditions, providing a reference for evaluating compliant deflection.

[0039] It may be provided that the target component is a compliance assembly of the robot and the controller can determine, based on a pose of the compliance assembly, a compliant deflection as a result of an external force acting on a robot component, such as on the robotic arm, the end effector, the charging connector or any moveable component of the robot. Preferably, the controller is configured to determine a compliant deflection by visually recognizing features of the compliance assembly or the charging connector, estimating the respective pose, and comparing that visually estimated pose to a pose where the parametric model would expect the pose of such component to be, such as an expected pose based on sensors which track joint positions. By observing pose variation in the compliance assembly, the system can assess and quantify the impact of external forces, such as those exerted on the robotic arm, end effector, charging connector, or other movable components.

[0040] It may be provided that determining compliant deflection includes determining a displacement or a rotation of the compliance assembly, or any component associated thereto. Thereby, the system can react to external forces influencing the charging connector or a robot component, including collisions with the electric vehicle or EV inlet, the dynamics of the charging cable's weight and movement, wind forces, and other environmental factors. In some embodiments, the controller is adapted to determine an external force acting the EV charging connector or any moveable component of the robot, and causing the robot to perform a positioning motion.

[0041] It may be provided that the pose of the target component is a current pose or a past pose. Herein, "current pose" refers to the real-time position and orientation of the target component in the charging system, while "past pose" signifies a historical position and orientation of the same component at a previous moment in time. This arrangement allows considering the target component's movement history and facilitates trajectory predictions.

[0042] It may be provided that the pose of the target component is determined by further utilizing a previously acquired image of the target component. In this context, "previously acquired image" refers to an image captured and stored during a prior instance of the charging operation. This arrangement allows for utilizing historical visual data to enable pose determination. By incorporating information from past images, the system improves its ability to handle scenarios where real-time visibility might be compromised. It may be provided that the controller determines the pose of the target component by utilizing received image(s) in conjunction with pre-existing data recorded from the target component. This data can include a range of parameters, such as historical pose information, geometric characteristics, or other relevant attributes of the target component.

[0043] It may be provided that the pose of the target component is determined by further utilizing a moving speed of the target component.

[0044] It may be provided that the pose of the target component is determined by further utilizing a correlation between a previously acquired image and a known past pose of the target component. In this arrangement, the system utilizes historical data, establishing a correlation between past poses and corresponding images. This correlation enables the controller to predict the current pose of the target component based on the recognized patterns, contributing to pose determination.

[0045] It may be provided that the pose of the target component is determined by utilizing previously acquired information, including a relative position between the target and auxiliary component. In this arrangement, the system relies on historical data encompassing the relative positions of the target and auxiliary components. This information serves as a reference point for deducing the pose of the target component in real-time scenarios.

[0046] It may be provided that the controller simultaneously and continuously identifies a pose of the target component while said target component undergoes a motion. Thereby, the controller can determine, e.g., the pose of the charging connector while such charging connector is being moved (manipulated) by the robot. By such an implementation, the controller can acquire updated information regarding unexpected changes in the pose of a target component, such as when the charging connector is subject to external forces.

[0047] It may be provided that at least two or more images are received and analysed to determine a pose of the target component.

[0048] It may be provided that the controller determines, based on the received image or images, if a target component is out of the field of view and to trigger a motion command so that the target component is within the camera field of view. This configuration may facilitate, for example, to initiate a mating cycle when the system detects the presence of an EV, or an EV inlet within the robot workspace and trigger the robotic arm to bring the charging connector into proximity of the EV inlet and such, within the camera field of view.

[0049] It may be provided that the controller operates the end-effector upon detecting that a target component, such as the charging connector, has shifted outside the camera's field of view after an initial pose estimation of such target component. In such a case, the controller may manipulate the end effector so that the charging connector is brought back withing the camera field of view. This configuration may facilitate the utilization of the camera to discern unexpected movements on the robotic arm as well as reducing dependence on motion sensors embedded in the robot.

[0050] It may be provided that the controller assesses a property of a received image or the recognizable feature within the image. Preferably, the controller utilizes a first property threshold that defines a first acceptable limit of an property; a second property threshold that defines an acceptance range based upon the first property threshold; and determines if a value of the property in a received image is within the acceptance range. If the property value is within the accepted range, the image is deemed suitable for pose estimation. Conversely, if the property value falls outside the accepted range, the controller triggers a corrective action. The nature of the corrective action is determined as a function of the assessed property. Thereby, the suitability of a received image for enabling a pose estimation of a target or an auxiliary component can be assessed. . This limitation may stem from various factors, such as the received image lacking clarity in the recognizable features, a pixel density falling below a predefined threshold, or instances where the target component is (partially) obstructed within the field of view. Where a resolution falls outside of the acceptable range, the controller may conduct further processing so that the recognizable features are better delineated or emphasize a more detailed aspect of the target component. This may involve employing image enhancement techniques, refining edge detection algorithms, or adjusting the focal length of the camera to capture finer details, thereby enhancing the overall quality of the image for more accurate pose estimation.

[0051] BRIEF DESCRIPTION OF THE DRAWINGS

[0052] FIG. 1 is a schematic view illustrating an example of a charging system in accordance with one embodiment.

[0053] FIG. 2 is a schematic view illustrating an example of a charging system and a field of view in accordance with one embodiment.

[0054] FIG. 3 is a schematic view illustrating an example of a charging system in accordance with one embodiment.

[0055] FIG. 4 depicts a flowchart illustrating a method in accordance with the present disclosure

[0056] FIG. 5 is a schematic view illustrating an example of a charging system in accordance with one embodiment.

[0057] FIG. 6 is a schematic view illustrating an example of an EV inlet and associated recognizable features.

[0058] FIG. 7a and FIG 7b are a schematic view recognizable features coupled to a compliance system.

[0059] FIG. 8. is a block diagram identifying some components of a system for supporting the autonomous connection of an EV charging connector into an EV inlet in accordance with one embodiment.

[0060] FIG. 9 depicts a flowchart illustrating a method in accordance with the present disclosure

[0061] DETAILED DESCRIPTION OF THE INVENTION

[0062] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which currently preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided for thoroughness and completeness, and fully convey the scope of the invention to the skilled person. When used herein, the term robot or robotic may, but not necessarily must, conform to the traditional industry standards definition of a complete autonomous robot with sensory perception, decision-making capabilities, and mobility. Instead, in the context of the present invention, the term "robot" primarily refers to a controlled actuated mechanism or arm designed for specific tasks, such as the ones in accordance with the present invention. As such, the term "robot" in this context emphasizes the mechanized and programmable nature of the system rather than the strict dependence on a fully autonomous robot with general-purpose capabilities.

[0063] When used herein, an autonomous charging device (or simply, a robot in accordance with the present disclosure) denotes a system comprising comprise several hardware and software components. In particular, the ACD may comprise an end effector for supporting and moving a vehicle charging connector, a motion control unit and a computer vision unit for pose estimation which utilizes a pose recognition algorithm. Preferably, the computer vision unit part of, or in communication with the controller and is supported by a pose recognition algorithm and neural network trained to determine the pose of an object through an analysis of one or more images. The robot preferably comprises an end effector comprising means to support, either in a releasable manner or not, a vehicle charger connector, and enables such autonomous charging process. The robot may further include one or more links with one or more joints, and actuators (e.g., electric motors, stepper motors, and solenoids) coupled and operable to move the links in response to control or drive signals

[0064] When used herein, the motion control unit denotes a component to control the motion of the robot for, among others, mating the connector into a EV inlet.

[0065] When used herein, mating and unmating denote the processes of connecting and disconnecting, respectively, the charging connector with / from the electric vehicle's (EV) charging inlet. Mating involves the alignment and engagement of the charging connector with the EV inlet for the transfer of electrical energy during the charging cycle. Unmating, on the other hand, refers to the disengagement and disconnection of the charging connector from the EV inlet once the charging process is complete.

[0066] When used herein, the term "pose" of an object refers to its spatial orientation in three-dimensional space. The pose encompasses three dimensions of information, describing the object's rotation concerning the camera perspective. Additionally, or alternatively, the pose may involve three dimensions of information representing the object's translation relative to the camera. Within the scope of this disclosure, a pose determination is conducted for a target component integral to the mating and / or unmating process.

[0067] When used herein, an imaging sensor is communication with the robot, the imaging sensor being adapted to capture image data within a field of view. A controller in accordance with several embodiments is configured to analysing an image received from the image sensor and determining, based on at least the image and a pose recognition algorithm, a pose of the target component. The controller is further configured to controlling the robot based on the determined pose.

[0068] When used herein, a target component refers to any component relevant for enabling or supporting the autonomous mating or unmating, including but not limited to the EV charging connector, EV inlet, or any movable part of the robot. In a preferred embodiment of the invention, the target component is at least one of the EV charging connector and any movable part of the robot. Obtaining pose information about a target component can enable the mating and unmating of the connector and the EV inlet and determining corrective actions, such as the positioning the connector at a desired angle or at a certain distance relative to the EV inlet. In certain instances, determining pose information of a movable component of the robot may eliminate or reduce the need for force sensors and / or positioning actuators. When used herein, an auxiliary component is understood as a secondary element mechanically coupled to the target component, and its pose is utilized as a reference for determining the pose of the target component. An auxiliary component can be a robot component selected from at least one of the end-effector, a compliance mechanism, a sensor, a locking mechanism, a connector attachment device, a robot arm actuator, and a joint assembly. Likewise, an auxiliary component can be an EV component selected from at least one of an inlet cover, an inlet locking mechanism, an inlet light indicator, an inlet port frame, an inlet port housing.

[0069] When used herein, a direct pose estimation involves determining the target component current pose based on an obtained image wherein said target component, or a recognizable feature thereof, is visible in such image.

[0070] When used herein, an indirect pose estimation involves inferring the target component's pose through the analysis of contextual information, external references, or the relative position and orientation with respect to other components within the system, such as utilizing an auxiliary component, a referential marker, or a combination thereof.

[0071] When used herein, a referential marker, or simply, a marker is defined as a distinctive visual indicator or identifier physically linked to the target component, serving as a reference point for pose determination. A referential marker can include purposeful fiducial markers like checkerboard patterns, ArUco markers, or AprilTags.

[0072] When used herein, a mating of the charging connector can also include allowing the robot to carry out the necessary actions to move the charging connector relative the EV inlet.

[0073] When used herein, a predetermined position range for mating denotes a pose or alignment at which the charging connector should be positioned relative to the EV inlet to enable successful mating during the charging process. This may involve specific ranges of angular orientations, spatial distances, or alignment parameters of a target component of from one target component relative to another one.

[0074] When used herein, a charging cycle includes at least one of a home or idle stage, a positioning stage, a mating stage, a charging stage, an unmating stage. During the home or idle stage, the robotic arm and the charging connector assume an initial or home pose, signifying the starting position of the charging operation. In the positioning stage, the robot manipulates the charging connector into a predetermined position range, setting the stage for the subsequent mating process. At the mating stage, the connector is manipulated and engaged with the inlet. The charging stage includes the transfer of electrical energy to the vehicle post-successful mating. The unmating stage involves the disengagement of the charging connector from the EV inlet. In certain scenarios, the robotic arm may disengage the charging connector while it remains mated into the EV inlet. In all stages, the controller is adapted to acquire image information of any target component in order to facilitate accurate pose determination, adjustment, and real-time decision-making during each phase of the charging operation.

[0075] When used herein, a compliance assembly denotes a mechanism designed to absorb external forces and facilitate controlled movement. A compliance assembly is preferably a robot component which facilitates absorbing a collision or even impact between the connector and the vehicle or vehicle inlet, and allows the connector to move along with the vehicle within certain boundaries once connected. As used herein, a compliance assembly includes but is not limited to a compliant joint, a spring-loaded mechanism, a flexible structure, or any other suitable mechanism designed to provide flexibility and absorb shocks or impacts while maintaining the structural integrity of the system. As used herein, a compliant deflection is the amount of deformation or movement in response to the applied force. When used herein, image property can include various factors critical for accurate image processing, such as white balance, color saturation, image brightness, exposure, and gamma correction. Furthermore, the image property can encompass factors related to object visibility and clarity, such as occlusion value, perspective angle, distortion value, and pixel density. Other relevant properties may involve contrast, sharpness, resolution, and signal-to-noise ratio.

[0076] When used herein, image resolution can be defined by factors such as the number of pixel rows, the number of pixel columns, pixel density value, total pixel count, or any similar indicator. Pixel density, in this context, refers to the number of image pixels per unit area.

[0077] When used herein, robot internal knowledge refers to information stored within the robot's memory, potentially including predefined spatial arrangements, historical data, or any data accessible to the robot's control system, including a robot model.

[0078] Several algorithms have been described in the prior art which may be suitable for pose recognition making use of a single, or multiple images. Examples include convolutional neural network algorithms or "You Only Look Once" (YOLO) models, SolvePnP or Ransac, among others. Generally, the pose detection algorithm analyses geometric elements and / or surface patterns and / or other features of the target component. By comparing this information with known features of the target component, or with previously acquired information regarding the target component, the pose may be determined.

[0079] When used herein, robot model denotes an internal representation encompassing one or more functions designed to furnish the pose of robot components, such as the charging connector, within a coordinate system pertinent to the automated charging process— typically a coordinate system anchored to the robot's base, a robot component, or the EV's charging inlet. This determination is based on sensor information, notably data derived from joint sensors like encoders or potentiometers. Specifically, robot models may encompass functions delineating the interplay and behaviour of joints. These functions could take the form of parametric functions or black-box functions, such as neural networks. They elucidate the robot's kinematics, incorporating factors like bending (or other forms of compliant deflection), backlash, play, hysteresis, or control inaccuracies, contributing to a precise estimation of the pose of the robot components. In some cases, the robot model abstains from utilizing visual data, such as vision-based pose estimation of robot components. In some cases, controlling the robot includes adjusting the robot model utilizing the pose of the target component.

[0080] FIG. 1 is a diagram illustrating a system 10 for the autonomous connection of an EV charging connector into an EV inlet according to various embodiment of the present invention. A system illustrated in FIG. 1 comprises a robotic system including a robotic arm 20 adapted to support a charging connector 40 and an image acquisition device (a camera) 30. FIG. 1 also depicts an end-effector 23 and an EV inlet 50. A camera 30 is positioned on an upper section of the robotic arm 20, such as on the arm base 22, defining a camera's field of view 70 as the area captured by the camera.

[0081] In reference to FIG. 2, the camera 30 field of view 70 encompasses a portion of the charging connector 40 and an anticipated position where the charging connector 40, 41 is expected to be. Field of view 70 also encompasses the EV inlet 50. The field of view can also include a view of a moveable component 23 of the robot. The controllable robot may be translatable and / or rotatable along / around an axis in at least one to six degrees of freedom. The controllable robot 20 may be affixed to the fixed world.

[0082] FIG. 1 schematically depicts an embodiment where a camera 30 is mounted on the robot base (22). Notably, the camera's position exhibits variability, as depicted in both FIG. 1 and FIG. 2. The camera may be situated at different points on the robot 30, 31. Alternatively, in specific instances, it can be positioned externally 32, defining an alternative field of view. In all cases, the camera maintains communication with the controller.

[0083] In reference to FIG. 1 and FIG. 2, the camera is fixed to a supporting base 22 of the robot and remains stationary relative to the motions of the robot, the robotic arm, the end effector, and / or the charging connector. The camera 30 position remains substantially unchanged when the charging connector is manipulated by the robotic arm 21. In some cases, the camera is fixed to a moveable section of the robot and at least one moveable joint (not shown) is positioned between the camera and the section to which the camera is affixed. Thereby, the camera can remain stationary relative to the motion of the connector.

[0084] In reference to FIG. 3, the camera 31 may be attached to a movable section of the robot, such as the robotic arm 21. In this case, the camera moves along the robotic arm in line with the degrees of freedom encompassed by the robotic arm. In some cases, the

[0085] FIG. 4 is a flowchart illustrating a method 200 in accordance with several embodiments of the present invention. The method includes 210 determining the presence of a target component selected from at least one of the EV charging connector and a moveable component of the robot; 220 when the target component is found to be present, analysing an image received from the image sensor and determining, based on at least the image and a pose recognition algorithm, a pose of the target component; and 230 controlling the robot based on the determined pose. In some embodiments (not shown) the method 200 comprises determining the pose of the target component if one or more recognizable features of the target component are not visible in the received image by utilizing a referential marker associated to such target component. In some embodiments (not shown) the method 200 comprises determining the pose of the target component by identifying recognizable features associated to such target component and wherein if one or more recognizable features of the target component are not visible in the received image, the pose of the target component is determined by utilizing a pose of an auxiliary component. In some embodiments (not shown) the target component is a compliance assembly of the robot the method 200 comprises determining, based on a pose of the compliance assembly, a compliant deflection as a result of an external force acting on the robot, such as the robotic arm, the end effector, the charging connector or any moveable component of the robot.

[0086] Fig. 5 depicts a view of a system 10 in accordance with several embodiments of the present invention, including target component 100. In some cases, the target component can be a moveable component of the robot 120. In FIG. 5, a moveable component of the robot is depicted as 120, which can represent, for example a joint coupling two segments of the robotic arm. As it will be described in several passages of this disclosure, a target component may be another component, whether from the robot or from the EV, in accordance with the definitions given in the present disclosure. In some embodiments, the controller is configured to obtain pose information of a target component 100, such as the EV inlet, as depicted in FIG. 5.

[0087] In reference to FIG. 5, an exemplary embodiment of the present disclosure can be described in detail. The controller can determine the presence of the EV inlet 110 and determine its pose by utilizing recognizable features visible in an image received by camera 30. Once such pose is determined, the controller triggers the motion of the robotic arm in order to position the connector at a predetermined position range for mating. In several cases, once the connector 100 is positioned in the proximity of the EV inlet 110, the EV inlet 110, or some of its recognizable features may be obstructed by the charging connector 100 in the camera field of view. As such, it is desirable to obtain pose information of the charging connector 100 and based on that information, assess from the camera perspective whether the connector 100 is aligned with the inlet. In case of a satisfactory alignment, the controller can proceed to execute the mating of the connector into the EV inlet. Otherwise, the controller may generate a corrective motion so that the connector is positioned at a predetermined position range for mating. By such implementation, the mating of the charging connector can be completed even when some of the EV inlet recognizable features, i.e., pins and holes, are not visible in the camera field of view.

[0088] FIG. 6 exemplifies a target component 80, i.e., EV inlet including its pins 81 and holes 82, which are generally utilized as recognizable features to make a pose estimation of such EV inlet. Auxiliary components include EV inlet frame 83, EV inlet cover 84, and a referential marker, such as a QR code 85, whose position and / or relative position is known.

[0089] FIG. 7 depicts an exemplary embodiment where the camera field of view 70 includes a compliance assembly 110. The compliance assembly includes a spring loaded assembly 113, a compliance base 111 and a compliance effector 112. In the camera field of view, compliance base 111 and compliance effector 112 are readily visible, as depicted with arrows 71 and 72. When an external force acts on the connector or any moveable component of the robot, the compliance assembly 111 is adapted to compensate for such force and the relative position between compliance base 111 and compliance effector 112 shall change. The controller utilizes such information to assess the deflection of the compliance assembly and the external force acting on it.

[0090] In other cases, the compliance assembly 110 may include visual markers 30, 31, 32 such as depicted in FIG. 7b, which are visible from the camera perspective. By such implementation, the controller is adapted to receive an image with a field of view including at least one or more of such visual markers mechanically coupled to the compliance mechanism and to determine a deflection acting on the compliance mechanism. Visual markers can be positioned at several parts of the compliance assembly, but they are preferably positioned at the compliance base and compliance effector. The combination of two, three or more visual markers positioned at a defined position and orientation can indicate the magnitude and direction of the force in up to six degrees of freedom. Compliant deflection can include a rotational and a translational deflection, depending on the degrees of freedom associated to the system compliance.

[0091] FIG. 8 is a block diagram identifying some components of a system in accordance with the invention. The system includes a controllable robot 20 an imaging device 30 and controller 60. The controller 60 includes, or is in communication with, a processor 61, a communication interface 62, a memory 63, a user interface 64, an imaging processing unit 65 and a motion control unit 66. The processor 61 includes an image processing unit 65 that processes image data received from the imaging device 30 to derive computer vision feedback and a motion control unit 66 that sends robot motion commands to the robot based on the computer vision feedback. The controllable robot 20 may include a supporting base 22 and a robot arm 21 as depicted in FIG. 1. The motion control unit is configured (e.g., programmed) to provide commands that enable the robot 20 to perform the at least one of the following actions or motions: mating, unmating, rotating, retracting, locking, unlocking, wiggling, aligning, repositioning, vibrating, or a combination thereof.

[0092] FIG. 9 depicts a flowchart illustrating a method in accordance with the present disclosure. Where a pose estimation 200 includes a direct pose 210 estimation of a target component, such as the EV inlet 300, the charging connector 301 or a moving component 302 of the robot. Provided that such pose estimation is acceptable, the robot is controlled to perform at least one of a corrective action 400, a mating 401 or an unmating 402. In case of a direct pose estimation is not feasible or unreliable, an indirect pose estimation 220 can be triggered by obtaining pose information of an auxiliary component 221, which can facilitate the pose estimation of the target component.

[0093] Exemplary embodiments Features shown in relation to one embodiment can also be applied to or combined with other embodiments and therefore do not limit the scope of the present invention as defined in the attached claims.

[0094] As an exemplary embodiment, the controller visually determines the pose of the charging connector by receiving an image wherein the end effector and the charging connector are visible in such image. Recognizable features of the charging connector include geometric elements, surface patterns and contours which the pose detection algorithm has been trained with, and optionally referential markers such as checkerboard patterns, Arllco markers, or AprilTags. In this exemplary embodiment, the controller obtains via a communication means the pose of the EV inlet. The controller visually determines the pose of the charging connector and executes a positioning motion to align the charging connector at a predetermined position range for mating relative to the EV inlet.

[0095] As an exemplary embodiment, the controller visually determines the pose of the vehicle inlet. Subsequently, the controller manipulates the charging connector towards the inlet. Optionally, the controller performs a new determination of the pose of the inlet, for example through direct visual pose estimation or through indirect visual pose estimation using a pose of an auxiliary component. Subsequently, the controller visually determines the pose of charging connector and aligns the charging connector with the inlet at a predetermined position range for mating. Optionally, the controller verifies the alignment visually and repeats the alignment process if confirmation is not achieved. Finally, the controller executes the mating process.

[0096] As an exemplary embodiment, the controller can determine, by utilizing a received image, an alignment between the charging connector and the EV inlet. The pose detection algorithm analyses geometric elements and surface patterns of the charging connector in the image. By comparing this information with known features of the EV inlet, or with previously acquired information regarding the inlet pose, the controller assesses the alignment accuracy. If the alignment is within predetermined tolerances, the controller proceeds with the mating process. In cases where misalignment is detected, the controller initiates corrective motions to achieve optimal alignment between the charging connector and the EV inlet

[0097] As an exemplary embodiment, the controller can determine the pose of a partially occluded target component, such as the EV inlet. While the connector is manipulated based on an initial pose estimation of the EV inlet, the controller may receive a subsequent image where not all or none of the recognizable features of the EV inlet (such as the pins and holes) are visible in the camera field of view, or where such features are fully or partially occluded. This may be the case if the connector is positioned between the camera and the EV inlet. To ensure that EV inlet pose has not changed after an initial pose determination, a new pose estimation is desirable. In such case, the controller can utilize an image containing a recognizable feature of the EV inlet which remains visible in the camera field of view, such as (some of) the pins and holes, and features of auxiliary components such as the EV inlet cover, or the edges of the EV inlet housing. The controller can utilize said image in combination with previously acquired information, such as a relative position from the EV inlet cover to the socket pins and holes to predict the pose of the EV inlet at its current position. Additionally, or alternatively, the controller can utilize a prediction model to predict the current pose of the EV inlet based on the recognizable features present in the received image.

[0098] Additionally, or alternatively the controller may determine that features of auxiliary components have moved, compared to the previous image, indicating that the inlet has moved, and move the charging connector back to undo the occlusion and allow for a new visual pose estimation of the charging inlet. In a particular embodiment hereof, a static camera provides a first image containing a charging inlet and the outline of the door normally covering the charging inlet, based on which the pose of the charging inlet is estimated. Based on this first estimation the connector is instructed to move towards the inlet. In a second image the charging connector may now occlude the charging inlet, but the outline of the door is still visible. Based on whether or not the outline of the door has moved in the image, the controller may conclude whether or not the inlet has moved.

[0099] As an exemplary embodiment, if some of the recognizable features of the charging connector, such as the alignment markers or specific surface contours, are not clearly visible, are affected by hard shadows or heavy lighting conditions, the controller can utilize a received image including an auxiliary component mechanically coupled to the charging connector, such as the end effector or another similar type of recognizable feature as an auxiliary component. In such a case, a relative pose between the target (charging connector) and the auxiliary component (end effector) is known. In other cases, such a relative pose is determined, predicted, or received by the system. Alongside this pose information, the controller may utilize data from robot encoders to obtain information about the position of the robot actuators and its joints. By acquiring pose information related to such an auxiliary component, the controller can determine the current pose of the charging connector, even when it is not visible or partially occluded in the camera field of view

[0100] As an exemplary embodiment, the controller may determine the position of the charging connector while the same is mated into the EV inlet and as such, (some of) its recognizable features may not be readily visible in the camera field of view. The robotic arm is generally equipped with an end-effector supporting the charging connector. The relative pose between the end-effector and the charging connector can be known by the system controller. By acquiring pose information of the end-effector, the controller can determine the current pose of the charging connector, even when the same is not visible or is (partially) occluded in the camera field of view.

[0101] As an exemplary embodiment, the controller can determine, based on a pose of a target component, a compliant deflection as a result of an external force acting on the robot component, such as the robotic arm, the end effector, the charging connector or any moveable component of the robot. Thereby, in an instance where the charging connector is in its initial pose, also referred to herein as the home pose, the controller can utilize pre-existing information about such expected home pose, assuming no manipulation by the robot. If, despite being in the home pose and no manipulation by the robot, the received image indicates that the connector is not in the expected home pose, the presence of an external force acting on the connector can be expected, causing it to deviate from the expected home pose. In several embodiments, such information can be acquired by receiving and analysing an image including a recognizable feature of the charging connector within the field of view, determining its actual pose and comparing said pose with a previously known expected home pose. Therefore, the system can assess the impact of the external force on a compliance mechanism of the robot and adjust accordingly to compensate for this external force.

[0102] As an exemplary embodiment, the controller can determine, based on a pose of the compliance assembly, a compliant deflection as a result of an external force acting on the robot, such as the robotic arm, the end effector, the charging connector or any moveable component of the robot. In some cases, a compliance assembly of the robot is readily visible from an externally mounted camera. In other cases, the compliance assembly is not visible from an externally mounted camera and the system can utilize visual markers which can provide a direct indication of the pose of the target component and consequently, any deflection on the compliance mechanism.

[0103] As an exemplary embodiment, the controller can visually determine deviations of the pose of a robot component compared to where the robot parametric model would expect that component to be based on other sensor information. As an exemplary embodiment, the controller can determine compliant deflection, by visually recognizing features of the compliance assembly or the charging connector and estimating the according pose, and comparing that visually estimated pose with where the parametric model would expect the compliance assembly or charging connector to be based on sensors that track joint positions.

[0104] As an exemplary embodiment, when the connector is mated into the EV inlet and is not readily visible for a direct pose estimation, it is desirable to assess the connector position and to derive whether the EV is being subject to a motion which can negatively impact the robot or which requires a preventive correction. In such a case, the controller can derive the pose of the charging connector from observation of the compliance assembly, i.e., a target and auxiliary component, respectively in this case. The controller determine the position of the compliance assembly and derive the relative position between the charging connector and the compliance effector and based on such information, achieve an estimation of the connector current pose. The evaluation of these external forces is facilitated by monitoring visual markers mechanically connected to the compliance system. This implementation is advantageous when the connector is mated into the EV inlet, and due to movements in the EV, it becomes stuck and cannot be easily unmated. With this information, the controller can initiate a corrective action, such as a slight repositioning of the charging connector, making it easier for the robot to disconnect it.

[0105] As another exemplary embodiment, the controller assesses a perspective angle of the image. Here, the perspective angle denotes the viewing angle at which the camera observes a recognizable feature of a target component. Preferably, a perspective angle should be greater than zero. In cases where it is not, a corrective action involves adjusting the robot's position to ensure that the charging connector or a recognizable feature is within the camera's field of view at a perspective angle greater than zero.

[0106] As another exemplary embodiment, the image property is a occlusion value for the target component, the occlusion value indicating a degree to which the target component is occluded.

[0107] As another exemplary embodiment, historical data and / or predictive modelling to predict and compensate for the effects of an image being unsuitable for target component pose determination.

[0108] As an exemplary embodiment, the controller can utilize a moving speed of the target component to determine its current pose. Thereby, where a moving speed of the EV is known, a current pose of the EV inlet may be obtained even if such EV inlet is (partially) occluded by, for example, the EV charging connector. In such a case, the controller acquires information about a moving speed of the EV and at least one additional means, such one or more past images of the EV inlet to determine a current pose of the inlet. Additional information may include a correlation between a previously acquired image of the target component and a known past pose of the target component.

[0109] Additional information may also be utilized, such as parking positioning information from the EV.

[0110] The person skilled in the art realizes that the present invention by no means is limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.

Claims

CLAIMS1. A system for supporting the autonomous connection of an EV charging connector into an EV inlet, the system comprising:- a controllable robot adapted to support the EV charging connector;- an imaging sensor in communication with the robot, the imaging sensor adapted to capture image data within a field of view; and- a controller including a processor and a non-transitory memory storing instructions executed by the processor to configure the controller, the controller configured to:- determining the presence of at least one target component within the field of view, the target component selected from at least one of the EV charging connector and a moveable component of the robot,- when the target component is found to be present, analysing an image received from the image sensor and determining a pose of the target component, utilizing at least the received image and a pose recognition algorithm, and- controlling the robot based on the determined pose.

2. System in accordance with claim 1, wherein the controller is configured to determine the pose of the EV charging connector.

3. System in accordance with any of claims 1 or 2, wherein the controller is configured to determine the pose of the moveable component of the robot.

4. System in accordance with any of claims 2 or 3, wherein the controller is configured to visually determine if the EV charging connector is at a predetermined position range for mating.

5. System in accordance with claim 1, wherein controlling the robot includes at least one of: i. Determining a positioning motion and causing the robot to perform such motion, ii. Performing a mating of the charging connector into the EV inlet when the determined pose is within a predetermined position range for mating; and iii. Performing an unmating of the charging connector.

6. System in accordance with any of the preceding claims, wherein the controller is configured to simultaneously and continuously determine a pose of the EV charging connector or the moveable component of the robot while each of them undergoes a positioning motion.

7. System in accordance with any of the preceding claims, wherein if one or more recognizable features of the target component are not suitable or visible in the received image for pose determination, the pose of the target component is determined by utilizing a referential marker associated to such target component.

8. System in accordance with any of the preceding claims, wherein if one or more recognizable features of the target component are not suitable or visible in the received image for pose determination, the pose of the target component is determined by analyzing in the received image a recognizable feature of an auxiliary component of such target component and a relative position between such target and auxiliary component, wherein the auxiliary component represents the pose of the target component, in particular by being mechanically coupled thereto, and wherein the auxiliary component is a robot component selected from at least one of the end-effector, a compliancemechanism, a sensor, a locking mechanism, a connector attachment device, a robot arm actuator, a joint assembly, or a combination thereof.

9. System in accordance with claim 8, wherein the controller is configured for determine the position of the EV charging connector while the EV charging connector is mated into the EV inlet and not visible in the received image for pose determination by analyzing in the received image a recognizable feature of the connector attachment device and a known relative pose between the connector attachment and the EV charging connector.

10. System in accordance with any of the preceding claims, wherein the controller is further configured to determine the presence of the EV inlet and to visually determine its pose utilizing at least the received image and the pose recognition algorithm.

11. System in accordance with claim 10, wherein the controller is configured to determine a pose of the EV inlet, positioning the charging connector towards the EV inlet, visually determining a pose of the EV charging connector or a moveable component of the robot, positioning the EV charging connector at a predetermined pose for mating, determining if the EV charging connector is at a predetermined position range for mating, and performing a mating of the connector into the EV inlet.

12. System in accordance with claim 10, wherein the controller is configured to i. determine if features of the EV inlet are partially occluded in a received image for pose determination; and ii. if features of the EV inlet are partially occluded, visually determining a pose of the EV charging by utilizing an image containing an auxiliary component of the EV inlet which remains visible in the camera field of view wherein the auxiliary component of the EV inlet is an EV component selected from at least one of an inlet cover, an inlet locking mechanism, an inlet light indicator, an inlet port frame, and an inlet port housing, or a combination thereof.

13. System in accordance with claim 10, wherein the controller is configured to i. determine if features of the EV inlet auxiliary component have moved compared to a previous image, indicating that the EV inlet has moved; ii. repositioning the EV charging connector to allow the features of the EV inlet to be visible in a received image for pose determination; and. iii. performing a pose determination of the EV inlet.

14. System in accordance with any of the preceding claims, wherein the controller is configured to determine a compliant deflection by (i) visually recognizing features of a robot compliance assembly and estimating the respective pose of such robot compliance assembly, or (ii) determining the pose of the target component; and (iii) comparing that visually estimated pose to a pose where a robot model would expect the pose of such compliance assembly or target component to be, wherein a compliant deflection comprises a deflection as a result of an external force acting on a robot component, such as on the robotic arm, the end effector, the charging connector or any moveable component of the robot, and wherein determining a compliant deflection comprises determining a displacement or a rotation of the compliance assembly.

15. System in accordance with claim 14, wherein the controller is configured to determine a compliant deflection when the EV charging connector is mated into the EV inlet.

16. Method for controlling a robot for the autonomous connection of an EV charging connector into an EV inlet, the method comprising: i. determining the presence of a target component within a field of view, the target component selected from an EV charging connector and a moveable component of the robot, ii. when the target component is found to be present, analyzing an image received from an image sensor adapted to capture image data within said field of view and determining, based on at least the received image and a pose recognition algorithm, a pose of the target component, and controlling the robot based on the determined pose, wherein controlling the robot includes at least one of: a) Determining a positioning motion and causing the robot to perform such motion, b) Performing a mating of the charging connector into the EV inlet when the determined pose is within a predetermined position range for mating; or c) Performing an unmating of the charging connector.

17. A non-transitory computer program product, comprising program code which is stored on a machine-readable medium, and having computer-executable instructions for performing a method in accordance with claim 16.

Images