Systems and method for the autonomous charging of electric vehicles

Abstract

Autonomous charging device comprising a controllable actuated mechanism for supporting an electric vehicle charging connector and for moving the charging connector towards a vehicle's charging port; a camera, positioned on or associated with the autonomous charging device for acquiring images of the charging port wherein a computer vision module determines, based on the images, a pose of the vehicle charging port; the pose including a position and orientation of the vehicle charging port socket and / or socket pins relative to the camera, a motion module, a data communication module for communicating the images and operational data of the autonomous charging device to and from a common data-storage-and-processing module, the computer vision module comprising a neural network subject to training by a training module. A system for supporting the connection of an electric vehicle connector into an electric vehicle charging port by an autonomous charging device. Training module for training a neural network of a computer vision system for an autonomous charging device

Description

[0001] Systems and Method for the Autonomous Charging of Electric Vehicles

[0002] TECHNICAL FIELD OF THE INVENTION

[0003] The present invention generally relates to actuated autonomous charging devices and methods and systems related thereto. Methods and systems in accordance with the invention aim to improve the actuated autonomous charging devices performance and reliability. The present invention also relates to systems and methods for training a neural network of a computer vision system for an autonomous charging device.

[0004] BACKGROUND OF THE INVENTION

[0005] Improving performance and reliability in robotic systems, and in particular in robotic autonomous charging devices (ACDs) remains an area of interest.

[0006] Robots are widely used in many different industries, such as the assembly or production lines to automate manufacturing procedures today. In recent years, robots have been utilized in the automotive industry to automate the charging of the batteries that power electric vehicles (EVs). In order to ensure a successful connection between the socket and the connector, it is desired for robots to accurately determine the position of the charging port and to align the connector for a continuous and reliable process.

[0007] Since several components may contribute to an unsuccessful plug in and to an undesirable operational performance, further improvements are still necessary. Furthermore, the challenge in making the physical connection is the precise positioning of the connector in the socket before plugging it in. This must be done with an accuracy in the range smaller than a few millimetres, which implies that the accuracy is met by the device for plugging the connector in, and not by a vehicle, which at present cannot be positioned automatically with an accuracy of a few centimetres.

[0008] Several solutions have been proposed, such as the use of magnetic coupling to allow for an accurate connection between the connector and the socket. However, many of these techniques require making modifications to either the EV or to the connector which may affect the solution's scalability and accessibility in the market.

[0009] Other improvements have been made in recent years and certain ACDs currently known in the art are able to locate the position of an electric vehicle's socket and direct and plug the charger's connector to the charging port without intervention of an operator or the modification of the vehicle or its charging port. These improvements rely on the use of computer vision and neural networks in order to accurately identify the vehicle or its charging port so that the ACD can reliably complete the plug in and plug out process. Devices for this purpose are known in the art, for instance from the international patent applications PCT / NL2020 / 050266, PCT / NL2021 / 050115, PCT / NL2021 / 050410,

[0010] PCT / NL2021 / 050495, PCT / NL2021 / 05061, from the same applicant, all of which are herein incorporated by reference. ACD devices in accordance with the present disclosure can include compliance mechanisms to support several connectors, including but not limited to CCS-1 / 2 connectors.

[0011] Neural networks are used for carrying out complex tasks, such as classification tasks in recognizing patterns or objects in images, natural language processing, computer vision, speech recognition, bioinformatics, and other applications. The quality of the output of a neural network depends on the quality of training of the neural network. Further, the training of neural networks requires collecting and annotating of large amount of training data in order to build a suitable training dataset However, many training datasets misrepresent or underrepresent the variation in conditions in they are intending to train neural networks for.

[0012] Autonomous charging devices for electric vehicles have the added difficulty that they are often commissioned in outdoor areas with different conditions that affect the reproducibility of the process. Such conditions may include lightning conditions, variations in the charging port and connector (OEM- specific features, wear and tear), movements and vibrations of the ACD or vehicle and weather conditions, amongst others. Even a slight shadow on the vehicle socket may affect the pose detection and prevent the ACD to complete the plug-in process. Given the large variation in conditions that autonomous charging devices and systems face, a one for all computer vision module and / or neural network does not necessarily lead to improved operational performance. This technical problem is not generally present in robots used for industrial purposes, such as in manufacturing facilities, which are bounded to more controlled conditions and thus higher process repeatability.

[0013] It is desired to provide an autonomous charging device and system with a computer vision component having robust neural networks that support their operation under a wide array of conditions, and that also consider the instances of the device.

[0014] It is particularly desired that the computer vision component enables the ACD to achieve an improved pose detection after its installation and operation, in order to support a successful connection of the connector with the vehicle charging port in an autonomous and reliably way. In some cases, many ACD's are simultaneously deployed and recurrent on site or remote update on the computer vision component may be cumbersome, thus achieving a reliable computer vision component in a relatively short timeframe or that requires less training cycles is desirable. It is furthermore desired to provide methods and systems that enable the testing and deployment of retrained neural networks without interrupting the ACD's normal operation or that enable the operation without recurrent system calibration.

[0015] It is furthermore desired to provide methods and systems utilizing computer vision means provided with robust neural networks for the operation of autonomous charging devices.

[0016] BRIEF DESCRIPTION OF THE PRIOR ART

[0017] Some existing systems have various shortcomings relative to certain applications. Accordingly, there remains a need for further contributions in this area of technology.

[0018] Document WO2020142496A1 describes a method for training a robot coupled with a camera, the process comprising setting a robot or camera parameter; capturing a training image of a training object with the camera using the robot or camera parameter; changing the setting and capturing another training image and repeating such setting and capturing to obtain a plurality of training images based on different settings; training a system to recognize the training object based on the plurality of training images; and evaluating the system using pre-selected test images. This document describes the training of an industrial robot aimed at completing a manufacturing process. Industrial robots operate generally under controlled conditions and are subject to little or low variability on their target object and can be intensively trained under scenarios that represent all possible conditions that the industrial robot will face during its operation. Industrial robots generally incorporate sophisticated hardware and software which may not be suitable for the implementation in an autonomous charging device system.

[0019] Document US 2022 355 692 Al describes a system that autonomously charges an electric vehicle (EV). The method includes: obtaining a trained machine learning (ML) model from a back-end server; capturing an image using an image capturing device of the charging system, wherein a portion of the image comprises the EV charging portal; inputting the image into the trained ML model to determine one or more regions of interest associated with the EV charging portal within the image; determining a location of the EV charging portal based on the one or more determined regions of interest and one or more image processing techniques; and providing information to maneuver the robotic arm of the charging system to a physical position based on the determined location of the EV charging portal.

[0020] SUMMARY OF THE INVENTION

[0021] In an embodiment, the present disclosure relates to an autonomous charging device (ACD) for connecting a charging connector into to a vehicle charging port. In an embodiment, the present disclosure relates to a system for supporting the connection of an electric vehicle connector into an electric vehicle charging port by an autonomous charging device (ACD).

[0022] In an embodiment, the present invention relates to a method for training a neural network of a computer vision system of an autonomous charging device.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure la is a diagram illustrating an autonomous charging device.

[0025] Figure lb is a diagram illustrating a charging port.

[0026] Figure 2 is a diagram illustrating an autonomous charging device and components thereof according with some embodiments.

[0027] Figure 3 is a diagram illustrating a system for supporting the operation a plurality of autonomous charging devices according with some embodiments.

[0028] Figure 4 is a diagram illustrating an autonomous charging device and components thereof device according with some embodiments.

[0029] Figure 5 is a diagram illustrating an autonomous charging device and components thereof device according with some embodiments.

[0030] Figure 6 is a diagram illustrating an autonomous charging device and components thereof device according with some embodiments.

[0031] Figure 7 is a diagram illustrating a configuration of a training module according with some embodiments.

[0032] Figure 8 is a diagram illustrating a configuration of a training module according with some embodiments.

[0033] Figure 9 depicts images of several charging ports.

[0034] DETAILED DESCRIPTION OF THE INVENTION

[0035] In order to improve performance, reliability and a continued operation of autonomous charging devices for electric vehicles, also referred herein to as ACD, it is desirable to provide for method and systems that improve the ACD autonomy and reliability, in relation to their computer vision component. In accordance with the present disclosure, a computer vision component includes at least one computer vision neural network that enables the estimation of the pose of the charging port, including the socket and pins, and for a subsequent plug-in / plug-out process completion. Disclosed herein are robotic autonomous charging devices (ACDs) and method and systems related thereto to improve their performance and reliability. Methods and systems in accordance with the present disclosure support the autonomous charging of one or a plurality of electric vehicles.

[0036] Also disclosed herein are methods and systems for training computer vision neural networks configured to support the estimation of a pose of an electric vehicle charging port by an autonomous charging device, the pose including the charging port position and orientation. Methods and systems may include the assessment of a target metric score, collecting training data, generating a training set including the training data, and using the training set for training neural networks for supporting the estimation of the pose of a vehicle charging port (e.g., determining the position and / or orientation of the charging port sockets and / or pins).

[0037] In some embodiments neural networks for charging port pose estimation can be trained and retrained using a training set including images from a large set of ACD instances. The neural networks may be subsequently retained by using case-specific data (such as data gathered by one specific ACD or a group of ACDs), non-case-specific data (such as data gathered by ACDs to which the neural network is not expected to be initially deployed to) and metadata (such as data gathered by other sources).

[0038] Also disclosed herein are methods and systems for testing, validating and / or deploying neural networks into a computer vision module, into the computer vision component associated to the autonomous charging device. Preferably, the methods and systems allow for a remote testing, validating and deploying of neural networks into an ACD in order to allow its continuous and autonomous operation.

[0039] Also disclosed herein are methods and systems utilizing computer vision systems provided with robust neural networks for improving pose determination or estimation of a vehicle charging port, wherein the methods and systems are preferably suited to use case of the device and the vehicle.

[0040] An autonomous charging device as referred hereinto comprises at least a controllable actuated mechanism comprising means to support, either in a releasable manner or not, a vehicle charger connector, and is able to carry out an autonomous charging process, including the sequence of steps carried out by the ACD to complete the plug-in and plug-out of the vehicle charging connector to enable the vehicle to be charged. The process comprises at least the steps of determining the pose of a vehicle charging port (including the socket and / or its pins), moving a vehicle charging connector towards the charging port, allowing the charging of the vehicle to take place and disconnecting the charging connector. The pose of the vehicle charging port is preferably determined by the ACD by a computer vision module which utilizes image data from the charging port to estimate a position and orientation in relation to a sensor, such as a camera, which is used to record the image data. Performance of an autonomous charging device is dependent on many factors. In accordance with the present disclosure, performance can be influenced by the individual performance of various components of the ACD, such as the performance of the computer vision component, mechatronics, system and component calibration (such as a camera's intrinsic and extrinsic calibration), wear and tear, amongst others. Additional factors external to the ACD may also influence performance, such as weather conditions, lighting conditions and temperature conditions among others.

[0041] In accordance with the present disclosure, the ACD reliability and autonomous operation may be improved by improving the robustness of the computer vision component, which is a component able to determine the relative position and orientation of the charging port. The computer vision component comprises, or is based on, a neural network model, which is configured for providing an output including an estimation of the position and orientation of the vehicle charging port and / or its pins in relation to the camera position. Given that the performance of the computer vision component is generally dependent on the neural network that it is built on, it is desirable to provide the system with a robust neural network suited to the ACD system.

[0042] Actual information on performance may be useful to determine the need of an update, training or retraining of a neural network. In some cases, an update or training may not be triggered only by an actual performance indicator but based on the availability of data suited to the ACD instance which is expected to improve the ACD performance, efficiency, safety, or any other element.

[0043] An autonomous charging device may be confronted to a complete different set of operational conditions than another one, such as different climate conditions, charger port type, etc. Creating a neural network model that enables a reliable operation for a group of ACDs poses significant challenges and in some cases a particular model should be crafted for a group of ACDs having similar or equal operational conditions.

[0044] For the purposes of promoting an understanding of the principles of the invention, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended. Any alterations and further modifications in the described embodiments, and any further applications of the principles of the invention as described herein are contemplated as would normally occur to one skilled in the art to which the invention relates

[0045] In reference to Figure la, an autonomous charging device (100) is depicted. Figure la also illustrates a vehicle (10), a charging connector (20) and a charging port (30).

[0046] In reference to Figure lb, a charging port (30) is depicted, including charging port pins (31). In reference to Figure 2a, an autonomous charging device (100) is depicted and Figure 2b further depicts a system (200) for supporting the operation of an autonomous charging device (100).

[0047] In reference to Figure 3, a plurality of autonomous charging devices (100) is depicted, as well as a system (200) for supporting the operation of said devices.

[0048] In reference to Figure 4, a schematic diagram of an autonomous charging device is depicted. Depicted is a controllable actuated mechanism (110), a camera (120), a motion module (140), a data communication module (160), a processor (150), a computer vision module (130), a common data- storage-and-processing module (170), a training module (180).

[0049] In reference to Figure 5, an autonomous charging device is depicted, where the device includes more than one controllable actuated mechanism. Depicted is also a controllable actuated mechanism (110), a camera (120), a motion module (140), a data communication module (160), a processor (150), a computer vision module (130), a common data-storage-and-processing module (170), a training module (180).

[0050] In reference to Figure 6, in addition to reference to an autonomous charging device (100), a system for supporting its operation (200) is depicted. A processor (150) comprises a memory (151), a storage (152) and a controller (153). The computer vision module (130), motion module (140), data communication module (160) and other modules may be located or hosted independently from the ACD processor (150), or partially or fully embedded therewithin.

[0051] In reference to Figures 4 to 5, one or more depicted components or modules may be physically integrated within the autonomous charging device. One or more of the depicted components or modules may be in communication or interact with the autonomous charging device to perform one or more of their functions.

[0052] In an embodiment, the present disclosure relates to an autonomous charging device (ACD) comprising:

[0053] - At least one controllable actuated mechanism for supporting an electric vehicle charging connector and for moving the charging connector towards a vehicle charging port;

[0054] - At least one camera, positioned on or associated with the autonomous charging device for acquiring images of the vehicle charging port,

[0055] - A computer vision module positioned on and / or coupled with the autonomous charging device for determining, based on a neural network model and the images, a pose of the vehicle charging port; the pose including a position and orientation of the vehicle charging port socket and / or socket pins; - A motion module for receiving or having access to the determined pose of the vehicle charging port, configured for moving the charging connector towards the vehicle charging port, based on said pose;

[0056] - A data communication module for communicating the images and an operational data of the autonomous charging device to and from at least one common data-storage-and-processing module, the common data-storage-and-processing module for receiving or having access to at least one of case-specific images and non-case-specific images, the case specific images including images communicated by the autonomous charging device, and the non-case specific images including images communicated by at least another autonomous charging device or network of autonomous charging devices; and

[0057] - A processor configured to control at least one of the controllable actuated mechanisms, the camera, the motion module, the computer vision module, and the data communication module.

[0058] In some cases, the computer vision comprises the neural network, which is subject to training by a training module, the training module configured for

[0059] - Acquiring annotated images from the common data-storage-and-processing module, each image including a functional descriptor;

[0060] - Generating a training dataset comprising the annotated images so that the combination of case specific and / or non-case specific images results in a multidimensional distribution of functional descriptors; and

[0061] - Training the neural network using the generated training dataset.

[0062] In some cases, the computer vision comprises the neural network, which is subject to training by a training module, the training module configured for

[0063] - Receiving a score of a target metric associated to the autonomous charging device and triggering a training of the neural network if the score is below a threshold;

[0064] - Acquiring annotated images from the common data-storage-and-processing module, each image including a functional descriptor;

[0065] - Generating a training dataset comprising the annotated images so that the combination of case specific and / or non-case specific images results in a multidimensional distribution of functional descriptors as a function of the target metric; and

[0066] - Training the neural network using the generated training dataset.

[0067] In an embodiment, the present disclosure relates to a system for supporting the operation of one or more autonomous charging devices for, the system comprising: - A data communication module for communicating images and operational data to and from the autonomous charging device;

[0068] - A common data storage processing module configured to:

[0069] - processing images and metadata from at least one autonomous charging device;

[0070] - processing operational data from at least one autonomous charging device;

[0071] - generating based on the processed image and operational data, case specific images and non-case specific images associated to at least one functional descriptor;

[0072] - A neural network training module configured to training and deploying a neural network into a computer vision module for at least one autonomous charging device,

[0073] - wherein the training module is configured for

[0074] - Receiving a score of a target metric associated to the autonomous charging device and triggering a training of the neural network if the score is below a threshold;

[0075] - Acquiring annotated images from the common data-storage-and-processing module, each image including a functional descriptor;

[0076] - Generating a training dataset comprising the annotated images so that the combination of case specific and / or non-case specific images results in a multidimensional distribution of functional descriptors as a function of the target metric; and

[0077] - Training the neural network using the generated training dataset.

[0078] In an embodiment, the present disclosure relates a method for training a neural network of a computer vision system of an autonomous charging device, the method comprising: a) Receiving a score of a target metric associated to the autonomous charging device and triggering a training of the neural network if the score is below a threshold; b) Acquiring annotated images from the common data-storage-and-processing module, each image including a functional descriptor, c) Generating a training dataset comprising the annotated images so that the combination of selected case specific and / or non-case specific images results in a multidimensional distribution of functional descriptors, wherein said multidimensional distribution is a function of the target metric; and d) Training the neural network using the generated training dataset.

[0079] A controllable actuated mechanism (110), also referred herein to as a manipulator or a robotic arm, is an actuated component of the device that is configured to support, whether in a releasable manner or not, a vehicle charging connector and to direct the connector towards the vehicle charging port in order to complete the plug-in and / or plug-out process.

[0080] Sensors can be mounted in or around the ACD and be used to capture data of the vehicle, the vehicle socket or its surroundings. Multiple sensors can be mounted on the same ACD and can be of various types to gather several types of part / object information. Sensors include sensors collecting data, such as a 2D or 3D camera that is configured to gather image, video and / or audio, however, such data may also be obtained from sensors positioned outside of the ACD system. Sensors in accordance with the present disclosure further including force, light, wind, geographic position and orientation, temperature and pressure sensors, as well as any combinations thereof. Sensors in accordance with the invention may also provide date and time on which the data was captured.

[0081] The ACD comprises a computer system, also referred herein to as a processor (150), suitable to control the at least one of the actuated mechanisms (110), the camera (120), the computer vision module (130), the motion module (140) and the data communication module (160). The processor can be any type of suitable computer system, such as an edge computer that is able to control either directly or via a controller, one or various components of the ACD.

[0082] The ACD preferably comprises a controller (153) to control at least one of a camera (120), computer vision (130) and motion module (140) among others. Modules may be controlled by the processor (150) or embedded within the processor (150). The controller may be one or more different devices configured to control one or more of the above-mentioned elements.

[0083] The processor (150) may include or be in communication or interact with a series of modules, that may be controlled by the processor and one or more controllers. Modules include a computer vision module (130), a motion module (140) and a data communication module (160). Further modules include a common data-storage-and-processing module (170) and a training module (180), both being hosted within the processor (150) or within separate processors within the autonomous charging device and / or within the system (200) which supports the operation of the device (100), or in network connection with any of the device (100) or (200).

[0084] The computer vision module (130, 230) based on a neural network model. In some embodiments, the computer vision model is configured to host one or more computer vision neural network, or simply, neural network. The computer vision module can be in communication with, or host, the training module and / or a testing module. The training module and / or testing module may be hosted independently from the computer vision module. In some embodiments, the computer system further comprises an edge computer hosting the training module and / or testing module, which can run machine learning algorithms based on collected data to shape the knowledge of neural network model for a general or a case specific application.

[0085] The neural network may be any kind of neural network that may be used in image processing. For example, the second neural network may be a feed-forward neural network, a regulatory feedback neural network, a convolutional neural network, a recurrent neural network

[0086] The motion module (140) preferably includes the algorithms and steps of necessary to effect movement and actions of the ACD.

[0087] The training module (180) preferably includes a training algorithm such as a machine learning algorithm including but not limited to a backpropagation algorithm, a gradient descent algorithm, a newton method algorithm, a conjugate gradient algorithm, a quasi-newton algorithm, and a Levenberg algorithm. The training module (180) may be positioned in, or in communication with the autonomous charging device (100), the system for supporting its operation (200) or the autonomous charging system (300).

[0088] The ACD may be in communication with at least one system (200) comprising a common data-storage- and-processing module (270) that is configured to cooperate with, and / or supports the operation of said ACD via a network environment. Suitable network environments include enterprise-wide computer networks, intranets, local area networks, wide area networks, personal area networks, cloud computing networks, crowd-sourced computing networks, the Internet, and the world wide web. The network may be a wireless network, a wired network or any other type of communication network.

[0089] The system (200) preferably also comprises a processor, (250) which may further include a memory and a storage. The processor (250) may comprise, host or be in communication with, a computer vision module (230), a training module (280) and a data-storage-and-processing module (270).

[0090] Computer vision module (130), motion module (140), data communication module (160) and other modules referred herein to can be saved in an individual or in a common memory device whether of the volatile or non-volatile type and can be expressed in any suitable type such as but not limited to source code, object code, and machine code.

[0091] In some embodiments, the ACD comprises an illumination system to irradiate light into the charging port in order to support the operation of the computer vision system and / or camera component.

[0092] In some embodiments, the vehicle is an electric vehicle which may be any vehicle that is at least partially driven by electric energy and that includes a rechargeable battery. Hybrid vehicles are also contemplated within the present disclosure. In other embodiments, the vehicle may be a hydrogen driven vehicle, a solar power-driven vehicle or any other vehicle having a charging port. Furthermore, the vehicle can be of any type, such a car, a passenger car, a transportation car, a truck, an industrial vehicle, a sports vehicle, a multi-wheeled vehicle, a ship or ferry, a utility task vehicle (UTV) or "side- by-side" and an aircraft, among others.

[0093] Electric vehicle charging ports, also referred hereto as sockets, or simply, sockets are generally standardized in their dimensions and functions. The socket may be any of a variety of connectors including, but not limited to AC or DC connectors, including but not limited to J1772 - Type 1, GB / T, CCS - Type 1 and Type 2 (Combined Charging System), a SAE Combo plug, an International Electrotechnical Commission (IEC) 62196 plug, or the like.

[0094] Figure lb depicts a diagram of a CCS type 2 connector. A socket is generally composed of different pins, whose layout and size depends on the specific type, such as a type-2 connector comprising seven contact places: two small and five larger. The top row consists of two small contacts for signalling, the middle row contains three pins, the centre pin is used for earthing, while the outer two pins used for the power supply, optionally in conjunction with the two pins on the bottom row which are also for power supply. For the purpose of pose detection, the layout of the socket and pins plays a significant role. Figures 9a-9d depict images of vehicle socket under different conditions.

[0095] The pose of an object describes how an object is placed in the three-dimensional space it uses. An object's pose may be determined with respect to a perspective, such as a camera perspective. The object's pose may include three dimensions of information characterizing the object's rotation with respect to the camera perspective. Alternately, or additionally, an object's pose may include three dimensions of information characterizing the object's translation with respect to the camera perspective.

[0096] In the context of the present disclosure, a pose determination is made of an electric vehicle charging port but is not limited thereto. Preferably, the pose of an electric vehicle charging port may be understood as a position and an orientation of the vehicle's charging port and is in some embodiments represented by a 2D Cartesian position and a yaw of the socket (x, y, 0). However, in some embodiments, the pose is a 6D pose where the position is defined by a 3D Cartesian position and the orientation is defined by a roll, pitch, and yaw of the socket.

[0097] Determination of the electric vehicle charging port pose is useful for describing the pose and orientation of the charging port relative to a camera position in order to direct an EV connector towards a charging port and completing the plug-in process.

[0098] Pose determination procedures described herein may provide improved techniques for determining the pose of an object from visual data. The pose of an object may be obtained from a single image, multiple images from assumably the same pose, multiple images from assumably different poses, making use of earlier determined poses. Alternatively, or additionally, the pose of an object may be obtained from a multi-view image or a video.

[0099] In several embodiments of the present disclosure, pose determination of a vehicle or a vehicle charging port may be done by a neural network on which a computer vision module is based on. Preferably, a neural network may be trained to determine the estimated vehicle charging port position and orientation through an analysis of one or more images. The estimated socket pose may include estimates about the socket dominant axes, roll, elevation, angular position, attitude, and azimuth angle.

[0100] In other embodiments of the present disclosure, pose determination of a vehicle or a vehicle charging port may be done supported by a neural network. Preferably, a neural network may be trained to determine the location of geometric features of the socket, to be used as fiducial marker, based on which an algorithm, such as a perspective-n-point algorithm like solvepnp or ransac, can estimate the pose of the socket. This may make use of a single, or multiple images.

[0101] A neural network's training process normally determines the quality of the network output. Generally, large amounts of training data are collected and manually or automatically annotated. However, many training datasets may incorrectly or inadequately represent the data that neural networks are supposed to be trained on.

[0102] In the context of this invention, data may refer to an image, optionally and preferably including an annotation, a metadata, and / or a classification, such as a functional descriptor.

[0103] Annotation in the context of the present disclosure refers to pose-relevant information, such as the position and orientation of the socket and / or socket pins in the image or the pixel location of recognizable features of the socket, which may be used as a fiducial marker.

[0104] Metadata in the context of the present disclosure refers information not directly related to the pose of the socket, such as conditions in the image, or under which the image was recorded.

[0105] Images may be preferably labelled, classified, categorized or the like using functional descriptors, which include data obtained from the annotation process, from the ACD operation and / or from metadata. Classification may denote a qualitative or quantitative subdivision of metadata.

[0106] Functional descriptors, in the context of the present disclosure are characteristics or features associated with the operation and environment of the ACD, derived from images or metadata. These descriptors may be qualitative, quantitative, or a combination thereof, providing details about ACD performance, image properties, weather conditions, light conditions, socket position, vehicle attributes, and geographic location, amongst others. Functional descriptors can be generated manually, automatically, or through algorithms and are utilized for the training of a neural network for the ACD. Constructing a training set for training a neural network may present certain challenges. The construction of a training set is a critical step in the training of a neural network and thus the successful operation of a neural network is largely dependent on it. The amount of data needed can be fairly large, such as 10s or 100s of 1000s, millions, or more datapoints. A network can learn, using the training set, to correctly generalize its learning to predict the proper outputs for inputs.

[0107] It is an object of the invention to provide an autonomous charging device, a system supporting it and a method whereby a computer vision module is supported by a neural network trained based on a crafted dataset that sufficiently allows the computer vision module to perform adequately in most of all relevant use cases. To this end, the present invention aims at providing a suitable training method and system, so that a suitable balance of data is built into the training dataset to ensure that relevant cases are properly represented.

[0108] In some embodiments, a computer-vision neural network is trained by providing it with input data and target output data that correspond to each other. A training set is all this data, including example inputs and target outputs. Through training, the network's weights may be incrementally or repeatedly adjusted so that, given a specific input from the training set, the network's output approaches (e.g., as closely as possible, desirable, or practicable) the goal output corresponding to that specific input data.

[0109] In accordance with the present disclosure, the construction of a training dataset may utilize data obtained from an individual source or from a plurality of sources. Preferably, the training dataset includes data from one ACD or a particular group of ACDs (case-specific) for which a training is to be triggered, or from a plurality of non-particular ACDs (non-case-specific) or from ACD data combined with metadata.

[0110] A neural network may be subsequently retrained, enhanced, or customized for a more particular instance or set of instances using images from a single ACD or from a determined group of ACDs having a common instance where a retrained neural network is deemed to be suitable (such as a group of ACD experiencing a low performance under sunny conditions or at low temperatures). Thus, the retrained neural network can expect an improved performance over the existing neural network for pose estimation under that instance and external affecting factors.

[0111] In some embodiments, training may continue indefinitely with an unlimited-size dataset. However, the time for training for neural networks that need to be deployed is generally limited. In limited training time, the neural network may only be trained on a finite number of images, i.e., a limited-size dataset. One aspect of the invention pertains to the generation of a training dataset so that it comprises a selection of images in the limited-size dataset, such that the neural network is trained to perform well for varying images taken under varying conditions. In practice, the size of the dataset is balanced with complexity of the training and network, with time, and with computational resources.

[0112] Preferably, the training and evaluation of the network is driven by specific application cases and thus dataset, training and evaluation are optimized based on the requirements of specific application cases. Preferably, the trained or retrained neural network are adapted or customized from the more general to a degree of partial specialization toward the instance of the ACD. The trained or retrained neural network may be used to support the pose estimation of an EV's socket with improved performance (e.g., higher accuracy), which may result in better reliability.

[0113] Furthermore, depending on the application purposes, different kinds of training criteria and methods may be used for training and evaluation of the neural network. In some embodiments, the training is driven by specific application cases and may incorporate datasets stemming from other specific application cases.

[0114] The output of a neural network may change, or deviate, over time. In the context of the present disclosure, where an ACD performs a task utilizing a neural network either with or without a user's knowledge, or potentially without any user involved at all, a change or deviation in a neural network behaviour may impact the ACD operation. A slight deviation in the neural network output, such as a pose estimation may be sufficient to impact performance of the ACD process.

[0115] The decision of triggering a training session for a neural network is an important step to ensure continued reliability of the ACD process. In reference to Figure 7, embodiments in accordance with the present disclosure include receiving, by a training module, a score of a target metric associated to the autonomous charging device and triggering a training of the neural network if the score is below a threshold. In some cases, the training module also, or alternatively, receives a digital case representation of the autonomous charging device.

[0116] The neural network may be an existing network, either previously trained, retrained or untrained before. A neural network may be trained, potentially using new(er) data provided by one or more ACD's, so that the retrained network or an entirely new model may be deployed that provides better estimations and addresses deviations.

[0117] According to various embodiments, the training module receives a score of a target metric associated to the autonomous charging device and triggers a training of the neural network if the score is below a threshold. If a score related to an operational metric, also referred to as a target metric or assessed metric is below a threshold, the training module triggers a training of the neural network. In some cases, the training module determines an expected score for the autonomous charging device metric upon deployment of the trained neural network.

[0118] The score of the target metric may be determined by the common data-storage-and-processing module (170, 270). The score of target metric can be determined via several means. The score of target metric can be determined by assessing a change in a statistical moment of one or more operational metrics. A statistical moment includes a mean, a covariance, a variance, a skewness, a kurtosis, or a combination thereof. The common data-storage-and-processing module, using a neural network, automatically or guided by an operator may determine score of a target metric. The score may be determined using a mathematical operation and / or a machine learning algorithm. Preferred machine learning algorithms include a random decision forest, a regression algorithm, among others. Mathematical operations can include a distribution based on the Softmax operator.

[0119] The common data-storage-and-processing module is configured to receiving data related to a target metric, processing said data and determining a score based on said data. A score can be a numerical value or an array of numerical values, such as between 0 and 1, that indicates, for example, the score associated to the performance of the autonomous charging device. A score may be a combined score when it is associated to one or more metrics. In some cases, when one or more metrics are considered, the module determines a critical metric corresponding to the one having the biggest weight on the determination of the score. The score of a target metric may refer to current, future, predicted, estimated or anticipated score.

[0120] The training module may trigger a training instance when a score or a combined score is lower than 0,9; lower than 0,8; lower than 0,7; lower than 0,6; lower than 0,5; lower than 0,4; lower than 0,3; lower than 0,2; or lower than 0,1.

[0121] A score in certain embodiments may consider one or more than one target metrics and, in some cases, classifications associated to each metric. Target metrics in accordance with the present disclosure include at least one of the following metrics:

[0122] - a performance metric of the autonomous charging device;

[0123] - a quantitative and qualitative metric of training data;

[0124] - a computer vision module specificity metric;

[0125] - a neural network training elapsed time,

[0126] - a change in a target object, and

[0127] - combinations thereof.

[0128] A performance metric of the autonomous charging device relates to a metric stemming from performance data collected from the ACD and includes but is not limited to pose estimation rates, successful plug-in attempt rates, unsuccessful plug-in attempts rates, false negatives rates, false positives rates, and estimations associated thereto.

[0129] In non-limiting example, the training module receives a performance score of a target metric associated to the autonomous charging device and triggers a training of the neural network if the score is below a threshold based on:

[0130] The training module may trigger a training decision when a score or a combined score of a performance metric is lower than 0,9; lower than 0,8; lower than 0,7; lower than 0,6; lower than 0,5; lower than 0,4; lower than 0,3; lower than 0,2; lower than 0,1. Preferably, the training module is configured to trigger a training decision when a score or a combined score of a performance metric is lower than 0.6.

[0131] A quantitative and qualitative metric of training data refers to the expected improvement on the output of the neural network that training data can impart. The autonomous charging device captures and generates images preferably on a constant basis. The data-storage-and-processing module can assign a score related to the expected improvement on the output of the neural network that said data may impart. Conveniently, a decision is trained when an expected improvement is observed even if a performance score of the autonomous charging device is now under a threshold that would trigger a training instance.

[0132] The training module may trigger a training decision when a score or a combined score of a quantitative and qualitative metric of available data is lower than 1; lower than 0,9; lower than 0,8; lower than 0,7; lower than 0,6; lower than 0,5; lower than 0,4; lower than 0,3; lower than 0,2; lower than 0,1. Preferably, the training module may trigger a training decision when a score or a combined score of a quantitative and qualitative metric of available data is lower than 0.4. In this case, a high or very high improvement on the output of the neural network based on the available data would be expected.

[0133] A computer vision module specificity metric relates to a distribution of case-specific to non-case- specific data used to train an existing neural network. Very low specificity and very high specificity may not necessarily have an immediate impact in the operation or operation of the ACD, however, based on a specificity score, the training module may trigger a training session in order to compensate for an unsuitable distribution of data.

[0134] For example, a specificity-related threshold may be increased as more case-specific data becomes available over the operational time of a particular ACD or a particular group of ACDs, which triggers training. The training module may trigger a training decision when a score or a combined score of a computer vision module specificity metric a is lower than 1; lower than 0,9; lower than 0,8; lower than 0,7; lower than 0,6; lower than 0,5; lower than 0,4; lower than 0,3; lower than 0,2; lower than 0,1. Preferably, the training module may trigger a training decision when a score or a combined score of a computer vision module specificity metric data is lower than 0.4. In some cases, where a performance score and specificity score are lower than a preferred threshold, the training module may trigger a training session. A training elapsed time of at least one ACD according to the present disclosure relates to a metric of the time elapsed since a last training session for a particular neural network for a particular ACD or network of ACD's. The training module may trigger a training decision when a score or a combined score of a training elapsed time is lower than 1; lower than 0,9; lower than 0,8; lower than 0,7; lower than 0,6; lower than 0,5; lower than 0,4; lower than 0,3; lower than 0,2; lower than 0,1. Preferably, the training module may trigger a training decision when a score or a combined score of a training elapsed time is lower than 0.4.

[0135] A new target object variation according to the present disclosure relates to the introduction of a new charging port one which the network has not been trained yet and for which the ACD is expected to operate with. A new target object variation may also relate to a change in the position of a socket in a vehicle, or a change in the area around a previously known socket. Examples thereof include other geometric features like additional flaps, indicator lights, or new material or colour.

[0136] The common data-storage-and-processing module may be further configured to assign a score and / or a classification to a target metric or in some cases, a combined score of all assessed metrics. In some cases, the score may be assigned by the training module. The training module receives a score of a target metric associated to the autonomous charging device and triggers a training of the neural network if the score is below a threshold. A score of a target metric associated may be determined, calculated, defined, estimated, predicted via different means, preferably via an algorithm, a computer-based model, a human operator or a combination thereof. A threshold in accordance with various embodiments can be a static threshold or a dynamic threshold.

[0137] The data communication module is configured for communicating the images and an operational data of the autonomous charging device to and from at least one common data-storage-and-processing module, the common data-storage-and-processing module for receiving or having access to at least one of case-specific images and non-case-specific images, the case specific images including images communicated by the autonomous charging device, and the non-case specific images including images communicated by at least another autonomous charging device or network of autonomous charging devices.

[0138] The target metric and / or its score may be an input for the subsequent steps of acquiring annotated images, generating a training dataset, training the neural network and optionally testing an output of the neural network.

[0139] The score of a target metric may include the systematic combination of classification and scores related to a metric or group of metrics into a single global score that enables the determination of a score and subsequent triggering of a training. The training may be initiated by a training module at the ACD, at a network environment or at a centralized system. The training may be automatically triggered by the training decision module, either within one specific ACD or within a network environment or a combination thereof. The training may be triggered by an operator and the training decision module may allow the operator to validate or reject a training decision. The training decision module may also enable the operator to adjust the network heuristics before triggering a training decision.

[0140] According to various embodiments, the input of / for a neural network in accordance with the present disclosure can be any data on which the neural network is able to support the determination of a pose of an object, preferably the pose of a vehicle or its charging port, or the outline of geometric features which may be used as a fiducial marker. For the purpose of illustration, the input can be at least one or a range of images, and the output can be a determination of the socket shape and segmentation of pins for each individual image.

[0141] Output may include the pose (position and orientation) of the socket in the image with respect to the camera.

[0142] Data, in the context of the present invention refers to data related to a vehicle, a vehicle's charging port or any other data that can support in the estimation of the pose of the socket.

[0143] In the present disclosure, operational data one or more operation parameters of the autonomous charging device. In some embodiments, the one or more operation parameters include at least one operation parameter of any of the controllable actuated mechanism, camera, computer vision module, motion module, data communication module, data-storage-and-processing module, processor, training module. In some embodiments, operational data includes a signal transmitted by a sensor, the signal corresponding to a measurement by the sensor of at least one operating parameter.

[0144] In the present disclosure, metadata refers to data that is not directly derived from image data or operational data, such as geographical data, meteorological data, positioning data, amongst others. Metadata may be used to generate functional descriptors which may be associated to image data and operational data.

[0145] According to the present disclosure, a dataset is referred to a set of data gathered by the ACD or any other source about an ACD, a plurality of ACDs, a vehicle, a vehicle socket, its surroundings and combinations thereof. A training dataset is referred to as dataset that is generated with the purpose of training a neural network. According to various embodiments, the computer vision module comprises a neural network subject to training by a training module, the training module further configured for:

[0146] - Acquiring annotated images from the common data-storage-and-processing module, each image including a functional descriptor,

[0147] - Generating a training dataset comprising the annotated images so that the combination of selected case specific and / or non-case specific images results in a multidimensional distribution of functional descriptors, wherein said multidimensional distribution is a function of the target metric; and

[0148] - Training the neural network using the generated training dataset.

[0149] The training module may also be configured to generate a testing dataset. The testing dataset may be generated by splitting the annotated data to generate a training dataset and a testing dataset. The training module may also be configured to testing the output of the trained network using the testing dataset.

[0150] The training module may also be configured to test the output of the trained neural network against a benchmark dataset including a set of poses validated against a ground truth.

[0151] The training module may also be configured to

[0152] Deploying the trained neural network into the computer vision module if an improvement in the output of the trained network is observed;

[0153] Repeating the step of generating a new training dataset and / or retraining the neural network using the generated dataset until an improvement in the output of the trained network is observed; and

[0154] Discontinuing the training of the neural network.

[0155] A training dataset may be generated from newly gathered data, data stemming from an existing dataset (i.e., a sliced dataset), metadata or from a combination thereof. A training dataset may be generated automatically, either in a supervised or unsupervised form, by a computer-based algorithm or by an operator.

[0156] The training module is configured for acquiring annotated images from the common data-storage- and-processing module, each image including a functional descriptor and generating a dataset for training a neural network based on said images. The training dataset may be generated by including images with functional descriptors so that the combination of selected case-specific and non-case- specific images results in a multidimensional distribution of functional descriptors of said images. The training module preferably acquires / receives annotated images from the common data-storage- and-processing module, each image preferably including and / or associated to a functional descriptor. The training module may also or alternatively acquire or receive images and / or other data by collecting said image and data from the operation of the ACD or group of ACDs. The data may include images of the ACD, the vehicle, its charging port and / or combinations thereof. The autonomous charging device is preferably configured to capture and collect the data on a continuous basis, the data being stored in the common data-storage-and-processing module which is in communication with the training module.

[0157] Case-specific data in accordance with the present disclosure refers to data acquired by a particular autonomous charging device having preferably a common set of operational features or that are expected to operate under similar conditions, such as installation site, weather conditions, lighting conditions, among others. Due to the common set of operational features, it is expected that a trained neural network deployed in said group of devices, provides substantially the same output type. As such, a trained neural network may be deployed into a set of devices positioned in different locations but having substantially similar operational features.

[0158] Non-case-specific data in accordance with the present disclosure refers to data acquired by other ACDs which operate generally under different conditions than the ACD or group of ACDs for which a training decision is to be made

[0159] The training module is configured to acquire annotated images from the common data-storage-and- processing module, each image including a functional descriptor. The annotated images may include case-specific and non-case-specific images, each image including a functional descriptor. Functional descriptors may be associated to each image by the common data-storage-and-processing module. In some cases, functional descriptors may be associated to each image by the training module. In all cases, functional descriptors may be associated to each image automatically, either in a supervised or unsupervised form, by a computer-based algorithm or by an operator.

[0160] Case-specific data and non-case-specific data preferably include annotated data and one or more associated functional descriptors. In certain embodiments, annotated data including functional descriptions may be acquired and no additional functional descriptors may be needed. In certain embodiments, annotating the data with functional descriptors can be made to that describe at least one, and more preferably a plurality of characteristics of the data. Functional descriptors may be subsequently clustered in the dataset construction to represent an ACD instance or a sub-instance.

[0161] In some embodiments, the common data-storage-and-processing module is configured for receiving or having access to case-specific images and / or non-case-specific images, the case specific images including images communicated by the autonomous charging device, and the non-case specific images including images communicated from at least another autonomous charging device or network of autonomous charging devices. In some embodiments, the data-storage-and-processing module is configured for annotating the images with geometric features of the charging port socket and pins to generate a fiducial marker as an input for the neural network. Annotating images includes the step of annotating a dataset of images, either by an operator or by a computer, to create ground truth data and may include manually identifying and annotating a shape of the vehicle socket, its connection pins or any component thereof.

[0162] Each image may have at least one qualitative or quantitative descriptor. A single image can also have multiple descriptors, each pertaining to another category. In other words, each image may be described (qualitative or quantitative) along multiple dimensions. By extension, a collection of images, i.e., a dataset, with multidimensional descriptions has a distribution of descriptions along each dimension.

[0163] Case-specific images and non-case-specific images including annotated images and / or one or more associated functional descriptors may be generated, gathered, collected, retrieved and / or processed either automatically by a computer-based algorithm or by an operator, either by / at the training module or by / at the common data-storage-and-processing module.

[0164] In some embodiments, the conditions on which an ACD operates can be captured by an image, or implied, derived from it in order to create suitable functional descriptors. Functional descriptors may be estimated or predicted based on other metadata and may be preferably categorized. As such, a functional descriptor of an image may be: "red vehicle" and a classification for said functional descriptor may be "vehicle colour". Subclassifications may be made and are all contemplated within the scope of the present disclosure.

[0165] Functional descriptors may have a qualitative nature, a qualitative nature or a combination thereof. Functional descriptors may be descriptors of device performance, image properties, weather conditions, light source, light conditions, light direction, socket position, socket angle, image orientation, vehicle brand, vehicle type, vehicle colour, geographic location, customer name, customer project, date and time of the image recording, socket state, socket visibility, camera properties, among many others. Functional descriptors may be constantly created, either manually, automatically or via an algorithm. Other functional descriptors, not described herein which are associated to a feature of the device, the vehicle, or the conditions in or around them are deemed to be part of the present disclosure. Functional descriptors may have a qualitative nature, a qualitative nature or a combination thereof. Functional descriptors may be descriptors of device performance (such as successful or unsuccessful plug-in values), image properties (such as brightness, contrast, temperature, tin, hue, saturation, gamma, blur, etc.), weather conditions (such as snow, rain, wind, thunder, etc.), light conditions (such as bright light, dark light, hard shadows, soft shadows, etc.), light direction (such as upwards lightning, downward lightning, etc.), socket position (such as a distance from the ground to the socket position), socket angle (such as 5 degrees angle, 10 degrees angle, 15 degrees angle, etc.), image orientation (such as vertical, horizontal, etc.), vehicle brand (such as Toyota, BMW, Audi, etc.), vehicle type (such as industrial truck, passenger vehicle, etc.), vehicle colour (such as white, black, grey, etc.), geographic location (such as a specific country, a city, a town, etc.), customer name (such as Customer A, Customer B, Customer C, etc), customer project (such as Project Al, Project B2, etc.) date and time of the image recording, socket state (such as damaged socket, obstructed socket, manipulated socket, etc.), socket visibility (such as fully visibly socket, partially visible socket, etc.), camera properties (such as exposure, aperture, ISO, shutter speed, etc.), among many others. Functional descriptors may be constantly created, either manually, automatically or via an algorithm.

[0166] One of the objects of the present invention is that the neural network the network is trained on a training dataset that is suitable for the autonomous charging device specifics on which the network is to be deployed, in order to improve reliability and confidence in the operation of the ACD. The present invention comprises providing a robust neural network model that considers shaping the data distribution of the training dataset, subset, or slice on which the neural network is trained. The training module is configured to so that the combination of selected case specific and / or non-case specific images results in a multidimensional distribution of functional descriptors, wherein said multidimensional distribution is a function of the target metric. As an exemplary realization, where the target metric is a computer vision module specificity metric, and where a score of the specificity metric is below a threshold, the training module is configured for acquiring annotated images and generating a training dataset comprising the annotated images so that the combination of case specific and / or non-case specific images results in a multidimensional distribution of functional descriptors as a function of the specificity metric. The training module thus creates a multidimensional distribution where the distribution of case-specific images is increased against non-case-specific images and may subsequently test the trained neural network against a testing dataset or a benchmark dataset to evaluate the improvement of the output under said training dataset generated as a function of the specificity metric.

[0167] The training module and / or the data-storage-and-processing module may be further configured to receive data, including image data, operational data and metadata related to an ACD to generate an ACD digital case representation, or ACD digital case model, wherein the ACD digital case representation includes parameters representing the actual operational conditions of the ACD. An ACD digital case representation may include functional descriptors, which may be distributed in the model to be a representation of the operational conditions of the ACD. The training module is configured to receive said digital case representation and generate, based on said feature model and / or based on the target metric, a training dataset comprising the annotated images so that the combination of selected case specific and / or non-case specific images results in a multidimensional distribution of functional descriptors. Said multidimensional distribution can be a function of the target metric, the digital case representation or a combination thereof. Said multidimensional distribution is such that the trained neural network delivers an optimal performance in an intended use case. Preferably, a training dataset is generated so that the combination of selected case-specific and non-case-specific image results in a multidimensional distribution of functional descriptors. The ACD digital case representation, or ACD digital case model may be generated automatically, either in a supervised or unsupervised form, by a computer-based algorithm or by an operator. Preferably, the ACD digital case representation is generated by the data-storage-and-processing module

[0168] The training dataset may be generated by the training module based on an algorithm, such as a logistic regression, a generative adversarial network, a decision tree, random forest, naive bayes, k-nearest neighbours and gradient boosting algorithms. The training dataset generation may be done in a supervised or semi-supervised mode, where training module may be supervised, directed, approved and / or rejected by a human operator.

[0169] The training module is configured for training the neural network using the generated training dataset. In some cases, the training of the neural network is made within the training module. In some cases, the training is made within a network environment.

[0170] The multidimensional distribution of functional descriptors is preferably and generally not a static distribution as it may be dependent on the actual instances of the ACD, which may also change throughout time. In some cases, the multidimensional distribution may be a fixed distribution. Likewise, the distribution of case-specific images and non-case-specific images is not a static distribution as this may depend on the availability of data. In some instances, generally upon commissioning, little to no case-specific data may be available, which may become available later during operation, which may trigger a training session and the adjustment of the image distribution in the dataset. Furthermore, as further case-specific images and operational data becomes available, a training of the network may become suitable. Images and operational data related to an underperformance of the system may be of particular interest. In certain embodiments, a training dataset is constructed in a way that it comprises at least 20% of case-specific data, at least 40% of case-specific data, at least 60% of case-specific data, at least 80% of case-specific data.

[0171] In some cases, the training dataset is generated so that the neural network is optimized for performance for all conditions that an ACD or plurality of ACDs systems may encounter, as opposed to optimized towards the most common conditions, by having all relevant conditions uniformly represented in the dataset.

[0172] In some embodiments, suitable images, either case-specific or non-case-specific may be already present in an existing dataset, but the neural network has not been trained under said images yet. Appending existing data to a training dataset, instead of newly acquired data, might result in a more favourable distribution of functional descriptors.

[0173] The training dataset is generated in order to account for a data gap, where the data cap denotes conditions related to the ACD which are not sufficiently represented in the training dataset on which the neural network has been initially trained. The training module is further configured to proactively trigger a training instruction in order to fill a data gap, for example, for a particular vehicle, socket type, light condition, etc. In doing so, even though the system incidentally already performs well on the existing network and conditions, a proactive data gap fill aims to enhance the robustness of the system.

[0174] A training dataset may be generated iteratively, usually over operational time of an ACD, and there might not be a static optimum to iterate towards. Before or during installation of one or a set of ACDs, a set of conditions may be observed, based on which a multidimensional distribution of functional descriptors is generated. A multidimensional distribution of functional descriptors considering casespecific and non-case-specific images that represent the actual working conditions of the ACD or set of ACDs may be used to generate a dataset.

[0175] Where no case-specific images are available for a condition that should be represented in the dataset, non-case-specific images may be used if available. Case-specific or non-case-specific images may become available over time and the availability of newly gathered images and data may warrant a training decision. If the training decision is based on the availability of newly acquired data, this newly acquired data will typically be included in the dataset. During operation of the ACD or set of ACDs the operational conditions may shift or evolve, which may trigger training or retraining. In either scenario, the training module generates a balanced dataset with case-specific and non-case-specific image to allow for a robust and sufficiently performing neural network. In some cases, where a training decision is triggered due to a confidence value being under a threshold due to a performance metric, image data where the ACD or set of ACDs is underperforming may be used to shape the multidimensional distribution of functional descriptors, i.e., unsuccessful plug-in attempt. Preferably, a training dataset contains more case-specific data than the initial existing dataset.

[0176] In some embodiments, the training module is further configured to split the annotated data to generate a training dataset and a testing dataset and testing the output of the trained network using the testing dataset.

[0177] In some embodiments, the training module is further configured to test the output of the trained neural network against a benchmark dataset including a set of validated poses.

[0178] In some embodiments, the training module is further configured to

[0179] - Deploying the trained neural network into the computer vision module if an improvement in the output of the trained network is observed;

[0180] - Repeating the step of generating a new training dataset and / or retraining the neural network using the generated dataset until an improvement in the output of the trained network is observed; and

[0181] - Discontinuing the training of the neural network.

[0182] The testing of a trained or retrained neural network can take place at an ACD level or at a central server level.

[0183] In some embodiments, the testing of the trained neural network comprises testing the neural network against a validation dataset. In some embodiments, the testing of the trained neural network further comprises testing the neural network against a testing dataset.

[0184] In some embodiments, the method and system further comprise testing the trained neural network against an existing neural network and preferably determining a target metric change. Preferably, the testing further comprises testing the existing network against an existing dataset, testing the existing network against the new dataset, testing the retrained network against the existing dataset and testing the retrained network against the new dataset.

[0185] In a non-limiting example according to the invention, the training module receives a digital case representation model for an ACD where a performance score has been determined to be below a threshold. The digital case representation model includes a series of functional descriptors. The digital case representation modules includes a high predominance black and grey colours and high predominance of operational time between 08:00 and 13:00 and 13:00 and 18:00. The training module generates a training dataset using the following multidimensional distribution of functional descriptors.

[0186] The training module subsequently trained the neural network and tested the trained neural network against a testing dataset and observed an improvement in the intersection over union and mean central distance in the pose estimation of the charging port. The training module subsequently sent a neural network update instruction to the computer vision module.

[0187] Subsequently, during regular operation, a signal is received indicating that a performance score was determined by the training module to be lower than a threshold, wherein the performance score was determined to be lower for white vehicles.

[0188] A training is triggered by the training module including the following adjusted multidimensional distribution of functional descriptors.

[0189] The training module can be configured to adjusting the distribution regarding vehicle colour to include more data with white vehicles compared to black vehicles (i.e., 35% black, 45% white), while maintaining the multidimensional distribution for other functional descriptors until an improvement in the output of the neural network is determined. For a fixed-size dataset this may result in images from the initial training dataset not being included in the new training dataset, because there is also a distribution of time-of-day to account for. For an unlimited-size dataset, this means more images across all classifications will likely have to be added to meet the target distribution.

[0190] In some embodiments, a dataset or a training dataset is at least partially generated using data that is artificially created, preferably to fill a data gap. The dataset construction may take place in various manners so that based on existing data, preferably, images, are altered to change a particular condition. As an exemplary embodiment, an image that was originally recorded under sunny conditions; may be altered to represent a snowy condition, or an image with a vehicle of a particular colour, may be altered to remain the same except for the vehicle's colour.

[0191] These variations may be a of implementation of image augmentation software or dedicated neural networks to alter images. Data may be fully digitally constructed. For example, using 3D models of a vehicle (the socket) within a simulated environment, or through a more complex generative neural network.

[0192] Any techniques and methods useful to conduct leaning of the training object are contemplated herein. In some embodiments, the neural network training is generally performed separate or remotely from the ACD, such as in a network environment, but in certain embodiments it may also be performed within the ACD domain, such as in an edge computer. In some embodiments, the testing further comprises the validation of the trained network against a benchmark dataset. A benchmark dataset may include ground truth data including a set of validated pose determinations. In some embodiments, a benchmark. In some embodiments, various benchmark datasets may be construed, and a retrained network may be validated against a benchmark set that is case-specific to the ACD instance.

[0193] For the purposes of promoting an understanding of the principles of the invention, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended. Any alterations and further modifications in the described embodiments, and any further applications of the principles of the invention as described herein are contemplated as would normally occur to one skilled in the art to which the invention relates

[0194] In reference to the drawings, dashed lines represent components that may or may not be optional.

[0195] In some instances, one or more components may be referred to herein as "configured to," "configured by," "configurable to," "operable / operative to," "adapted / adaptable," "able to," "conformable / conformed to," etc. Those skilled in the art will recognize that such terms (for example "configured to") generally encompass active-state components and / or inactive-state components and / or standby-state components, unless context requires otherwise.

[0196] Conditional language used herein, such as, among others, "can", "could", "might," "may,""e.g." and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and / or steps.

[0197] Thus, such conditional language is not generally intended to imply that features, elements and / or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and / or steps are included or are to be performed in any embodiment. The terms "comprising," "including," "having," and the like are synonymous and are used inclusively, in an open- ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term "or" means one, some, or all the elements in the list. In addition, the articles "a," "an," and "the" as used in this application and the appended claims are to be construed to mean "one or more" or "at least one" unless specified otherwise.

[0198] As used herein, a phrase referring to "at least one of" or "and / or" a list of items refers to any combination of those items, including single members. Similarly, while operations may be depicted in the drawings in a particular order, it is to be recognized that such operations need not be performed in the order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Further, the drawings may schematically depict one more example process in the form of a flowchart. However, other operations that are not depicted can be incorporated in the example methods and processes that are schematically illustrated. For example, one or more additional operations can be performed before, after, simultaneously, or between any of the illustrated operations. Additionally, the operations may be rearranged or reordered in other implementations. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

[0199] It will be appreciated that the detailed description set forth above is merely illustrative in nature and variations that do not depart from the gist and / or spirit of the claimed subject matter are intended to be within the scope of the claims. Such variations are not to be regarded as a departure from the spirit and scope of the claimed subject matter.

[0200] It is noted that the processes, methods, actions, instructions described herein may be embodied in executable instructions stored in a computer readable medium for use by or in connection with a processor-based instruction execution machine, system, apparatus, or device. It will be appreciated by those skilled in the art that, for some embodiments, various types of computer-readable media can be included for storing data. As used herein, a "computer-readable medium" includes one or more of any suitable media for storing the executable instructions of a computer program such that the instruction execution machine, system, apparatus, or device may read (or fetch) the instructions from the computer-readable medium and execute the instructions for carrying out the described embodiments. Suitable storage formats include one or more of an electronic, magnetic, optical, and electromagnetic format. A non-exhaustive list of conventional exemplary computer-readable medium includes: a portable computer diskette; a random-access memory (RAM); a read-only memory (ROM); an erasable programmable read only memory (EPROM); a flash memory device; and optical storage devices, including a portable compact disc (CD), a portable digital video disc (DVD), and the like.

Claims

CLAIMS1. An autonomous charging device (ACD) comprising:At least one controllable actuated mechanism for supporting an electric vehicle charging connector and for moving the charging connector towards a vehicle charging port;At least one camera, positioned on or associated with the autonomous charging device for acquiring images of the vehicle charging port,A computer vision module positioned on and / or coupled with the autonomous charging device for determining, based on a neural network model and the images, a pose of the vehicle charging port; the pose including a position and orientation of the vehicle charging port socket and / or socket pins;A motion module for receiving or having access to the determined pose of the vehicle charging port, configured for moving the charging connector towards the vehicle charging port, based on said pose;A data communication module for communicating the images and an operational data of the autonomous charging device to and from at least one common data-storage- and-processing module, the common data-storage-and-processing module for receiving or having access to at least one of case-specific images and non-case-specific images, the case-specific images including images communicated by the autonomous charging device, and the non-case specific images including images communicated by at least another autonomous charging device or network of autonomous charging devices; andA processor configured to control at least one of the controllable actuated mechanisms, the camera, the motion module, the computer vision module, and the data communication module; wherein the computer vision module comprises the neural network, which is subject to training by a training module, the training module configured for- Receiving a score of a target metric associated to the autonomous charging device and triggering a training of the neural network if the score is below a threshold;- Acquiring annotated images from the common data-storage-and- processing module, each image including a functional descriptor;- Generating a training dataset comprising the annotated images so that the combination of case specific and / or non-case specific imagesresults in a multidimensional distribution of functional descriptors as a function of the target metric; and- Training the neural network using the generated training dataset.

2. The autonomous charging device according to claim 1, wherein the target metric is selected from:- a performance metric of the autonomous charging device;- a quantitative and qualitative metric of training data;- a computer vision module specificity metric;- a neural network training elapsed time,- a change in a target object, and- combinations thereof.

3. The autonomous charging device according to any of the preceding claims, wherein the operational data includes operation parameters of the autonomous charging device and preferably at least one operation parameter of any of the controllable actuated mechanism, camera, computer vision module, motion module, data communication module, data-storage-and-processing module, processor, training module.

4. The autonomous charging device according to any of the preceding claims, wherein the operational data includes a signal transmitted by a sensor, the signal corresponding to a measurement by the sensor of at least one operating parameter.

5. The autonomous charging device according to according to any of the preceding claims, wherein the neural network is configured for providing an output including an estimation of the position and orientation of the vehicle charging port in relation to the camera position and / or a location of the charging port pins in the image, preferably in relation to the camera position.

6. The autonomous charging device according to according to any of the preceding claims, wherein the data-storage-and-processing module is configured for annotating the images with geometric features of the charging port socket and / or pins as input for the neural network training.

7. The autonomous charging device according to any of the preceding claims, wherein the functional descriptors include descriptions or representations of: device performance, image properties, weather conditions, light source, light conditions, light direction, socket position, socket angle, image orientation, vehicle brand, vehicle type, vehicle color, geographic location, customer name, customer project, date and time of the image recording, socket state, socket visibility, camera properties, and combinations thereof.

8. The autonomous charging device according to any of the preceding claims, wherein the training dataset is generated based on an ACD digital case representation including functional descriptors representing the actual operational conditions of the ACD.

9. The autonomous charging device according to claim 8, wherein the ACD digital case representation includes a multidimensional distribution of functional descriptors.

10. The autonomous charging device according to any claims 8-9, wherein the training dataset is generated based on the ACD digital case representation and the target metric.

11. The autonomous charging device according to any of the preceding claims, wherein the data-storage-and-processing module is configured to associate the functional descriptors to the images.

12. The autonomous charging device according to any of the preceding claims, wherein the data communication module is in network connection with the data-storage-and- processing module and the training module.

13. The autonomous charging device according to any of the preceding claims, wherein the training module is configured to generate, based on the training dataset, a testing dataset, and testing the output of the trained network using the testing dataset.

14. The autonomous charging device according to any of the preceding claims, wherein the training module is configured to test the output of the trained neural network against a benchmark dataset, the benchmark dataset including a set of pose determinations validated against a ground truth.

15. The autonomous charging device according to any of the preceding claims, wherein the training module is configured for testing the neural network by testing an intersection over union and / or a mean central distance for an output of the neural network.

16. The autonomous charging device according to any of the preceding claims, wherein the training module configured to:- Deploying the trained neural network into the computer vision module if an improvement in the output of the trained network is observed upon testing against at least one of the testing dataset and benchmark dataset;- Repeating the step of generating a new training dataset and / or retraining the neural network using the generated dataset until an improvement in the output of the trained network is observed upon testing against at least one of the testing dataset and benchmark dataset; and- Discontinuing the training of the neural network.

17. The autonomous charging device according to any of the preceding claims, wherein the common data-storage-and-processing module is configured to assign a score to the target metric and provide said score to the training module.

18. The autonomous charging device according to any of the preceding claims, wherein the target metric is a performance metric of at least one autonomous charging device and wherein the training dataset is generated so that it comprises a relative amount of images associated to one or more functional descriptors that is inversely related to an expected performance associated to said one or more functional descriptors.

19. The autonomous charging device according to any of the preceding claims, wherein the target object includes a charging port type, the workspace around it or the vehicle on which it is mounted.

20. The autonomous charging device according to any of the preceding claims, wherein the training module is configured for generating the training dataset by appending a set of images to an existing dataset.

21. The autonomous charging device according to any of the preceding claims, wherein the training module is configured for generating the training dataset by adjusting the multidimensional distribution of functional descriptors in an existing dataset.

22. The autonomous charging device according to any of the preceding claims, wherein the training module is configured for determining a bias deviation between the trained neural network and a neural network previously installed within the computer vision module.

23. The autonomous charging device according to any of the preceding claims, wherein the training module is further configured to compare a metric of the autonomous charging device operation based on the trained neural network and deploying the trained neural network into the computer vision module if an improvement in the metric is observed.

24. The autonomous charging device according to any of the preceding claims, wherein the training dataset comprises case specific and non-case specific images.

25. A system for supporting the operation of one or more autonomous charging devices, the system comprising:A data communication module for communicating images and operational data to and from the autonomous charging device according to claim 1;A common data-storage-and-processing module configured for:- processing images and metadata from at least one autonomous charging device;- processing operational data from at least one autonomous charging device;- generating based on the processed image and operational data, case specific images and non-case specific images associated to at least one functional descriptor;A neural network training module configured to training and deploying a neural network into a computer vision module for at least one autonomous charging device, wherein the training module is configured for- Receiving a score of a target metric associated to the autonomous charging device and triggering a training of the neural network if the score is below a threshold;- Acquiring annotated images from the common data-storage-and-processing module, each image including a functional descriptor;- Generating a training dataset comprising the annotated images so that the combination of case specific and / or non-case specific images results in a multidimensional distribution of functional descriptors as a function of the target metric; and- Training the neural network using the generated training dataset.

26. An autonomous charging system comprising an autonomous charging device according to any of claims 1-24 and a system for supporting the operation of one or more autonomous charging devices according to claim 25.

27. A method for training a neural network of a computer vision system of an autonomous charging device, the method comprising: a. Acquiring annotated images from a data-storage-and-processing module, each image including a functional descriptor, b. Generating a training dataset comprising the annotated images so that the combination of selected case specific and / or non-case specific images results in a multidimensional distribution of functional descriptors; and c. Training the neural network using the generated training dataset.

28. Method according to claim 27, the method further comprises receiving a score of a target metric associated to the autonomous charging device and triggering a training of the neural network if the score is below a threshold, wherein the multidimensional distribution is a function of the target metric.

Images