System and method for autonomous charging of an electric vehicle

Abstract

An autonomous charging apparatus includes a controllable actuation mechanism for supporting an electric vehicle charging connector and for moving the charging connector toward a charging port of a vehicle; a camera positioned on or associated with the autonomous charging device for acquiring an image of the charging port, where the computer vision module determines a pose of the vehicle charging port based on the image; the posture comprises the position and orientation of the vehicle charging port socket and / or the socket pin relative to the camera; a motion module; and a data communication module for transmitting the image and operational data of the autonomously charged device to and from the common data storage and processing module, the computer vision module comprising a neural network undergoing the training by the training module. A system for supporting connection of an electric vehicle connector to an electric vehicle charging port by an autonomous charging device. The training module is to train a neural network of a computer vision system for autonomously charging a device.

Description

[0001] Field of the Invention

[0002] The present invention generally relates to actuating an autonomous charging device and related methods and systems. The methods and systems according to the present invention are aimed at improving the performance and reliability of actuating an autonomous charging device. The present invention also relates to systems and methods for training a neural network for a computer vision system for an autonomous charging device. Background of the Invention

[0004] Improving performance and reliability in robotic systems, particularly in robotic autonomous charging devices (ACDs), remains an area of interest.

[0005] Robots are widely used in many different industries, such as in the assembly or production lines of today's automated manufacturing processes. In recent years, robots have been used in the automotive industry to automate the charging of batteries that power electric vehicles (EVs). To ensure a successful connection between the socket and the connector, it is desirable for the robot to accurately determine the position of the charging port and align the connector for a continuous and reliable process.

[0006] Due to several components that may result in unsuccessful insertions and suboptimal operational performance, further improvements are still needed. Additionally, the challenge of making a physical connection is to precisely position the connector within the socket prior to insertion. This must be done with a precision within a range of less than a few millimeters, meaning that this precision is met by the device used to insert the connector rather than by the vehicle, which currently cannot automatically position with a precision of a few centimeters.

[0007] Several solutions have been proposed, such as using magnetic coupling to allow for precise connection between the connector and the socket. However, many of these technologies require modifications to the EV or the connector, which may affect the scalability and accessibility of the solution in the market.

[0008] In recent years, other improvements have been made, and certain ACDs currently known in the art are capable of locating the socket of an electric vehicle and guiding and inserting the connector of a charger into the charging port without the intervention of an operator or modification of the vehicle or its charging port. These improvements rely on the use of computer vision and neural networks in order to accurately identify the vehicle or its charging port, enabling the ACD to reliably complete the plug in and plugout process. Devices for this purpose are known in the art, such as International Patent Applications PCT / NL2020 / 050266, PCT / NL2021 / 050115, PCT / NL2021 / 050410, PCT / NL2021 / 050495, PCT / NL2021 / 05061 from the same applicant, all of which are incorporated herein by reference. The ACD device according to the present disclosure may include a compliance mechanism for supporting a number of connectors, including but not limited to CCS-1 / 2 connectors.

[0009] Neural networks are used to perform complex tasks such as pattern recognition in images or classification tasks of objects, natural language processing, computer vision, speech recognition, bioinformatics, and other applications. The output quality of a neural network depends on the quality of its training. In addition, the training of a neural network requires the collection and annotation of a large amount of training data in order to construct a suitable training dataset. However, many training datasets misrepresent or fail to adequately represent the variations in the conditions they are intended to train the neural network for.

[0010] Autonomous charging devices for electric vehicles have additional difficulties in that they are typically commissioned in outdoor areas with different conditions, which affect the reproducibility of the process. These conditions can include lighting conditions, variations in charging ports and connectors (OEM-specific features, wear), movement and vibration of the ACD or the vehicle, and weather conditions, among others. Even a slight shadow on the vehicle socket can affect pose detection and hinder the ACD from completing the insertion process. Given the large variations in the conditions faced by autonomous charging devices and systems, general-purpose computer vision modules and / or neural networks do not necessarily improve operational performance. This technical problem generally does not exist in robots used for industrial purposes, such as robots used in manufacturing facilities, which are restricted to more controlled conditions and thus have higher process reproducibility.

[0011] There is a desire to provide an autonomous charging device and system having a computer vision component with robust neural networks that support their operation under a wide variety of conditions and also take into account the instances of the device.

[0012] It is particularly desirable that the computer vision component enables the ACD to achieve improved pose detection after its installation and operation, so as to support the successful connection of the connector to the vehicle charging port in an autonomous and reliable manner. In some cases, many ACDs are deployed simultaneously, and regular on-site or remote updates of the computer vision component can be cumbersome. Therefore, it is desirable to implement a reliable computer vision component or a computer vision component that requires fewer training cycles within a relatively short time frame.

[0013] In addition, it is desirable to provide methods and systems that can test and deploy retrained neural networks without interrupting the normal operation of the ACD, or methods and systems that can operate without repeating system calibration.

[0014] In addition, it is desirable to provide methods and systems that utilize a computer vision device provided with a robust neural network for the operation of an autonomous charging device.

[0015] Brief description of the prior art

[0016] Some existing systems have various drawbacks relative to certain applications. Therefore, there is still a need for further contributions in this technical field.

[0017] Document WO2020142496A1 describes a method for training a robot coupled with a camera. The process includes setting robot or camera parameters; using the robot or camera parameters to capture training images of a training object with the camera; changing the settings and capturing another training image, and repeating such settings and captures to obtain multiple training images based on different settings; training the system to recognize the training object based on the multiple training images; and evaluating the system using pre-selected test images. This document describes the training of industrial robots aimed at completing the manufacturing process. Industrial robots typically operate under controlled conditions and are subject to little or low variability on their target objects, and can be intensively trained in scenarios representing all possible conditions that the industrial robot will face during its operation. Industrial robots typically include complex hardware and software, which may not be suitable for implementation in an autonomous charging device system.

[0018] Document US2022355692A1 describes a system for autonomous charging of an electric vehicle (EV). The method includes: obtaining a trained machine learning (ML) model from a backend server; capturing an image using an image capture device of a charging system, where a portion of the image includes an EV charging portal; inputting the image into the trained ML model to determine one or more regions of interest within the image associated with the EV charging portal; determining the location of the EV charging portal based on the determined one or more regions of interest and one or more image processing techniques; and providing information based on the determined location of the EV charging portal to maneuver a robotic arm of the charging system to a physical location. SUMMARY OF THE INVENTION

[0020] In an embodiment, the present disclosure relates to an autonomous charging device (ACD) for connecting a charging connector to a vehicle charging port.

[0021] In an embodiment, the present disclosure relates to a system for supporting the connection of an electric vehicle connector to an electric vehicle charging port by an autonomous charging device (ACD).

[0022] In an embodiment, the present invention relates to a method for training a neural network of a computer vision system of an autonomous charging device. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1a is a diagram showing an autonomous charging device.

[0025] Figure 1b is a diagram showing a charging port.

[0026] FIG. 2 is a diagram showing an autonomous charging device and its components according to some embodiments.

[0027] Figure 3 is a diagram showing a system for supporting the operation of multiple autonomous charging devices according to some embodiments.

[0028] Figure 4 is a diagram showing an autonomous charging device and its device components according to some embodiments.

[0029] Figure 5 is a diagram showing an autonomous charging device and its device components according to some embodiments.

[0030] Figure 6 is a diagram showing an autonomous charging device and its device components according to some embodiments.

[0031] Figure 7 is a diagram showing the configuration of a training module according to some embodiments.

[0032] Figure 8 FIG. is a diagram showing the configuration of a training module according to some embodiments.

[0033] FIG. 9 depicts an image of a plurality of charging ports.

[0034] DETAILED DESCRIPTION OF THE INVENTION

[0035] To improve the performance, reliability, and continuous operation of an autonomous charging device (also referred to herein as ACD) for an electric vehicle, it is desirable to provide methods and systems that improve the autonomy and reliability of the ACD relative to its computer vision components. According to the present disclosure, the computer vision components include at least one computer vision neural network that enables the estimation of the pose of a charging port (including the socket and pins) and is used for subsequent insertion / removal process completion.

[0036] Disclosed herein are robotic autonomous charging devices (ACDs) and associated methods and systems to improve their performance and reliability. The methods and systems according to the present disclosure support the autonomous charging of one or more electric vehicles.

[0037] Also disclosed herein are methods and systems for training a computer vision neural network configured to support the estimation of the pose of a charging port of an electric vehicle by an autonomous charging device, the pose including the position and orientation of the charging port. The methods and systems may include evaluating a target metric score, collecting training data, generating a training set including the training data, and using the training set to train the neural network to support the estimation of the pose of the vehicle charging port (e.g., determining the position and / or orientation of the charging port socket and / or pins).

[0038] In some embodiments, a training set including a large set of images from ACD instances may be used to train and retrain a neural network for charging port pose estimation. Subsequently, the neural network may be retained by using case-specific data (such as data collected by a particular ACD or a group of ACDs), non-case-specific data (such as data collected by an ACD that the neural network was not initially expected to be deployed to), and metadata (such as data collected by other sources).

[0039] Also disclosed herein are methods and systems for testing, validating a neural network, and / or deploying the neural network into a computer vision module and into a computer vision component associated with an autonomous charging device. Preferably, these methods and systems allow for remote testing, validation, and deployment of the neural network into the ACD to allow for its continuous and autonomous operation.

[0040] The present disclosure also discloses methods and systems for improving the pose determination or estimation of a vehicle charging port using a computer vision system provided with a robust neural network, where these methods and systems are preferably suitable for use cases of devices and vehicles.

[0041] An autonomous charging device as described herein includes at least a controllable actuation mechanism that includes means for supporting a vehicle charger connector in a releasable or non-releasable manner and is capable of performing an autonomous charging process that includes a series of steps performed by the ACD to complete the insertion and removal of the vehicle charging connector so that the vehicle can be charged. The process includes at least the following steps: determining the pose of the vehicle charging port (including the socket and / or its pins), moving the vehicle charging connector towards the charging port, allowing charging of the vehicle to occur, and disconnecting the charging connector. The pose of the vehicle charging port is preferably determined by the ACD through a computer vision module that uses image data from the charging port to estimate the position and orientation relative to a sensor (such as a camera) used to record the image data.

[0042] The performance of an autonomous charging device depends on many factors. According to the present disclosure, the performance may be affected by the individual performance of the various components of the ACD, such as the performance of the computer vision component, mechatronics, system and component calibration (such as the internal and external calibration of the camera), wear, etc. Additional factors external to the ACD may also affect performance, such as weather conditions, lighting conditions, and temperature conditions, etc.

[0043] According to the present disclosure, the reliability and autonomous operation of the ACD can be improved by enhancing the robustness of the computer vision component, which is the component capable of determining the relative position and orientation of the charging port. The computer vision component includes a neural network model or is based on a neural network model that is configured to provide an output including an estimate of the position and orientation of the vehicle charging port and / or its pins relative to the position of the camera. Given that the performance of the computer vision component typically depends on the neural network on which it is built, it is desirable to provide the system with a robust neural network suitable for the ACD system.

[0044] Actual information regarding performance can be used to determine the need for updating, training, or retraining of the neural network. In some cases, the update or training may be triggered not only by actual performance metrics but also based on the availability of data suitable for the ACD instance, which is expected to improve ACD performance, efficiency, safety, or any other element.

[0045] An autonomous charging device may face a completely different set of operating conditions compared to another charging device, such as different climate conditions, charger port types, etc. Creating a neural network model that enables a group of ACDs to operate reliably presents significant challenges, and in some cases, a specific model should be crafted for a group of ACDs with similar or identical operating conditions.

[0046] To enhance the understanding of the principles of the present invention, embodiments illustrated in the accompanying drawings will now be cited and specific language will be used to describe the embodiments. However, it should be understood that the scope of the present invention is not limited thereby. Any changes and further modifications in the described embodiments, as well as any further applications of the principles of the present invention as described herein, are considered to be commonly contemplated by those skilled in the art to which the present invention pertains.

[0047] Reference Figure 1a , depicts an autonomous charging device (100). Figure 1a A vehicle (10), a charging connector (20), and a charging port (30) are also shown.

[0048] Reference Figure 1b , depicts a charging port (30), including charging port pins (31).

[0049] Reference Figure 2a , depicts an autonomous charging device (100), and Figure 2b further depicts a system (200) for supporting the operation of the autonomous charging device (100).

[0050] Reference Figure 3 , depicts a plurality of autonomous charging devices (100) and a system (200) for supporting the operation of the devices.

[0051] Reference Figure 4 , depicts a schematic diagram of an autonomous charging device. A controllable actuation mechanism (110), a camera (120), a motion module (140), a data communication module (160), a processor (150), a computer vision module (130), a common data storage and processing module (170), a training module (180) are depicted.

[0052] Reference Figure 5 , depicts an autonomous charging device, wherein the device includes more than one controllable actuation mechanism. A controllable actuation mechanism (110), a camera (120), a motion module (140), a data communication module (160), a processor (150), a computer vision module (130), a common data storage and processing module (170), a training module (180) are also depicted.

[0053] Reference Figure 6, in addition to referring to the autonomous charging device (100), a system (200) for supporting its operation is also depicted. The processor (150) includes a memory (151), a storage device (152), and a controller (153). The computer vision module (130), the motion module (140), the data communication module (160), and other modules may be located or hosted independently of the ACD processor (150), or partially or fully embedded therein.

[0054] Reference Figures 4 to 5 , one or more of the depicted components or modules may be physically integrated within the autonomous charging device. One or more of the depicted components or modules may communicate or interact with the autonomous charging device to perform one or more of its functions.

[0055] In an embodiment, the present disclosure relates to an autonomous charging device (ACD) comprising:

[0056] - at least one controllable actuating mechanism for supporting an electric vehicle charging connector and for moving the charging connector towards a vehicle charging port;

[0057] - at least one camera positioned on or associated with the autonomous charging device for acquiring an image of the vehicle charging port,

[0058] - a computer vision module positioned on and / or coupled to the autonomous charging device for determining the pose of the vehicle charging port based on a neural network model and an image; the pose includes the position and orientation of the vehicle charging port socket and / or socket pins;

[0059] - a motion module for receiving or accessing the determined pose of the vehicle charging port, configured to move the charging connector towards the vehicle charging port based on the pose;

[0060] - a data communication module for transmitting and receiving images and operation data of the autonomous charging device to and from at least one common data storage and processing module, the common data storage and processing module for receiving or accessing at least one of case-specific images and non-case-specific images, the case-specific images including images transmitted by the autonomous charging device, and the non-case-specific images including images transmitted by at least another autonomous charging device or a network of autonomous charging devices; and

[0061] - a processor configured to control at least one of the controllable actuating mechanism, the camera, the motion module, the computer vision module, and the data communication module.

[0062] In some cases, computer vision includes a neural network that is trained by a training module configured to:

[0063] - Obtain annotated images from a common data storage and processing module, each image including a functional descriptor;

[0064] - Generate a training data set including the annotated images such that a combination of case-specific images and / or non-case-specific images results in a multi-dimensional distribution of functional descriptors; and

[0065] - Train the neural network using the generated training data set.

[0066] In some cases, computer vision includes a neural network that is trained by a training module configured to:

[0067] - Receive a score of a target metric associated with an autonomous charging device and, if the score is below a threshold, trigger training of the neural network;

[0068] - Obtain annotated images from a common data storage and processing module, each image including a functional descriptor;

[0069] - Generate a training data set including the annotated images such that a combination of case-specific images and / or non-case-specific images results in a multi-dimensional distribution of functional descriptors as a function of the target metric; and

[0070] - Train the neural network using the generated training data set.

[0071] In an embodiment, the present disclosure relates to a system for supporting the operation of one or more autonomous charging devices, the system including:

[0072] - A data communication module for transmitting images and operation data to and from the autonomous charging device;

[0073] - A common data storage and processing module configured to:

[0074] - Process images and metadata from at least one autonomous charging device;

[0075] - Process operation data from at least one autonomous charging device;

[0076] - Generate case-specific images and non-case-specific images associated with at least one functional descriptor based on the processed images and operation data;

[0077] - A neural network training module configured to train a neural network and deploy the neural network into a computer vision module for at least one autonomous charging device,

[0078] - Among them, the training module is configured to:

[0079] - Receive a score of a target metric associated with the autonomous charging device, and if the score is below a threshold, trigger the training of the neural network;

[0080] - Obtain annotated images from the common data storage and processing module, each image including a functional descriptor;

[0081] - Generate a training data set including the annotated images such that a combination of case-specific images and / or non-case-specific images produces a multi-dimensional distribution of functional descriptors as a function of the target metric; and

[0082] - Train the neural network using the generated training data set.

[0083] In an embodiment, the present disclosure relates to a method for training a neural network of a computer vision system for an autonomous charging device, the method including:

[0084] a) Receive a score of a target metric associated with the autonomous charging device, and if the score is below a threshold, trigger the training of the neural network;

[0085] b) Obtain annotated images from the common data storage and processing module, each image including a functional descriptor,

[0086] c) Generate a training data set including the annotated images such that a combination of selected case-specific images and / or non-case-specific images produces a multi-dimensional distribution of functional descriptors, wherein the multi-dimensional distribution is a function of the target metric; and

[0087] d) Train the neural network using the generated training data set.

[0088] The controllable actuation mechanism (110), also referred to herein as a manipulator or robotic arm, is the actuating component of the device, which is configured to support (whether in a releasable manner or not) the vehicle charging connector and guide the connector towards the vehicle charging port in order to complete the insertion and / or extraction process.

[0089] Sensors can be installed in or around the ACD and are used to capture data on the vehicle, vehicle socket, or their surroundings. Multiple sensors can be installed on the same ACD and can be of various types to collect several types of component / object information. The sensors include sensors for collecting data, such as 2D or 3D cameras configured to collect images, videos, and / or audio. However, such data can also be obtained from sensors located outside the ACD system. The sensors according to the present disclosure also include force sensors, light sensors, wind sensors, geographical location and orientation sensors, temperature sensors, and pressure sensors, and any combination thereof. The sensors according to the present invention can also provide the date and time of the captured data.

[0090] The ACD includes a computer system, also referred to herein as a processor (150), which is adapted to control at least one of the actuation mechanism (110), camera (120), computer vision module (130), motion module (140), and data communication module (160). The processor can be any type of suitable computer system, such as an edge computer capable of directly or via a controller controlling one or each component of the ACD.

[0091] The ACD preferably includes a controller (153) to control at least one of the camera (120), computer vision (130), and motion module (140), etc. The modules can be controlled by the processor (150) or embedded within the processor (150). The controller can be one or more different devices configured to control one or more of the above elements.

[0092] The processor (150) can include a series of modules or communicate or interact with them, and these modules can be controlled by the processor and one or more controllers. The modules include a computer vision module (130), a motion module (140), and a data communication module (160). Additional modules include a common data storage and processing module (170) and a training module (180), both hosted within the processor (150) or within a separate processor within the autonomous charging device and / or within the system (200) supporting the operation of the device (100), or hosted in a network connection with either the device (100) or (200).

[0093] The computer vision modules (130, 230) are based on a neural network model. In some embodiments, the computer vision model is configured to host one or more computer vision neural networks or simply neural networks. The computer vision module may communicate with or host the training module and / or the testing module. The training module and / or the testing module may be hosted independently of the computer vision module. In some embodiments, the computer system further includes an edge computer that hosts the training module and / or the testing module, which may run machine learning algorithms based on the collected data to shape the knowledge of the neural network model for general applications or case-specific applications.

[0094] The neural network can be any type of neural network that can be used in image processing. For example, the second neural network can be a feedforward neural network, a recurrent feedback neural network, a convolutional neural network, or a recurrent neural network.

[0095] The motion module (140) preferably includes the algorithms and steps necessary to implement the motions and actions of the ACD.

[0096] The training module (180) preferably includes training algorithms, such as machine learning algorithms, including but not limited to backpropagation algorithm, gradient descent algorithm, Newton's method algorithm, conjugate gradient algorithm, quasi - Newton algorithm, and Levenberg algorithm. The training module (180) may be located in or communicate with the autonomous charging device (100), the system (200) for supporting its operation, or the autonomous charging system (300).

[0097] The ACD may communicate with at least one system (200) including a common data storage and processing module (270), which is configured to cooperate with the ACD and / or support the operation of the ACD via a network environment. Suitable network environments include enterprise - level computer networks, intranets, local area networks, wide area networks, personal area networks, cloud computing networks, crowdsourcing computing networks, the Internet, and the World Wide Web. The network may be a wireless network, a wired network, or any other type of communication network.

[0098] The system (200) preferably further includes a processor (250), which may also include a memory and a storage device. The processor (250) may include, host, or communicate with the computer vision module (230), the training module (280), and the data storage and processing module (270).

[0099] The computer vision module (130), the motion module (140), the data communication module (160), and other modules mentioned herein can be stored in separate or common memory devices, whether volatile or non-volatile in type, and can be expressed in any suitable type (such as but not limited to source code, object code, and machine code).

[0100] In some embodiments, the ACD includes an illumination system to shine light into the charging port to support the operation of the computer vision system and / or the camera assembly.

[0101] In some embodiments, the vehicle is an electric vehicle, which can be any vehicle that is at least partially driven by electric energy and includes a rechargeable battery. Hybrid vehicles are also contemplated in the present disclosure. In other embodiments, the vehicle can be a hydrogen-powered vehicle, a solar-powered vehicle, or any other vehicle having a charging port. Additionally, the vehicle can be of any type, such as a car, a passenger vehicle, a transport vehicle, a truck, an industrial vehicle, a sports vehicle, a multi-wheeled vehicle, a ship or ferry, a utility task vehicle (UTV) or "side-by-side", and an aircraft, etc.

[0102] An electric vehicle charging port (also referred to herein as a socket or simply a socket) is generally standardized in its size and function. The socket can be any one of various connectors, including but not limited to AC or DC connectors, including but not limited to J1772-Type 1, GB / T, CCS-Type 1 and Type 2 (Combined Charging System), SAE combined plug, International Electrotechnical Commission (IEC) 62196 plug, etc.

[0103] Figure 1b A diagram depicting a CCS Type 2 connector. The socket generally consists of different pins, and its layout and size depend on the specific type. For example, a Type 2 connector includes seven contact points: two small contact points and five large contact points. The top row consists of two small contacts for signal transmission, the middle row contains three pins, the center pin is for grounding, and the two outer pins are for power supply, optionally in combination with the two pins also for power supply on the bottom row. For the purpose of pose detection, the layout of the socket and pins plays an important role. Figures 9a - 9d An image of a vehicle socket under different conditions is depicted.

[0104] The pose of an object describes the way the object is placed in the three-dimensional space in which it is used. The pose of the object can be determined relative to a perspective such as the camera view. The pose of the object can include three-dimensional information characterizing the rotation of the object relative to the camera view. Alternatively or additionally, the pose of the object can include three-dimensional information characterizing the translation of the object relative to the camera view.

[0105] In the context of the present disclosure, pose determination is performed for, but not limited to, an electric vehicle charging port. Preferably, the pose of an electric vehicle charging port can be understood as the position and orientation of the charging port of the vehicle, and in some embodiments is represented by the 2D Cartesian position and yaw angle (x, y, θ) of the socket. However, in some embodiments, the pose is a 6D pose, where the position is defined by a 3D Cartesian position and the orientation is defined by the roll, pitch, and yaw of the socket.

[0106] Determination of the pose of an electric vehicle charging port is useful for describing the pose and orientation of the charging port relative to the camera position in order to guide the EV connector towards the charging port and complete the insertion process.

[0107] The pose determination process described herein can provide an improved technique for determining the pose of an object based on visual data. Using the previously determined pose, the pose of an object can be obtained from a single image, multiple images assumed to be of the same pose, multiple images assumed to be of different poses. Alternatively or additionally, the pose of an object can be obtained from multi-view images or video.

[0108] In several embodiments of the present disclosure, determination of the pose of a vehicle or a vehicle charging port can be accomplished by a neural network on which a computer vision module is based. Preferably, the neural network can be trained to determine the estimated position and orientation of the vehicle charging port by analyzing one or more images. The estimated socket pose can include estimates of the socket spindle, roll, elevation angle, angular position, pose, and azimuth angle.

[0109] In other embodiments of the present disclosure, determination of the pose of a vehicle or a vehicle charging port can be accomplished with the support of a neural network. Preferably, the neural network can be trained to determine the position of the geometric features of the socket to be used as fiducial markers, based on which an algorithm such as a perspective-n-point algorithm like solvepnp or ransac can estimate the pose of the socket. This can utilize a single or multiple images.

[0110] The training process of a neural network generally determines the quality of the network output. Typically, a large amount of training data is collected and annotated manually or automatically. However, many training data sets may not correctly or adequately represent the data on which the neural network should be trained.

[0111] In the context of the present invention, data can refer to images, optionally and preferably including annotations, metadata, and / or classifications such as functional descriptors.

[0112] In the context of the present disclosure, annotations refer to pose-related information, such as the position and orientation of a socket and / or socket pins in an image or the pixel positions of recognizable features of the socket, which can be used as fiducial markers.

[0113] In the context of the present disclosure, metadata refers to information that is not directly related to the pose of the socket, such as conditions in the image or conditions under which the image was recorded.

[0114] An image can preferably be labeled, classified, categorized, etc. using a functional descriptor that includes data obtained from the annotation process, from ACD operations, and / or from metadata. Classification can represent a qualitative or quantitative breakdown of the metadata.

[0115] In the context of the present disclosure, a functional descriptor is a characteristic or feature derived from an image or metadata that is associated with the operation and environment of the ACD. These descriptors can be qualitative, quantitative, or a combination thereof, providing details about ACD performance, image attributes, weather conditions, lighting conditions, socket location, vehicle attributes, and geographical location, etc. Functional descriptors can be generated manually, automatically, or algorithmically, and are used to train the neural network of the ACD. There can be certain challenges in constructing a training set for training the neural network. The construction of the training set is a key step in neural network training, and thus the successful operation of the neural network depends to a large extent on it. The amount of data required can be quite large, for example, 10 or 100 thousands, millions, or more data points. The network can use the training set to learn to generalize its learning correctly so as to predict the correct output for an input.

[0116] The object of the present invention is to provide an autonomous charging device, a system and a method for supporting the device, whereby the computer vision module is supported by a neural network trained based on a well-crafted data set that fully allows the computer vision module to perform adequately in most all relevant use cases. To this end, the present invention aims to provide a suitable training method and system such that a suitable data balance is built into the training data set to ensure that relevant cases are appropriately represented.

[0117] In some embodiments, the computer vision neural network is trained by providing input data and target output data that correspond to each other to the computer vision neural network. The training set is all this data, including example inputs and target outputs. Through training, the weights of the network can be adjusted incrementally or repeatedly such that, given a particular input from the training set, the output of the network (e.g., as closely as possible, desirably, or feasibly) approaches the target output corresponding to that particular input data.

[0118] According to the present disclosure, the construction of the training dataset can utilize data obtained from a single source or from multiple sources. Preferably, the training dataset includes data from one ACD or a specific set of ACDs (case-specific) for which training is to be triggered, or data from multiple non-specific ACDs (not case-specific), or data from ACD data combined with metadata.

[0119] Subsequently, the neural network can be retrained, enhanced, or customized for more specific instances or a set of instances using images from a single ACD or from a set of determined ACDs having common instances, where the retrained neural network is considered suitable (such as a set of ACDs that experience low performance under sunny conditions or low temperatures). Thus, for pose estimation under this instance and external influencing factors, the retrained neural network can be expected to have improved performance over existing neural networks.

[0120] In some embodiments, training can continue indefinitely using a dataset of infinite size. However, the training time of the neural network to be deployed is typically limited. Within the limited training time, the neural network can be trained only on a limited number of images (i.e., a dataset of finite size). One aspect of the present invention relates to the generation of a training dataset such that it includes a selection of images in a dataset of finite size, so that the neural network is trained to perform well on different images taken under different conditions. In practice, the size of the dataset is balanced with the complexity of training and the network, time, and computational resources.

[0121] Preferably, the training and evaluation of the network are driven by a specific application case and, therefore, the dataset, training, and evaluation are optimized based on the requirements of the specific application case. Preferably, the trained or retrained neural network is adjusted or customized from a more general to a partially specialized degree for ACD instances. The trained or retrained neural network can be used to support pose estimation of the EV's socket with improved performance (e.g., higher accuracy), which can lead to better reliability.

[0122] In addition, different kinds of training criteria and methods can be used to train and evaluate the neural network according to the application purpose. In some embodiments, the training is driven by a specific application case and can incorporate datasets derived from other specific application cases.

[0123] The output of the neural network can change or deviate over time. In the context of the present disclosure, in cases where the ACD utilizes the neural network to perform tasks with or without user knowledge, or potentially without any user involvement at all, the change or deviation in the behavior of the neural network may affect the ACD operation. A slight deviation in the neural network output (such as pose estimation) may be sufficient to affect the performance of the ACD process.

[0124] The decision to trigger a training session of the neural network is an important step in ensuring the continued reliability of the ACD process. Refer to Figure 7 , an embodiment according to the present disclosure includes receiving, by a training module, a score of a target metric associated with an autonomous charging device and triggering training of the neural network if the score is below a threshold. In some cases, the training module also or alternatively receives a digital case representation of the autonomous charging device.

[0125] The neural network can be an existing network that has been previously trained, retrained, or not previously trained. New (newer) data provided by one or more ACDs can potentially be used to train the neural network such that a retrained network or a brand-new model that provides better estimates and addresses biases can be deployed.

[0126] According to various embodiments, the training module receives a score of a target metric associated with an autonomous charging device and triggers training of the neural network if the score is below a threshold. If the score associated with an operation metric (also referred to as a target metric or an evaluation metric) is below a threshold, the training module triggers training of the neural network. In some cases, the training module determines an expected score of the autonomous charging device metric when deploying the trained neural network.

[0127] The score of the target metric can be determined by a common data storage and processing module (170, 270). The score of the target metric can be determined by several methods. The score of the target metric can be determined by evaluating a change in statistical moments of one or more operation metrics. Statistical moments include mean, covariance, variance, skewness, kurtosis, or a combination thereof. The common data storage and processing module can determine the score of the target metric automatically or under the guidance of an operator using the neural network. Mathematical operations and / or machine learning algorithms can be used to determine the score. Preferred machine learning algorithms include random decision forests, regression algorithms, etc. Mathematical operations can include a distribution based on the Softmax operator.

[0128] The common data storage and processing module is configured to receive data related to the target metric, process the data, and determine a score based on the data. The score can be a numerical value or an array of numerical values, such as between 0 and 1, which indicates, for example, a score associated with the performance of the autonomous charging device. When the score is associated with one or more metrics, the score can be a combined score. In some cases, when considering one or more metrics, the module determines a key metric that corresponds to the metric having the greatest weight in determining the score. The score of the target metric can refer to a current, future, predicted, estimated, or expected score.

[0129] When the score or combined score is below 0.9, below 0.8, below 0.7, below 0.6, below 0.5, below 0.4, below 0.3, below 0.2, or below 0.1, the training module may trigger a training instance.

[0130] In some embodiments, the score may consider one or more than one objective metric, and in some cases, the classification associated with each metric. Objective metrics according to the present disclosure include at least one of the following metrics:

[0131] - Performance metrics of the autonomous charging device;

[0132] - Quantitative and qualitative metrics of the training data;

[0133] - Computer vision module specific metrics;

[0134] - Training elapsed time of the neural network,

[0135] - Changes in the target object, and

[0136] - Combinations of the above items.

[0137] The performance metrics of the autonomous charging device relate to metrics derived from performance data collected from the ACD and include, but are not limited to, the pose estimation rate, the successful insertion attempt rate, the unsuccessful insertion attempt rate, the false negative rate, the false positive rate, and the associated estimates.

[0138] In a non-limiting example, the training module receives a performance score of an objective metric associated with the autonomous charging device, and if the score is below a threshold based on the following items, triggers the training of the neural network:

[0139] Performance metric - Classification Performance metric - Score Very low performance 0-0.2 Low performance 0.2-0.4 Medium performance 0.4-0.6 High performance 0.6-0.8 Very high performance 0.8-1

[0140] When the score or combined score of the performance metric is below 0.9, below 0.8, below 0.7, below 0.6, below 0.5, below 0.4, below 0.3, below 0.2, below 0.1, the training module may trigger a training decision. Preferably, the training module is configured to trigger a training decision when the score or combined score of the performance metric is below 0.6.

[0141] The quantitative and qualitative metrics of the training data refer to the expected improvement in the neural network output that the training data can confer. The autonomous charging device preferably captures and generates images on a constant basis. The data storage and processing module may assign a score related to the expected improvement in the neural network output that the data can confer. Conveniently, when the expected improvement is observed, a training decision is made even if the performance score of the autonomous charging device is now below the threshold that would otherwise trigger a training instance.

[0142] Quantitative and qualitative metrics of available data - Classification Quantitative and qualitative metrics of available data - Score Very low improvement 0.8-1 Low improvement 0.6-0.8 Moderate improvement 0.4-0.6 High improvement 0.2-0.4 Very high improvement 0-0.2

[0143] When the score or combined score of the quantitative and qualitative metrics of the available data is below 1, below 0.9, below 0.8, below 0.7, below 0.6, below 0.5, below 0.4, below 0.3, below 0.2, or below 0.1, the training module may trigger a training decision. Preferably, when the score or combined score of the quantitative and qualitative metrics of the available data is below 0.4, the training module may trigger a training decision. In such a case, a high or very high improvement in the output of the neural network based on the available data is expected.

[0144] The computer vision module specificity metric relates to the distribution of case-specific data and non-case-specific data used to train an existing neural network. Very low specificity and very high specificity may not necessarily have a direct impact on the operation or performance of the ACD. However, based on the specificity score, the training module may trigger a training session to compensate for an inappropriate distribution of data.

[0145] Specificity - Classification Specificity - Score Very low specificity 0-0.2 Low specificity 0.2-0.4 Medium specificity 0.4-0.6 High specificity 0.6-0.8 Very high specificity 0.8-1

[0146] For example, as more case-specific data becomes available during the operation time of a particular ACD or a set of particular ACDs, the threshold related to specificity for triggering training may be increased. When the score or combined score of the computer vision module specificity metric is below 1, below 0.9, below 0.8, below 0.7, below 0.6, below 0.5, below 0.4, below 0.3, below 0.2, or below 0.1, the training module may trigger a training decision. Preferably, when the score or combined score of the computer vision module specificity metric data is below 0.4, the training module may trigger a training decision. In some cases, when the performance score and the specificity score are below the preferred threshold, the training module may trigger a training session.

[0147] The time elapsed since the training of at least one ACD according to the present disclosure involves a measure of the time elapsed since the last training session of a particular neural network for a particular ACD or network of ACDs. When the score or combined score of the time elapsed since training is below 1, below 0.9, below 0.8, below 0.7, below 0.6, below 0.5, below 0.4, below 0.3, below 0.2, or below 0.1, the training module may trigger a training decision. Preferably, when the score or combined score of the time elapsed since training is below 0.4, the training module may trigger a training decision.

[0148] New target object changes according to the present disclosure involve the introduction of new charging ports for which the new charging port network has not been trained and the ACD is expected to operate with the charging ports. New target object changes may also involve changes in the location of sockets in a vehicle or changes in the area around previously known sockets. Examples thereof include other geometric features, such as additional flaps, indicator lights, or new materials or colors.

[0149] The common data storage and processing module may also be configured to assign scores and / or classifications to target metrics, or in some cases, assign combined scores to all evaluated metrics. In some cases, the scores may be assigned by the training module. The training module receives scores for target metrics associated with the autonomous charging device and, if the score is below a threshold, triggers the training of the neural network. The scores for the associated target metrics may be determined, calculated, defined, estimated, predicted via different means, preferably via algorithms, computer-based models, human operators, or combinations thereof. The threshold according to various embodiments may be a static threshold or a dynamic threshold.

[0150] The data communication module is configured to transmit images and operation data of the autonomous charging device to and from at least one common data storage and processing module, the common data storage and processing module being configured to receive or access at least one of case-specific images and non-case-specific images, the case-specific images including images transmitted by the autonomous charging device, and the non-case-specific images including images transmitted by at least another autonomous charging device or a network of autonomous charging devices.

[0151] The target metric and / or its score may be an input for subsequent steps of obtaining annotated images, generating a training dataset, training a neural network, and optionally testing the output of the neural network.

[0152] The score for the target metric may include systematically combining the classification and scores associated with a metric or a set of metrics into a single global score that enables the determination of the score and subsequent triggering of training.

[0153] Training may be initiated by the training module at the ACD, in the network environment, or at a centralized system. Training may be automatically triggered by a training decision module within a specific ACD, within the network environment, or a combination thereof. Training may be triggered by an operator, and the training decision module may allow the operator to verify or reject the training decision. The training decision module may also enable the operator to adjust network heuristics before triggering the training decision.

[0154] According to various embodiments, the input to the neural network according to the present disclosure / the input for the neural network can be any data that the neural network is capable of supporting the determination of the pose of an object (preferably the pose of a vehicle or its charging port or the contour of a geometric feature that can be used as a fiducial marker). For illustrative purposes, the input can be at least one image or a series of images, and the output can be the determination of the socket shape and the segmentation of the pins for each individual image.

[0155] The output can include the pose (position and orientation) of the socket in the image relative to the camera.

[0156] In the context of the present invention, data refers to data related to a vehicle, the charging port of a vehicle, or any other data that can support the estimation of the pose of a socket.

[0157] In the present disclosure, operation data is one or more operation parameters of an autonomous charging device. In some embodiments, the one or more operation parameters include at least one operation parameter of any one of a controllable actuation mechanism, a camera, a computer vision module, a motion module, a data communication module, a data storage and processing module, a processor, a training module. In some embodiments, the operation data includes a signal transmitted by a sensor, and the signal corresponds to a measurement of at least one operation parameter by the sensor.

[0158] In the present disclosure, metadata refers to data that is not directly derived from image data or operation data, such as geographical data, meteorological data, location data, etc. The metadata can be used to generate functional descriptors that can be associated with the image data and the operation data.

[0159] According to the present disclosure, a data set refers to a set of data collected by an ACD or any other source regarding an ACD, multiple ACDs, vehicles, vehicle sockets, their surrounding environments, and their combinations. A training data set is referred to as a data set generated for the purpose of training a neural network.

[0160] According to various embodiments, the computer vision module includes a neural network trained by a training module, and the training module is further configured to:

[0161] - Obtain annotated images from a common data storage and processing module, each image including a functional descriptor,

[0162] - Generate a training data set including the annotated images such that a combination of selected case-specific images and / or non-case-specific images produces a multi-dimensional distribution of functional descriptors, where the multi-dimensional distribution is a function of a target metric; and

[0163] - Train the neural network using the generated training data set.

[0164] The training module can also be configured to generate a test data set. The test data set can be generated by splitting the annotated data to generate a training data set and a test data set. The training module can also be configured to use the test data set to test the output of the trained network.

[0165] The training module can also be configured to test the output of the trained neural network against a benchmark data set, the benchmark data set including a set of poses verified against ground truth.

[0166] The training module can also be configured to:

[0167] - If an improvement in the output of the trained network is observed, deploy the trained neural network into the computer vision module;

[0168] - Repeat the steps of generating a new training data set and / or retraining the neural network using the generated data set until an improvement in the output of the trained network is observed; and

[0169] - Stop the training of the neural network.

[0170] The training data set can be generated from newly collected data, data derived from existing data sets (i.e., sliced data sets), metadata, or a combination thereof. The training data set can be automatically generated by a computer-based algorithm or by an operator in a supervised or unsupervised form.

[0171] The training module is configured to obtain annotated images from a common data storage and processing module, each image including a functional descriptor, and generate a data set for training a neural network based on the images. The training data set can be generated by including images with functional descriptors such that a combination of case-specific and non-case-specific images selected produces a multi-dimensional distribution of the functional descriptors of the images.

[0172] The training module preferably obtains / receives annotated images from a common data storage and processing module, each image preferably including a functional descriptor and / or associated with a functional descriptor. The training module can also or alternatively obtain or receive the images and / or other data by collecting images and data from the operation of an ACD or a set of ACDs. The data can include images of the ACD, the vehicle, its charging port, and / or a combination thereof. The autonomous charging device is preferably configured to capture and collect data on a continuous basis, and the data is stored in a common data storage and processing module in communication with the training module.

[0173] Case-specific data according to the present disclosure refers to data acquired by a specific autonomous charging device, which preferably has a common set of operating characteristics or is expected to operate under similar conditions, such as installation location, weather conditions, lighting conditions, etc. Due to a common set of operating characteristics, the trained neural network deployed in the set of devices is expected to provide substantially the same type of output. Therefore, the trained neural network can be deployed to a set of devices located at different positions but having substantially similar operating characteristics.

[0174] Non-case-specific data according to the present disclosure refers to data acquired by other ACDs that typically operate under different conditions compared to the ACD or set of ACDs for which training decisions are to be made.

[0175] The training module is configured to obtain annotated images from the common data storage and processing module, each image including a functional descriptor. The annotated images can include case-specific images and non-case-specific images, each image including a functional descriptor. The functional descriptor can be associated with each image by the common data storage and processing module. In some cases, the functional descriptor can be associated with each image by the training module. In all cases, the functional descriptor can be automatically associated with each image by a computer-based algorithm or by an operator in a supervised or unsupervised form.

[0176] Case-specific data and non-case-specific data preferably include annotated data and one or more associated functional descriptors. In certain embodiments, annotated data including functional descriptions can be acquired and additional functional descriptors may not be required. In certain embodiments, data can be annotated with functional descriptors to describe at least one, and more preferably multiple, characteristics of the data. The functional descriptors can then be clustered in dataset construction to represent ACD instances or sub-instances.

[0177] In some embodiments, the common data storage and processing module is configured to receive or access case-specific images and / or non-case-specific images, where the case-specific images include images transmitted by autonomous charging devices, and the non-case-specific images include images transmitted from at least another autonomous charging device or a network of autonomous charging devices. In some embodiments, the data storage and processing module is configured to annotate the images with geometric features of the charging port socket and pins to generate fiducial markers as inputs to the neural network. Annotating the images includes the steps of annotating a dataset of images by an operator or computer to create ground truth data, and can include manually identifying and annotating the shape of a vehicle socket, its connection pins, or any of its components.

[0178] Each image can have at least one qualitative or quantitative descriptor. A single image can also have multiple descriptors, each belonging to another category. In other words, each image can be described along multiple dimensions (qualitative or quantitative). By extension, a set of images with multi-dimensional descriptions, i.e., a data set, has a description distribution along each dimension.

[0179] Case-specific and non-case-specific images including annotated images and / or one or more associated functional descriptors can be automatically generated, aggregated, collected, retrieved, and / or processed by a computer-based algorithm or by an operator, or generated, aggregated, collected, retrieved, and / or processed by a training module / at a training module or by a common data storage and processing module / at a common data storage and processing module.

[0180] In some embodiments, the conditions for ACD operation can be captured by, or implied from, the image to create a suitable functional descriptor. The functional descriptor can be estimated or predicted based on other metadata and can preferably be classified. Thus, the functional descriptor of an image can be: "red vehicle" and the classification of the functional descriptor can be "vehicle color". Sub-classifications can be made and are all contemplated within the scope of the present disclosure.

[0181] The functional descriptor can have a qualitative nature, a quantitative nature, or a combination thereof. The functional descriptor can be a descriptor of device performance, image attributes, weather conditions, light source, lighting conditions, lighting direction, socket location, socket angle, image orientation, vehicle brand, vehicle type, vehicle color, geographical location, customer name, customer project, date and time of image recording, socket status, socket visibility, camera attributes, etc. The functional descriptor can be created manually, automatically, or continuously via an algorithm. Other functional descriptors associated with the characteristics of the device, vehicle, or conditions in or around them not described herein are considered part of the present disclosure.

[0182] The functional descriptors can have qualitative nature, quantitative nature or a combination thereof. The functional descriptors can be descriptors of the following: device performance (such as successful or unsuccessful insertion values), image attributes (such as brightness, contrast, color temperature, hue, chroma, saturation, gamma, blur, etc.), weather conditions (such as snow, rain, wind, thunder, etc.), lighting conditions (such as strong light, weak light, hard shadow, soft shadow, etc.), lighting direction (such as upward illumination, downward illumination, etc.), socket position (such as the distance from the ground to the socket position), socket angle (such as 5-degree angle, 10-degree angle, 15-degree angle, etc.), image orientation (such as vertical, horizontal, etc.), vehicle brand (such as Toyota, BMW, Audi, etc.), vehicle type (such as industrial truck, passenger car, etc.), vehicle color (such as white, black, gray, etc.), geographical location (such as a specific country, city, town, etc.), customer name (such as customer A, customer B, customer C, etc.), customer project (such as project A1, project B2, etc.), date and time of image recording, socket status (such as a damaged socket, a blocked socket, a manipulated socket, etc.), socket visibility (such as a fully visible socket, a partially visible socket, etc.), camera attributes (such as exposure, aperture, ISO, shutter speed, etc.), etc. The functional descriptors can be created manually, automatically or continuously via an algorithm.

[0183] An object of the present invention is that the neural network is trained on a training data set suitable for the specific details of the autonomous charging device on which the network is to be deployed, so as to improve the reliability and credibility of the operation of the ACD. The present invention includes providing a robust neural network model that takes into account shaping the data distribution of the training data set, subset or slice for training the neural network. The training module is configured such that a combination of selected case-specific images and / or non-case-specific images results in a multi-dimensional distribution of functional descriptors, where the multi-dimensional distribution is a function of the target metric. As an exemplary implementation, in the case where the target metric is a computer vision module-specific metric and where the score of the specific metric is below a threshold, the training module is configured to obtain annotated images and generate a training data set including the annotated images such that a combination of case-specific images and / or non-case-specific images results in a multi-dimensional distribution of functional descriptors as a function of the specific metric. Thus, the training module creates a multi-dimensional distribution in which the distribution of case-specific images increases relative to non-case-specific images, and the trained neural network can subsequently be tested against a test data set or a benchmark data set to evaluate the improvement of the output under the training data set generated according to the specific metric.

[0184] The training module and / or the data storage and processing module may also be configured to receive data including image data, operation data, and metadata related to the ACD to generate an ACD digital case representation or an ACD digital case model, where the ACD digital case representation includes parameters representing the actual operating conditions of the ACD. The ACD digital case representation may include functional descriptors, which may be distributed in the model as a representation of the operating conditions of the ACD. The training module is configured to receive the digital case representation and generate a training data set including annotated images based on the feature model and / or based on a target metric such that a combination of selected case-specific images and / or non-case-specific images results in a multi-dimensional distribution of the functional descriptors. The multi-dimensional distribution may be a function of the target metric, the digital case representation, or a combination thereof. The multi-dimensional distribution enables the trained neural network to provide optimal performance in the expected use cases. Preferably, the training data set is generated such that a combination of selected case-specific images and non-case-specific images results in a multi-dimensional distribution of the functional descriptors. The ACD digital case representation or the ACD digital case model may be automatically generated by a computer-based algorithm or by an operator in a supervised or unsupervised form. Preferably, the ACD digital case representation is generated by the data storage and processing module.

[0185] The training data set may be generated by the training module based on algorithms such as logistic regression, generative adversarial networks, decision trees, random forests, naive Bayes, k-nearest neighbors, and gradient boosting algorithms. The training data set generation may be done in a supervised or semi-supervised mode, where the training module may be supervised, guided, approved, and / or rejected by a human operator.

[0186] The training module is configured to use the generated training data set to train a neural network. In some cases, the training of the neural network is performed within the training module. In some cases, the training is performed within a network environment.

[0187] The multi-dimensional distribution of the functional descriptors is preferably and typically not a static distribution as it may depend on the actual instances of the ACD, which may also change over time. In some cases, the multi-dimensional distribution may be a fixed distribution. Similarly, the distribution of case-specific images and non-case-specific images is not a static distribution as this may depend on the availability of data. In some cases, typically during debugging, little or no case-specific data is available, which may become available during later operations, which may trigger an adjustment of the training session and the image distribution in the data set. Additionally, as additional case-specific images and operation data become available, the training of the network may become appropriate. Images and operation data related to poor system performance may be of particular interest.

[0188] In some embodiments, the training dataset is constructed in such a way that it includes at least 20% case-specific data, at least 40% case-specific data, at least 60% case-specific data, at least 80% case-specific data.

[0189] In some cases, a training dataset is generated such that the neural network is optimized for performance across all conditions that an ACD or multiple ACD systems may encounter, by uniformly representing all relevant conditions in the dataset, rather than for the most common conditions.

[0190] In some embodiments, suitable images (whether case-specific or not) may already exist in an existing dataset, but the neural network has not been trained on said images. Appending existing data rather than newly acquired data to the training dataset may result in a more favorable distribution of functional descriptors.

[0191] The training dataset is generated to fill a data gap, where the data gap represents conditions related to the ACD that were not adequately represented in the training dataset on which the neural network was initially trained. The training module is also configured to actively trigger training instructions to fill the data gap, for example, for a specific vehicle type, socket type, lighting conditions, etc. In doing so, even if the system happens to perform well on existing networks and conditions, the active data gap filling is aimed at enhancing the robustness of the system.

[0192] The training dataset can be generated iteratively, typically during the operation time of the ACD, and there may be no static optimal value for iteration. Before or during the installation of an ACD or a set of ACDs, a set of conditions can be observed, and a multi-dimensional distribution of functional descriptors can be generated based on these conditions. A dataset can be generated using the multi-dimensional distribution of functional descriptors that takes into account case-specific and non-case-specific images representing the actual operating conditions of the ACD or a set of ACDs.

[0193] In cases where there are no case - specific images available for the conditions that should be represented in the dataset, non - case - specific images (if available) can be used. Case - specific images or non - case - specific images may become available over time, and the availability of newly collected images and data may require making training decisions. If a training decision is based on the availability of newly acquired data, that newly acquired data will typically be included in the dataset. During the operation of an ACD or a group of ACDs, the operating conditions may change or evolve, which may trigger training or retraining. In either scenario, the training module generates a balanced dataset with case - specific images and non - case - specific images to achieve a robust and well - performing neural network. In some cases, in situations where a training decision is triggered due to a confidence value being below a threshold as a result of a performance metric, image data of poorly performing ACDs or a group of ACDs can be used to shape the multi - dimensional distribution of the feature descriptors (i.e., unsuccessful insertion attempts). Preferably, the training dataset contains more case - specific data than the initial existing dataset.

[0194] In some embodiments, the training module is further configured to split the annotated data to generate a training dataset and a test dataset, and use the test dataset to test the output of the trained network.

[0195] In some embodiments, the training module is further configured to test the output of the trained neural network against a benchmark dataset that includes a set of verified poses.

[0196] In some embodiments, the training module is further configured to:

[0197] - If an improvement in the output of the trained network is observed, deploy the trained neural network into the computer vision module;

[0198] - Repeat the steps of generating a new training dataset and / or retraining the neural network using the generated dataset until an improvement in the output of the trained network is observed; and

[0199] - Stop the training of the neural network.

[0200] Testing of the trained or retrained neural network can be performed at the ACD level or at the central server level.

[0201] In some embodiments, testing the trained neural network includes testing the neural network against a validation dataset. In some embodiments, testing the trained neural network further includes testing the neural network against a test dataset.

[0202] In some embodiments, the method and system further include testing the trained neural network against an existing neural network and, preferably, determining a target metric change. Preferably, the testing also includes testing the existing network against an existing dataset, testing the existing network against a new dataset, testing the retrained network against the existing dataset, and testing the retrained network against the new dataset.

[0203] In a non-limiting example according to the present invention, the training module receives a digital case representation model for ACD, where the performance score has been determined to be below a threshold. The digital case representation model includes a series of functional descriptors. The digital case representation module includes a high proportion of black and gray, as well as a high proportion of operating hours between 08:00 and 13:00 and between 13:00 and 18:00. The training module uses the following multidimensional distribution of the functional descriptors to generate a training dataset.

[0204] Function descriptors associated with the image Weights in the training dataset dimension Vehicle color: Black 40％ Vehicle color: White 40％ Vehicle color: Colors other than black and white 20％ Subtotal 100％ Time of day: 08:00 to 13:00 30％ Time of day: 13:00 to 18:00 30％ Time of day: Times other than 08:00 to 18:00 40％ Subtotal 100％

[0205] Subsequently, the training module trains the neural network and tests the trained neural network against a test dataset, and improvements in the intersection over union and the average center distance are observed in the pose estimation of the charging port. Subsequently, the training module sends a neural network update instruction to the computer vision module.

[0206] Subsequently, during normal operation, a signal indicating that the performance score determined by the training module is below the threshold is received, where the performance score of the white vehicle is determined to be low.

[0207] Training is triggered by the training module, which includes the following multidimensional distribution of the adjusted functional descriptors.

[0208] Function descriptors associated with the image Weights in the training dataset dimension Vehicle color: Black 35％ Vehicle color: White 45％ Vehicle color: Colors other than black and white 20％ Subtotal 100％ Time of day: 08:00 to 13:00 30％ Time of day: 13:00 to 18:00 30％ Time of day: Times other than 08:00 to 18:00 40％ Subtotal 100％ Case - specific images 70％ Non - case - specific images 30％

[0209] The training module can be configured to adjust the distribution regarding the vehicle color to include more data for white vehicles compared to black vehicles (i.e., 35% black, 45% white), while maintaining the multidimensional distribution of the other functional descriptors until an improvement in the output of the neural network is determined. For a dataset of fixed size, this may result in images from the initial training dataset not being included in the new training dataset, as the distribution of the time of day also needs to be considered. For a dataset of infinite size, this means that more images of all classifications may have to be added to meet the target distribution.

[0210] In some embodiments, the dataset or training dataset is generated using at least partially artificially created data, preferably for filling data gaps. The dataset construction can be carried out in various ways such that, based on the existing data, preferably, the images are altered to change specific conditions. As an exemplary embodiment, an image initially recorded under sunny conditions can be altered to represent snowy conditions, or an image of a vehicle having a specific color can be altered while keeping everything else the same except for the color of the vehicle.

[0211] These alterations can be implemented by image enhancement software or a dedicated neural network to change the images. The data can be constructed entirely digitally. For example, 3D models of vehicles (sockets) are used in a simulation environment, or by more sophisticated generative neural networks.

[0212] Any techniques and methods for performing learning of the training object are contemplated herein. In some embodiments, neural network training is typically performed separately or remotely from the ACD, for example, in a network environment, but in certain embodiments, it can also be performed within the ACD domain, for example, in an edge computer.

[0213] In some embodiments, the testing also includes validating the trained network against a benchmark dataset. The benchmark dataset can include ground truth data that includes a set of verified pose determination results. In some embodiments, the benchmark. In some embodiments, various benchmark datasets can be constructed, and the retrained network can be validated against a benchmark set that is specific to the ACD instance case.

[0214] To enhance the understanding of the principles of the present invention, embodiments illustrated in the accompanying drawings will now be referred to and specific language will be used to describe the embodiments. However, it should be understood that the scope of the present invention is not limited thereby. Any changes and further modifications in the described embodiments, as well as any further applications of the principles of the present invention as described herein, are considered to be commonly contemplated by those skilled in the art to which the present invention pertains.

[0215] Referring to the accompanying drawings, the dashed lines indicate components that may or may not be optional.

[0216] In some cases, one or more components may be referred to herein as "configured to", "configured by...", "can be configured to", "operable / can operate as", "adapted / can be adapted", "able to", "conformable / can conform", etc. Those skilled in the art will recognize that such terms (e.g., "configured to") generally cover active state components and / or inactive state components and / or standby state components unless the context requires otherwise.

[0217] Conditional language, such as, among others, "can", "could", "might", "may", used herein, unless expressly stated otherwise or otherwise understood in the context in which it is used, is generally intended to convey that certain embodiments include certain features, elements, and / or states, while other embodiments do not include certain features, elements, and / or states.

[0218] Accordingly, such conditional language is generally not intended to imply that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether such features, elements, and / or steps are included in any particular embodiment or are to be performed in any particular embodiment, with or without author input or prompting. The terms "comprising", "including", "having", etc. are synonymous and are used inclusively in an open-ended manner and do not exclude additional elements, features, acts, operations, etc. Further, the term "or" is used in its inclusive sense (and not in its exclusive sense) such that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, unless otherwise specified, the articles "a", "an", and "the" used in this application and the appended claims should be construed to mean "one or more" or "at least one".

[0219] As used herein, a phrase referring to "at least one" or "and / or" in a list of items refers to any combination of those items, including a single member.

[0220] Similarly, although operations may be depicted in a particular order in the figures, it should be recognized that such operations need not be performed in the sequence or order shown, or that all illustrated operations need to be performed to achieve the desired result. Additionally, the figures may schematically depict one or more example processes in the form of a flowchart. However, other operations not depicted may be incorporated into the example methods and processes schematically shown. For example, one or more additional operations may be performed before, after, concurrently with, or between any of the operations shown. Further, in other embodiments, the operations may be rearranged or reordered. In certain instances, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the above-described embodiments should not be understood to be required in all embodiments, and the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. Additionally, other embodiments are within the scope of the appended claims. In some cases, the acts recited in the claims may be performed in a different order and still achieve the desired result.

[0221] It should be understood that the detailed description set forth above is merely illustrative in nature and that variations that do not depart from the gist and / or spirit of the claimed subject matter are intended to fall within the scope of the claims. Such variations should not be regarded as a departure from the spirit and scope of the claimed subject matter.

[0222] It should be noted that the processes, methods, actions, instructions described herein can be embodied in executable instructions stored on a computer-readable medium for use by or in conjunction with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will appreciate that for some embodiments, various types of computer-readable media for storing data can be included. As used herein, "computer-readable medium" includes one or more of any suitable medium for storing executable instructions of a computer program such that the instruction execution machine, system, apparatus, or device can read (or extract) the instructions from the computer-readable medium and execute the instructions for performing the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer floppy disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; and optical storage devices, including portable compact discs (CDs), portable digital video discs (DVDs), and the like.

Claims

1. An autonomous charging device (ACD) comprising: - at least one controllable actuation mechanism configured to support an electric vehicle charging connector and to move the charging connector towards a vehicle charging port; - at least one camera positioned on or associated with the autonomous charging device for acquiring an image of the vehicle charging port, - a computer vision module positioned on and / or coupled with the autonomous charging device for determining an attitude of the vehicle charging port based on a neural network model and the image; the attitude including the position and orientation of a vehicle charging port socket and / or socket pins; - a motion module for receiving or accessing the determined attitude of the vehicle charging port, the motion module being configured to move the charging connector towards the vehicle charging port based on the attitude; - a data communication module for transmitting the image and operational data of the autonomous charging device to and from at least one common data storage and processing module, the common data storage and processing module being configured to receive or access at least one of case-specific images and non-case-specific images, the case-specific images including the images transmitted by the autonomous charging device, and the non-case-specific images including the images transmitted by at least one other autonomous charging device or a network of autonomous charging devices; and - a processor configured to control at least one of the controllable actuation mechanism, the camera, the motion module, the computer vision module, and the data communication module; wherein the computer vision module includes the neural network, the neural network being subject to training by a training module configured to: -- receive a score of a target metric associated with the autonomous charging device and trigger training of the neural network if the score is below a threshold; -- obtain annotated images from the common data storage and processing module, each image including a functional descriptor; -- generate a training data set including the annotated images such that a combination of case-specific images and / or non-case-specific images results in a multi-dimensional distribution of functional descriptors as a function of the target metric; and -- train the neural network using the generated training data set.

2. The autonomous charging device according to claim 1, wherein, The target metric is selected from the following: - a performance metric of the autonomous charging device; - quantitative and qualitative metrics of training data; - computer vision module specificity metrics; - time elapsed for neural network training, - changes in target objects, and - combinations of the above items.

3. The autonomous charging device according to any one of the preceding claims, wherein, The operational data includes the operational parameters of the autonomous charging device and preferably includes at least one operational parameter of any one of the controllable actuation mechanism, the camera, the computer vision module, the motion module, the data communication module, the data storage and processing module, the processor, the training module.

4. The autonomous charging device according to any one of the preceding claims, wherein, The operation data includes signals transmitted by sensors, the signals corresponding to measurements by the sensors of at least one operation parameter.

5. The autonomous charging device according to any one of the preceding claims, wherein, The neural network is configured to provide an output that includes an estimate of the position and orientation of the charging port of the vehicle relative to the camera position and / or an estimate of the position of the charging port pins in the image, preferably relative to the camera position.

6. The autonomous charging device according to any one of the preceding claims, wherein, The data storage and processing module is configured to annotate the image with geometric features of the charging port socket and / or pins as an input for neural network training.

7. The autonomous charging device according to any one of the preceding claims, wherein, The functional descriptor includes a description or representation of the following items: device performance, image attributes, weather conditions, light sources, lighting conditions, lighting direction, socket position, socket angle, image orientation, vehicle brand, vehicle type, vehicle color, geographical location, customer name, customer project, date and time of image recording, socket status, socket visibility, camera attributes, and combinations of the foregoing items.

8. The autonomous charging device according to any one of the preceding claims, wherein, The training data set is generated based on an ACD digital case representation that includes functional descriptors representing the actual operating conditions of the ACD.

9. The autonomous charging device according to claim 8, wherein, The ACD digital case representation includes a multi-dimensional distribution of functional descriptors.

10. The autonomous charging device according to any one of claims 8-9, wherein, The training data set is generated based on the ACD digital case representation and the target metric.

11. The autonomous charging device according to any one of the preceding claims, wherein, The data storage and processing module is configured to associate the functional descriptor with the image.

12. The autonomous charging device according to any one of the preceding claims, wherein, The data communication module is network-connected to the data storage and processing module and the training module.

13. The autonomous charging device according to any one of the preceding claims, wherein, The training module is configured to generate a test data set based on the training data set and use the test data set to test the output of the trained network.

14. The autonomous charging device according to any one of the preceding claims, wherein, The training module is configured to test the output of the trained neural network against a benchmark data set that includes a set of pose determination results verified against ground truth.

15. The autonomous charging device according to any one of the preceding claims, wherein, The training module is configured to test the neural network by testing the intersection over union and / or average center distance of the output of the neural network.

16. The autonomous charging device according to any one of the preceding claims, wherein, The training module is configured to: - If an improvement in the output of the trained neural network is observed when testing against at least one of the test data set and the benchmark data set, deploy the trained network to the computer vision module; - Repeat the steps of generating a new training data set and / or retraining the neural network using the generated data set until an improvement in the output of the trained network is observed when testing against at least one of the test data set and the benchmark data set; and - Stop the training of the neural network.

17. The autonomous charging device according to any one of the preceding claims, wherein, The common data storage and processing module is configured to assign a score to the target metric and provide the score to the training module.

18. The autonomous charging device according to any one of the preceding claims, wherein, The target metric is a performance metric of at least one autonomous charging device, and wherein the training data set is generated such that the training data set includes a relative amount of images associated with one or more functional descriptors, the relative amount of these images being inversely correlated with the expected performance associated with the one or more functional descriptors.

19. The autonomous charging device according to any one of the preceding claims, wherein, The target object includes the type of charging port, the working space around it, or the vehicle on which it is installed.

20. The autonomous charging device according to any one of the preceding claims, wherein, The training module is configured to generate the training dataset by attaching a set of images to an existing dataset.

21. The autonomous charging device according to any one of the preceding claims, wherein, The training module is configured to generate the training dataset by adjusting the multi-dimensional distribution of the functional descriptors in the existing dataset.

22. The autonomous charging device according to any one of the preceding claims, wherein, The training module is configured to determine the bias deviation between the trained neural network and the neural network previously installed in the computer vision module.

23. The autonomous charging device according to any one of the preceding claims, wherein, The training module is further configured to compare the metrics of the autonomous charging device operation based on the trained neural network, and if an improvement in the metrics is observed, deploy the trained neural network into the computer vision module.

24. The autonomous charging device according to any one of the preceding claims, wherein, The training dataset includes case-specific images and non-case-specific images.

25. A system for supporting the operation of one or more autonomous charging devices, the system comprising: - A data communication module for transmitting images and operation data to and from the autonomous charging device according to claim 1; - A common data storage and processing module configured to: -- Process images and metadata from at least one autonomous charging device; -- Process operation data from at least one autonomous charging device; -- Generate case-specific images and non-case-specific images associated with at least one functional descriptor based on the processed images and operation data; - A neural network training module configured to train a neural network and deploy the neural network into the computer vision module for at least one autonomous charging device, - wherein the training module is configured to -- Receive a score of a target metric associated with the autonomous charging device, and trigger the training of the neural network if the score is below a threshold; -- Obtain annotated images from the common data storage and processing module, each image including a functional descriptor; -- Generate a training dataset including the annotated images such that a combination of case-specific images and / or non-case-specific images produces a multi-dimensional distribution of the functional descriptor as a function of the target metric; and -- Train the neural network using the generated training dataset.

26. An autonomous charging system comprising the autonomous charging device according to any one of claims 1-24 and the system for supporting the operation of one or more autonomous charging devices according to claim 25.

27. A method for training a neural network of a computer vision system of an autonomous charging device, the method comprising: a. Obtain annotated images from a data storage and processing module, each image including a functional descriptor, b. Generate a training dataset including the annotated images such that a combination of selected case-specific images and / or non-case-specific images produces a multi-dimensional distribution of the functional descriptor; and c. Train the neural network using the generated training dataset.

28. The method according to claim 27, the method further comprising receiving a score of a target metric associated with the autonomous charging device, and if the score is below a threshold, triggering training of the neural network, wherein the multidimensional distribution is a function of the target metric.

Images