Systems and methods for autonomous charging of electric vehicles

Abstract

and a controllable actuation mechanism for supporting the electric vehicle charging connector and moving the charging connector toward the vehicle's charging port; a camera disposed on or associated with the autonomous charging device for capturing images of the charging port, wherein a computer vision module determines an attitude of the vehicle charging port based on the images, the attitude including a position and orientation of the vehicle charging port socket and / or socket pins relative to the camera; a motion module; and a data communications module for communicating the images and operational data of the autonomous charging device between the camera and a common data storage and processing module, wherein the computer vision module comprises a neural network that is trained by a training module. A system for supporting connection of an electric vehicle connector to an electric vehicle charging port by an autonomous charging device. A training module for training the neural network of a computer vision system for the autonomous charging device.

Description

Detailed Description of the Invention

[0001] [Technical field of the invention] The present invention relates generally to an autonomous charging device and related methods and systems. The methods and systems according to the present invention aim to improve the performance and reliability of an autonomous charging device. The present invention also relates to a system and method for training a neural network of a computer vision system for an autonomous charging device.

[0002] [Background of the invention] Improving performance and reliability in robotic systems, and specifically robotic autonomous charging devices (ACDs), remains an area of ​​concern.

[0003] Robots are now widely used in many different industries to automate manufacturing procedures, such as on assembly or production lines. Recently, robots have been utilized in the automotive industry to automate the charging of batteries that power electric vehicles (EVs). To ensure a reliable connection between the socket and the connector, it is desirable for the robot to accurately locate the charging port and position the connector for a continuous and reliable process.

[0004] Further improvements are still needed, as several factors can contribute to plug-in failures and undesirable operational performance. Moreover, a challenge in establishing a physical connection is precisely positioning the connector within the socket before plugging it in. This must be done with an accuracy in the range of less than a few millimeters, which implies that the accuracy is met by the device for plugging in the connector, and not by the vehicle, which currently cannot be automatically positioned with an accuracy of a few centimeters.

[0005] Several solutions have been proposed to enable a precise connection between the connector and the socket, such as the use of magnetic coupling, but many of these techniques require modifications to either the EV or the connector, which can affect the scalability and accessibility of the solution in the market.

[0006] Other improvements have been made in recent years, and some ACDs currently known in the art are capable of locating an electric vehicle socket and guiding and inserting a charger connector into a charging port without operator intervention or modification of the vehicle or its charging port. These improvements rely on the use of computer vision and neural networks to accurately identify the vehicle or its charging port so that the ACD can reliably complete the plug-in and plug-out process. Devices for this purpose are known in the art, for example, from commonly assigned International Patent Applications PCT / NL2020 / 050266, PCT / NL2021 / 050115, PCT / NL2021 / 050410, PCT / NL2021 / 050495, and PCT / NL2021 / 05061, all of which are incorporated herein by reference. ACD devices according to the present disclosure may include a compliance mechanism to support several connectors, including, but not limited to, CCS-1 / 2 connectors.

[0007] Neural networks are used to perform complex tasks, such as classification tasks in recognizing patterns or objects in images, natural language processing, computer vision, speech recognition, bioinformatics, and other applications. The quality of a neural network's output depends on the quality of the neural network's training. Furthermore, training a neural network requires the collection and annotation of large amounts of training data to build a suitable training dataset. However, many training datasets do not accurately represent or underestimate the variability of the conditions for which the neural network is intended to be trained.

[0008] Autonomous charging devices for electric vehicles present an additional challenge: they are often operated in outdoor areas with varying conditions that affect process repeatability. Such conditions can include lighting conditions, charging port and connector variations (OEM-specific characteristics, wear and tear), movement and vibration of the ACD or vehicle, and weather conditions, among others. Even a slight shadow on the vehicle socket can affect orientation detection and prevent the ACD from completing the plug-in process. Given the large variation in conditions faced by autonomous charging devices and systems, a single computer vision module and / or neural network that fits all may not necessarily lead to improved operational performance. This technical issue does not generally exist for robots used for industrial purposes, such as in manufacturing facilities, which are limited to more controlled conditions and therefore higher process repeatability.

[0009] It would be desirable to provide an autonomous charging device and system that has a computer vision component with a robust neural network that supports operation under a wide range of conditions and also takes into account the instance of the device.

[0010] It is particularly desirable for the computer vision component to enable improved ACD posture detection after the ACD is installed and operational to support reliable connection between the connector and the vehicle charging port in an autonomous and reliable manner. In some cases, many ACDs are deployed simultaneously, and periodic on-site or remote updates of the computer vision component can be cumbersome; thus, it is desirable to achieve a reliable computer vision component in a relatively short time, or one that requires fewer training cycles.

[0011] Moreover, it would be desirable to provide a method and system that allows for testing and deployment of retrained neural networks without interrupting normal operation of the ACD or allowing operation without repeated system calibration.

[0012] Moreover, it would be desirable to provide a method and system that utilizes computer vision tools with robust neural networks for the operation of autonomous charging devices. [Brief Description of the Prior Art] Some existing systems have various shortcomings for some applications, and therefore there remains a need for further contributions in this field.

[0013] WO2020142496A1 describes a method for training a robot coupled to a camera, the process comprising: setting parameters of the robot or camera; capturing training images of a training object with the camera using the parameters of the robot or camera; changing the settings and capturing another training image; repeating such setting and capturing to obtain multiple training images based on different settings; training the system to recognize the training object based on the multiple training images; and evaluating the system using preselected test images. This document describes training an industrial robot intended to complete a manufacturing process. Industrial robots generally operate under controlled conditions, with little or no variation regarding their target object, and can be intensively trained under scenarios representing all possible conditions the industrial robot will encounter during its operation. Industrial robots generally include sophisticated hardware and software that may not be suitable for implementation in an autonomous charging device system.

[0014] Document US2022355692A1 describes a system for autonomously charging electric vehicles (EVs), the method including obtaining a trained machine learning (ML) model from a backend server, capturing an image using an image capture device of the charging system, a portion of the image comprising an EV charging portal, inputting the image into the trained ML model to identify one or more regions of interest associated with the EV charging portal in the image, identifying a location of the EV charging portal based on the one or more identified regions of interest and one or more image processing techniques, and providing information for moving a robotic arm of the charging system to a physical location based on the identified location of the EV charging portal.

[0015] [Summary of the Invention] In one embodiment, the present disclosure relates to an autonomous charging device (ACD) for connecting a charging connector to a vehicle charging port.

[0016] In one embodiment, the present disclosure relates to a system for supporting connection of an electric vehicle connector to an electric vehicle charge port by an autonomous charging device (ACD). In one embodiment, the present invention relates to a method for training a neural network of a computer vision system of an autonomous charging device. [Brief explanation of the drawings]

[0017] [Figure 1a] FIG. 1 illustrates an autonomous charging device. [Figure 1b] FIG. 1 illustrates a charging port. [Figure 2] FIG. 1 illustrates an autonomous charging device and its components, according to some embodiments. [Figure 3] FIG. 1 illustrates a system for supporting operation of multiple autonomous charging devices, according to some embodiments. [Figure 4] FIG. 1 illustrates an autonomous charging device and its components, according to some embodiments. [Figure 5] FIG. 1 illustrates an autonomous charging device and its components, according to some embodiments. [Figure 6] FIG. 1 illustrates an autonomous charging device and its components, according to some embodiments. [Figure 7] FIG. 1 illustrates the configuration of a training module, according to some embodiments. [Figure 8] FIG. 1 illustrates the configuration of a training module, according to some embodiments. [Figure 9] It depicts some images of charging ports. DETAILED DESCRIPTION OF THE INVENTION

[0018] [Detailed Description of the Invention] To improve the performance, reliability, and continued operation of autonomous charging devices for electric vehicles, also referred to herein as ACDs, it is desirable to provide methods and systems for improving ACD autonomy and reliability with respect to their computer vision components. According to the present disclosure, the computer vision component includes at least one computer vision neural network that enables estimation of the pose of a charging port, including its socket and pins, and subsequent completion of the plug-in / plug-out process.

[0019] Disclosed herein are robotic autonomous charging devices (ACDs) and related methods and systems for improving their performance and reliability. The methods and systems of the present disclosure support autonomous charging of one or more electric vehicles.

[0020] Also disclosed herein are methods and systems for training a computer vision neural network configured to support estimation of an electric vehicle charge port pose by an autonomous charging device, the pose including the charge port's location and orientation. The methods and systems may include assessing a target metric score, collecting training data, generating a training set including the training data, and using the training set to train a neural network to support estimation of the vehicle charge port's pose (e.g., identifying the location and / or orientation of the charge port's sockets and / or pins).

[0021] In some embodiments, a neural network for charging port pose estimation can be trained and retrained using a training set containing images from multiple ACD instances. The neural network can then be maintained by using case-specific data (e.g., data collected by one particular ACD or group of ACDs), non-case-specific data (e.g., data collected by ACDs on which the neural network was not originally intended to be deployed), and metadata (e.g., data collected by other sources).

[0022] Also disclosed herein are methods and systems for testing, validating, and / or deploying neural networks to computer vision modules, i.e., computer vision components associated with autonomous charging devices. Preferably, the methods and systems enable remote testing, validation, and deployment of neural networks to ACDs to enable continuous and autonomous operation of the ACDs.

[0023] Also disclosed herein are methods and systems that utilize computer vision systems with robust neural networks to improve vehicle charging port pose identification or estimation, preferably suited for device and vehicle use cases.

[0024] The autonomous charging device referred to herein comprises at least a controllable actuation mechanism comprising means for detachably or non-detachably supporting the vehicle charger connector, and is capable of executing an autonomous charging process including a series of steps executed by the ACD to complete plugging-in and plugging-out of the vehicle charging connector to allow the vehicle to be charged. The process includes at least the steps of determining the orientation of the vehicle charging port (including the socket and / or its pins), moving the vehicle charging connector toward the charging port, allowing charging of the vehicle to occur, and disconnecting the charging connector. The orientation of the vehicle charging port is preferably determined by the ACD by a computer vision module that utilizes image data from the charging port to estimate its position and orientation relative to a sensor, such as a camera, used to record the image data.

[0025] The performance of an autonomous charging device depends on many factors. According to the present disclosure, its performance may be affected by the individual performance of various components of the ACD, such as the performance of computer vision components, mechatronics, system and component calibration (e.g., internal and external calibration of cameras), damage and wear, etc. Additional factors external to the ACD, such as weather, lighting, and temperature conditions, may also affect performance.

[0026] According to the present disclosure, the reliability and autonomous operation of an ACD can be improved by improving the robustness of a computer vision component capable of identifying the relative position and orientation of a charging port. The computer vision component comprises or is based on a neural network model, and the neural network model is configured to provide an output including an estimate of the position and orientation of the vehicle charging port and / or its pins relative to the camera position. Because the performance of a computer vision component generally depends on the neural network it is built on, it is desirable to provide a system with a robust neural network that is suitable for an ACD system.

[0027] Actual information about performance can be useful in determining the need for updating, training, or retraining of neural networks. In some cases, updating or training can be triggered not only by actual performance indicators, but also based on the availability of data appropriate to the ACD instance that is expected to improve ACD performance, efficiency, safety, or any other factor.

[0028] An autonomous charging device may face a completely different set of operating conditions than another autonomous charging device, such as different climate conditions, charger port types, etc. Creating a neural network model that allows reliable operation for a group of ACDs poses significant challenges, and in some cases requires creating specific models for groups of ACDs that have similar or comparable operating conditions.

[0029] For the purposes of promoting an understanding of the principles of the invention, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe them, without, however, intending to limit the scope of the invention. Any alterations and further modifications in the described embodiments, and any further applications of the principles of the invention as described herein, are contemplated as would normally occur to one skilled in the art to which the invention pertains.

[0030] Referring to Figure 1a, an autonomous charging device 100 is depicted. Figure 1a also illustrates a vehicle 10, a charging connector 20, and a charging port 30. Referring to Figure 1b, a charging port (30) including charging port pins (31) is depicted.

[0031] Referring to FIG. 2a, an autonomous charging device (100) is depicted, and FIG. 2b further depicts a system (200) for supporting the operation of the autonomous charging device (100).

[0032] Referring to FIG. 3, there is depicted a number of autonomous charging devices (100) as well as a system (200) for supporting the operation of those devices. Referring to Figure 4, a schematic diagram of an autonomous charging device is depicted, including a controllable actuation mechanism (110), a camera (120), a motion module (140), a data communication module (160), a processor (150), a computer vision module (130), a common data storage and processing module (170), and a training module (180).

[0033] Referring to Figure 5, an autonomous charging device is depicted that includes two or more controllable actuation mechanisms. Also depicted are a controllable actuation mechanism (110), a camera (120), a motion module (140), a data communication module (160), a processor (150), a computer vision module (130), a common data storage and processing module (170), and a training module (180).

[0034] Referring to Figure 6, with reference to the autonomous charging device (100), a system (200) for supporting its operation is depicted. A processor (150) includes memory (151), storage (152), and a controller (153). The computer vision module (130), motion module (140), data communication module (160), and other modules may be located or hosted independently of the ACD processor (150), or may be partially or fully incorporated therein.

[0035] 4-5, one or more of the depicted components or modules may be physically integrated into an autonomous charging device, and one or more of the depicted components or modules may communicate or interact with the autonomous charging device to perform one or more of their functions.

[0036] In an embodiment, the present disclosure provides an autonomous charging device (ACD), comprising: at least one controllable actuation mechanism for supporting the electric vehicle charging connector and for moving the charging connector toward the vehicle charging port; at least one camera disposed on or associated with the autonomous charging device for capturing images of the vehicle charging port; a computer vision module disposed on and / or coupled to the autonomous charging device for identifying a vehicle charge port pose based on a neural network model and images, the pose including a position and orientation of a vehicle charge port socket and / or socket pins; a motion module for receiving or accessing the determined orientation of the vehicle charge port, the motion module configured to move a charge connector toward the vehicle charge port based on the orientation; a data communications module for communicating the images and autonomous charging device operational data to and from at least one common data storage and processing module, the common data storage and processing module for receiving or accessing at least one of case-specific images and non-case-specific images, the case-specific images including images communicated by the autonomous charging device and the non-case-specific images including images communicated by at least one other autonomous charging device or by a network of multiple autonomous charging devices; a processor configured to control at least one of a controllable actuation mechanism, a camera, a motion module, a computer vision module, and a data communication module; The present invention relates to an autonomous charging device (ACD) comprising:

[0037] In some cases, the computer vision comprises a neural network, the neural network being trained by a training module, the training module comprising: retrieving annotated images from a common data storage and processing module, each image including a feature descriptor; generating a training dataset including annotated images such that a combination of case-specific and / or non-case-specific images results in a multidimensional distribution of feature descriptors; Using the generated training dataset, train the neural network. It is configured to:

[0038] In some cases, the computer vision comprises a neural network, the neural network being trained by a training module, the training module comprising: receiving a score for a target metric associated with the autonomous charging device, and triggering training of a neural network if the score is below a threshold; retrieving annotated images from a common data storage and processing module, each image including a feature descriptor; generating a training dataset containing annotated images, such that a combination of case-specific and / or non-case-specific images results in a multidimensional distribution of feature descriptors as a function of the target metric; Using the generated training dataset, train the neural network. It is configured to:

[0039] In one embodiment, the present disclosure provides a system for supporting operation of one or more autonomous charging devices, the system comprising: a data communications module for communicating image and operational data to and from the autonomous charging device; A common data storage processing module, processing images and metadata from at least one autonomous charging device; processing operational data from at least one autonomous charging device; generating case-specific and non-case-specific images associated with at least one feature descriptor based on the processed images and operational data; a common data storage and processing module configured to: a neural network training module configured to train a neural network and deploy it to a computer vision module for at least one autonomous charging device; and Equipped with Training modules include: receiving a score for a target metric associated with the autonomous charging device, and triggering training of a neural network if the score is below a threshold; retrieving annotated images from a common data storage and processing module, each image including a feature descriptor; generating a training dataset containing annotated images, such that a combination of case-specific and / or non-case-specific images results in a multidimensional distribution of feature descriptors as a function of the target metric; Using the generated training dataset, train the neural network. The present invention relates to a system configured to perform the above.

[0040] In an embodiment, the present disclosure provides a method for training a neural network of a computer vision system of an autonomous charging device, the method comprising: a) receiving a score for a target metric associated with the autonomous charging device, and triggering training of a neural network if the score is below a threshold; b) retrieving annotated images from a common data storage and processing module, each image including a feature descriptor; c) generating a training dataset including annotated images such that a combination of selected case-specific and / or non-case-specific images results in a multidimensional distribution of functional descriptors, the multidimensional distribution being a function of the target indicator; d) training a neural network using the generated training dataset; and The present invention relates to a method comprising:

[0041] A controllable actuation mechanism (110), also referred to herein as a manipulator or robotic arm, is an actuation component of the device configured to support the vehicle charging connector, whether in a detachable or removable manner, and to guide the connector toward the vehicle charging port to complete the plug-in and / or plug-out process.

[0042] Sensors may be mounted within or around the ACD and used to acquire data about the vehicle, vehicle socket, or its surroundings. Multiple sensors may be mounted on the same ACD, and the multiple sensors may be of various types to gather multiple types of part / object information. Sensors include sensors that collect data, such as 2D or 3D cameras configured to acquire images, video, and / or audio, although such data may also be obtained from sensors installed outside the ACD system. Sensors according to the present disclosure further include force, light, wind, geographic location and orientation, temperature, and pressure sensors, as well as any combination thereof. Sensors according to the present disclosure may also provide the date and time the data was acquired.

[0043] The ACD comprises a computer system, also referred to herein as a processor (150), suitable for controlling at least one of the actuation mechanism (110), the camera (120), the computer vision module (130), the motion module (140), and the data communication module (160). The processor may be any type of suitable computer system, such as an edge computer, capable of controlling one or various components of the ACD directly or via a controller.

[0044] The ACD preferably includes a controller (153) for controlling at least one of the camera (120), computer vision (130), and motion module (140), etc. The modules may be controlled by or incorporated within the processor (150). The controller may be one or more different devices configured to control one or more of the above-mentioned elements.

[0045] The processor (150) may include, communicate with, or interact with a series of modules that may be controlled by the processor and one or more controllers, including a computer vision module (130), a motion module (140), and a data communication module (160). The modules further include a common data storage and processing module (170) and a training module (180), both hosted within the processor (150) or a separate processor, or networked with either of the devices (100) or (200), the separate processor being within the autonomous charging device and / or the system (200) supporting the operation of the device (100).

[0046] The computer vision module (130, 230) is based on a neural network model. In some embodiments, the computer vision model is configured to host one or more computer vision neural networks, or simply neural networks. The computer vision module may be in communication with or host a training module and / or a testing module. The training module and / or the testing module may be hosted independently of the computer vision module. In some embodiments, the computer system further includes an edge computer that hosts the training module and / or the testing module, and the edge computer can run machine learning algorithms based on the collected data to form knowledge for a general-purpose or application-specific neural network model.

[0047] The neural network can be any type of neural network that can be used in image processing, for example, the second neural network can be a feedforward neural network, a regulatory feedback neural network, a convolutional neural network, or a recurrent neural network.

[0048] The motion module (140) preferably contains the algorithms and processes necessary to implement the movement and operation of the ACD. The training module (180) preferably includes a training algorithm, such as, for example, a machine learning algorithm, including, but not limited to, a backpropagation algorithm, a gradient descent algorithm, a Newton method algorithm, a conjugate gradient algorithm, a quasi-Newton algorithm, and a Levenberg algorithm. The training module (180) may be located within or in communication with the autonomous charging device (100), a system for supporting its operation (200), or the autonomous charging system (300).

[0049] The ACD is in communication with at least one system (200) comprising a common data storage and processing module (270), the at least one system configured to cooperate with and / or support the operation of the ACD via a network environment. Suitable network environments include an enterprise-wide computer network, an intranet, a local area network, a wide area network, a personal area network, a cloud computing network, a crowdsourced computing network, the Internet, and the World Wide Web. The network can be a wireless network, a wired network, or any other type of communication network.

[0050] The system 200 also preferably includes a processor 250, which may further include memory and storage, and may include, host, or be in communication with a computer vision module 230, a training module 280, and a data storage and processing module 270.

[0051] The computer vision module (130), motion module (140), data communication module (160), and other modules mentioned herein may be stored in separate or common memory devices, either of a volatile or non-volatile type, and may be expressed in any suitable format, for example, but not limited to, source code, object code, and machine code.

[0052] In some embodiments, the ACD includes a lighting system for illuminating the charging port to support the operation of the computer vision system and / or camera component.

[0053] In some embodiments, the vehicle is an electric vehicle, which can be any vehicle that is at least partially powered by electric energy and includes a rechargeable battery. Hybrid vehicles are also intended to be included in this disclosure. In other embodiments, the vehicle can be a hydrogen-powered vehicle, a solar-powered vehicle, or any other vehicle with a charging port. Furthermore, the vehicle can be of any type, such as an automobile, a passenger car, a transportation vehicle, a truck, an industrial vehicle, a sports vehicle, a multi-wheeled vehicle, a ship or ferry, a utility task vehicle (UTV) or "side-by-side," and an aircraft.

[0054] Electric vehicle charging ports, also referred to herein as sockets, are generally standardized in their dimensions and functionality. The sockets can be any of a variety of connectors, including, but not limited to, AC or DC connectors, including, but not limited to, J1772-Type 1, GB / T, CCS-Type 1 and Type 2 (Combined Charging System), SAE Combo Plug, International Electrotechnical Commission (IEC) 62196 Plug, etc.

[0055] Figure 1b shows a diagram of a CCS Type 2 connector. The socket generally consists of several different pins, the layout and size of which depend on the specific type, such as a Type 2 connector with seven contact points: two small and five larger. The top row has two small contacts for signals, the middle row has three pins, the center pin is used for ground, and the outer two pins are used for power, optionally with two pins on the bottom row that are also used for power. For posture detection purposes, the socket and pin layout play an important role. Figures 9a-9d show images of vehicle sockets under different conditions.

[0056] An object's pose indicates the object's placement within the three-dimensional space it occupies. An object's pose may be specified relative to a viewpoint, such as a camera viewpoint. An object's pose may include three-dimensional information characterizing the object's rotation relative to the camera viewpoint. Alternatively, or in addition, an object's pose may include three-dimensional information characterizing the object's translation relative to the camera viewpoint.

[0057] While this disclosure provides for, but is not limited to, determining the attitude of an electric vehicle charging port, preferably, the attitude of an electric vehicle charging port can be understood as the position and orientation of the charging port on the vehicle, and in some embodiments, is represented by a 2D Cartesian position and yaw (x, y, θ) of the socket. However, in some embodiments, the attitude is a 6D attitude, where the position is defined by a 3D Cartesian position and the orientation is defined by the roll, pitch, and yaw of the socket.

[0058] Identifying the electric vehicle charge port attitude is useful for describing the attitude and orientation of the charge port relative to the camera position in order to guide the EV connector towards the charge port and complete the plug-in process.

[0059] The pose determination procedures described herein may provide improved techniques for determining the pose of an object from visual data. The pose of the object may be obtained from a single image, multiple images, possibly from the same pose, multiple images, possibly from different poses, or by utilizing a previously determined pose. Alternatively, or in addition, the pose of the object may be obtained from multi-view images or video.

[0060] In some embodiments of the present disclosure, vehicle or vehicle charge port pose determination may be performed by a neural network on which a computer vision module is based. Preferably, the neural network may be trained to determine an estimated vehicle charge port location and orientation through analysis of one or more images. The estimated socket pose may include estimates for socket principal axis, roll, elevation, angular position, attitude, and azimuth.

[0061] In other embodiments of the present disclosure, vehicle or vehicle charging port pose determination may be supported by a neural network. Preferably, the neural network may be trained to identify the locations of geometric features of the socket that should be used as fiducial markers based on which an algorithm, such as a perspective n-point algorithm like solvepnp or ransac, can estimate the socket pose. This may use a single image or multiple images.

[0062] The training process of a neural network usually determines the quality of the network output. Typically, large amounts of training data are collected and annotated manually or automatically. However, many training datasets may inaccurately or insufficiently represent the data on which the neural network is to be trained.

[0063] In the context of the present invention, data represents an image and may optionally and preferably include annotations, metadata, and / or classifications such as feature descriptors. In the context of this disclosure, annotation refers to pose-related information, such as the position and orientation of the socket and / or socket pins in the image, or the pixel locations of recognizable features of the socket that can be used as fiducial markers.

[0064] In the context of this disclosure, metadata refers to information that is not directly related to the pose of the socket, such as the conditions in the image or the conditions when the image was recorded. The images are preferably labeled, classified, categorized, etc. using functional descriptors, which include data obtained from the annotation process, ACD operations, and / or metadata. The classifications may indicate qualitative or quantitative subdivisions of the metadata.

[0065] In the context of this disclosure, feature descriptors are characteristics or features associated with the ACD's operation and environment, derived from images or metadata. These descriptors can be qualitative, quantitative, or a combination thereof, providing details about ACD performance, image properties, weather conditions, lighting conditions, socket locations, vehicle attributes, and geographic location, among other things. Feature descriptors can be generated manually, automatically, or through an algorithm and are utilized to train a neural network for the ACD. Constructing a training set for training a neural network can present some challenges. Building a training set is a critical step in training a neural network, and therefore, the proper operation of the neural network primarily depends on the construction of the training set. The amount of data required can be very large, such as tens of thousands, hundreds of thousands, millions, or more data points. The network learns using the training set, thereby generalizing its learning correctly to predict appropriate outputs for inputs.

[0066] It is an object of the present invention to provide an autonomous charging device, a system and method for supporting the autonomous charging device, whereby a computer vision module is supported by a neural network, and the neural network is trained based on a dataset designed to enable the computer vision module to perform sufficiently well in almost all relevant use cases. To this end, the present invention aims to provide a suitable training method and system such that a suitable balance of data is constructed in the training dataset to ensure that relevant cases are adequately represented.

[0067] In some embodiments, a computer vision neural network is trained by providing the computer vision neural network with corresponding input data and target output data. A training set is all this data, including example inputs and target outputs. Through training, the network weights may be adjusted sequentially or iteratively so that, given a particular input from the training set, the network's output approaches (e.g., as closely as possible, desirable, or practical) the target output corresponding to that particular input data.

[0068] According to the present disclosure, construction of the training data set may utilize data obtained from an individual source or from multiple sources. Preferably, the training data set includes data from one ACD or a specific group of ACDs for which training should be triggered (case-specific), or data from multiple non-specific ACDs (non-case-specific), or data from ACD data combined with metadata.

[0069] The neural network can then be retrained, enhanced, or customized for a more specific instance or set of instances using images from a single ACD or from a specific group of ACDs with common instances for which the retrained neural network is deemed suitable (e.g., a group of ACDs that experience poor performance under clear skies or at low temperatures), thereby enabling the retrained neural network to expect improved performance over the existing neural network in pose estimation for that instance and under external influences.

[0070] In some embodiments, training can continue indefinitely using a dataset of unlimited size. However, the training time for a neural network that needs to be deployed is generally limited. With limited training time, the neural network can only be trained on a finite number of images, in other words, a dataset of limited size. One aspect of the present invention involves generating a training dataset to include a selection of images in a dataset of limited size so that the neural network can be trained to perform well on a variety of images taken under various conditions. In practice, the size of the dataset is balanced with training and network complexity, time, and computational resources.

[0071] Preferably, the training and evaluation of the network are performed based on a specific application, whereby the dataset, training, and evaluation are optimized based on the requirements of the specific application. Preferably, the trained or retrained neural network is adapted or customized from a more general one to a partially specialized one for the ACD instance. The trained or retrained neural network can be used to support EV socket pose estimation with improved performance (e.g., higher accuracy), which can result in better reliability.

[0072] Moreover, depending on the application, different types of training criteria and methods may be used for training and evaluating neural networks. In some embodiments, training is based on a specific application and may incorporate datasets from other specific applications.

[0073] The output of a neural network may change or deviate over time. In the context of this disclosure, when an ACD utilizes a neural network to perform a task, with or without the user's knowledge or potentially without any user involvement, changes or deviations in the behavior of the neural network may affect the operation of the ACD. Small deviations in a neural network output, such as a pose estimate, may be sufficient to affect the performance of the ACD process.

[0074] The decision to trigger a training session for the neural network is a critical step to ensure the continued reliability of the ACD process. Referring to FIG. 7 , an embodiment according to the present disclosure includes receiving, by a training module, a score of a target metric associated with the autonomous charging device and triggering training of the neural network if the score is below a threshold. In some cases, the training module additionally or alternatively receives a digital case representation of the autonomous charging device.

[0075] The neural network can be an existing network that has been previously trained, retrained, or not previously trained. The neural network can be trained potentially using new (more recent) data provided by one or more ACDs, so that a retrained network or an entirely new model can be developed that provides better estimation and addresses deviations.

[0076] According to various embodiments, a training module receives scores for target metrics associated with the autonomous charging device and triggers training of a neural network if the scores are below a threshold. The training module triggers training of the neural network if scores related to operational metrics, also referred to as target metrics or assessment metrics, are below a threshold. In some cases, the training module determines expected scores for the autonomous charging device metrics upon deployment of the trained neural network.

[0077] The scores of the target indicators may be determined by the common data storage and processing module (170, 270). The scores of the target indicators may be determined through multiple means. The scores of the target indicators may be determined by evaluating the change in statistical moments of one or more operational indicators. Statistical moments include mean, covariance, variance, skewness, kurtosis, or combinations thereof. The common data storage and processing module may determine the scores of the target indicators automatically or under operator guidance using neural networks. The scores may be determined using mathematical operations and / or machine learning algorithms. Preferred machine learning algorithms include random decision forests, regression algorithms, etc. The mathematical operations may include distributions based on the Softmax operator.

[0078] The common data storage and processing module is configured to receive data related to the target indicator, process the data, and determine a score based on the data. The score may be a number or an array of numbers, such as between 0 and 1, indicating a score associated with the performance of the autonomous charging device. The score may be an integrated score if it is associated with one or more indicators. In some cases, when one or more indicators are considered, the module identifies a significant indicator corresponding to the one having the greatest weight in determining the score. The score of the target indicator may refer to a current score, a future score, a predicted score, an estimated score, or an expected score.

[0079] The training module may trigger a training instance when the score or combined score is less than 0.9, less than 0.8, less than 0.7, less than 0.6, less than 0.5, less than 0.4, less than 0.3, less than 0.2, or less than 0.1.

[0080] The score in certain embodiments may take into account one or more target indicators, and in some cases the classification associated with each indicator. Target indicators according to the present disclosure include the following indicators: ·Performance indicators of autonomous charging devices; Quantitative and qualitative metrics of training data, Computer vision module specificity index, Neural network training time elapsed, · Target object changes, and · A combination of these It includes at least one of the following:

[0081] Performance metrics for the autonomous charging device relate to metrics derived from performance data collected from the ACD, including, but not limited to, attitude estimation rate, plug-in attempt success rate, plug-in attempt failure rate, false negative rate, false positive rate, and their associated estimates.

[0082] In a non-limiting example, the training module receives a performance score for a target metric associated with the autonomous charging device and triggers training of the neural network if the score is below a threshold based on:

[0083] [Table 1]

[0084] The training module may trigger a training decision when the score of the performance index or the combined score is less than 0.9, less than 0.8, less than 0.7, less than 0.6, less than 0.5, less than 0.4, less than 0.3, less than 0.2, or less than 0.1. Preferably, the training module is configured to trigger a training decision if the score of the performance index or the combined score is less than 0.6.

[0085] The quantitative and qualitative metrics of the training data refer to the expected improvement to the neural network's output that the training data can bring. The autonomous charging device preferably continuously captures and generates images. The data storage and processing module can assign a score related to the expected improvement to the neural network's output that the data can bring. Advantageously, the decision is trained when an expected improvement is observed, even if the autonomous charging device's performance score is below a threshold that triggers a training instance.

[0086] [Table 2]

[0087] The training module may trigger a training decision when the score or combined score of the quantitative and qualitative indicators of the available data is less than 1, less than 0.9, less than 0.8, less than 0.7, less than 0.6, less than 0.5, less than 0.4, less than 0.3, less than 0.2, or less than 0.1. Preferably, the training module may trigger a training decision when the score or combined score of the quantitative and qualitative indicators of the available data is less than 0.4. In this case, a high or very high improvement in the output of the neural network based on the available data is expected.

[0088] The computer vision module specificity metric relates to the distribution of case-specific and non-case-specific data used to train the existing neural network. Although very low and very high specificity do not necessarily have an immediate impact on the operation or performance of the ACD, based on the specificity score, the training module may trigger a training session to compensate for the improper distribution of data.

[0089] [Table 3]

[0090] For example, as more case-specific data becomes available over time for a particular ACD or a particular group of ACDs, the specificity-related threshold may be increased, which triggers training. The training module may trigger a training decision when the score or combined score of the computer vision module specificity index data is less than 1, less than 0.9, less than 0.8, less than 0.7, less than 0.6, less than 0.5, less than 0.4, less than 0.3, less than 0.2, or less than 0.1. Preferably, the training module may trigger a training decision when the score or combined score of the computer vision module specificity index data is less than 0.4. In some cases, the training module may trigger a training session when the performance score and specificity score are below a preferred threshold.

[0091] The training elapsed time of at least one ACD according to the present disclosure relates to an indication of the time elapsed since the most recent training session for a particular neural network for a particular ACD or network of ACDs. The training module may trigger a training decision if the training elapsed time score or combined score is less than 1, less than 0.9, less than 0.8, less than 0.7, less than 0.6, less than 0.5, less than 0.4, less than 0.3, less than 0.2, or less than 0.1. Preferably, the training module may trigger a training decision if the training elapsed time score or combined score is less than 0.4.

[0092] A new target object variation according to the present disclosure relates to the introduction of a new charging port for which the network has not yet been trained and with which the ACD is expected to operate. A new target object variation may also relate to a change in the location of the socket on the vehicle or a change in the area around an existing socket. Examples include additional flaps, indicator lights, or other geometric features such as new materials or colors.

[0093] The common data storage and processing module may be further configured to assign a score and / or classification to the target metric, or in some cases, to the combined score of all assessed metrics. In some cases, the score may be assigned by a training module. The training module receives the score of the target metric associated with the autonomous charging device and triggers training of the neural network if the score is below a threshold. The score of the associated target metric may be determined, calculated, defined, estimated, or predicted through different means, preferably via an algorithm, a computer-based model, a human operator, or a combination thereof. The threshold value according to various embodiments may be a static threshold or a dynamic threshold.

[0094] The data communications module is configured to communicate images and autonomous charging device operational data to and from at least one common data storage and processing module, the common data storage and processing module being for receiving or having access to at least one of case-specific images and non-case-specific images, the case-specific images including images communicated by the autonomous charging device, and the non-case-specific images including images communicated by at least one other autonomous charging device or a network of multiple autonomous charging devices.

[0095] The target metrics and / or their scores can be inputs for subsequent steps such as obtaining annotated images, generating a training dataset, training a neural network, and testing the output of any neural network.

[0096] Scoring a target indicator involves systematically combining classifications and scores for an indicator or set of indicators into a single global score that allows for scoring and subsequent triggering of training.

[0097] Training can be initiated by a training module in an ACD, in a network environment, or in a centralized system. Training can be automatically triggered by a training decision module either within one particular ACD, in a network environment, or a combination thereof. Training can be triggered by an operator, and the training decision module can allow the operator to approve or reject the training decision. The training decision module can also allow the operator to adjust the network heuristics before triggering the training decision.

[0098] According to various embodiments, the input of / for a neural network according to the present disclosure can be any data that can support the neural network in identifying the pose of an object, preferably the pose of a vehicle or its charging port, or the contours of geometric features that can be used as fiducial markers. By way of example, the input can be at least one or multiple images, and the output can be the identification of the socket shape and pin segmentation for each individual image.

[0099] The output may include the pose (position and orientation) of the socket in the image relative to the camera. Data, in the context of the present invention, refers to data relating to the vehicle, data relating to the vehicle's charging port, or any other data that can support the estimation of the socket's attitude.

[0100] In this disclosure, operational data refers to one or more operating parameters of the autonomous charging device. In some embodiments, the one or more operating parameters include at least one operating parameter of a controllable actuation mechanism, a camera, a computer vision module, a motion module, a data communication module, a data storage and processing module, a processor, or a training module. In some embodiments, the operational data includes a signal transmitted by a sensor, the signal corresponding to a measurement of the at least one operating parameter by the sensor.

[0101] In this disclosure, metadata refers to data that is not directly derived from image data or operational data, such as geographic data, meteorological data, location data, etc. Metadata may be used to generate functional descriptors, which may be associated with image data and operational data.

[0102] According to this disclosure, a dataset refers to a set of data collected by an ACD or any other source regarding an ACD, multiple ACDs, a vehicle, a vehicle socket, its surroundings, and combinations thereof. A training dataset refers to a dataset generated for the purpose of training a neural network.

[0103] According to various embodiments, the computer vision module comprises a neural network that is trained by a training module, the training module comprising: Obtaining annotated images from a common data storage and processing module, each image containing a feature descriptor; generating a training dataset including annotated images such that a combination of selected case-specific and / or non-case-specific images results in a multidimensional distribution of feature descriptors, the multidimensional distribution being a function of the target metrics; Use the generated training dataset to train a neural network It is further configured for:

[0104] The training module may also be configured to generate a test data set. The test data set may be generated by splitting the annotated data to generate a training data set and a test data set. The training module may also be configured to test the output of the trained network using the test data set.

[0105] The training module may also be configured to test the output of the trained neural network against a benchmark dataset that includes a set of poses that have been validated against ground truth.

[0106] The training module also Deploying the trained neural network to a computer vision module if an improvement in the output of the trained network is observed; repeating the steps of generating a new training dataset and / or retraining the neural network using the generated dataset until an improvement in the output of the trained network is observed; Stopping neural network training and The method may be configured to:

[0107] The training dataset can be generated from newly collected data, data derived from an existing dataset (i.e., a sliced ​​dataset), metadata, or a combination thereof. The training dataset can be generated automatically by a computer-based algorithm or an operator, in either a supervised or unsupervised format.

[0108] The training module is configured to retrieve the annotated images from the common data storage and processing module and generate a dataset for training the neural network based on the images, each image including a functional descriptor. The training dataset can be generated by including images with functional descriptors such that a combination of selected case-specific and non-case-specific images results in a multidimensional distribution of the functional descriptors for the images.

[0109] The training module preferably acquires / receives annotated images from a common data storage and processing module, each image preferably including and / or associated with a functional descriptor. The training module may also, or alternatively, acquire or receive images and / or other data by collecting such images and data from the operation of an ACD or fleet of ACDs. Such data may include images of the ACD, the vehicle, its charging port, and / or combinations thereof. The autonomous charging device is preferably configured to continuously capture and collect data, which is stored in a common data storage and processing module in communication with the training module.

[0110] Case-specific data according to the present disclosure refers to data acquired by specific autonomous charging devices, which preferably have a common set of operational characteristics or are expected to operate under similar conditions, such as installation location, weather conditions, lighting conditions, etc. Due to the common set of operational characteristics, a trained neural network deployed on the group of devices is expected to provide substantially the same output type. Thus, a trained neural network may be deployed to a set of devices located in different locations but having substantially similar operational characteristics.

[0111] Non-case-specific data according to this disclosure generally refers to data acquired by other ACDs operating under different conditions than the ACD or ACDs for which the training decisions are made. The training module is configured to retrieve annotated images from the common data storage and processing module, each image including a functional descriptor. The annotated images may include case-specific images and non-case-specific images, each image including a functional descriptor. The functional descriptor may be associated with each image by the common data storage and processing module. In some cases, the functional descriptor may be associated with each image by the training module. In all cases, the functional descriptor may be associated with each image automatically by a computer-based algorithm or by an operator, either in a supervised or unsupervised manner.

[0112] The case-specific data and non-case-specific data preferably include annotated data and one or more associated functional descriptors. In certain embodiments, annotated data including functional descriptors may be obtained, and additional functional descriptors may not be required. In certain embodiments, annotating the data with functional descriptors may be performed for functional descriptors that represent at least one characteristic, and more preferably multiple characteristics, of the data. The functional descriptors may then be clustered in dataset construction to represent ACD instances or sub-instances.

[0113] In some embodiments, the common data storage and processing module is configured to receive or have access to case-specific images and / or non-case-specific images, where the case-specific images include images communicated by the autonomous charging device and the non-case-specific images include images communicated from at least one other autonomous charging device or from a network of autonomous charging devices. In some embodiments, the data storage and processing module is configured to annotate the images with geometric features of the charging port socket and pins to generate fiducial markers as input for the neural network. Annotating the images includes annotating a dataset of images by an operator or a computer to create ground truth data, and may include manually identifying and annotating the shape of the vehicle socket, its connecting pins, or any components thereof.

[0114] Each image may have at least one qualitative or quantitative descriptor. A single image may have multiple descriptors, each associated with a different category. In other words, each image may be described (qualitatively or quantitatively) along multiple dimensions. By extension, a collection of images with multidimensional descriptors, in other words, a dataset with multidimensional descriptions, has a distribution of descriptors along each dimension.

[0115] The case-specific and non-case-specific images, including the annotated images and / or one or more associated functional descriptors, may be generated, collected, collected, retrieved, and / or processed automatically by a computer-based algorithm or by an operator by / in a training module or by / in a common data storage and processing module.

[0116] In some embodiments, the conditions under which the ACD operates may be captured by, or implied or derived from, an image to create a suitable functional descriptor. The functional descriptor may be estimated or predicted based on other metadata, and preferably may be categorized. Thus, a functional descriptor for an image may be "red vehicle," and the classification of that functional descriptor may be "vehicle color." Sub-classifications may be created, all of which are within the scope of this disclosure.

[0117] Functional descriptors may have qualitative properties, quantitative properties, or a combination thereof. Functional descriptors may be descriptors of device performance, image properties, weather conditions, light source, light conditions, light direction, socket location, socket angle, image orientation, vehicle brand, vehicle type, vehicle color, geographic location, customer name, customer project, date and time of image recording, socket status, socket visibility, camera properties, among many others. Functional descriptors may be created manually, automatically, or continuously via an algorithm. Other functional descriptors associated with device features, vehicle features, or condition features in or around them not described herein are considered part of this disclosure.

[0118] The functional descriptors may have qualitative properties, quantitative properties, or a combination thereof. The functional descriptors may include device performance (e.g., a plug-in success value or a plug-in failure value), image properties (e.g., brightness, contrast, temperature, tint, hue, saturation, gamma, blur, etc.), weather conditions (e.g., snow, rain, wind, thunder, etc.), light conditions (e.g., bright light, dark light, strong shadows, weak shadows, etc.), light direction (e.g., upward lighting, downward lighting, etc.), socket location (e.g., the distance from the ground to the socket location), socket angle (e.g., 5 degrees, 10 degrees, 15 degrees, etc.), image orientation (e.g., vertical, horizontal, etc.), vehicle brand (e.g., Toyota, BMW, Audi, etc.), manufacturer (e.g., BMW, BMW X5, BMW X6, BMW X7, BMW X8, BMW X9, BMW X9, BMW X5, BMW X6, BMW X7 ... The descriptors can be many, such as vehicle type (e.g., commercial truck, passenger car, etc.), vehicle color (e.g., white, black, gray, etc.), geographic location (e.g., a particular country, city, town, etc.), customer name (e.g., Customer A, Customer B, Customer C, etc.), customer project (e.g., Project A1, Project B2, etc.), date and time of image recording, socket status (e.g., damaged socket, blocked socket, altered socket, etc.), socket visibility (e.g., fully visible socket, partially visible socket, etc.), camera properties (e.g., exposure, aperture, ISO, shutter speed, etc.), etc. Feature descriptors can be created manually, automatically, or continuously via an algorithm.

[0119] One of the objectives of the present invention is to train a neural network on a training dataset to improve the reliability and dependability of ACD operation, the training data being suited to the characteristics of the autonomous charging device on which the network is deployed. The present invention comprises providing a robust neural network model, which takes into account forming a data distribution of a training dataset, subset, or slice, on which the neural network is trained. The training module is configured to obtain a multidimensional distribution of functional descriptors by combining selected case-specific and / or non-case-specific images, the multidimensional distribution being a function of a target indicator. In an exemplary implementation, if the target indicator is a computer vision module specificity indicator, and if the score of the specificity indicator is below a threshold, the training module is configured to obtain annotated images and generate a training dataset comprising the annotated images, such that the combination of the case-specific and / or non-case-specific images results in a multidimensional distribution of functional descriptors as a function of the specificity indicator. This allows the training module to create a multidimensional distribution in which the distribution of case-specific images is increased relative to non-case-specific images, and then test the trained neural network against a test or benchmark dataset to evaluate the improvement in output under the generated training dataset as a function of the specificity metric.

[0120] The training module and / or the data storage and processing module may be further configured to receive data including image data, operational data, and metadata related to the ACD to generate an ACD digital case representation or an ACD digital case model. The ACD digital case representation may include parameters representing actual operating conditions of the ACD. The ACD digital case representation may include functional descriptors, which may be distributed in the model to represent the operating conditions of the ACD. The training module is configured to receive the digital case representation and generate a training dataset comprising annotated images based on the feature model and / or based on target indicators, such that a combination of selected case-specific and / or non-case-specific images results in a multidimensional distribution of functional descriptors. The multidimensional distribution may be a function of the target indicator, the digital case representation, or a combination thereof. The multidimensional distribution is such that the trained neural network performs optimally in the intended use case. Preferably, the training dataset is generated such that a combination of selected case-specific and / or non-case-specific images results in a multidimensional distribution of functional descriptors. The ACD digital case representation, or ACD digital case model, may be generated automatically by a computer-based algorithm or by a human operator, either in a supervised or unsupervised manner. Preferably, the ACD digital case representation is generated by a data storage and processing module.

[0121] The training dataset may be generated by the training module based on algorithms such as logistic regression, generative adversarial networks, decision trees, random forests, naive Bayes, k-nearest neighbors, and gradient boosting algorithms. The training dataset generation may be performed in a supervised or semi-supervised mode, where the training module may be monitored, directed, approved, and / or rejected by a human operator.

[0122] The training module is configured to train the neural network using the generated training dataset. In some cases, training of the neural network occurs within the training module. In some cases, training occurs within the network environment.

[0123] The multidimensional distribution of feature descriptors is preferably and generally not a static distribution, since it may depend on the actual instance of ACD, which may change over time. In some cases, the multidimensional distribution may be a fixed distribution. Similarly, the distribution of case-specific images and non-case-specific images is not a static distribution, since it may depend on data availability. In some instances, typically at the start of operation, little or no case-specific data may be available, but may become available later during operation, which may trigger a training session and adjustment of the image distribution in the dataset. Furthermore, as more case-specific image and operational data become available, network training may become appropriate. Image and operational data related to system performance degradation may be particularly important.

[0124] In some embodiments, the training data set is constructed in such a way that it comprises at least 20% case-specific data, at least 40% case-specific data, at least 60% case-specific data, or at least 80% case-specific data.

[0125] In some cases, the training data set is generated such that all relevant conditions are equally represented in the data set so that the neural network is not optimized for the most common conditions, but rather performance is optimized for all conditions that the ACD or ACDs may encounter.

[0126] In some embodiments, suitable images, either case-specific or non-case-specific, may already exist in an existing dataset, but the neural network has not yet been trained under those images. Adding existing data to the training dataset instead of newly acquired data may result in a more favorable distribution of feature descriptors.

[0127] The training data set is generated to account for data gaps, where the data gaps indicate conditions related to ACD that are not well represented in the training data set for which the neural network was originally trained. The training module is further configured to proactively trigger training instructions to fill data gaps, for example, for particular vehicles, socket types, lighting conditions, etc. In doing so, the proactive data gap filling aims to enhance the robustness of the system, even if the system happens to already be working well with existing networks and conditions.

[0128] The training dataset may typically be generated iteratively over the operational life of the ACD, and there may not be a static optimum condition to iterate over. A set of conditions may be observed before or during installation of an ACD or a group of ACDs, and a multidimensional distribution of functional descriptors may be generated based on the set of conditions. A multidimensional distribution of functional descriptors that takes into account case-specific and non-case-specific images that represent the actual operating conditions of the ACD or group of ACDs may be used to generate the dataset.

[0129] If case-specific images are not available for the conditions to be represented in the dataset, non-case-specific images may be used if available. Case-specific or non-case-specific images may become available over time, and the availability of newly collected images and data may be the basis for the training decision. If the training decision is based on the availability of newly acquired data, this newly acquired data is generally included in the dataset. During operation of the ACD or ACD cluster, operational conditions may change or evolve, which may trigger training or retraining. In either scenario, the training module generates a balanced dataset with case-specific and non-case-specific images to enable a robust and well-functioning neural network. In some cases, if the training decision is triggered due to a performance metric causing a confidence value to fall below a threshold, image data for which the ACD or ACD cluster is performing poorly—in other words, image data for which plugging attempts failed—may be used to form a multidimensional distribution of feature descriptors. Preferably, the training dataset contains more case-specific data than the initial existing dataset.

[0130] In some embodiments, the training module is further configured to split the annotated data to generate a training data set and a test data set, and to test the output of the trained network using the test data set.

[0131] In some embodiments, the training module is further configured to test the output of the trained neural network against a benchmark dataset comprising a set of validated poses.

[0132] In some embodiments, the training module comprises: Deploying the trained neural network to a computer vision module if an improvement in the output of the trained network is observed; repeating the steps of generating a new training dataset and / or retraining the neural network using the generated dataset until an improvement in the output of the trained network is observed; Stopping neural network training and The device is further configured to:

[0133] Testing of the trained or retrained neural network can occur at the ACD level or at the central server level. In some embodiments, testing the trained neural network comprises testing the neural network against a validation data set. In some embodiments, testing the trained neural network further comprises testing the neural network against a test data set.

[0134] In some embodiments, the methods and systems further comprise testing the trained neural network against an existing neural network, preferably determining a change in the target metric. Preferably, the testing further comprises testing the existing network against an existing dataset, testing the existing network against a new dataset, testing the re-trained network against the existing dataset, and testing the re-trained network against the new dataset.

[0135] In a non-limiting example according to the present invention, a training module receives a digital case representation model for an ACD whose performance score has been determined to be below a threshold. The digital case representation model includes a set of functional descriptors. The digital case representation model includes a high percentage of black and gray colors and a high percentage of operating hours between 08:00 and 13:00 and between 13:00 and 18:00. The training module generates a training data set using the following multidimensional distribution of the functional descriptors:

[0136] [Table 4]

[0137] The training module then trained the neural network, tested the trained neural network on the test dataset, and observed improvements in intersection-over-union and mean center distance in charging port pose estimation. The training module then sent neural network update instructions to the computer vision module.

[0138] Thereafter, during normal operation, a signal is received indicating that the performance score has been determined by the training module to be below a threshold, and the performance score has been determined to be low for white vehicles.

[0139] The training module triggers training that includes the following adjusted multidimensional distributions of feature descriptors:

[0140] [Table 5]

[0141] The training module may be configured to adjust the distribution for vehicle color to include more data for white vehicles compared to black vehicles (e.g., 35% black, 45% white), while maintaining the multidimensional distributions for the other feature descriptors, until improvements in the neural network's output are observed. For fixed-size datasets, this may result in images from the initial training dataset not being included in the new training dataset, as there is also a time-of-day distribution to consider. For datasets of unlimited size, this may mean that more images must be added across all classifications to meet the target distribution.

[0142] In some embodiments, the dataset or training dataset is generated at least in part using artificially created data, preferably to fill data gaps. Dataset construction can be done in a variety of ways, based on existing data, preferably where images are altered to modify specific conditions. As an example embodiment, images originally recorded under clear skies can be altered to represent snowy conditions, or images with vehicles of a particular color can be altered to remain the same except for the color of the vehicle.

[0143] These variations can be implementations of image augmentation software or specialized neural networks to modify the image. The data can be constructed entirely digitally, for example, using a 3D model of the vehicle (socket) in a simulated environment, or through more complex generative neural networks.

[0144] Techniques and methods useful for conducting learning of training objects are contemplated herein. In some embodiments, neural network training is generally performed separately or remotely from the ACD, such as in a network environment, although in some embodiments it may be performed within the ACD domain, such as on an edge computer.

[0145] In some embodiments, the testing further comprises validating the trained network against a benchmark dataset. The benchmark dataset may include ground truth data including a set of validated pose-specifics. In some embodiments, the benchmark. In some embodiments, various benchmark datasets may be constructed, and the retrained network may be validated against a benchmark set that is case-specific to the ACD instance.

[0146] For the purposes of promoting an understanding of the principles of the invention, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the embodiments. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended. Any alterations and further modifications in the described embodiments, and any further applications of the principles of the invention as described herein, are contemplated as would normally occur to one skilled in the art to which the invention pertains.

[0147] Referring to the drawings, dashed lines represent components that may or may not be optional. In some instances, one or more components may be referred to herein as being "configured to," "configured by," "configurable to," "operable to," "adapted," "capable of," "adaptable," etc. Those skilled in the art will recognize that such terms (e.g., "configured to") generally encompass active and / or inactive and / or standby components unless the context requires otherwise.

[0148] In particular, conditional language used herein such as "can," "could," "could possibly," "could," "for example," and the like, is intended to generally indicate that certain embodiments include particular features, elements and / or improvements, while other embodiments do not include particular features, elements and / or steps, unless expressly stated otherwise or understood otherwise by context.

[0149] Such conditional language is not thereby generally intended to imply that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining, with or without author input or prompting, whether these features, elements, and / or steps are included in or should be performed in any embodiment. The terms "comprising," "including," "having," and the like are synonymous and are used open-ended and inclusively, and do not exclude additional elements, features, acts, operations, etc. Also, the term "or," when used, for example, to connect a list of elements, is used in its inclusive sense to mean one, some, or all of the elements in the list (rather than in its exclusive sense). Additionally, the articles "a," "an," and "the," as used in this specification and the appended claims, should be interpreted to mean "one or more" or "at least one," unless otherwise specified.

[0150] As used herein, a phrase referring to "at least one of" or "and / or" a list of items refers to any combination of those items, including single members. Similarly, while operations may be depicted in the figures in a particular order, it should be understood that such operations need not be performed in the order shown or in sequential order, or that all of the depicted operations need not be performed, to achieve desirable results. Furthermore, the figures may generally depict one or more exemplary processes in the form of a flowchart. However, other operations not depicted may be incorporated into the exemplary methods and processes generally depicted. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the depicted operations. Additionally, operations may be rearranged or reordered in other implementations. Under certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations; the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

[0151] The detailed description set forth above is exemplary only, and variations that do not depart from the spirit and / or scope of the claimed subject matter are intended to be within the scope of the claims. Such variations should not be considered a departure from the spirit and scope of the claimed subject matter.

[0152] It should be noted that the processes, methods, acts, and instructions described herein may be embodied as executable instructions stored on a computer-readable medium for use by or in association with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will appreciate that various types of computer-readable media for storing data may be included for some embodiments. As used herein, "computer-readable medium" includes one or more of any suitable medium for storing executable instructions of a computer program such that an instruction execution machine, system, apparatus, or device may read (or fetch) the instructions from the computer-readable medium and execute the instructions to implement the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes portable computer diskettes, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory devices, and optical storage devices including portable compact discs (CDs), portable digital video discs (DVDs), and the like.

Claims

1. 1. An autonomous charging device (ACD), comprising: at least one controllable actuation mechanism for supporting the electric vehicle charging connector and for moving said charging connector towards the vehicle charging port; at least one camera disposed on or associated with the autonomous charging device for capturing images of the vehicle charging port; a computer vision module disposed on and / or coupled to the autonomous charging device for determining a posture of the vehicle charge port based on a neural network model and the image, the posture including a position and orientation of the vehicle charge port socket and / or socket pins; and a motion module for receiving or accessing the identified orientation of the vehicle charge port, the motion module configured to move the charging connector toward the vehicle charge port based on the orientation; a data communications module for communicating the images and operational data of the autonomous charging device to and from at least one common data storage and processing module, the common data storage and processing module for receiving or accessing at least one of case-specific images and non-case-specific images, the case-specific images including images communicated by the autonomous charging device, and the non-case-specific images including images communicated by at least one other autonomous charging device or by a network of multiple autonomous charging devices; and a processor configured to control at least one of the controllable actuation mechanism, the camera, the motion module, the computer vision module, and the data communication module; Equipped with The computer vision module comprises the neural network, the neural network being trained by a training module, the training module comprising: receiving a score for a target metric associated with the autonomous charging device and triggering training of the neural network if the score is below a threshold; - retrieving annotated images from the common data storage and processing module, each image including a feature descriptor; generating a training dataset comprising said annotated images, such that a combination of case-specific and / or non-case-specific images results in a multidimensional distribution of functional descriptors as a function of said target indicators; training the neural network using the generated training data set; An autonomous charging device (ACD) configured to:

2. 10. The autonomous charging device of claim 1, The target indicator is - a performance indicator of the autonomous charging device; Quantitative and qualitative metrics of the training data, - Computer vision module specificity index, - neural network training time elapsed, - Changes in the target object, and ・Combinations of these Select from: autonomous charging devices.

3. 10. An autonomous charging device according to any of the preceding claims, The operational data includes operating parameters of the autonomous charging device, preferably at least one operating parameter of any of the controllable actuation mechanism, camera, computer vision module, motion module, data communication module, data storage and processing module, processor, and training module.

4. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the operational data includes a signal transmitted by a sensor, the signal corresponding to a measurement by the sensor of at least one operating parameter.

5. 10. An autonomous charging device according to any of the preceding claims, An autonomous charging device, wherein the neural network is configured to provide an output including an estimate of the position and orientation of the vehicle charge port relative to the camera position and / or the location of the charge port pins in the image, preferably relative to the camera position.

6. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the data storage and processing module is configured to annotate the images with geometric features of the charging port socket and / or pins as input for training the neural network.

7. 10. An autonomous charging device according to any of the preceding claims, The functional descriptors include descriptions or representations of device capabilities, image properties, weather conditions, light source, light conditions, light direction, socket location, socket angle, image orientation, vehicle brand, vehicle type, vehicle color, geographic location, customer name, customer project, date and time of image recording, socket status, socket visibility, camera properties, and combinations thereof.

8. 10. An autonomous charging device according to any of the preceding claims, The training data set is generated based on an ACD digital case representation including functional descriptors that represent actual operating conditions of the ACD.

9. 9. The autonomous charging device of claim 8, The ACD digital instance representation includes a multi-dimensional distribution of functional descriptors.

10. 10. The autonomous charging device according to claim 8 or claim 9, The training dataset is generated based on the ACD digital case representation and the target indicators.

11. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the data storage and processing module is configured to associate the functional descriptor with the image.

12. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the data communication module is networked with the data storage and processing module and the training module.

13. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the training module is configured to generate a test data set based on the training data set and test the output of the trained network using the test data set.

14. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the training module is configured to test the output of the trained neural network against a benchmark dataset, the benchmark dataset including a set of pose specifics that have been validated against ground truth.

15. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the training module is configured to test the neural network by testing intersection-over-union and / or mean center distance on the output of the neural network.

16. 10. An autonomous charging device according to any of the preceding claims, The training module comprises: deploying the trained neural network to the computer vision module if an improvement in the output of the trained network is observed when tested on at least one of the test data set and a benchmark data set; repeating the steps of generating new training datasets and / or retraining the neural network using the generated datasets until an improvement in the output of the trained network is observed in tests on at least one of the test dataset and a benchmark dataset; - stopping the training of the neural network; an autonomous charging device configured to:

17. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the common data storage and processing module is configured to assign a score to the target indicator and provide the score to the training module.

18. 10. An autonomous charging device according to any of the preceding claims, The target indicator is a performance indicator of at least one autonomous charging device, and the training dataset is generated to include images associated with one or more functional descriptors in a proportion that is inversely proportional to the expected performance associated with the one or more functional descriptors.

19. 10. An autonomous charging device according to any of the preceding claims, The target object is an autonomous charging device, including a charging port type, the workspace around it, or the vehicle to which it is attached.

20. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the training module is configured to generate the training dataset by appending a set of images to an existing dataset.

21. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the training module is configured to generate the training dataset by adjusting the multidimensional distribution of feature descriptors in an existing dataset.

22. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the training module is configured to determine a bias deviation between the trained neural network and a neural network previously installed in the computer vision module.

23. 10. An autonomous charging device according to any of the preceding claims, The autonomous charging device, wherein the training module is further configured to compare indicators of the autonomous charging device operation based on the trained neural network, and if an improvement in the indicators is observed, deploy the trained neural network to the computer vision module.

24. 10. An autonomous charging device according to any of the preceding claims, The training dataset comprises case-specific images and non-case-specific images of an autonomous charging device.

25. 1. A system for supporting operation of one or more autonomous charging devices, the system comprising: a data communication module for communicating image and operational data to and from the autonomous charging device of claim 1; a common data storage and processing module, - processing images and metadata from at least one autonomous charging device; - processing operational data from at least one autonomous charging device; generating case-specific and non-case-specific images associated with at least one functional descriptor based on the processed images and operational data; a common data storage and processing module configured to: a neural network training module configured to train and deploy a neural network to a computer vision module for at least one autonomous charging device; Equipped with The training module comprises: receiving a score for a target metric associated with the autonomous charging device and triggering training of the neural network if the score is below a threshold; - retrieving annotated images from the common data storage and processing module, each image including a feature descriptor; generating a training dataset comprising said annotated images, such that a combination of case-specific and / or non-case-specific images results in a multidimensional distribution of functional descriptors as a function of said target indicators; training the neural network using the generated training data set; A system configured to:

26. An autonomous charging system, An autonomous charging system comprising an autonomous charging device according to any one of claims 1 to 24 and a system for supporting the operation of one or more autonomous charging devices according to claim 25.

27. 1. A method for training a neural network of a computer vision system of an autonomous charging device, the method comprising: a. Retrieving annotated images from a data storage and processing module, each image including a feature descriptor; b. generating a training dataset including the annotated images such that a combination of selected case-specific and / or non-case-specific images results in a multi-dimensional distribution of feature descriptors; c. training the neural network using the generated training data set; A method comprising:

28. 28. The method of claim 27, The method further comprises receiving a score of a target metric associated with the autonomous charging device, and triggering training of the neural network if the score is below a threshold, wherein the multidimensional distribution is a function of the target metric.

Images