AUTOMATED SYSTEM FOR THE DETECTION OF PUBLIC CHARGING CHARACTERISTICS

Abstract

A charger attribute table describes which charging stations possess which of a multitude of attributes. Vehicle data is collected for each attribute at each charging station, with the vehicle data predefined with respect to sensor types from vehicles with known configurations. A data request specifying the vehicle data to be collected is transmitted to the vehicles. New vehicle data is received from the vehicles in response to the data request. A multimodal model, trained to recognize the multitude of attributes using the sensor types of vehicles with known configurations, is used to determine an updated value for the attribute, at least partially, based on the new vehicle data. The charger attribute table is then updated to include the updated value for the attribute.The charger feature table is used to identify charging stations for a requesting vehicle.

Description

AREA OF TECHNOLOGY

[0001] Aspects of the disclosure generally concern an automated system for detecting public loading features. GENERAL STATE OF THE ART

[0002] The increased availability of electric vehicles has led to a rise in the number of charging stations required for vehicle use. Charging stations can have different attributes, such as charger plug type, maximum charging speed, charge capacity, availability, reliability, and location. SUMMARY

[0003] In one or more illustrative examples, a procedure for using a multimodal model to update a charger attribute table includes the following: for each charging station specified by a charger attribute table that describes which charging stations have which of a multitude of attributes, and for each attribute of the charging station attributes: if a value of the attribute is unknown or outdated: identifying vehicle data to be collected according to the attribute, where the vehicle data is predefined with respect to sensor types of vehicles with known configurations; transmitting a data request specifying the vehicle data to be collected to the vehicles; receiving new vehicle data from the vehicles in response to the data request;Using the multimodal model, which is trained to recognize the multitude of features using the sensor types of vehicles with known configurations, to determine an updated value for the feature, at least partially, based on the new vehicle data; and updating the charger feature table to include the updated value for the feature; and using the charger feature table to identify charging stations for a requesting vehicle.

[0004] In one or more illustrative examples, a system for using a multimodal model to update a charger attribute table comprises the following: a charger monitoring server comprising one or more hardware processors configured for each charging station specified by a charger attribute table describing which charging stations possess which of a multitude of attributes, and for each attribute of the charging station attributes, to: when a value of the attribute is unknown or outdated: identify vehicle data to be collected according to the attribute, where the vehicle data is predefined with respect to sensor types of vehicles with known configurations; transmit a data request specifying the vehicle data to be collected to the vehicles; receive new vehicle data from the vehicles in response to the data request; and use the multimodal model.which is trained to recognize the multitude of features using the sensor types of vehicles with known configurations, in order to determine an updated value for the feature at least partially based on the new vehicle data, and to update the charger feature table to include the updated value for the feature; and to use the charger feature table to identify charging stations for a requesting vehicle.

[0005] In one or more illustrative examples, a non-transient computer-readable medium includes instructions for using a multimodal model to update a charger attribute table that describes which charging stations have which of a multitude of attributes. When executed by one or more hardware processors of a charger monitoring server, these instructions cause the charger monitoring server to perform operations that include: for each charging station specified by a charger attribute table, and for each attribute of the charging station attributes: if a value of the attribute is unknown or outdated: identifying vehicle data to be collected according to the attribute, the vehicle data being predefined with respect to sensor types of vehicles with known configurations; and transmitting a data request specifying the vehicle data to be collected to the vehicles.Receiving new vehicle data from the vehicles in response to the data request; using the multimodal model, trained to recognize the multitude of features using the sensor types of the vehicles with known configurations, to determine an updated value for the feature, at least partially, based on the new vehicle data; and updating the charger feature table to include the updated value for the feature; and using the charger feature table to identify charging stations for a requesting vehicle. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates an exemplary system for machine-learned dynamic charging station evaluation for vehicles; Fig. Figure 2 illustrates an exemplary data flow of the operation of the multimodal model for updating the charger feature table; Fig.Figure 3 illustrates an exemplary process for the operation of the system when updating the charger feature table; Fig. 4A illustrates an example of charging stations with an existing roof; Fig. Figure 4B illustrates an example of charging stations without an existing roof; Fig. Figure 5 illustrates an exemplary semantic segmentation of vehicle data for an area surrounding a set of charging stations; Fig. Figure 6A illustrates an example of charging stations with vertical parking spaces; Fig. Figure 6B illustrates an example of charging stations with parallel parking spaces; and Fig. Figure 7 illustrates an exemplary computing device for implementing a machine-learned dynamic charging station evaluation for vehicles. DETAILED DESCRIPTION

[0006] Depending on the requirements, detailed embodiments of the present invention are disclosed here; however, it is understood that the disclosed embodiments are merely exemplary of the invention, which can be implemented in various and alternative forms. The figures are not necessarily to scale; some features may be greatly enlarged or reduced to show details of specific components. Therefore, specific structural and functional details disclosed in this document are not to be interpreted as limiting, but merely as a representative basis to teach those skilled in the art the diverse applications of the present invention.

[0007] With the increasing popularity of electric vehicles (EVs), the user experience at public charging stations is becoming ever more important. EV drivers may prefer a straightforward and efficient charging process, expecting a successful connection on their first attempt. Given potential charging wait times, access to accurate information about the best charging locations is essential.

[0008] Customers can use various factors to define a high-quality charging experience. These factors can include environmental aspects such as the presence of lighting at night, cleanliness of the charging area, a rain shelter over the chargers, cleanliness of the surrounding area, and sufficient parking space size. They can also include amenities such as the availability of restrooms, cafes / restaurants, the number of chargers, and the presence of grocery stores and / or retail outlets. Static websites that evaluate charger features tend to become outdated. Furthermore, such websites may lack complete information.

[0009] Aspects of this disclosure concern the identification and cataloging of EV charging stations. The approach integrates vehicle sensor data into cloud computing by employing a suite of sensors to collect data from both electric and non-electric vehicles, which is particularly effective when a vehicle is parked at or near a charging point. This approach utilizes vehicle sensors to gather data and transmits the collected information, which may include signals, images, or videos, to a cloud system. For example, when an EV approaches a charging station and / or charges at a public charging station, or even when non-EVs park or drive past EV chargers in a store parking lot within a usable line of sight, the EV charging stations can be classified, even though the vehicles taking the measurements do not plug themselves in.

[0010] Additionally, algorithms based on multimodal generative artificial intelligence (AI) can be used to combine the collected vehicle data with additional sources, such as customer feedback, visual content, and charging performance assessments, to create a comprehensive and continuously updated feature list for each charging location. This approach improves the EV charging experience by providing manufacturers and users with valuable insights and contributing to the efficiency of the smart transportation ecosystem.

[0011] In some examples, the approach can identify infrastructure health issues, such as cracks and potholes in the road surface, damage to signage or obstacles, structural problems with roofs or canopies, lighting deficiencies, accessibility barriers, environmental hazards, etc. These problems can be proactively addressed through automated maintenance requests to improve the overall operation and functionality of the charging station.

[0012] Fig.Figure 1 illustrates an exemplary system 100 for machine-learned dynamic charging station evaluation for vehicles 102. The vehicle 102 includes components such as a telematics control unit (TCU) 104 configured to communicate via a communication network 106, sensors 108, a controller 110 of a global navigation satellite system (GNSS), and a controller 112 of a human-machine interface (HMI). If the vehicle 102 is an EV, it may also include a charging controller 114 for communication with various charging stations 116. The vehicles 102, the charging station 116, mobile devices 120, and a charger monitoring server 122 may be configured to communicate via the communication network 106.The charger monitoring server 122 can run a charger service 124 and can be configured to receive vehicle data 118 from the vehicles 102. The charger monitoring server 122 can also be configured to receive third-party data 126 from other sources, such as a third-party data server 128. Using the vehicle data 118 and the third-party data 126, a multimodal model 130 can be configured to update a charger feature table 132 that describes which charging stations 116 have which features. A charger application 134 can be installed in the vehicles 102. Using the charger application 134, the vehicle 102 can send a charger request 136 to the charger monitoring server 122. The charger service 124 can process the request to offer charger recommendations 138 based on the information in the charger feature table 132.The charger characteristic table 132 can accordingly be used to provide users of the vehicles 102 with charger recommendations 138. In some examples, the multimodal model 130 may have limited information available to update the charger characteristic table 132. In such cases, the charger service 124 can send a data request 140 for additional vehicle data 118 and / or third-party data 126 to update the charger characteristic table 132. It should be noted that the system 100 is only an exemplary implementation and more, fewer, and / or other elements of the system 100 may be used.

[0013] Vehicle 102 can include various types of automobiles, soft-roaders (crossover utility vehicles - CUVs), off-road vehicles (sport utility vehicles - SUVs), trucks, recreational vehicles (RVs), boats, aircraft, or other mobile machinery for transporting people or goods. In many cases, Vehicle 102 can be a battery electric vehicle (BEV), powered by a traction battery and one or more electric motors. Alternatively, Vehicle 102 can be a hybrid electric vehicle, powered by an internal combustion engine, a traction battery, and one or more electric motors. Hybrid Vehicles 102 can take various forms, such as a series hybrid electric vehicle, a parallel hybrid electric vehicle, or a parallel / series hybrid electric vehicle.Since the type and configuration of Vehicle 102 can vary, its capabilities can also vary. For example, Vehicle 102s may have different capabilities in terms of passenger capacity, towing capacity, and storage volume. For registration, inventory, and other purposes, Vehicle 102s may be assigned unique identifiers, such as Vehicle Identification Numbers (VINs), Globally Unique Identifiers (GUIDs), customer or fleet accounts, etc.

[0014] The vehicle 102 can include a variety of components configured to perform and manage various functions of the vehicle 102 using the power of the vehicle battery and / or the power transmission. As shown, the exemplary vehicle components are represented as discrete controllers (e.g., the TCU 104, the sensors 108, the GNSS controller 110, the HMI controller 112, the charging controller 114, etc.). However, the components of the vehicle 102 can share physical hardware, firmware, and / or software, so the functionality of several controllers can be integrated into a single controller, and the functionality of several such controllers can be distributed across a variety of controllers.

[0015] The TCU 104 can be used by the vehicle 102 for communication via the communication network 106. The TCU 104 can include network hardware configured to facilitate communication between the vehicle 102 and other devices of the system 100. For example, the TCU 104 can include a cellular modem configured to facilitate communication with, or otherwise access, the communication network 106. The communication network 106 can include one or more interconnected communication networks, such as, but not limited to, the internet, a cable television distribution network, a satellite connection network, a local area network, and a telephone network.

[0016] The sensors 108 can include various hardware components of the vehicle 102, which are used to gather information about its environment and status. In some non-restrictive examples, the vehicle sensors 108 can include one or more cameras (e.g., cameras of an advanced driver assistance system (ADAS)), ultrasonic sensors, radio detection and ranging (RADAR) systems, and / or light detection and ranging (LIDAR) systems.

[0017] The GNSS controller 110 can be configured to provide information indicating the current location of vehicle 102. For example, the GNSS controller 110 can be responsible for receiving signals from a GNSS constellation of satellites. This allows the GNSS controller 110 to receive time information and determine the precise location of vehicle 102. The location determined by the GNSS controller 110 can be used for various tasks, such as navigation or other location-based services.

[0018] The HMI controller 112 can be configured to provide an interface through which vehicle occupants can interact with the vehicle 102. This interface can include a touchscreen display, voice commands, and physical controls such as buttons and knobs. The HMI controller 112 can be configured to receive user input via various buttons or other controls and to provide a driver with status information such as fuel level, engine operating temperature, and the current location of the vehicle 102. The HMI controller 112 can also be configured to display information on various screens within the vehicle 102, such as a center console touchscreen, an instrument cluster screen, etc.Accordingly, the HMI control 112 can enable the occupants of the vehicle 102 to access and control various systems, such as navigation, entertainment and climate control.

[0019] The charging controller 114 (included in those vehicles 102 that accept charging) can be configured to manage battery charging, including monitoring the charging status, managing the flow of electricity, and communicating with the power grid. The charging controller 114 can communicate with a charging station 116 via a cable or other device, enabling the vehicle 102 to be charged from an external power source. The charging stations 116 can be configured to conduct and manage the transfer of energy between a power source and the vehicle 102. An external power source can supply electrical power to the charging stations 116 using direct current (DC) or alternating current (AC).The charging stations 116, in turn, can have a charging plug for insertion into a respective charging port of the vehicle 102. The charging port can be any type of port configured to transfer power from the charging stations 116 to the vehicle 102. Alternatively, the charging stations 116 can be configured to transfer power using other methods, such as wireless inductive coupling. The charging stations 116 are connected to the charging controller 114; the charging stations 116 can include circuitry and controls for directing and managing the transfer of energy between the power source and the vehicle 102.

[0020] The vehicle data 118 can include various pieces of information collected by the vehicle 102 using the sensors 108 and / or the charging controller 114. Regardless of whether the vehicle 102 is an EV, the vehicle data 118 can include images, sound, point clouds, etc., captured by the sensors 108 of the vehicle 102 when the vehicle 102 is near the charging stations 116.

[0021] For electric vehicles 102, the vehicle data 118 can also include information relating to charging sessions carried out by the vehicle 102. This can include, for example, status information transmitted between the vehicles 102 and the charging stations 116 and forwarded to the charger monitoring server 122 in various ways. In one example, the vehicles 102 can send the vehicle data 118 in response to the occurrence of an event, such as a completed charging event, an event where the charger is plugged in, etc. In another example, the vehicles 102 can send the vehicle data 118 to the charger services 124 at regular intervals, at night, or at another time when network connectivity is available, such as when the vehicle 102 is at a home location and connected to a home network instead of a cellular network.

[0022] The Mobile Device 120 can be any of the various types of portable computing devices, such as mobile phones, tablet computers, smartwatches, laptop computers, portable music players, or other devices that possess processing and communication capabilities. The Mobile Device 120 can include one or more processors configured to execute computer instructions and a storage medium on which the computer-executable instructions and / or data can be stored.

[0023] The charger monitoring server 122 can be an example of a networked computing device that vehicles 102, mobile devices 120, charging stations 116, and / or other devices can access via the communication network 106. The charger monitoring server 122 can be configured to run the charger service 124 to perform the operations of the charger monitoring server 122 discussed in this document. The vehicle 102 can send data from its sensors 108 to the charger monitoring server 122 via the communication network 106. In the case of an EV, the vehicle 102 can monitor its use of the charging station 116 and can send its vehicle data 118 to the charger monitoring server 122 via the communication network 106. In another EV example, the charger monitoring server 122 can be configured to receive vehicle data 118 from the charging stations 116 via the communication network 106 (e.g.to be received as part of a billing process for the use of charging station 116 or as a separate process).

[0024] In yet another example, the charger monitoring server 122 can generate a distribution of the query status and / or overload status for the charging stations 116 based on the communication traffic in the network of vehicles 102, which exchange information at different times of day, in different months, seasons, etc. This information exchange can take into account factors such as the type of vehicle 102, e.g., ICE, BEV, PHEV, etc., since different types may exhibit different patterns of information exchange due to aspects such as a charging schedule. This exchange of information can be used to determine the best times to receive the vehicle data 118.

[0025] In addition to the vehicle data 118 from the vehicles 102 and / or charging stations 116, the third-party data 126 may refer to data from any of several different sources. This may include, but is not limited to, user reviews of the charging stations 116, social media posts relating to the charging stations 116, internet searches of the charging stations 116 and / or their surroundings, image searches of the charging stations 116 and / or their surroundings, etc. In some examples, the third-party data 126 may be received by the charger monitoring server 122 from one or more third-party data servers 128.

[0026] The charger monitoring server 122 can be configured to use a multimodal model 130 to process the vehicle data 118. The multimodal model 130 can be a machine learning (ML) model trained on information from multiple modalities, such as images, videos, text, and audio. For example, the multimodal model 130 can process the different modalities of data together. Accordingly, the multimodal model 130 can be able to process and find patterns across the different modalities of data. The charger service 124 can also be configured to generate and / or update a charger attribute table 132 based on the output of the multimodal model 130. The charger attribute table 132 can describe the attributes derived for the various charging stations 116.Further aspects of the operation of the multimodal model 130 are discussed with reference to . Fig. 2 discussed.

[0027] A charger application 134 can be installed in the vehicle 102. Using the charger application 134, the vehicle 102, if the user has consented, can send the vehicle data 118 to the charger service 124. Additionally, the user can send a charger request 136 to the charger service 124 to request that one or more charging stations 116 be recommended for charging the vehicle 102. The charger request 136 can include information such as the location of the vehicle 102 and an identifier of the user or the vehicle 102. In response to receiving the charger request 136 from a vehicle 102, the charger service 124 can access the charger attribute table 132 to provide charger recommendations 138 for the vehicle 102 requesting to be charged.

[0028] It should be noted that in some cases, the charger service 124 may lack sufficient data about one or more of the charging stations 116. In such a case, the charger service 124 may request the vehicles 102 and / or the third-party data server 128 to collect additional data to be used by the multimodal model 130 to update the charger attribute table 132. For example, if the multimodal model 130 has sufficient confidence in one or more aspects of the charger attribute table 132, the charger service 124 may send a data request 140 for the vehicles 102 (whether EV or not) to collect image or other data of the charging stations 116 to enable a more accurate update of the charger attribute table 132. In another example, the charger service 124 may request an input, such as...Photos of the mobile devices 120 of the users, as another form of data request 140, to collect image or other data of the charging stations 116 and / or their surroundings.

[0029] Fig. Figure 2 illustrates an example data flow 200 of the operation of the multimodal model 130 for updating the charger characteristic table 132. As shown, the data sources 202 can contain data of various data types 204 that are input into the multimodal model 130. The multimodal model 130 uses the inputs to derive information for addition to the charger characteristic table 132.

[0030] As noted in this document, the data sources 202 can include the charging stations 116, the vehicles 102, and various third-party data servers 128. The information from the data sources 202 can include the vehicle data 118 and / or the third-party data 126, as noted above. In one example, the vehicle data 118 can include data from the sensors 108 of the vehicle 102. In another example, the vehicle data 118 can include data about charging sessions that can be received by the charging stations 116 and / or by the vehicles 102. As some other examples, the third-party data 126 can include ratings 206 received from users of various rating services for charging stations 116, social media posts 208 from various social media pages 208, and / or internet data 210 from various websites.

[0031] It should be noted that the contents of the vehicle data 118 and the third-party data 126 can include various data types 204. As shown, these data types 204 can include, for example, image data 212, text data 214, structured data 216, geographic location data 218, such as GNSS locations, and / or audio data 220.

[0032] The image data 212 can include images, videos, point clouds, etc., captured by the vehicle's sensors 108. In another example, the image data 212 can include photos specifically requested by users' mobile devices. In yet another example, the image data 212 can include photos captured by users who provide ratings 206, images from social media 208, and / or image searches from internet queries, such as the internet, and / or amenities around the charging station. In yet another example, the image data 212 can include images taken by the charging station's equipment 116 and / or its surroundings. In many examples, the image data 212 includes location metadata that can be used to filter the images to those corresponding to the locations of the analyzed charging station 116.

[0033] The text data 214 can include written information describing the charging stations 116. In one example, the text data 214 could be created by the owners or operators of the charging stations 116. In another example, the charging stations 116 could include text from users who submit reviews 206, post them on social media 208, and / or text from internet queries related to the charging stations 116 that are being analyzed.

[0034] The structured data 216 can include database data, such as existing databases with information about the charging station 116. For example, the structured data 216 can be made available for analysis by downstream data consumers via various third-party data servers 128.

[0035] The geographic location data 218 may include GNSS data acquired by the GNSS controller 110 of the vehicles 102. The geographic location data 218 may also include GNSS data included as metadata in the image data 212, and / or other types of data that have embedded or otherwise associated location metadata or location data.

[0036] The audio data 220 can include audio recorded by microphones or other sensors 108 of the vehicle 102. In another example, the audio data 220 can include audio captured by users who provide audio ratings 206, post audio on social media 208, audio clips from internet data searches 210, etc.

[0037] The different data types 204 can be tokenized together as training material for training the multimodal model 130. Additionally, the different data types 204 can be tokenized together as material for determining inferences about the charging stations 116. By using these different types of data together, a more complete view of the charging stations 116 can be modeled.

[0038] The multimodal model 130 can utilize various techniques for processing the data sources 202 of the different data types 204. In one example, the multimodal model 130 can employ various techniques for machine vision 222, aspect-based semantic analysis 224, and / or retrieval augmented generation (RAG) techniques 226.

[0039] Machine vision 222 can involve various techniques for performing an analysis of the image data 212 using the multimodal model 130. These techniques may include, for example, semantic segmentation to identify and classify objects in images or videos and to determine object boundaries. In another example, the techniques may involve deep learning to identify image features in order to classify the image data 212 (e.g., illuminance and / or weather conditions). As yet another example, the techniques may involve feature extraction to determine image shapes, colors, textures, etc., in order to convert the images into features for further processing. In a still other example, captioning techniques may be used to apply text or other markers to the identified features or segmentation to facilitate further analysis of the data.

[0040] Semantic analysis 224 can involve various techniques to determine a sentiment from vehicle data 118 and / or third-party data 126. For example, a simple semantic analysis 224 can be performed to treat a text or other data element as a whole and assign it a single sentiment marker (e.g., positive, negative, or neutral). In another example, aspect-based sentiment analysis (ABSA) can be performed to determine the sentiment related to a specific aspect (e.g., a feature of the charging station 116) for use when updating the charger feature table 132.

[0041] RAG 226 can be used to improve the results of the multimodal model 130 by drawing on facts from various known good data sources 202 and enhancing contextual knowledge. RAG 226 can be used to expand the dataset and / or features. For example, the system 100 can directly improve the charging success rate and reliability of vehicles 102 and charging stations 116, and RAG 226 can support the extraction of these information elements into the charger feature table 132. For example, RAG 226 can be useful for expanding human-verified information in a charger feature table 132 and / or other high-reliability data, ensuring that additional inferences are aligned with this specific good data.

[0042] The result of operating the multimodal model 130 is the data in the charger characteristic table 132. The charger characteristic table 132 can include information relating to the charging stations 116, such as reliability, feel, lighting, cleanliness, presence / absence of rain shelters, parking styles (e.g., parallel, perpendicular, etc.), presence / absence of toilets, charging costs, points of interest (POI) near the charging stations 116 (and in some cases ratings for these POIs in addition to information on their presence), etc.

[0043] An example section of a charger characteristic table 132 is shown in Table 1: Table 1 - Example charger feature table Charging location Lighting during the night Cleanliness of the loading area Rain canopy Park size / style ... Snow removal A YES YES NO UNKNOWN YES B NO NO UNKNOWN YES NO C YES NO NO YES UNKNOWN D NO NO YES UNKNOWN UNKNOWN

[0044] As shown in Table 1, each row of the charger feature table 132 can correspond to a specific charging station 116 and / or specific charging stations 116. For example, each row can contain a charging location identifier that identifies the specific charging station 116 and / or specific charging stations 116. While shown as A, B, C, D, this identifier can be a latitude, a longitude, a city, a state, an address, a charging network, etc.

[0045] Additionally, each line can contain information describing the various charging location characteristics. These characteristics can include various environmental features that can be detected by the multimodal model 130. Some non-restrictive examples of these characteristics include nighttime lighting, cleanliness of the charging area, a rain shelter over the charger, the cleanliness of the surroundings, the size of the parking space, designated outdoor seating, etc.

[0046] Fig. Figure 3 illustrates an exemplary process 300 for the operation of system 100 when updating the charger characteristic table 132. In one example, process 300 in connection with system 100 can be carried out by the charger monitoring server 122, which executes the charger service 124.

[0047] In process 302, the charger monitoring server 122 identifies a charging station 116 to be analyzed. In one example, the charger monitoring server 122 can iterate through the rows of the charger attribute table 132. In another example, the charger monitoring server 122 can receive requests to add a new charging station 116 and can then perform process 300 on these newly added charging stations 116.

[0048] In process 304, the charger monitoring server 122 determines whether more features of the charging station 116 need to be analyzed. For example, the charger monitoring server 122 can iterate through each column of the current row of the charger feature table 132 to determine which features need to be updated. If more features need to be analyzed, the controller proceeds to process 306. Once all features have been analyzed and / or are up to date, the controller returns to process 302 to analyze the next charging station 116.

[0049] In process 306, the charger monitoring server 122 determines whether the charging station 116 characteristic in the charger characteristic table 132 is known or unknown. For example, the charger monitoring server 122 can determine whether the current row and column of the charger characteristic table 132 indicate that the value is known. If so, the controller proceeds to process 308 to determine whether this known characteristic requires an update. If not, the characteristic is identified as unknown, and the controller proceeds to process 310.

[0050] In process 308, the charger monitoring server 122 determines whether the known attribute of charging station 116 requires an update. For example, each attribute can have a validity period value (e.g., one hour, one day, etc.) after which the attribute is considered outdated and must be updated. In some implementations, this value is the same for every attribute; however, in other implementations, this value varies depending on the attribute, as some attributes (e.g., covered in snow) change more frequently than others (e.g., charging stations 116 with perpendicular versus parallel parking). If the attribute has expired due to its validity period, the controller proceeds to process 310. If the attribute is still considered valid, the controller returns to process 304 to analyze the next attribute.

[0051] In some examples, if new vehicle data 118 and / or third-party data 126 have been received relating to a feature, this feature may be considered obsolete due to the new data. In such a situation, the control may proceed to operation 310, even if the feature would otherwise still be considered known.

[0052] In process 310, the charger monitoring server 122 identifies vehicle data 118 to be collected. In one example, the charger monitoring server 122 can determine which vehicle data 118 to collect according to the unknown feature. In another example, the charger monitoring server 122 can determine a sampling rate for the vehicle data 118 to be collected. For example, some data can be collected once to meet the requirement, while other data may require multiple samples over time to ensure a reliable value. For example, certain features may be best identified using image data 212 (such as the presence or absence of a roof), while other features may be best identified using video data over time (such as which locations tend to be exposed to direct sunlight).Other features, such as ambient noise levels, can best be identified using audio data 220, etc.

[0053] In process 312, the charger monitoring server 122 transmits a data request 140 for vehicle data 118. This request can be sent in various ways. In one example, the data request 140 can be sent to all vehicles 102 (e.g., those that have agreed to respond to data requests 140). In another example, the data request 140 can be sent to vehicles 102 in a fleet. In yet another example, the data request 140 can be transmitted locally to the charging station 116 to allow vehicles 102 near the charging station 116 to collect the information and respond.

[0054] In process 314, the charger monitoring server 122 receives the vehicle data 118. In one example, the vehicles 102, which have collected the requested data, can send the data in a response to the data request 140. In some examples, the data request 140 can include a response address for sending the data, such as the address of the charger monitoring server 122. In other examples, the data request 140 can be sent by the vehicles 102 to an address pre-stored by the vehicles 102 for responding to data requests 140. In some examples, the data can be sent by the vehicle 102 as it is collected and / or as soon as it has been collected according to the sampling rate of the vehicle data 118.In other examples, the data can be offloaded from the vehicle 102 at a later time, such as when the vehicle 102 is connected to WiFi, is connected to the internet while charging and / or is connected via a home connection, in order to avoid additional overloading of the communication network 106, which is used for moving vehicles 102.

[0055] In process 316, the charger monitoring server 122 uses the multimodal model 130 to analyze the feature of the charging station 116. In one example, the vehicle data 118 can be analyzed as above with respect to the data flow 200 from Fig. 2. The analysis will be discussed. The result of the analysis can be a derived value for the characteristic being analyzed.

[0056] In operation 318, the charger monitoring server 122 determines whether the analysis result has a confidence level of at least one predefined threshold. For example, the multimodal model 130 can determine a confidence level of the achieved value in combination with the value. For instance, the multimodal model 130 can provide a vector output of each possible value for the feature and a probability for each possible value. It can be deduced that the most probable value above the predefined threshold is the value of the feature. If such a value is determined, the controller proceeds to operation 320 to apply this value to the row and column of the charger feature table 132, which is updated. After operation 302, the controller proceeds to operation 304 to process additional features and / or charging stations 116.

[0057] However, if no value is determined that exceeds the threshold, the control progresses from operation 318 to operation 322. In operation 322, the charger monitoring server 122 identifies a low-confidence root cause. For example, the received image data 212 may have been blocked (e.g., by an obstructing truck), so the information that should have been collected was not actually collected. In some examples, the low-confidence results may be flagged for human review to identify the root cause. Regardless, after operation 322, the control progresses to operation 310 to identify additional data to be collected.

[0058] Process 300 can proceed to update all characteristics of all charging stations 116. It should be noted that, while Process 300 is shown iterating simultaneously at a single charging station 116, multiple charging stations 116 can be processed concurrently and / or one or more operations of Process 300 can be performed concurrently and / or in a different order than shown. Additionally, the operations performed in Process 300 indicate the traffic that was present at multiple charging stations 116. This data, combined with the vehicle data 118, enables the charger monitoring server 122 to generate a distribution of the query and overload status for the charging stations 116 in the charger characteristic table 132.

[0059] For example, batch processing can be used as a cost-effective approach to provide a higher degree of confidence in the results. Real-time data collection involves analyzing live data as it comes in, which can be resource-intensive, requiring significant network bandwidth and computing power to manage the constant stream of information. In contrast, batch processing can utilize data collected over a specific period and process it in groups, enabling efficient feature detection by analyzing multiple datasets. However, batch processing may require the charger monitoring server to use additional memory to manage the data while it is being batched for processing.

[0060] Fig.Figures 4A-4B together illustrate a specific example of the presence and absence of a roof over charging stations 116. Fig. 4A illustrates an example of 400A for charging stations 116 with an existing roof, whereas Fig. 4B illustrates an example of 400B for charging stations 116 without an existing roof. In Fig. 4B is noted by the rain cloud, which, compared to Fig. 4A may have less protection from the elements at charging stations 116, which are located in Fig. 4B are shown.

[0061] To update the feature corresponding to the rain canopy, the charger monitoring server 122 can request vehicle data 118, including ADAS front-end and surrounding camera sensors 108, as well as data from the vehicle's rain sensors 108. This information can enable the model to both see and track moisture near the charging stations 116 to determine whether or not a canopy is present.

[0062] Fig.Figure 5 illustrates an exemplary semantic segmentation 500 of vehicle data 118 of an area surrounding a set of charging stations 116. As shown, the charging stations 116 and a vehicle 102 near the charging stations 116 are identified. Additionally, a roof is detected with a high probability over the charging stations 116. In this case, however, the multimodal model 130 may incorrectly infer, due to the perspective of the image, that these charging stations 116 have a covering roof, although they do not in fact and are instead located in front of other buildings that do have roofs.

[0063] Other variations in false detection are possible. For example, on a cloudy day, a gray sky might be incorrectly marked as a roof by the multimodal model 130 when no roof is present. Or, if an image is taken from inside the cabin of vehicle 102, the rearview mirror in the resulting image might also be incorrectly interpreted as a roof by the multimodal model 130.

[0064] To address these types of problems, the disclosed approach ensures that the data collected by the vehicles 102 conform to specific known properties regarding height, field of view, image depth, etc., to guarantee more consistent data acquisition. This is because the data is collected by the vehicles 102 and the configurations of the sensors 108 are predefined and known according to the implementation. For example, the data request 140 from the charger monitoring server 122 may require the use of calibrated camera images from the camera sensor 108 on the front grille and / or the camera sensor 108 on the windshield of the vehicle 102. This approach can lead to misdetection problems with crowdsourced data taken from unusual angles and / or from inside the passenger compartment of the vehicle 102, without ensuring that mirrors and other obstructions can be prevented.

[0065] Furthermore, the multimodal model 130 of the charger monitoring server 122 can utilize rain sensors 108 or light sensors 108 to ensure favorable weather conditions. The light sensors 108 can be located on the instrument panel, the rearview mirror, or the exterior surface of the vehicle 102, enabling them to accurately measure the ambient light intensity around the vehicle 102. These sensors 108 can employ various technologies to detect light and convert it into an electrical signal, which is then processed by the vehicle 102's light control system. Similarly, the rain sensors 108 can be positioned near the windshield, for example, on or adjacent to the rearview mirror mount, to avoid obstructing the windshield.These sensors 108 can be configured to detect, for example, a change in capacitance, which can be used to infer the presence of rain.

[0066] Since the multimodal model 130 can directly utilize several sensor data types 204, the system 100 can directly use rain sensors 108 and / or light sensors 108 to confirm the presence of a canopy. For example, if the rain sensors 108 are activated, indicating that it is raining, and the windshield wipers have started operating, it can be inferred that the parking space does not have a canopy, even though, for example, the image data 212 for the location of the vehicle 102 is not unambiguous. Conversely, if the light sensors 108 indicate a bright condition, this can be used as an additional data point to indicate that there is no canopy at the location of the vehicle 102.

[0067] Fig.6A-6B together illustrate a specific example of perpendicular parking compared to parallel parking of the charging stations 116. Fig. 6A illustrates an example of 600A for charging stations 116 with perpendicular parking (although other examples may use angled parking, such as 30-degree or 45-degree parking), while Fig. 6B illustrates an example 600B for charging stations 116 with parallel parking spaces.

[0068] Drivers of electric trucks may find it particularly relevant whether a parking space allows for trailer parking. Similarly, drivers of electric vans may need information about whether the parking space is large enough. By utilizing the vehicle's camera sensors (108) and calibrated photos, the multimodal model (130) can identify these factors.

[0069] Furthermore, additional signals from sensors 108 in the vehicle data 118, such as the trailer parking assist signal, can be used to supplement the multimodal model 130 with the inputs. For example, if an EV truck 102 has the trailer parking assist feature activated when it enters a charging station 116, this implies that parking at this charging station 116 supports a vehicle 102 in a articulated configuration.

[0070] Similar to what was discussed above regarding rain canopies, the derivation of parking size, style, and height can depend on the image quality used. If the multimodal model 130 relies solely on customer photos, determining the orientation of the photo can be difficult. This can lead to significant uncertainty when applying image classification algorithms. However, by using the sensors 108 of the vehicles 102, a consistent known camera orientation and direction of movement can be input into the multimodal model 130 when deriving parking size, style, and height. This can increase the accuracy of the results compared to previous systems. For example, the charger monitoring server 122 can request data from the known camera sensor 108 of the vehicles 102 when performing the parking orientation analysis.

[0071] As some further examples, Table 2 illustrates an assignment of features of the charging stations 116 to potential sensors 108, from which vehicle data 118 can be requested in data requests 140 to update the respective feature: Table 2 - Assignment of features to vehicle sensors feature Sensors Snow removal Ambient temperature sensor + rain sensor Night light light sensor Cleanliness of the surroundings camera sensor Designated seating in the outdoor area camera sensor Visibility for the main highway, for other users camera sensor Condition of the road surface camera sensor

[0072] In some examples, the health status of the infrastructure with respect to the charging stations 116 can be updated using the system 100. This can be done in addition to updating the charger characteristic table 132, which describes which charging stations 116 have which characteristics. Table 3 illustrates some examples of characteristics and a sensor 108 that can be used to identify such infrastructure problems.

[0073] In one example, the vehicle's camera sensors 108 can be used to identify cracks, potholes, or uneven surfaces in the road surface around the charging station 116. This can enable the identification and mitigation of hazards for EV drivers and pedestrians, particularly when entering and exiting the charging station 116.

[0074] In another example, the camera sensors 108 of vehicle 102 can be used to detect damage to signage, including missing letters, faded colors, or physical damage due to vandalism or weather. Clear and visible signage is useful for guiding EV drivers to charging station 116 and providing essential information about usage, rates, and other guidelines.

[0075] In yet another example, the vehicle's camera sensors 108 can identify structural problems, such as cracks, leaks, or sagging in roofs or protective canopies covering the loading area. Problems with these structures can pose issues for users and can also impair the functionality of loading equipment when exposed to the elements.

[0076] As another example, the vehicle's light level sensors 108 and camera sensors 108 can detect a malfunction or insufficient lighting around the charging station 116, including streetlights, pathway lights, or signage lighting. Proper lighting is useful for ensuring user safety, especially during nighttime charging sessions.

[0077] In yet another example, the vehicle's camera, radar, and / or ultrasonic sensors 108 can identify obstacles or barriers that may impede access to the charging station 116, such as uneven sidewalks, high curbs, or blocked paths. Ensuring accessibility for all users, including those with disabilities, is essential for promoting inclusivity and compliance with accessibility standards.

[0078] Furthermore, the vehicle's camera, radar, and / or ultrasonic sensors 108 can be used to detect overgrown vegetation or foliage that may obstruct visibility, signage, or access routes around the charging station 116. Regular vegetation maintenance ensures clear lines of sight and unobstructed access to the charging infrastructure.

[0079] In another example, the vehicle's camera sensors 108 can be used to detect environmental problems, such as puddles of water, patches of ice, or accumulations of dirt around the charging station 116. These hazards can create slip and fall risks for users and can also damage equipment if left unaddressed.

[0080] Thus, the sensors 108 of the vehicle 102 can be used to detect environmental features of the charging stations 116. Additionally, the disclosed approach could potentially offer EV drivers another digital service by providing incentives for them to explore new and less popular public charging locations.

[0081] System 100 can be scaled to cover many or even all charging stations 116, including potentially restricted charging locations such as corporate charging sites, commercial charging sites, and hotel charging sites. The revealed approach also leverages cloud computing and generative AI technologies to improve the capability and accuracy of feature detection.

[0082] Furthermore, the system 100 does not require vehicles 102 to contain all sensors 108, as the system 100 includes an interface configured to communicate with a variety of vehicles 102, some of which may possess the necessary data. Moreover, the disclosed approach offers the flexibility to utilize any sensors 108 of the vehicle 102, regardless of whether they are built-in sensors 108 or plug-and-play sensors 108, such as dashcams.

[0083] Fig.Figure 7 illustrates an exemplary computing device 702 for implementing a machine-learned dynamic assessment of a charging station 116 for vehicles 102. With reference to Fig. 7 and with reference to Fig.In Figures 1-6B, the vehicles 102, the TCU 104, the communication network 106, the sensors 108, the GNSS controller 110, the HMI controller 112, the charging controller 114, the charging stations 116, the mobile devices 120, the charger monitoring server 122, and the third-party data server 128 are examples of such computing devices 702. The computing devices 702 generally contain computer-executable instructions, such as those of the charger service 124 and the charger application 134, the instructions being executable by one or more computing devices 702. Computer-executable instructions can be compiled or interpreted by computer programs created using a variety of programming languages ​​and / or technologies, including, but not limited to, Java™, C, C++, C#, Visual Basic, JavaScript, Python, Perl, etc., either individually or in combination. Generally, a processor (e.g.,a microprocessor) receives instructions, e.g., from memory, a computer-readable medium, etc., and executes these instructions, thereby carrying out one or more processes that include one or more of the processes described in this document. Such instructions and other data, such as the vehicle data 118, the third-party data 126, the multimodal model 130, and the charger characteristic table 132, can be stored and transmitted using a variety of computer-readable media.

[0084] As shown, the computing device 702 can include a processor 704, which is operatively connected to a memory 706, a network device 708, an output device 810, and an input device 712. It should be noted that this is only an example and computing devices 702 can be used with more, fewer, or different components.

[0085] The 704 processor can include one or more integrated circuits that implement the functionality of a central processing unit (CPU) and / or graphics processing unit (GPU). In some examples, the 704 processors are a system-on-a-chip (SoC) that integrates the functionality of both the CPU and the GPU. The SoC may optionally include other components, such as the 706 data storage device and the 708 networking device, in a single integrated device. In other examples, the CPU and GPU are interconnected via a peripheral interconnect device, such as Peripheral Component Interconnect Express (PCI Express) or another suitable peripheral data connection.In one example, the CPU is a commercially available central processing device that executes a set of instructions, such as one from the x86, ARM or Power instruction set family, or a Microprocessor without Interlocked Pipeline Stages (MIPS) instruction set family.

[0086] Regardless of its specific features, the 704 processor executes stored program instructions during operation, which are retrieved from the 706 memory. The stored program instructions accordingly contain software that controls the operation of the 704 processors to perform the operations described in this document. The 706 memory can contain both non-volatile and volatile memory devices. The non-volatile memory includes solid-state memory, such as emergency NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the system is powered off or its power supply is interrupted. The volatile memory includes static and dynamic random-access memory (RAM) that stores 100 program instructions and data during system operation.

[0087] The GPU can include hardware and software for displaying at least two-dimensional (2D) and optionally three-dimensional (3D) graphics on an output device 710. The output device 710 can include a graphical or visual display device, such as an electronic display screen, a projector, a printer, or any other suitable device that reproduces a graphical display. As another example, the output device 710 can include an audio device, such as a loudspeaker or headphones. As yet another example, the output device 710 can include a tactile device, such as a mechanically liftable device, which in one example can be configured to display Braille or other physical output that can be touched to provide information to a user.

[0088] The input device 712 can include any of the various devices that enable the computing device 702 to receive control inputs from users. Examples of suitable input devices 712 that receive inputs via a human interface can include keyboards, mice, trackballs, touchscreens, microphones, graphics tablets, and the like.

[0089] The Network Devices 708 can each include any of the various devices that enable the described components to send and / or receive data from external devices over networks. Examples of suitable Network Devices 708 include an Ethernet interface, a Wi-Fi transceiver, a cellular transceiver, a Bluetooth or Bluetooth Low Energy (BLE) transceiver, or any other network adapter or peripheral connection device that receives data from another computer or external storage device, which can be useful for efficiently receiving large datasets.

[0090] With regard to the processes, systems, procedures, heuristics, etc., described in this document, it is understood that, although the steps of such processes, etc., have been described as following a certain orderly sequence, such processes could be implemented in practice, with the described steps being carried out in a sequence that differs from the sequence described in this document. Furthermore, it is understood that certain steps could be carried out simultaneously, other steps added, or certain steps described herein omitted. In other words, the descriptions of processes herein serve the purpose of illustrating certain embodiments and should in no way be interpreted as limiting the patent claims.

[0091] Accordingly, it is understood that the foregoing description is intended to be illustrative and not limiting. Many other embodiments and applications beyond the examples provided will become apparent from reading the preceding description. The scope should not be determined by reference to the foregoing description, but instead by reference to the attached claims, together with the full scope of equivalents to which these claims entitle. It is anticipated and intended that there will be future developments in the prior art discussed in this document and that the disclosed systems and methods will be incorporated into such future embodiments. Overall, it is understood that the application may be modified and varied.

[0092] All terms used in the claims shall be understood to have their most comprehensive and comprehensible constructions and general meanings as they would be known to those skilled in the art in the art of the techniques described herein, unless expressly stated otherwise. In particular, the use of singular articles such as "a", "an", "the", "a", etc., shall be understood to refer to one or more of the elements indicated, unless a claim expressly limits this to the contrary.

[0093] The summary of disclosure is provided to enable the reader to quickly grasp the nature of the technical disclosure. It is submitted on the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, it is evident from the preceding detailed description that, for the purpose of simplifying the disclosure, various features in different embodiments have been grouped together. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly mentioned in each claim. Rather, as reflected in the following claims, the subject matter of the invention consists of fewer than all the features of any single disclosed embodiment.The following patent claims are hereby included in the detailed description, each patent claim being a separately claimed subject matter.

[0094] While exemplary embodiments have been described above, it is not intended that these embodiments describe all possible forms of the disclosure. Rather, the terms used in the description are descriptive rather than limiting, and it is understood that various modifications can be made without departing from the spirit and scope of the disclosure. Furthermore, the features of different implementing embodiments can be combined to form further embodiments of the disclosure.

[0095] According to the present invention, a method for using a multimodal model to update a charger feature table comprises the following: for each charging station specified by a charger feature table that describes which charging stations have which of a plurality of features, and for each feature of the charging station features: if a value of the feature is unknown or outdated: identifying vehicle data to be acquired according to the feature, wherein the vehicle data is predefined with respect to sensor types of vehicles with known configurations; transmitting a data request specifying the vehicle data to be acquired to the vehicles; receiving new vehicle data from the vehicles in response to the data request;Using the multimodal model, which is trained to recognize the multitude of features using the sensor types of vehicles with known configurations, to determine an updated value for the feature, at least partially, based on the new vehicle data; and updating the charger feature table to include the updated value for the feature; and using the charger feature table to identify charging stations for a requesting vehicle.

[0096] In one aspect of the invention, the method includes the following: using an assignment of features to the sensor types to identify the new vehicle data requested for acquisition for the feature as information acquired from the specified sensor types.

[0097] In one aspect of the invention, a validity period is defined for each feature of the plurality of features, and further comprising: identifying the feature as obsolete on the basis that the feature has been updated for longer than the validity period.

[0098] In one aspect of the invention, the vehicle data includes a plurality of data types, wherein the plurality of data types includes at least two of image data, text data and audio data.

[0099] In one aspect of the invention, the multimodal model further uses third-party data in combination with the vehicle data to recognize the features, wherein the third-party data includes one or more of the following: ratings of the charging stations, posts about the charging stations on social media and / or results of internet searches or image searches regarding the charging stations and / or amenities around the charging stations.

[0100] In one aspect of the invention, the method includes one or more of the following: performing semantic segmentation to identify and classify objects in images or videos and to determine one or more of object boundaries, lighting levels, and / or weather conditions; performing aspect-based sentiment analysis (ABSA) to determine a sentiment related to the charging station feature; and / or performing extended retrieval generation (RAG) to improve the updated value by drawing on facts from various known good data sources.

[0101] In one aspect of the invention, the sensor types include at least two of image sensors, light level sensors and / or humidity sensors.

[0102] According to the present invention, a system for using a multimodal model to update a charger feature table is provided, comprising: a charger monitoring server comprising one or more hardware processors configured for each charging station, as specified by a charger feature table describing which charging stations have which of a plurality of features, and for each feature of the charging station features, to: if a feature value is unknown or outdated: identify vehicle data to be acquired according to the feature, wherein the vehicle data is predefined with respect to sensor types of vehicles with known configurations, transmit a data request specifying the vehicle data to be acquired to the vehicles, receive new vehicle data from the vehicles in response to the data request, and use the multimodal model.which is trained to recognize the multitude of features using the sensor types of vehicles with known configurations, in order to determine an updated value for the feature at least partially based on the new vehicle data, and to update the charger feature table to include the updated value for the feature; and to use the charger feature table to identify charging stations for a requesting vehicle.

[0103] According to one embodiment, the charger monitoring server is further configured to: utilize an assignment of features to the sensor types to identify the new vehicle data requested for acquisition for the feature as information acquired from the specified sensor types.

[0104] According to one embodiment, a validity period is defined for each feature of the plurality of features, and the charger monitoring server is further configured to: identify the feature as obsolete based on the fact that the feature has been updated for longer than the validity period.

[0105] According to one embodiment, the vehicle data includes a multitude of data types, wherein the multitude of data types includes at least two of image data, text data and audio data.

[0106] According to one embodiment, the multimodal model further uses third-party data in combination with the vehicle data to recognize the features, wherein the third-party data includes one or more of the following: ratings of the charging stations, posts about the charging stations on social media and / or results of internet searches or image searches regarding the charging stations and / or amenities around the charging stations.

[0107] According to one embodiment, the charger monitoring server is further configured to perform one or more of the following: performing semantic segmentation to identify and classify objects in images or videos and to determine object boundaries, lighting levels, and / or weather conditions; performing aspect-based sentiment analysis (ABSA) to determine a sentiment related to the charging station feature; and / or performing extended retrieval generation (RAG) to improve the updated value by drawing on facts from various known good data sources.

[0108] According to one embodiment, the sensor types include at least two types: image sensors, light level sensors and / or humidity sensors.

[0109] According to the present invention, a non-transient, computer-readable medium is provided which includes instructions for using a multimodal model to update a charger feature table that describes which charging stations have which of a plurality of features. When executed by one or more hardware processors of a charger monitoring server, this table causes the charger monitoring server to perform operations that include: for each charging station specified by the charger feature table, and for each feature of the charging station features: if a feature value is unknown or outdated: identifying vehicle data to be acquired according to the feature, wherein the vehicle data is specified with respect to sensor types of vehicles with known configurations; transmitting a data request specifying the vehicle data to be acquired to the vehicles;Receiving new vehicle data from the vehicles in response to the data request; using the multimodal model, trained to recognize the multitude of features using the sensor types of the vehicles with known configurations, to determine an updated value for the feature, at least partially, based on the new vehicle data; and updating the charger feature table to include the updated value for the feature; and using the charger feature table to identify charging stations for a requesting vehicle.

[0110] According to one embodiment, the invention is further characterized by instructions which, when executed by one or more hardware processors of the charger monitoring server, cause the charger monitoring server to perform operations that include: using an assignment of features to the sensor types to identify the new vehicle data requested for acquisition for the feature as information acquired from the specified sensor types, wherein the sensor types include at least two of image sensors, light level sensors and / or humidity sensors.

[0111] According to one embodiment, a validity period is defined for each feature of the plurality of features, and further comprising instructions which, when executed by the one or more hardware processors of the charger monitoring server, cause the charger monitoring server to perform operations that include: identifying the feature as obsolete based on the fact that the feature has been updated for longer than the validity period.

[0112] According to one embodiment, the vehicle data includes a multitude of data types, wherein the multitude of data types includes at least two of image data, text data and audio data.

[0113] According to one embodiment, the multimodal model further uses third-party data in combination with the vehicle data to recognize the features, wherein the third-party data includes one or more of the following: ratings of the charging stations, posts about the charging stations on social media and / or results of internet searches or image searches regarding the charging stations and / or amenities around the charging stations.

[0114] According to one embodiment, the invention is further characterized by instructions which, when executed by the one or more hardware processors of the charger monitoring server, cause the charger monitoring server to perform operations that include one or more of the following: performing semantic segmentation to identify and classify objects in images or videos and to determine object boundaries, lighting levels, and / or weather conditions; performing aspect-based sentiment analysis (ABSA) to determine a sentiment related to the feature of the charging station; and / or performing extended retrieval generation (RAG) to improve the updated value by drawing on facts from various known good data sources.

Claims

[1] Method for using a multimodal model to update a charger feature table, comprising: for each charging station, which is specified by a charger characteristic table that describes which charging stations have which of a multitude of characteristics, and for each characteristic of the charging station characteristics: if a value of the feature is unknown or outdated: Identifying vehicle data to be captured according to the characteristic, whereby the vehicle data regarding sensor types of vehicles with known configurations are specified; Transmitting a data request specifying the vehicle data to be recorded to the vehicles; Receiving new vehicle data from the vehicles in response to the data request; Using the multimodal model, which is trained to recognize the multitude of features using the sensor types of vehicles with known configurations, to determine an updated value for the feature at least partially based on the new vehicle data; and Update the charger feature table to include the updated value for the feature; and Using the charger feature table to identify charging stations for a requesting vehicle. [2] The method of claim 1, further comprising: Utilizing an assignment of features to sensor types to identify the new vehicle data requested for acquisition for the feature as information acquired from the specified sensor types. [3] The method of claim 1, wherein a validity period is defined for each feature of the plurality of features, and further comprising: Identifying the feature as outdated based on the fact that the feature has been updated for longer than its validity period. [4] Method according to claim 1, wherein the vehicle data includes a plurality of data types, wherein the plurality of data types includes at least two of image data, text data and audio data. [5] Method according to claim 1, wherein the multimodal model further uses third-party data in combination with the vehicle data to detect the features, wherein the third-party data includes one or more of the following: Reviews of the charging stations, posts about the charging stations on social media and / or results of internet searches or image searches regarding the charging stations and / or amenities around the charging stations. [6] The method of claim 1, further comprising one or more of the following: Performing semantic segmentation to identify and classify objects in images or videos and to determine one or more of object boundaries, lighting levels and / or weather conditions; Conducting an aspect-based emotion analysis (ABSA) to determine an emotion related to the charging station feature; and / or Perform an extended retrieval generation (RAG) to improve the updated value by drawing on facts from several known good data sources. [7] Method according to claim 1, wherein the sensor types include at least two of image sensors, light level sensors and / or humidity sensors. [8] System for using a multimodal model to update a charger feature table, comprising: a charger monitoring server comprising one or more hardware processors configured to perform the following functions: for each charging station specified by a charger attribute table that describes which charging stations possess which of a multitude of attributes, and for each attribute of the charging station attributes: if a value of the feature is unknown or outdated: Identifying vehicle data to be captured according to the characteristic, whereby the vehicle data regarding sensor types of vehicles with known configurations are specified, Transmitting a data request specifying the vehicle data to be collected to the vehicles, Receiving new vehicle data from the vehicles in response to the data request, Using the multimodal model, which is trained to recognize the multitude of features using the sensor types of vehicles with known configurations, to determine an updated value for the feature at least partially based on the new vehicle data, and Update the charger feature table to include the updated value for the feature; and Using the charger feature table to identify charging stations for a requesting vehicle. [9] System according to claim 8, wherein the charger monitoring server is further configured to: Utilizing an assignment of features to sensor types to identify the new vehicle data requested for acquisition for the feature as information acquired from the specified sensor types. [10] System according to claim 8, wherein a validity period is defined for each feature of the plurality of features and wherein the charger monitoring server is further configured to: Identifying the feature as outdated based on the fact that the feature has been updated for longer than its validity period. [11] System according to claim 8, wherein the vehicle data includes a plurality of data types, wherein the plurality of data types includes at least two of image data, text data and audio data. [12] System according to claim 8, wherein the multimodal model further uses third-party data in combination with the vehicle data to recognize the features, wherein the third-party data includes one or more of the following: ratings of the charging stations, posts about the charging stations on social media and / or results of internet searches or image searches relating to the charging stations and / or amenities around the charging stations. [13] System according to claim 8, wherein the charger monitoring server is further configured to one or more of the following: Performing semantic segmentation to identify and classify objects in images or videos and to determine object boundaries, lighting levels and / or weather conditions; Conducting an aspect-based emotion analysis (ABSA) to determine an emotion related to the charging station feature; and / or Perform an extended retrieval generation (RAG) to improve the updated value by drawing on facts from several known good data sources. [14] System according to claim 8, wherein the sensor types include at least two of image sensors, light level sensors and / or humidity sensors. [15] Non-transient computer-readable medium comprising instructions for using a multimodal model to update a charger feature table that describes which charging stations have which of a variety of features which, when executed by one or more hardware processors of a charger monitoring server, cause the charger monitoring server to perform operations that include: for each charging station specified by the charger feature table, and for each feature of the charging station's features: if a value of the feature is unknown or outdated: Identifying vehicle data to be captured according to the characteristic, whereby the vehicle data regarding sensor types of vehicles with known configurations are specified; Transmitting a data request specifying the vehicle data to be recorded to the vehicles; Receiving new vehicle data from the vehicles in response to the data request; Using the multimodal model, which is trained to recognize the multitude of features using the sensor types of vehicles with known configurations, to determine an updated value for the feature at least partially based on the new vehicle data; and Update the charger feature table to include the updated value for the feature; and Using the charger feature table to identify charging stations for a requesting vehicle.

Images