Charging pile data capturing method based on python toolkit
The Python-based method for capturing charging pile data addresses the fragmented data sources by automating data acquisition and verification, enhancing database construction efficiency and accuracy through tools like mitmproxy and Plugshare.
Patent Information
- Application Number
- CN202411080721.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-03
AI Technical Summary
The lack of a unified global data source for overseas charging piles and the absence of clear standards in the construction of overseas charging pile databases hinder the development of the new energy vehicle market, as major operators independently manage their charging piles in different regions, leading to inefficiencies in data acquisition and database construction.
A method using a Python toolkit, specifically mitmproxy, to capture charging pile data from Plugshare, combined with tools like Google Maps and QGIS, for data cleaning and verification, and storage in standardized formats, including capturing URL requests and downloading images to create comprehensive databases.
Automates the acquisition of charging pile data, improving efficiency and speed in constructing global databases by providing accurate, real-time information on charging pile status and usage, overcoming manual recording limitations.
Smart Images

Figure CN119066246A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of new energy technology, and in particular to a method for capturing charging pile data based on a Python toolkit. Background Art
[0002] At present, the overseas new energy vehicle market is developing rapidly, and the demand for overseas charging pile database is increasing. However, there is no mature technical solution for the acquisition and database construction of overseas charging data. The main reason is that there is no unified data source for overseas charging piles. Major operators independently operate their own charging pile brands and models in different regions or countries, and there is no unified global charging pile data. Secondly, the overseas new energy vehicle market is in its infancy, and the industry is still in the exploratory stage for the construction of overseas charging pile databases, and there are no clear standards and methods. Summary of the invention
[0003] In view of the above problems, the present invention proposes a method for capturing charging pile data based on a python toolkit. Plugshare is used as the target charging pile website. Based on the mitmproxy toolkit, the overseas charging pile data is automatically captured. The captured information includes key information such as country code, administrative region, longitude and latitude, update time, score, photo, number of comments, etc. At the same time, Google map, QGIS, OpenStreetMap and other tools are combined to confirm the model and brand of the charging pile, complete data cleaning and data verification, and realize storage in accordance with standards.
[0004] The present invention is achieved through the following technical solutions:
[0005] A method for capturing charging pile data based on a python toolkit includes the following steps:
[0006] (1) Start the packet capture tool: confirm the target charging pile address, start the mitmproxy toolkit, set the data capture port number, and then start the capture script to start automatic capture of charging pile data;
[0007] (2) Capture URL requests: Capture website URL requests through the mitmproxy toolkit, read data stream information, write the request information into a file in json format, and name the json file with the site ID;
[0008] (3) Reading data stream information code: reading dictionary value content from data stream information;
[0009] (4) Configure the Internet Download Manager tool: By using the Internet Download Manager tool, download the images of the corresponding ID site to a folder and name the folder according to the ID. The Internet Download Manager tool can improve download efficiency and support breakpoint resume function to avoid downloading again after interruption.
[0010] (5) Run Main: The program runs and generates an initial xlsx file in a specific format, traverses all json files stored in the folder, processes the data flow information and images of each charging pile json file, writes them into the xlsx file, and moves the processed json file into the finished folder, indicating that the site data has been processed.
[0011] Through a lot of research, it was found that Plugshare has charging pile data in most countries or regions in the world and is the world's largest electric vehicle owner community. Car owners can upload photos and comments of on-site charging piles to help electric vehicle community owners understand the situation of on-site charging piles, whether the charging piles are damaged or updated, and other information. Car owners can filter the PlugShare map by plug type (including CHAdeMO and SAE / CCS) and charging speed (including: Level 1, Level 2 and DC fast chargers, such as Tesla Supercharger Stations). You can also filter by charging provider-PlugShare maps include detailed charging pile information of major electric vehicle charging networks in North America, Europe and most parts of the world, and the data is very comprehensive. Therefore, the target charging pile address of the present invention is based on the Plugshare charging pile website database.
[0012] Furthermore, the data stream information includes country code, first-level administrative region, second-level administrative region, ID, longitude, latitude, longitude and latitude, plus code, rating, total number of photos, total number of comments, total number of charging piles, creation time, last update time, Plugshare link, Google map link, abnormal situation of charging pile,
[0013] Among them, country code: can filter CCS charging pile data of different countries according to Asia, Europe and other regions.
[0014] First-level administrative region: You can filter the charging pile data of a province / county in a certain country.
[0015] Second-level administrative region: You can filter data for a certain province or region in a certain country.
[0016] ID: Each charging station has a unique ID code.
[0017] Longitude: Get the longitude coordinates of a specific charging station.
[0018] Latitude: Get the latitude coordinates of a specific charging station.
[0019] Latitude and longitude: the specific latitude and longitude coordinates of the charging pile,
[0020] plus code: the site code of a specific charging station.
[0021] Rating: The user's rating of the ease of using the charging pile at the station.
[0022] Total number of photos: The number of photos of the actual scene at a charging station, which can be used to confirm the on-site conditions, charging station model, number of charging stations, parking methods, etc.
[0023] Total number of comments: Users comment on the ease of using the charging piles at this station.
[0024] Number of charging piles: can be used to confirm the number of charging piles at a certain station.
[0025] Creation time: the time when a charging station data is updated.
[0026] Last updated time: the time when the data of a charging station is updated.
[0027] Plugshare link: You can jump to the Plugshare charging map for navigation, etc.
[0028] Google Maps link: You can jump to Google Maps for navigation, etc.
[0029] Abnormal conditions of charging piles: can be used to confirm whether the charging pile is under construction / maintenance / normal use, etc.
[0030] The beneficial effects of the present invention are: on the one hand, the present invention can automatically capture the charging pile information of PlugShare. Compared with the tedious steps of manually recording the charging piles, the automatic acquisition of the charging pile information greatly improves the efficiency of recording the charging piles in a certain country or region, and greatly speeds up the construction of the charging pile database of all countries in the world; on the other hand, the automatic acquisition can complete the update of the charging pile information more quickly, obtain the latest and accurate longitude and latitude of the charging pile, and at the same time can understand the current usage or abnormal situation of the on-site charging piles in real time, whether they are in use / under construction / faulty, etc., and can also understand the difficulty of using the charging piles through ratings or comments. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a flow chart of a method for capturing charging pile data based on a python toolkit of the present invention;
[0032] Figure 2 The following is a crawling flow chart of a specific embodiment of the present invention. DETAILED DESCRIPTION
[0033] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.
[0034] like Figure 1 The method for capturing charging pile data based on a python toolkit shown in the figure includes the following steps:
[0035] (1) Start the packet capture tool: confirm the target charging pile address, start the mitmproxy toolkit, set the data capture port number, and then start the capture script to start automatic capture of charging pile data;
[0036] (2) Capture URL requests: Capture website URL requests through the mitmproxy toolkit, read data stream information, write the request information into a file in json format, and name the json file with the site ID;
[0037] (3) Reading data stream information code: reading dictionary value content from data stream information;
[0038] (4) Configure the Internet Download Manager tool: By using the Internet Download Manager tool, download the images of the corresponding ID site to a folder and name the folder according to the ID. The Internet Download Manager tool can improve download efficiency and support breakpoint resume function to avoid downloading again after interruption.
[0039] (5) Run Main: The program runs and generates an initial xlsx file in a specific format, traverses all json files stored in the folder, processes the data flow information and images of each charging pile json file, writes them into the xlsx file, and moves the processed json file into the finished folder, indicating that the site data has been processed.
[0040] The following example illustrates this: Figure 2 As shown:
[0041] First, enter the mitmproxy command. Use the command prompt under the project file to enter the mitmproxy command, start the packet capture script, and set the corresponding port number. After confirmation, it prompts to start listening to the port number. At the same time, open the default browser and prompt "client connect" to indicate that the connection is successful.
[0042] Capture charging pile information. Open the https: / / www.plugshare.com website, select any charging pile site in the target area, and left-click the charging pile information. If the command prompt window prompts that the charging station ID information is successfully obtained, the data is successfully captured and the data information is stored in the folder in the form of a json file.
[0043] Run the Main method data processing script. The program runs to generate an xlsx file, and you can see that the data is completely written into the xlsx table. The processed json file is moved to the finished folder for storage.
[0044] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A method for capturing charging pile data based on a python toolkit, characterized in that: The steps include: (1) Start the packet capture tool: confirm the target charging pile address, start the mitmproxy toolkit, set the data capture port number, and then start the capture script to start automatic capture of charging pile data; (2) Capture URL requests: Capture website URL requests through the mitmproxy toolkit, read data stream information, write the request information into a file in json format, and name the json file with the site ID; (3) Reading data stream information code: reading dictionary value content from data stream information; (4) Configure the Internet Download Manager tool: Use the Internet Download Manager tool to download the images from the corresponding ID site to a folder and name the folder according to the ID. (5) Run Main: The program runs and generates an initial xlsx file in a specific format, traverses all json files stored in the folder, processes the data flow information and images of each charging pile json file, writes them into the xlsx file, and moves the processed json file into the finished folder, indicating that the site data has been processed.
2. The method for capturing charging pile data based on a python toolkit according to claim 1 is characterized in that: The target charging pile address is based on the Plugshare charging pile website database.
3. The method for capturing charging pile data based on a python toolkit according to claim 1 is characterized in that: The data stream information includes country code, first-level administrative region, second-level administrative region, ID, longitude, latitude, longitude and latitude, pluscode, rating, total number of photos, total number of comments, total number of charging piles, creation time, last update time, Plugshare link, Google map link, and abnormal conditions of charging piles.
