Pandas-based new energy industry data deep mining and visualization method and device

Abstract

The application provides a new energy industry data deep mining and visualization method and equipment based on Pandas, and relates to the technical field of new energy industry data mining and analysis. The method first uses a network mapping platform for deep data mining, imports a relational database, then uses Pandas to clean and secondarily develop the industry data to obtain high-quality data, constructs an industry database meeting the depth of new energy industry demand, and supports on-demand retrieval and data visualization. The application can accurately locate specific data assets in the new energy field by constructing an aggregation mapping search model based on industry basic words and sensitive words. This has significance for the new energy industry in that it can strip out vertical data with real technical value and commercial value from massive Internet noise, providing a high-quality data base for accurately grasping industry dynamics and researching technical routes.

Description

Technical Field

[0001] This invention relates to the field of data mining and analysis technology in the new energy industry, and more specifically, to a method and device for deep data mining and visualization in the new energy industry based on Pandas. Background Technology

[0002] The new energy industry is experiencing rapid growth in both scale and technological iteration. Accurately grasping industry dynamics, supply chain relationships, and potential technological assets is crucial for government decision-making and corporate strategy formulation. Therefore, the demand for in-depth data mining within the vertical industry of new energy is increasingly prominent. In the existing technological system, acquiring industry data typically relies on two main approaches: first, using cyberspace mapping platforms (such as Quake and Shodan) to scan wide area network assets to identify servers, ports, and service components; second, using general web crawling technology to batch-grab information displayed on web page front-ends by simulating HTTP requests. After acquiring the data, it is usually stored in traditional relational databases, and basic queries and statistics are performed using SQL statements, ultimately displayed through static charts.

[0003] However, the existing data acquisition and processing models exhibit multi-dimensional technical limitations when dealing with high-value-density data from the new energy industry. Firstly, regarding the depth and breadth of data acquisition, existing cyberspace mapping technologies primarily focus on discovering the exposure surfaces of network assets, i.e., focusing on "what devices are there" or "what vulnerabilities exist," while struggling to delve into sensitive business files (such as XLS files, SQL database backups, and PDF technical documents) stored in device or service directories. While general web crawling technologies can capture web page content, they often lack semantic understanding capabilities specific to vertical sectors, making it difficult to accurately locate high-value unstructured data resources through "industry-sensitive keywords," resulting in low signal-to-noise ratios and severe generalization of the acquired data.

[0004] Secondly, regarding the stability and concealment of data collection, with the widespread adoption of network security technologies, target data websites generally deploy Web Application Firewalls (WAFs), DDoS protection, and traffic behavior analysis systems. The high-concurrency, continuous request mode used by traditional web crawlers easily triggers the frequency limiting policies of target servers, leading to IP blocking or connection resets, thus preventing deep traversal of target directories and file downloads. How to ensure collection efficiency while using technical means to circumvent the feature recognition of security protection systems is a significant technical challenge currently facing industrial data mining.

[0005] Furthermore, in terms of data processing and analysis, new energy industry data typically exhibits multi-source heterogeneity, including structured tabular data, semi-structured logs, and unstructured text data. Traditional direct database-based operations (such as MySQL) are often limited by the expressive power of SQL and the I / O performance of databases when faced with complex data cleaning needs (such as multi-table deduplication, cross-database merging, and redundancy removal), making it difficult to efficiently achieve complex data cleaning and purification logic. Especially when multi-condition nested cascading queries are required across multiple related tables based on specific "industry-sensitive keywords," traditional methods suffer from low computational efficiency and struggle to generate high-value-added in-depth analytical data.

[0006] Finally, regarding data interaction and visualization, most existing industry data analysis platforms adopt static statistical reports, lacking the ability to dynamically display the layered value of data. Users find it difficult to directly drill down into the underlying data within the visualization interface, and the platforms also cannot integrate information extracted from PDF documents with structured data from databases, significantly reducing the readability and decision-making support value of the data. Therefore, there is an urgent need for a comprehensive data mining method that integrates network mapping and positioning, anti-detection acquisition mechanisms, efficient memory-based data cleaning (such as using the Pandas library), and interactive visualization to address these technical bottlenecks. Summary of the Invention

[0007] The present invention aims to solve at least one of the aforementioned technical problems existing in the prior art.

[0008] To this end, the first aspect of the present invention provides a method for deep mining and visualization of new energy industry data based on Pandas.

[0009] A second aspect of the present invention provides a computer device.

[0010] This invention provides a method for deep data mining and visualization of the new energy industry based on Pandas, including: S1. Construct an aggregated mapping search model, including: defining a set of basic terms for the new energy industry (Title) and a set of sensitive terms for the industry (Body), combining the set of network mapping platforms (S), constructing a search function (F(s), and using the search function (F(s)) to lock the access address U and title W of the target data object for the industry; S2. Intermittent data acquisition based on the scanner class, including: instantiating a network scanner object, performing directory traversal queries based on the access address U, using an intermittent request mechanism to circumvent security detection, retrieving a specific format data file containing the title field W and downloading it to the local storage path; S3. Sensitive feature amplification and secondary traversal, including: manually reviewing the file content obtained in step S2, amplifying the feature words in the industry sensitive word set Body, and repeating step S2 using the amplified set; S4. Data cleaning and database integration, including: cleaning and denoising downloaded data files, importing structured and unstructured data into a MySQL database, and establishing a database connection engine. S5. Deep value mining based on Pandas, including: using Pandas to read data through the database connection engine, merging multiple tables and removing redundancies based on the primary key, and performing multi-condition nested data cascading queries based on industry-sensitive keywords to generate high-value industry data tables. S6. Layered visualization display, including: connecting the high-value industry data table to the DataGear visualization platform and building layered visualization dashboards for different types of data sources.

[0011] The Pandas-based method for deep data mining and visualization of the new energy industry according to the above-described technical solution of the present invention may also have the following additional technical features: In the above technical solution, the set of industry-based terms defined in step S1 is represented as follows:

[0012] The defined set of industry-sensitive words is represented as follows:

[0013] The network mapping platform set S is a collection of several types of network mapping platforms; The construction of the search function F(s) includes: Define the constituent parameters of the search function F(s), which include the basic word parameter Title, the sensitive word parameter Body, the country parameter Country, the region parameter Region, and the constraint condition array D; The parameters in the constraint array D are used to represent specific constraint conditions that depend on the selected mapping platform S; The search function F(s) calculates the operational relationship between input variables t and b and array parameter d based on the selected platform parameter S, and outputs a set U containing the target URL and a set W containing the corresponding titles, that is: ,

[0014] in, M represents the output of the search function F(s).

[0015] In the above technical solution, in step S2, the instantiated network scanner object includes: Initialize the scanner class, and configure the basic URL parameters, maximum number of threads, and download path parameters; Establish an HTTP session and execute the scanning task concurrently under the constraint of the maximum number of threads; The specific format data files include files with the extensions xls, xlsx, sql, pdf, zip, doc, and docx. The scanner categorizes and stores them into the corresponding file type directories based on their file extensions.

[0016] In the above technical solution, step S4, the database connection engine includes: Create a database engine object using the sqlalchemy library; Configure the connection string, which includes the database type mysql, the driver pymysql, the username, the password, the host address, the port number, and the database name; The engine object establishes a communication channel between Pandas and the MySQL database.

[0017] In the above technical solution, step S5, reading data using Pandas includes: Write SQL query statements; Call the SQL query read function to directly convert the query results from the database into Pandas data frames.

[0018] In the above technical solution, step S5, the execution of multi-condition nested data cascading query based on industry-sensitive keyword groups includes: Based on the relationships between data tables, a multi-level query logic is constructed in Pandas. For two data tables with relationships, records containing different combinations of features from preset industry-sensitive keyword groups are filtered. The filtered records are then linked through primary keys or foreign keys to generate a high-value industry data table containing multi-dimensional sensitive features.

[0019] In the above technical solution, step S5, the multi-table merging and redundancy removal based on the primary key, includes: Identify the primary key fields of the tables to be merged; Using Pandas' merge function, multiple scattered data tables are joined vertically or horizontally based on the primary key field; duplicate value detection is performed on the joined data frame to remove completely duplicate rows, and logical verification and cleaning are performed on conflicting data.

[0020] In the above technical solution, step S6, the construction of the hierarchical visualization dashboard includes: For unstructured data in PDF and Word formats, extract text information and store it in a database, then display a text information statistics dashboard on the DataGear platform; For structured data in Excel and SQL formats, after processing in step S5, the industry value data chart dashboard is displayed on the DataGear platform. The DataGear platform supports access to SQL, CSV, Excel, HTTP interfaces, and JSON data sources, and supports managing imported data through the SQL workbench.

[0021] In the above technical solution, in step S2, the intermittent request mechanism includes: inserting random or fixed sleep time segments between consecutive file download requests, wherein the length of the sleep time segment is set to a threshold to prevent the target website's security system from identifying the request as a malicious crawler.

[0022] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any of the above technical solutions.

[0023] In summary, due to the adoption of the above-mentioned technical features, the beneficial effects of the present invention are: This invention solves the problem of "broad but not precise" data acquisition in the new energy industry, achieving accurate intelligence mining in vertical fields. Existing technologies mostly focus on broad-area scanning of macro-industries, making it difficult to reach specific business-sensitive data within the new energy industry. This invention, by constructing an aggregated mapping and search model based on industry-specific keywords and sensitive keywords, can accurately locate specific data assets in the new energy field (such as specific technical document PDFs, supply chain data XLS, etc.). The significance for the new energy industry lies in its ability to extract truly valuable vertical data with technological and commercial significance from massive amounts of internet noise, providing a high-quality data foundation for accurately grasping industry dynamics and analyzing technological routes.

[0024] This invention breaks through the security barriers of new energy industry websites, ensuring the continuity and integrity of industry intelligence acquisition. High-value data in the new energy industry is often heavily protected by website security systems, making it easy for traditional web crawlers to be blocked and cause data interruptions. This invention employs a strategy combining intermittent request mechanisms with specific file type traversal, achieving "low-frequency, deep" acquisition of target data files without triggering DDoS protection or WAF blocking on the target website. This is of great significance for the long-term monitoring of new energy industry data, ensuring the stable and continuous accumulation of key industry chain data even when facing highly protected industry websites.

[0025] This invention enables the value reconstruction and deep correlation of data across the new energy industry chain, aiding in the discovery of hidden industry logic. The new energy industry is characterized by a long industry chain and high correlation (covering materials, batteries, complete vehicles, and power grids). This invention utilizes Pandas for in-memory data cleaning and secondary development, particularly through "multi-condition nested data cascading queries based on industry-sensitive keywords," which allows for deep correlation of isolated information scattered across different data tables (such as upstream raw material prices and downstream production capacity data) based on business logic. This processing method effectively eliminates redundant noise, uncovers the supply and demand relationships and competitive intelligence hidden behind the data, and realizes the leap of data from "raw materials" to "high-value assets."

[0026] Additional aspects and advantages of the invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a method for deep data mining and visualization of the new energy industry based on Pandas, according to an embodiment of the present invention. Figure 2 This is a flowchart of the multi-condition nested data cascading query logic in a Pandas-based method for deep data mining and visualization of the new energy industry, according to an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the data visualization process in a Pandas-based method for in-depth data mining and visualization of the new energy industry, according to an embodiment of the present invention. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0029] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0030] The following reference Figures 1 to 3 This paper describes a method for deep data mining and visualization of the new energy industry based on Pandas, provided by some embodiments of the present invention.

[0031] Some embodiments of this application provide a method for deep data mining and visualization of the new energy industry based on Pandas.

[0032] like Figure 1 As shown, the first embodiment of the present invention proposes a method for deep mining and visualization of new energy industry data based on Pandas, including the following steps S1 to S6.

[0033] S1. Construct an aggregated mapping search model, including: defining a set of basic terms for the new energy industry (Title) and a set of sensitive terms for the industry (Body), combining the set of network mapping platforms (S), constructing a search function (F(s), and using the search function (F(s)) to lock the access address (U) and title (W) of the target data object for the industry.

[0034] In some embodiments, the set of industry-based terms defined in step S1 is represented as:

[0035] The defined set of industry-sensitive words is represented as follows:

[0036] The set S of network mapping platforms is a collection of several network mapping platforms; in a specific embodiment, the set of network mapping platforms includes Quake, Shodan, FOFA, and 360 Mapping Cloud, etc., and the set S is represented as: S={Quake, FOFA, Shodan, Hunter,...} The construction of the search function F(s) includes: Define the constituent parameters of the search function F(s), which include the basic word parameter Title, the sensitive word parameter Body, the country parameter Country, the region parameter Region, and the constraint condition array D, etc. The parameters in the constraint array D are used to represent specific constraints that depend on the selected mapping platform S; that is, the way the parameters in array D (such as a, b, c, d, etc.) are expressed depends on the selection of parameter S.

[0037] The search function F(s) calculates the operational relationship between input variables t and b and array parameter d based on the selected platform parameter S, and outputs a set U containing the target URL and a set W containing the corresponding titles, that is: ,

[0038] in, , , M represents the output of the search function F(s).

[0039] S2. Intermittent data acquisition based on the scanner class includes: instantiating a network scanner object, performing directory traversal queries based on the access address U, using an intermittent request mechanism to circumvent security detection, retrieving a specific format data file containing the title field W, and downloading it to the local storage path.

[0040] Based on the output parameters U and W from step S1, the GET request method is used to request parameter U, and a Dir directory traversal query of U is performed to retrieve common data and text storage information such as xls, xlsx, sql, pdf, zip, doc, and docx containing the field W.

[0041] In some embodiments, in step S2, the instantiated network scanner object includes: Initialize the scanner class, and configure the basic URL parameters, maximum number of threads, and download path parameters; Establish an HTTP session and execute the scanning task concurrently under the constraint of the maximum number of threads; The specific format data files include files with the extensions xls, xlsx, sql, pdf, zip, doc, and docx. The scanner categorizes and stores them into the corresponding file type directories based on their file extensions.

[0042] In one specific embodiment, the implementation code for step S2 is as follows: class XnWebScanner: def _init_(sel, base_url, max_threads=3); self.base_url = base_url.rstrip(' / ') / / Extract URL path self.visited_urls = set () self.files_downloaded = [] self.session = requests.Session() self.max_threads = max_threads ... / / Define the main function, specify parameters such as the scanning thread. self.download_dir = “C: / data” / / Define the data storage path self.target_extensions ={ '.pdf': 'PDF document', '.pdf': 'doc document', '.pdf': 'xls document', ...... / / Define the data type for mining } It is understandable that the above code is a code example with the same meaning as step S2, used to fully demonstrate the execution process (the same applies to the code examples that appear later).

[0043] In one specific embodiment, in step S2, the intermittent request mechanism includes: inserting random or fixed sleep time segments between consecutive file download requests, the length of which is set to a threshold to prevent the target website's security system from identifying the request as a malicious crawler.

[0044] S3. Sensitive feature amplification and secondary traversal, including: manually reviewing the file content obtained in step S2, amplifying the feature words in the industry sensitive word set Body, and using the amplified set to repeatedly execute the intermittent request method proposed in step S2 to traverse the target data website again.

[0045] S4. Data cleaning and database integration, including: cleaning and denoising downloaded data files, importing structured and unstructured data into a MySQL database, and establishing a database connection engine. Specifically, given the differences in data formats and varying data quality and value collected in steps S1-S3, it is necessary to clean, select, and store the data to ensure its reliability and value.

[0046] Data cleaning is the process of detecting and correcting dirty data, and it involves processing redundant data and other obviously noisy data to preserve their fidelity. It is one of the important means to improve the quality of data research. This invention manually cleans and selects downloaded relational data files such as xls, xlsx, and sql files before importing them into the Navicat database platform. Secondary data development is then performed using Pandas and a MySQL-driven data engine.

[0047] In some embodiments, in step S4, the database connection engine includes: Create a database engine object using the sqlalchemy library; Configure the connection string, which includes the database type mysql, the driver pymysql, the username, the password, the host address, the port number, and the database name; The engine object establishes a communication channel between Pandas and the MySQL database.

[0048] The core code implementation of MySQL data-driven connection is shown below: from sqlalchemy import create_engine engine=create_engine('mysql+pymysql: / / root:root@127.0.0.1:3306 / data',echo=True) sql='select from data' df=pd.read_sql_query (sql,engine) S5. Deep value mining based on Pandas, including: using Pandas to read data through the database connection engine, merging multiple tables and removing redundancies based on primary keys, and performing multi-condition nested data cascading queries based on industry-sensitive keywords to generate high-value industry data tables.

[0049] Pandas is a powerful toolset for analyzing structured data, supporting importing data from various file formats such as CSV, JSON, SQL, and Excel. Pandas can also perform various data operations. Specifically, the following operations can be performed on the imported data: Based on the data entered into the database, select a data form, use Pandas to perform secondary data deduplication and data merging, merge data from multiple data tables into one data table according to the primary key, perform conflict detection on redundant data values, and complete the value transfer of data.

[0050] Data on the new energy industry is filtered using MySQL, and Pandas is used to filter the cleaned data based on industry-sensitive word phrases (W).

[0051] Based on different combinations of the sensitive word W phrase and according to the relationships between tables, multi-condition nested data cascading queries are used to generate data table items that simultaneously contain multiple industry sensitive words.

[0052] In some embodiments, reading data using Pandas in step S5 includes: Write SQL query statements; Call the SQL query read function to directly convert the query results from the database into Pandas data frames.

[0053] In some embodiments, such as Figure 2 As shown, the multi-condition nested data cascading query based on industry-sensitive keyword phrases includes: Based on the relationships between data tables, a multi-level query logic is constructed in Pandas. For two data tables with relationships, records containing different combinations of features from preset industry-sensitive keyword groups are filtered. The filtered records are then linked through primary keys or foreign keys to generate a high-value industry data table containing multi-dimensional sensitive features.

[0054] Specifically, in step S5, the multi-table merging and redundancy removal based on the primary key includes: Identify the primary key fields of the tables to be merged; Using Pandas' merge function, multiple scattered data tables are joined vertically or horizontally based on the primary key field; duplicate value detection is performed on the joined data frame to remove completely duplicate rows, and logical verification and cleaning are performed on conflicting data.

[0055] S6. Layered visualization display, including: connecting the high-value industry data table to the DataGear visualization platform and building layered visualization dashboards for different types of data sources.

[0056] Specifically, the DataGear development platform boasts advantages such as ease of deployment, simple operation, and integration with visualization panels. It supports access to various data sources including SQL, CSV, Excel, HTTP interfaces, and JSON. It enables data import / export via an SQL workbench for data management and supports visualization of dataset management charts and dashboards. Therefore, using Pandas to access data on the DataGear platform, a data display panel was designed to complete the visualization of both structured and unstructured data. Its industry data visualization display includes... Figure 3 As shown.

[0057] In some embodiments, in step S6, constructing the hierarchical visualization dashboard includes: For unstructured data in PDF and Word formats, extract text information and store it in a database, then display a text information statistics dashboard on the DataGear platform; For structured data in Excel and SQL formats, after processing in step S5, the industry value data chart dashboard is displayed on the DataGear platform. The DataGear platform supports access to SQL, CSV, Excel, HTTP interfaces, and JSON data sources, and supports managing imported data through the SQL workbench.

[0058] Other embodiments of the present invention provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any of the above embodiments.

[0059] In this specification, the illustrative expressions of the terms used do not necessarily refer to the same embodiments or examples. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0060] Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this invention shall be included within the scope of protection of this invention.

Claims

1. A method for deep data mining and visualization of the new energy industry based on Pandas, characterized in that, include: S1. Construct an aggregated mapping search model, including: defining a set of basic terms for the new energy industry (Title) and a set of sensitive terms for the industry (Body), combining the set of network mapping platforms (S), constructing a search function (F(s), and using the search function (F(s)) to lock the access address U and title W of the target data object for the industry; S2. Intermittent data acquisition based on the scanner class, including: instantiating a network scanner object, performing directory traversal queries based on the access address U, using an intermittent request mechanism to circumvent security detection, retrieving a specific format data file containing the title field W and downloading it to the local storage path; S3. Sensitive feature amplification and secondary traversal, including: manually reviewing the file content obtained in step S2, amplifying the feature words in the industry sensitive word set Body, and repeating step S2 using the amplified set; S4. Data cleaning and database integration, including: cleaning and denoising downloaded data files, importing structured and unstructured data into a MySQL database, and establishing a database connection engine. S5. Deep value mining based on Pandas, including: using Pandas to read data through the database connection engine, merging multiple tables and removing redundancies based on the primary key, and performing multi-condition nested data cascading queries based on industry-sensitive keywords to generate high-value industry data tables. S6. Layered visualization display, including: connecting the high-value industry data table to the DataGear visualization platform and building layered visualization dashboards for different types of data sources.

2. The method for deep data mining and visualization of the new energy industry based on Pandas according to claim 1, characterized in that, In step S1, the defined set of industry-based terms is represented as follows: The defined set of industry-sensitive words is represented as follows: The network mapping platform set S is a collection of several types of network mapping platforms; The construction of the search function F(s) includes: Define the constituent parameters of the search function F(s), which include the basic word parameter Title, the sensitive word parameter Body, the country parameter Country, the region parameter Region, and the constraint condition array D; The parameters in the constraint array D are used to represent specific constraint conditions that depend on the selected mapping platform S; The search function F(s) calculates the operational relationship between input variables t and b and array parameter d based on the selected platform parameter S, and outputs a set U containing the target URL and a set W containing the corresponding titles, that is: ， in, M represents the output of the search function F(s).

3. The method for deep data mining and visualization of the new energy industry based on Pandas according to claim 1, characterized in that, In step S2, the instantiated network scanner object includes: Initialize the scanner class, and configure the basic URL parameters, maximum number of threads, and download path parameters; Establish an HTTP session and execute the scanning task concurrently under the constraint of the maximum number of threads; The specific format data files include files with the extensions xls, xlsx, sql, pdf, zip, doc, and docx. The scanner categorizes and stores them into the corresponding file type directories based on their file extensions.

4. The method for deep data mining and visualization of the new energy industry based on Pandas according to claim 1, characterized in that, In step S4, the database connection engine includes: Create a database engine object using the sqlalchemy library; Configure the connection string, which includes the database type mysql, the driver pymysql, the username, the password, the host address, the port number, and the database name; The engine object establishes a communication channel between Pandas and the MySQL database.

5. The method for deep data mining and visualization of the new energy industry based on Pandas according to claim 4, characterized in that, In step S5, reading data using Pandas includes: Write SQL query statements; Call the SQL query read function to directly convert the query results from the database into Pandas data frames.

6. The method for deep data mining and visualization of the new energy industry based on Pandas according to claim 1, characterized in that, In step S5, the multi-condition nested data cascading query based on industry-sensitive keyword groups includes: Based on the relationships between data tables, a multi-level query logic is constructed in Pandas. For two data tables with relationships, records containing different combinations of features from preset industry-sensitive keyword groups are filtered. The filtered records are then linked through primary keys or foreign keys to generate a high-value industry data table containing multi-dimensional sensitive features.

7. The method for deep data mining and visualization of the new energy industry based on Pandas according to claim 1, characterized in that, In step S5, the multi-table merging and redundancy removal based on the primary key includes: Identify the primary key fields of the tables to be merged; Using Pandas' merge function, multiple scattered data tables are joined vertically or horizontally based on the primary key field; duplicate value detection is performed on the joined data frame to remove completely duplicate rows, and logical verification and cleaning are performed on conflicting data.

8. The method for deep data mining and visualization of the new energy industry based on Pandas according to claim 1, characterized in that, In step S6, constructing the hierarchical visualization dashboard includes: For unstructured data in PDF and Word formats, extract text information and store it in a database, then display a text information statistics dashboard on the DataGear platform; For structured data in Excel and SQL formats, after processing in step S5, the industry value data chart dashboard is displayed on the DataGear platform. The DataGear platform supports access to SQL, CSV, Excel, HTTP interfaces, and JSON data sources, and supports managing imported data through the SQL workbench.

9. The method for deep data mining and visualization of the new energy industry based on Pandas according to claim 1, characterized in that, In step S2, the intermittent request mechanism includes inserting random or fixed sleep time segments between consecutive file download requests, wherein the length of the sleep time segment is set to a threshold that prevents the target website's security system from identifying the request as a malicious crawler.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 9.

Images