Method, device and equipment for analyzing data of multiple CSV files

Abstract

The invention belongs to the technical field of CSV file processing, and provides a method, a device and equipment for analyzing data of multiple CSV files. The method comprises the following steps: loading a plurality of CSV files to the same interface, detecting duplicate items in all column names, and performing duplicate removal operation on the duplicate items; identifying separators in the CSV file, determining a coding format, and analyzing file content; generating a time sequence diagram according to the analysis content; in response to user operation, performing viewing operation on the time sequence diagram; and merging and exporting the data of the plurality of CSV files. According to the method for analyzing the data of the multiple CSV files, the multiple CSV files can be loaded and processed to the same interface at the same time, and a user is supported to conduct unified screening, analysis or visualization operation on the integrated data columns.

Description

Technical Field

[0001] The present invention relates to the technical field of CSV file processing, and in particular to a method, device and equipment for analyzing data of multiple CSV files. Background Art

[0002] With the development of big data technology, CSV (Comma-Separated Values) is widely used in scientific research, business analysis and engineering practice as a lightweight data storage format. The most widespread application is to transfer tabular data between programs that operate on incompatible formats (often proprietary and / or unregulated formats). Because a large number of programs support some CSV variant, at least as an optional input / output format.

[0003] The CSV format is a delimited data format with comma characters separating fields / columns and newline characters separating records / rows. Fields containing special characters (commas, newline characters, or double quotes) must be enclosed in double quotes. Rows containing an item that is an empty string may be enclosed in double quotes. When a field value contains a double quote, double the double quote (as if treating a double quote as an escape character). The CSV file format does not require a specific character encoding, byte order, or row termination format.

[0004] However, current processing of CSV files may support batch loading, but most existing tools may still display the data of each file in a separate window or tab. When processing multiple CSV files, traditional methods usually rely on command lines or non-interactive tools, which have the following problems: 1. Inefficiency: Faced with multiple CSV files, users need to manually merge and extract key data columns, which is time-consuming and error-prone; 2. Poor interactivity: Most existing tools are command lines or simple scripts, which cannot provide intuitive chart interaction functions; 3. Limited scalability: lack of support for complex scenarios (such as multi-column data comparison analysis, real-time interactive data query); 4. Complex operation: For non-professional technicians, the learning and use thresholds of existing tools are high and it is difficult to meet actual needs. Summary of the invention

[0005] In view of the defects in the prior art, the present invention provides a method, device and equipment for analyzing data of multiple CSV files to solve the problems of low efficiency and poor interactivity caused by relying on command lines or non-interactive tools when processing multiple CSV files.

[0006] In a first aspect, the present invention provides a method for analyzing data of multiple CSV files, comprising: Load multiple CSV files into the same interface, detect duplicates in all column names, and deduplicate the duplicates; Identify the delimiter in the CSV file and determine the encoding format, and parse the file content; Generate a time series graph based on the analyzed content; In response to a user operation, viewing the time series graph; Merge and export data from multiple CSV files.

[0007] It can be seen from the above technical solution that the method for analyzing multiple CSV file data provided by the present invention can simultaneously load and process multiple CSV files into the same interface, and support users to perform unified screening, analysis or visualization operations on the integrated data columns.

[0008] Optionally, identifying the delimiter in the CSV file includes: Read the contents of the default line and determine the frequency of occurrence of various predicted delimiters; Matching a plurality of patterns of the predicted delimiters based on regular expressions; The predicted delimiter with the highest frequency is used as a delimiter to parse the CSV file; if the parsing fails, the predicted delimiter with the next frequency is tried as a delimiter for parsing; If the predicted separator cannot be determined as the separator, a prompt for the user to manually select is popped up.

[0009] It can be seen from the above technical solution that the delimiters in the CSV file can be automatically identified, making it easier to parse the file content.

[0010] Optionally, determining the encoding format includes: Get the detection result based on the encoding format of the file detected by the third-party library; According to the detection result, the file content is loaded in a corresponding encoding format; If the file encoding format is not UTF-8, convert the file to UTF-8 encoding format after loading the file content.

[0011] It can be seen from the above technical solution that the encoding format can be automatically determined to facilitate correct parsing of the encoding and avoid garbled characters or file reading failures.

[0012] Optionally, when parsing the file content, the file is also cleaned of null values, including: Detecting null values; the null values ​​include empty strings, special marks, and blank characters; Fill empty values ​​according to data type; numeric data is filled with the column average or median, time data is filled with the previous or next valid value, and text data is filled with the default value; When the proportion of null values ​​in any column or row exceeds the preset value, the corresponding row or column is deleted and a missing mark is added.

[0013] It can be seen from the above technical solution that there may be missing values ​​(NaN) or abnormal values ​​(such as values ​​outside a reasonable range) in the CSV file. Data cleaning is automatically completed to facilitate the subsequent automatic generation of charts.

[0014] Optionally, generating a time series graph according to the parsed content includes: Read the time column and data column of the parsed content, and sort the data column after matching it to the time column; A time series graph is drawn with the time column as the X-axis and the data column as the Y-axis; the time series graph is displayed with a single Y-axis or a double Y-axis.

[0015] Optionally, the viewing operation on the time series graph in response to a user operation includes: Monitoring a user's click operation and acquiring data information associated with the click operation; the data information includes coordinates of the click position; According to the coordinates of the clicked position, searching for the nearest data point, and determining the time and value information of the clicked position; Displays the found data points.

[0016] Optionally, the merging and exporting of data from multiple CSV files includes: Store the loaded multiple CSV files into memory; When the column names are consistent and the data structure is the same, multiple CSV files are merged vertically; otherwise, multiple CSV files are merged horizontally; Export according to the data format selected by the user.

[0017] Optionally, the method further includes log file generation and analysis, including: Record key information based on different log levels and generate log files; The log file is analyzed to generate an execution report; the log file is analyzed manually, automatically by Python script or by log management tools, and the execution report statistics key information including the number of errors and execution time.

[0018] In a second aspect, the present invention provides a device for analyzing data of multiple CSV files, comprising: A file loading module, used to load multiple CSV files into the same interface, detect duplicates in all column names, and perform deduplication operations on the duplicates; A data parsing module, used to identify the delimiters in the CSV file and determine the encoding format, and parse the file content; The drawing module is used to generate time series graphs based on the parsed content; An interactive module, used for viewing the time series graph in response to a user operation; The merge and export module is used to merge and export data from multiple CSV files.

[0019] Optionally, the data parsing module is specifically used to: Read the contents of the default line and determine the frequency of occurrence of various predicted delimiters; Matching a plurality of patterns of the predicted delimiters based on regular expressions; The predicted delimiter with the highest frequency is used as a delimiter to parse the CSV file; if the parsing fails, the predicted delimiter with the next frequency is tried as a delimiter for parsing; If the predicted separator cannot be determined as the separator, a prompt for the user to manually select is popped up.

[0020] Optionally, the data parsing module is further used to: Get the detection result based on the encoding format of the file detected by the third-party library; According to the detection result, the file content is loaded in a corresponding encoding format; If the file encoding format is not UTF-8, convert the file to UTF-8 encoding format after loading the file content.

[0021] Optionally, the data parsing module is further used to perform null value cleaning processing on the file, specifically including: Detecting null values; the null values ​​include empty strings, special marks, and blank characters; Fill empty values ​​according to data type; numeric data is filled with the column average or median, time data is filled with the previous or next valid value, and text data is filled with the default value; When the proportion of null values ​​in any column or row exceeds the preset value, the corresponding row or column is deleted and a missing mark is added.

[0022] Optionally, the drawing module is specifically used to: Read the time column and data column of the parsed content, and sort the data column after matching it to the time column; A time series graph is drawn with the time column as the X-axis and the data column as the Y-axis; the time series graph is displayed with a single Y-axis or a double Y-axis.

[0023] Optionally, the interaction module is specifically used to: Monitoring a user's click operation and acquiring data information associated with the click operation; the data information includes coordinates of the click position; According to the coordinates of the clicked position, searching for the nearest data point, and determining the time and value information of the clicked position; Displays the found data points.

[0024] Optionally, the merging and exporting module is specifically used to: Store the loaded multiple CSV files into memory; When the column names are consistent and the data structure is the same, multiple CSV files are merged vertically; otherwise, multiple CSV files are merged horizontally; Export according to the data format selected by the user.

[0025] Optionally, a log module is also included, specifically used for: Record key information based on different log levels and generate log files; The log file is analyzed to generate an execution report; the log file is analyzed manually, automatically by Python script or by log management tools, and the execution report statistics key information including the number of errors and execution time.

[0026] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0027] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer program instructions stored thereon, which implement the steps of any of the above methods when executed by a processor.

[0028] By adopting the above technical solution, the present application has the following beneficial effects: The present invention provides a method for analyzing data of multiple CSV files, which can simultaneously load and process multiple CSV files into the same interface, and support users to uniformly filter, analyze or visualize the integrated data columns; The present invention can automatically identify delimiters in CSV files, making it easier to parse the file content; there may be missing values ​​(NaN) or abnormal values ​​(such as values ​​outside a reasonable range) in CSV files, and data cleaning is automatically completed, making it easier to automatically generate charts later. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0030] Figure 1 A flowchart of a method for parsing data of multiple CSV files provided by an embodiment of the present invention is shown; Figure 2 A schematic diagram of a dual Y-axis display provided by an embodiment of the present invention is shown; Figure 3 A schematic diagram showing detailed information displayed in a time series graph according to a user click operation provided by an embodiment of the present invention is shown; Figure 4 A structural block diagram of a device for parsing data of multiple CSV files provided by an embodiment of the present invention is shown; Figure 5 A structural block diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0031] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention. It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the invention belongs.

[0032] like Figure 1 As shown, in one embodiment, a method for parsing data of multiple CSV files is provided, including: S1. Load multiple CSV files into the same interface, detect duplicates in all column names, and deduplicate the duplicates.

[0033] Existing tools may support batch loading of multiple CSV files (such as through programming or data analysis tools), but still display the data of each file in a separate window or tab. This embodiment integrates the data of multiple CSV files in one interface and supports users to perform unified screening, analysis or visualization operations on the integrated data columns.

[0034] Normally, the first row of a CSV file is considered as the column names (header row). By default, the system reads the first row of data and parses it into column names.

[0035] To avoid duplicate column names, all column names are checked and duplicates are automatically detected. Duplicate columns are renamed by adding suffixes (such as _1, _2). This embodiment can use the duplicated() method in pandas or other custom logic to implement this function. If the column name conflict cannot be automatically resolved, a manual operation prompt may pop up to prompt the user to manually adjust the column name.

[0036] S2. Identify the delimiters in the CSV file and determine the encoding format, and parse the file content.

[0037] The delimiters of CSV files may be commas (,), tabs (\t), semicolons (;), pipes (|), etc. These differences usually depend on the software or regional specifications that generate the file. Therefore, in order to correctly parse the file content, the system needs to automatically identify the delimiters.

[0038] Step S2 includes step S210 of identifying the delimiter in the CSV file, specifically including the following steps: S211. Read the content of the default row to determine the frequency of occurrence of multiple predicted delimiters. Open the CSV file and read the first few rows (e.g., 10 rows), treat the above symbols that may be delimiters as predicted delimiters, and count the frequency of occurrence of each predicted delimiter.

[0039] S212. Match multiple predicted delimiter patterns based on regular expressions.

[0040] S213. Use the predicted delimiter with the highest frequency as the delimiter to parse the CSV file; if the parsing fails, try the predicted delimiter with the next frequency as the delimiter for parsing.

[0041] Parse the file using the most frequent predicted delimiter as the delimiter, and if parsing fails, try the next possible delimiter.

[0042] S214. If the predicted delimiter cannot be determined as the delimiter, a prompt for the user to manually select is popped up.

[0043] CSV files may use different encoding formats (such as UTF-8, ANSI, GBK, etc.). If the encoding is not parsed correctly, garbled characters or file reading failure may occur. To solve the above problem, step S2 also includes step S220 to determine the encoding format, which specifically includes: S221. Detect the encoding format of the file based on a third-party library (such as chardet or charset-normalizer) and obtain the detection result.

[0044] S222. According to the detection result, load the file content in the corresponding encoding format.

[0045] Currently common encoding formats include UTF-8, GBK, ISO-8859-1, etc. S223. If the file encoding format is not UTF-8, convert the file to UTF-8 encoding format after loading the file content. If the file encoding is not UTF-8, convert it to UTF-8 after loading to ensure data consistency in subsequent processing.

[0046] Step S2 may also adopt the default encoding format of using UTF-8 to read the file. After the UTF-8 loading fails, other common encoding formats such as GBK and ISO-8859-1 are tried. After the attempt fails, a prompt pops up for the user to automatically select the encoding format.

[0047] While executing step S2, step S230 is also included to perform null value cleaning on the file, and step S230 includes: S231. Detect null values; null values ​​may include empty strings ("" or no content), special markers (NULL, NA, N / A, etc.) and whitespace characters (such as spaces, tabs, \t, etc.). The system will treat these values ​​as "missing values".

[0048] S232. Fill empty values ​​according to data types; among them, numeric data is filled with the average or median of the column, time data is filled with the previous or next valid value, and text data is filled with a default value (such as "unknown").

[0049] S233. When the proportion of null values ​​in any column or row exceeds a preset value (such as 50%), delete the corresponding row or column and add a missing mark. The user can also adopt a conservative strategy in step S233, such as filling without deleting, to avoid losing valid data.

[0050] There may be missing values ​​(NaN) or abnormal values ​​(for example, values ​​outside a reasonable range) in a CSV file. For other tools such as Excel, manual operations or scripts must be relied upon. This step automatically completes data cleaning, facilitating the subsequent automatic generation of charts.

[0051] S3. Generate a time series graph based on the analyzed content.

[0052] Specifically, step S3 includes: S301. Read the time column and data column of the parsed content, and sort the data column after matching it to the time column. Before this, ensure that the time column has been correctly parsed into a standardized time format (such as datetime).

[0053] S302. Draw a time series graph with the time column as the X-axis and the data column as the Y-axis; the time series graph is displayed with a single Y-axis or a double Y-axis.

[0054] Step S302 uses a drawing library (such as Matplotlib, Seaborn or Plotly) to draw a line graph or other styles with the time column as the X-axis and the data column as the Y-axis. Figure 2 The time series graph shown with dual Y axes also specifically performs the following annotations: obtaining the maximum, minimum or unknown mean of the data column and annotating it on the time series graph.

[0055] Since CSV files are in plain text format, there is no metadata describing the file content, and the data has no clear structural identification, it is impossible to determine which columns are suitable for dual Y-axis data. Therefore, the following information needs to be parsed when displaying dual Y-axis: Column data type and meaning: Automatically identify numeric data columns to avoid drawing errors caused by selecting non-numeric columns (such as text columns); Dimensions and ranges of column data: Determine which columns have different dimensions (such as voltage vs. current) and need to be mapped to different Y axes; User interaction: Provide an interface for users to easily select data columns on the left and right Y axes.

[0056] Compared with Excel files, there are the following differences: On the one hand, in CSV files, it is impossible to quickly locate the required columns through table metadata like in Excel files. The method provided in this embodiment automatically identifies column attributes through step S3, which is convenient for subsequent chart generation. On the other hand, Excel generates charts in a static operation mode, while this step S3 realizes the instantaneous update and generation of charts, and automatically sets the appropriate Y-axis scale and unit range for each column to prevent the data from being too large or too small and affecting the chart display.

[0057] The plain text format of the CSV file is an obstacle to steps S2 and S3. Based on S2 and S3 in this embodiment, the unstructured characteristics of the CSV file data are solved through intelligent processing logic, and the column name extraction, duplicate column name removal, missing value processing, data type identification and other tasks are automatically completed, which facilitates the drawing operation of step S3.

[0058] S4. In response to the user operation, the time series graph is viewed.

[0059] S401. Monitor the user's click operation and obtain data information associated with the click operation; the data information includes the coordinates of the click position; The monitoring can rely on the event monitoring function provided by the chart library (such as Matplotlib, Plotly, etc.) to monitor the user's click behavior; or rely on the mp_connect method provided by Matplotlib to capture the user's mouse click events, such as using matplotlib.backend_bases.MouseEvent to obtain the click position data.

[0060] S402. Find the nearest data point according to the coordinates of the clicked position, and determine the time and value information of the clicked position.

[0061] The event object of the click event range contains xdata and ydata properties, which represent the unknown X and Y axis coordinates of the click respectively. Based on these coordinates, the system needs to find the corresponding data point and extract relevant information. If the coordinates clicked by the user are (x, y), we can locate the corresponding value by finding the nearest data point. For example, for a time series graph, the clicked position x can be mapped to the time column, and y can be mapped to the corresponding data column (such as voltage or current).

[0062] S403. Display the found data points.

[0063] like Figure 3 As shown in the figure, once the corresponding data point is determined, the system will pop up a prompt box or directly display detailed information (such as time, value, unit, etc.) in the time series graph. For example, the detailed information can be displayed through the annotate() method of Matplotlib or other libraries (such as Tkinter or PyQt).

[0064] Steps S401-S403 specifically illustrate data display based on the user's click operation. However, data trend viewing, local zooming and data point annotation can also be achieved by hovering over the data. At the same time, the chart style (line color, line width, coordinate axis range, etc.) can be adjusted in real time according to user needs.

[0065] Since CSV files have higher flexibility and scalability and support more customization and automated processing, the method provided in this embodiment is particularly suitable for scenarios with large data volumes or the need to frequently process new data sources.

[0066] S5. Merge and export data from multiple CSV files.

[0067] S501. Store the loaded multiple CSV files into memory. Generally, libraries such as pandas are used to process data files.

[0068] S502. When the column names are consistent and the data structures are the same, multiple CSV files are merged vertically; otherwise, multiple CSV files are merged horizontally.

[0069] When merging, you need to specify the columns in the files that need to be merged (usually common column names). Common merging methods include horizontal merging (merging based on the content of the same column) and vertical merging (joining up and down). If the column names are consistent and the data structure is the same, you can directly merge vertically, that is, merge multiple files by row. If the column names are inconsistent or the data structure is different, you can merge horizontally, that is, connect based on the common columns.

[0070] Multiple CSV files will be merged according to the specified column names to generate a unified data set.

[0071] S503. Export according to the data format selected by the user.

[0072] This step supports exporting data from CSV files to formats such as Excel or JSON, mainly involving data format conversion and data adaptation. Although CSV, Excel, and JSON are all common data storage formats, there are certain differences in their data representation, so certain processing is required during conversion.

[0073] CSV to Excel conversion: A. Save CSV data as Excel file using pandas.to_excel() method. Excel format supports more styles, cell formatting, formulas, etc., which can be further customized through openpyxl or xlsxwriter library.

[0074] CSV to JSON conversion: Use pandas.to_json() method to convert CSV data to JSON format. JSON format has higher flexibility and is suitable for representing complex nested data structures. By setting different orient parameters, you can flexibly control the output JSON structure.

[0075] The method provided in this embodiment further includes step S6 of log file generation and analysis, and step S6 specifically includes: S601. Record key information based on different log levels and generate log files. Use the Python logging library to record key information during program execution at different log levels (INFO, ERROR, etc.) and save the logs to files. If the log file content is too large, you can configure the log file to roll and archive to prevent a single file from being too large and affecting performance.

[0076] S602. Analyze the log file and generate an execution report; analyze the log file manually, automatically with Python scripts, or with log management tools. The execution report statistics key information including the number of errors and execution time.

[0077] Log file analysis can be done manually, automatically through Python scripts, or by using professional log management tools (such as Logstash+Kibana, Splunk, etc.) for monitoring and visualization. Log files can be used to automatically generate execution reports and count key information (such as the number of errors, execution time, etc.) to help users quickly review and trace the source.

[0078] In one embodiment, Figure 4 As shown, a plurality of CSV file data analysis devices 70 are provided, including: The file loading module 701 is used to load multiple CSV files into the same interface, detect duplicate items in all column names, and perform deduplication operations on the duplicate items; The data parsing module 702 is used to identify the delimiters in the CSV file and determine the encoding format, and parse the file content; A drawing module 703, used to generate a time series graph according to the parsed content; An interactive module 704 is used to view the time series graph in response to a user operation; The merging and exporting module 705 is used to merge and export the data of multiple CSV files.

[0079] The multiple CSV file data analysis device 20 provided in the embodiment of the present application adopts the same inventive concept as the above-mentioned multiple CSV file data analysis method and can achieve the same beneficial effects, which will not be repeated here.

[0080] Optionally, the data parsing module is specifically used to: Read the contents of the default line and determine the frequency of occurrence of various predicted delimiters; Matching a plurality of patterns of the predicted delimiters based on regular expressions; The predicted delimiter with the highest frequency is used as a delimiter to parse the CSV file; if the parsing fails, the predicted delimiter with the next frequency is tried as a delimiter for parsing; If the predicted separator cannot be determined as the separator, a prompt for the user to manually select is popped up.

[0081] Optionally, the data parsing module is further used to: Get the detection result based on the encoding format of the file detected by the third-party library; According to the detection result, the file content is loaded in a corresponding encoding format; If the file encoding format is not UTF-8, convert the file to UTF-8 encoding format after loading the file content.

[0082] Optionally, the data parsing module is further used to perform null value cleaning processing on the file, specifically including: Detecting null values; the null values ​​include empty strings, special marks, and blank characters; Fill empty values ​​according to data type; numeric data is filled with the column average or median, time data is filled with the previous or next valid value, and text data is filled with the default value; When the proportion of null values ​​in any column or row exceeds the preset value, the corresponding row or column is deleted and a missing mark is added.

[0083] Optionally, the drawing module is specifically used to: Read the time column and data column of the parsed content, and sort the data column after matching it to the time column; A time series graph is drawn with the time column as the X-axis and the data column as the Y-axis; the time series graph is displayed with a single Y-axis or a double Y-axis.

[0084] Optionally, the interaction module is specifically used to: Monitoring a user's click operation and acquiring data information associated with the click operation; the data information includes coordinates of the click position; According to the coordinates of the clicked position, searching for the nearest data point, and determining the time and value information of the clicked position; Displays the found data points.

[0085] Optionally, the merging and exporting module is specifically used to: Store the loaded multiple CSV files into memory; When the column names are consistent and the data structure is the same, multiple CSV files are merged vertically; otherwise, multiple CSV files are merged horizontally; Export according to the data format selected by the user.

[0086] Optionally, a log module is also included, specifically used for: Record key information based on different log levels and generate log files; The log file is analyzed to generate an execution report; the log file is analyzed manually, automatically by Python script or by log management tools, and the execution report statistics key information including the number of errors and execution time.

[0087] Based on the same inventive concept as the above-mentioned method for analyzing data of multiple CSV files, the embodiment of the present application further provides an electronic device 80, such as Figure 5 As shown, the electronic device 80 may include a processor 801 and a memory 802 .

[0088] The processor 801 may be a general-purpose processor that can implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of the present invention. The steps of the methods disclosed in the embodiments of the present invention may be directly implemented as being executed by a hardware processor, or may be implemented as being executed by a combination of hardware and software modules in the processor.

[0089] The memory 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. The memory is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 802 in the embodiment of the present invention can also be a circuit or any other device that can realize a storage function, and is used to store program instructions and / or data.

[0090] An embodiment of the present invention provides a computer-readable storage medium for storing computer program instructions used by the above-mentioned electronic device, which includes a program for executing the above-mentioned multiple CSV file data analysis method.

[0091] The above embodiments are only used to introduce the technical solutions of the present application in detail, but the description of the above embodiments is only used to help understand the methods of the embodiments of the present invention and should not be understood as limiting the embodiments of the present invention. Any changes or substitutions that can be easily thought of by those skilled in the art should be included in the protection scope of the embodiments of the present invention.

Claims

1. A method for analyzing data of multiple CSV files, characterized in that: include: Load multiple CSV files into the same interface, detect duplicates in all column names, and deduplicate the duplicates; Identify the delimiter in the CSV file and determine the encoding format, and parse the file content; Generate a time series graph based on the analyzed content; In response to a user operation, viewing the time series graph; Merge and export data from multiple CSV files.

2. The method according to claim 1, characterized in that The identifying the delimiter in the CSV file includes: Read the contents of the default line and determine the frequency of occurrence of various predicted delimiters; Matching a plurality of patterns of the predicted delimiters based on regular expressions; The predicted delimiter with the highest frequency is used as a delimiter to parse the CSV file; if the parsing fails, the predicted delimiter with the next frequency is tried as a delimiter for parsing; If the predicted separator cannot be determined as the separator, a prompt for the user to manually select is popped up.

3. The method according to claim 2, characterized in that The determining of the encoding format comprises: Get the detection result based on the encoding format of the file detected by the third-party library; According to the detection result, the file content is loaded in a corresponding encoding format; If the file encoding format is not UTF-8, convert the file to UTF-8 encoding format after loading the file content.

4. The method according to claim 3, characterized in that: When parsing the file content, the file is also cleaned of null values, including: Detecting null values; the null values ​​include empty strings, special marks, and blank characters; Fill empty values ​​according to data type; numeric data is filled with the column average or median, time data is filled with the previous or next valid value, and text data is filled with the default value; When the proportion of null values ​​in any column or row exceeds the preset value, the corresponding row or column is deleted and a missing mark is added.

5. The method according to claim 2, characterized in that: The step of generating a time series graph according to the parsed content includes: Read the time column and data column of the parsed content, and sort the data column after matching it to the time column; A time series graph is drawn with the time column as the X-axis and the data column as the Y-axis; the time series graph is displayed with a single Y-axis or a double Y-axis.

6. The method according to claim 5, characterized in that The step of viewing the time series graph in response to a user operation includes: Monitoring a user's click operation and acquiring data information associated with the click operation; the data information includes coordinates of the click position; According to the coordinates of the clicked position, searching for the nearest data point, and determining the time and value information of the clicked position; Displays the found data points.

7. The method according to claim 6, characterized in that The merging and exporting of data from multiple CSV files includes: Store the loaded multiple CSV files into memory; When the column names are consistent and the data structure is the same, multiple CSV files are merged vertically; otherwise, multiple CSV files are merged horizontally; Export according to the data format selected by the user.

8. The method according to claim 6, characterized in that The method also includes log file generation and analysis, including: Record key information based on different log levels and generate log files; The log file is analyzed to generate an execution report; the log file is analyzed manually, automatically by Python script or by log management tools, and the execution report statistics key information including the number of errors and execution time.

9. A device for analyzing data of multiple CSV files, characterized in that: include: A file loading module, used to load multiple CSV files into the same interface, detect duplicates in all column names, and perform deduplication operations on the duplicates; A data parsing module, used to identify the delimiters in the CSV file and determine the encoding format, and parse the file content; The drawing module is used to generate time series graphs based on the parsed content; An interactive module, used for viewing the time series graph in response to a user operation; The merge and export module is used to merge and export data from multiple CSV files.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images