Text file processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a text file processing method and device, electronic equipment and a storage medium, and relates to the field of computer data processing.The method comprises the steps that a data processing requirement for a target text file is obtained, the data processing requirement is analyzed, and a data processing statement corresponding to the data processing requirement is obtained; determining a plurality of data processing plans corresponding to the data processing statement, determining data processing costs respectively corresponding to the plurality of data processing plans, determining a target data processing plan from the plurality of data processing plans according to the data processing costs, and determining a target data processing plan from the plurality of data processing plans based on the target data processing plan. And performing data processing on the target text file corresponding to the data processing demand to obtain a data processing result. By analyzing and optimizing the user data processing demand and adopting the efficient data processing plan to access the text file, the resource consumption is reduced, and the data processing performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer data processing, and in particular to a text file processing method and device, electronic equipment, and storage medium. Background Art

[0002] In today's big data era, text file formats such as CSV (Comma-Separated Values), Parquet, and JSON play a vital role in data storage and exchange. CSV files are known for their simplicity and wide compatibility, and are often used for data import, export, and data analysis. Parquet files are a columnar storage format designed for big data processing, providing efficient compression and query performance. JSON files are widely used in data exchange for Web services and APIs due to their flexible structure and easy human-readable features.

[0003] Although these text file formats have significant advantages in data storage and exchange, they also face some challenges in practical applications. The solutions of related technologies provide read and write operations for these text file formats, which solve the read and write problems of text file formats to a certain extent. However, when retrieving data, it is often necessary to traverse the entire file, resulting in low data processing capabilities. Especially in scenarios where data processing operations are frequent, the performance problem of data processing is more prominent. Summary of the invention

[0004] In view of the above problems, a text file processing method and device, electronic device, and storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] In a first aspect of the present invention, a text file processing method is provided, wherein the method comprises:

[0006] Obtaining data processing requirements for a target text file, and parsing the data processing requirements to obtain data processing statements corresponding to the data processing requirements;

[0007] Determine a plurality of data processing plans corresponding to the data processing statement, and determine data processing costs respectively corresponding to the plurality of data processing plans;

[0008] According to the data processing cost, a target data processing plan is determined from the multiple data processing plans, and based on the target data processing plan, data processing is performed on the target text file corresponding to the data processing requirement to obtain a data processing result.

[0009] Optionally, before obtaining the data processing requirements for the target text file, the method further includes:

[0010] Obtain a text file and a first index field set for the text file;

[0011] A first index is created for the text file based on the first index field set.

[0012] Optionally, the method further includes:

[0013] Obtaining historical data processing information and current index information of the text file at preset time intervals;

[0014] Determining a second index field set according to the historical data processing information, where the second index field set is used to represent historical commonly used fields;

[0015] Update target index information in the current index information that matches the second index field set.

[0016] Optionally, the determining a plurality of data processing plans corresponding to the data processing statement includes:

[0017] Determining whether the data processing statement needs to be rewritten;

[0018] When it is determined that the data processing statement needs to be rewritten, rewrite the data processing statement to obtain a rewritten data processing statement;

[0019] Determining target text file information corresponding to the data processing statement and target index information of the target text file information;

[0020] Based on the rewritten data processing statement and the target index information, multiple data processing plans corresponding to the data processing statement are determined.

[0021] Optionally, determining the data processing costs corresponding to the multiple data processing plans respectively includes:

[0022] Determine the execution time and system resource consumption cost corresponding to each of the data processing plans;

[0023] Determine the data processing costs respectively corresponding to the multiple data processing plans according to the execution time respectively corresponding to each of the data processing plans and the system resource consumption cost respectively corresponding to each of the data processing plans;

[0024] Determining a target data processing plan from the multiple data processing plans based on the data processing cost includes:

[0025] Determining a minimum data processing cost from the data processing costs respectively corresponding to the plurality of data processing plans;

[0026] The data processing plan corresponding to the minimum data processing cost is determined as the target data processing plan.

[0027] Optionally, after determining the target data processing plan from the multiple data processing plans, the method further comprises:

[0028] Determine target data corresponding to the target data processing plan, and obtain target data historical processing information for the target data;

[0029] Determining whether the target data processing plan satisfies the index adding strategy according to the target data historical processing information;

[0030] In the case where it is determined that the index adding strategy is satisfied, determining a third index field set based on the target data processing plan and the target data historical processing information;

[0031] Based on the third index field set, a third index is created for the target text file.

[0032] Optionally, obtaining data processing requirements for the target text file includes:

[0033] Obtain data processing requirements for target text files through a graphical data processing interface;

[0034] Or, through a command line interface, obtain data processing requirements for the target text file; wherein the graphical data processing interface includes a drag control; the drag control is used to drag the data processing information corresponding to the data processing requirements.

[0035] In a second aspect of the present invention, there is also provided a text file processing device, characterized in that it includes:

[0036] A first acquisition module is configured to acquire a data processing requirement for a target text file, and parse the data processing requirement to obtain a data processing statement corresponding to the data processing requirement;

[0037] A first determination module is configured to determine a plurality of data processing plans corresponding to the data processing statement, and determine data processing costs respectively corresponding to the plurality of data processing plans;

[0038] The first processing module is configured to determine a target data processing plan from the multiple data processing plans based on the data processing cost, and perform data processing on a target text file corresponding to the data processing requirement based on the target data processing plan to obtain a data processing result.

[0039] The embodiments of the present application also disclose a computer program product, including a computer program / instruction, which, when executed by a processor, implements a text file processing method as described in one or more of the embodiments of the present application.

[0040] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes a text file processing method as described in one or more of the embodiments of the present application.

[0041] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the text file processing method as described in one or more of the embodiments of the present application.

[0042] The embodiments of the present invention have the following advantages:

[0043] By obtaining the data processing requirements for the target text file and parsing the data processing requirements, the data processing statements corresponding to the data processing requirements are obtained, multiple data processing plans corresponding to the data processing statements are determined, and the data processing costs corresponding to the multiple data processing plans are determined. According to the data processing costs, a target data processing plan is determined from multiple data processing plans, and based on the target data processing plan, data processing is performed on the target text file corresponding to the data processing requirements to obtain the data processing results. By analyzing and optimizing the user's data processing requirements, an efficient data processing plan is used to access the text file, which reduces resource consumption and improves data processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0045] Figure 1 is a flowchart of a method for processing a text file provided by some embodiments of the present invention;

[0046] Figure 2 is a structural block diagram of a text file processing device provided by some embodiments of the present invention;

[0047] Figure 3 It is a code implementation framework diagram provided by some embodiments of the present invention;

[0048] Figure 4It is a sequence diagram between program modules provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Reference Figure 1 , shows a flowchart of a method for processing a text file provided by some embodiments of the present invention, which may specifically include the following steps:

[0051] Step 101: Obtain data processing requirements for a target text file, and parse the data processing requirements to obtain data processing statements corresponding to the data processing requirements.

[0052] Among them, the target text file can include Parquet, CSV and JSON files, and each file format has its specific storage structure and optimization method. For example, Parquet files are optimized for columnar storage. Parquet supports complex nested data structures such as arrays and nested dictionaries, which makes it very suitable for storing and processing structured data. In addition, Parquet files contain rich metadata that describes the structure and encoding method of the data, which helps to improve query efficiency and data parsing speed, and is suitable for big data analysis and data warehouses. CSV files use row storage, and each row of data is stored as a line of text. Fields are separated by commas or other delimiters. This storage method is simple and intuitive, easy to generate and parse. CSV files are in plain text format, easy to read and edit, and suitable for data exchange and import and export, but CSV files do not support complex nested data structures and are suitable for simple data exchange and import and export. JSON files use nested structures to store data and support complex nested data structures such as arrays, dictionaries and nested objects. They are very suitable for storing and processing structured data. JSON files are in plain text format, easy to read and edit, and suitable for data exchange and API responses. Therefore, JSON files have high flexibility and are suitable for complex data structures and dynamic data storage. In the specific implementation of the present invention, a unified API can be provided through the file system abstraction layer to process target text files of different formats, thereby hiding the underlying storage details of the text files, providing a unified query and operation interface for text files of different formats, simplifying the data access process, and reducing the user's learning cost.

[0053] The data processing requirements may be specifically data query requirements, data addition requirements, etc. The user's data query requirements may be obtained through a visual interface, and the user's data query requirements may also be obtained through a command line.

[0054] Specifically, the data processing demand can be an SQL statement, or a query demand, a new demand, an update demand, etc. that can be converted into an SQL statement. SQL (Structured Query Language) is a standard language for managing and operating relational databases. Standard SQL syntax includes common operations such as SELECT, INSERT, UPDATE, DELETE, JOIN, GROUP BY, ORDER BY, etc. The embodiment of the present invention uses an SQL engine to process data processing requirements. The SQL engine in the present application not only supports standard SQL syntax, but also extends the function to support operations specific to text file formats. That is to say, the SQL engine in the embodiment of the present application can not only process data in the database, but also process data stored in text files, and provide specific operations for these text file formats, so that users and developers with SQL knowledge background can quickly get started and use the processing method of the present invention to perform complex data processing. In a preferred embodiment of the present invention, the SQL engine can also support multiple SQL dialects or query languages, such as supporting Hive SQL, PrestoSQL, etc., to meet the needs of different users and data processing in different scenarios. The SQL engine can also allow users or developers to add custom functions and operators to the SQL engine to support data analysis needs of specific industries.

[0055] In the specific implementation, the SQL engine can process data stored in text files, such as Parquet, CSV, and JSON files, and can also process XML files, etc. These extended functions allow users to use SQL syntax to query and operate data in text files without importing the data into the database. Suppose we have a CSV file orders.csv, the SQL engine can support the following operations: 1. Read the CSV file and execute the query, SELECT*FROM orders.csv WHEREtotal_amount>100; In this query, the SQL engine will read the orders.csv file and perform the conditional filtering WHERE total_amount>100. For another example, the SQL engine supports the JSON file format. We can have a JSON file customers.json. The SQL engine can support the following operations: SELECT*FROM customers.json WHERE nameLIKE'J%'; In this query, the SQL engine will read the customers.json file and perform the conditional filtering WHERE name LIKE'J%'.

[0056] Furthermore, after obtaining the data processing requirements for the target text file, the data processing requirements can be parsed to obtain the data processing statements corresponding to the data processing requirements. For example, the user simply drags and clicks in the query interface, selects the condition "greater than 30 years old", the data type is "user information", and then submits the query requirement. It means that we have the following data processing requirements: the data source is a CSV file containing user information, named users.csv; read data from the users.csv file, filter out users over 30 years old, and display the results. The specific steps for parsing data processing requirements include: 1. Read data, which requires reading data from the CSV file. 2. Filter data: Users over 30 years old need to be filtered out. 3. Get the filtered data. This application will parse the user's query requirements and obtain the corresponding data query statement. The data query statement at this time can be "select*from users.csv where age>30".

[0057] In some embodiments of the present application, the following steps may be included before step 101:

[0058] S1: Acquire a text file and a first index field set for the text file;

[0059] Among them, the text file can be specifically CSV, parquet file, json file, XML (eXtensible Markup Language), Avro file, etc. Among them, XML is a markup language for storing and transmitting data, used to store structured data, and can be used for data exchange and configuration files. Avro file is a binary file format for storing and exchanging data. Compared with text formats (such as JSON and XML), it occupies less storage space and has faster reading and writing speed. Avro file contains the schema of the data, which describes the structure and type of the data, so that Avro file does not require additional metadata files when reading, and can directly parse the data. In addition, Avro supports a variety of data types, including basic data types (such as integers, strings, Boolean values) and complex data types (such as arrays, mappings, records). Avro files are widely used in the following scenarios: 1. Avro files are often used to store and process large-scale data sets; 2. Avro files can be used for data exchange between different systems, especially in scenarios where large amounts of data need to be efficiently transmitted and stored.

[0060] Index fields are usually one or more key fields in a text file. By creating indexes for these fields, the speed and efficiency of text search can be significantly improved, especially when processing large amounts of text data. Through the index, the system can quickly locate the data rows that meet the search criteria without scanning the entire text file. The index field can also be used to enforce uniqueness constraints to ensure that a certain field value in the text file is unique. Therefore, the embodiment of the present application can accelerate sorting and grouping operations by creating or regularly updating index fields, because the index can be directly used for sorting when performing data processing, without the need to sort the entire text file. In addition, the index field can reduce disk I / O operations because the system only needs to read the index file, not the entire text file.

[0061] S2: Creating a first index for the text file based on the first index field set.

[0062] From the above content, we can know that creating an index field for a text file means creating one or more fields in a text file, which can speed up data processing operations such as searching and querying text files. Specifically, the types of index fields can include multiple types: 1. Single-column index: an index that contains only one field; 2. Multi-column index (composite index): an index that contains multiple fields, which can be used to speed up queries involving multiple fields; 3. Unique index: ensure that the value of the index field is unique. 4. Full-text index: an index used to speed up text searches, usually used to process large amounts of text data. The specific implementation of creating index fields for text files can rely on specific data structures, such as B-trees, hash tables, inverted indexes, full-text indexes, etc.

[0063] Here are some common index implementations:

[0064] B-Tree: B-Tree is a self-balancing tree data structure suitable for range queries and sorting operations. B-tree indexes are usually used for primary key indexes and range queries.

[0065] Hash Table: A hash table is a data structure based on a hash function, which is suitable for equal value queries. Hash table indexes are usually used for unique indexes and equal value queries.

[0066] Inverted Index: An inverted index is a special type of index used to speed up text searches. Inverted indexes are commonly used in search engines and full-text indexes.

[0067] Full-text indexing: Full-text indexing is a technology used to accelerate text data queries. It preprocesses text data and builds indexes, allowing you to quickly locate documents or records containing specific keywords when you query. Full-text indexing usually includes preprocessing steps such as stemming, stop word filtering, and word segmentation to convert text data into an index structure that can be efficiently queried.

[0068] In a specific implementation, the first index field set for a text file can specifically be an index field actively selected by a user according to expected usage requirements when the text file is imported. The embodiment of the present invention can use an index builder to construct different types of indexes for text files using different data structures. For example, when a user imports a csv file, if the user expects a high frequency of querying the user ID, the user ID field can be actively selected to create a unique index, and then the unique index can be constructed through the index builder. It can be understood that the first index field set can be one or more, and the specific number is selected according to the actual needs of the user. When the first index field set is multiple, a multi-column index (composite index) can be established for the text file. The index construction method of the embodiment of the present application can significantly improve the data retrieval speed, especially when processing large-scale data sets, and can achieve fast response and efficient query.

[0069] In some embodiments of the present application, step 101 may include the following sub-steps:

[0070] Sub-step 11: Obtain data processing requirements for the target text file through a graphical data processing interface; or, obtain data processing requirements for the target text file through a command line interface; wherein the graphical data processing interface includes a drag control; the drag control is used to drag data processing information corresponding to the data processing requirements.

[0071] Among them, the graphical data processing interface refers to the visual way of data processing by dragging controls, clicking buttons, etc. The command line interface allows users to directly input commands and parameters, which is more flexible.

[0072] In a specific implementation, users can build data processing flows by dragging and dropping controls, and controls usually represent different data processing operations (such as reading data, filtering data, saving data, etc.). In addition, users can also intuitively see the data processing flow through a graphical interface, which is easy to understand and adjust. It can be understood that the graphical data processing interface is suitable for beginners and users who need to quickly build data processing flows. At the same time, in an embodiment of the present invention, users can also directly input data processing requirements through the command line, that is, users can automate data processing flows by writing scripts, and this method is suitable for batch processing and complex tasks. It can be understood that in the graphical data processing interface, users do not need to write code, and the operation is simple and intuitive; the command line interface requires users to be familiar with commands and parameters, and the learning curve is high, which is suitable for advanced users and users who need automated processing. In addition, through visualization tools, users can see the execution process and performance indicators of different query plans, so as to manually adjust the optimization strategy.

[0073] Step 102: Determine multiple data processing plans corresponding to the data processing statements, and determine the data processing costs corresponding to the multiple data processing plans respectively.

[0074] In a specific implementation, a data optimizer layer (SQL optimizer) can be constructed to determine multiple data processing plans corresponding to data processing statements. This process can be an automated and intelligent process aimed at selecting the optimal data processing strategy.

[0075] The data optimizer layer analyzes the structure and semantics of data processing statements, and generates multiple possible execution plans based on database statistics, index information, and system resources. It is understandable that each plan corresponds to a different data access path and operation sequence, with different execution times and resource consumption. Ultimately, the plan with the lowest cost is selected as the optimal execution strategy to improve data processing efficiency and performance.

[0076] In some embodiments of the present invention, step 102 may further include the following sub-steps:

[0077] Sub-step 21: Determine whether the data processing statement needs to be rewritten;

[0078] The purpose of determining whether data processing statements need to be rewritten is to improve performance, optimize query or processing logic through the optimizer layer, reduce resource consumption (such as CPU, memory, I / O), and shorten execution time. It can also simplify code logic to make it easier to understand and maintain, reduce maintenance costs, and improve development efficiency.

[0079] For example, if the optimizer layer receives the SQL statement "SELECT * FROM orders.csv WHERE customer_id = 123 AND order_date BETWEEN '2023-01-01' AND '2023-12-31';", the query may be slow to execute when querying a large text file because the existing orders table does not have a proper index. For another example, if the optimizer layer receives the SQL statement "select * from users.csv where age>35+3", if it is not rewritten, 35+3=38 must be calculated for each query of data, which will cause serious resource consumption when the data volume is large.

[0080] Sub-step 22: when it is determined that the data processing statement needs to be rewritten, rewrite the data processing statement to obtain a rewritten data processing statement;

[0081] For the example of sub-step 21, for the SQL statement "SELECT * FROM orders.csv WHERE customer_id = 123 AND order_date BETWEEN '2023-01-01' AND '2023-12-31';", a composite index can be created for the customer_id and order_date columns by rewriting, for example: CREATE INDEX idx_orders_customer_date ON orders(customer_id,order_date);

[0082] For the SQL statement "select * from users.csv where age>35+3", the SQL optimizer will identify and calculate the constant expression 35+3, and replace it with the calculation result 38. In other words, the original data processing statement is rewritten, and the rewritten statement is "SELECT * FROM users.csv WHERE age>38;".

[0083] Sub-step 23: determining the target text file information corresponding to the data processing statement and the target index information of the target text file information;

[0084] The target text file refers to the data source. For example, if the user needs to query order data, the data source (ie, the target text file) is orders.csv. If the user needs to query user data, the data source (ie, the target text file) is users.csv.

[0085] In the specific implementation, it is necessary to determine the target text file corresponding to the data processing statement or data query statement, and it is also necessary to determine the index information of the target text file. The index information of the target text file is also the target index information. For example, suppose we have the following data query statement: SELECT*FROM users.csv WHERE age>38, then the target text file is users.csv, and we need to understand the structure and content of this file. Assume that the users.csv file contains four columns: id, name, age, and email, and users.csv contains 10,000 pieces of data, that is, it contains user information of 10,000 different user identifiers. Further, determine that the index information corresponding to the text file is the mapping relationship between age and id.

[0086] Sub-step 24: Determine multiple data processing plans corresponding to the data processing statement based on the rewritten data processing statement and the target index information.

[0087] According to the rewritten data processing statement or data query statement and the index information of the target text file, multiple different data processing plans can be determined. The processing costs corresponding to the multiple different data processing plans may be different.

[0088] In a specific implementation, different data processing plans can be different data scanning ranges, such as full table scan or index scan; different data processing plans can also use different indexes for data processing, such as memory index or external index file. Among them, memory index is a technology that stores data index in memory, which is used to speed up data query. By maintaining an efficient index structure (such as hash table, B-tree, Trie tree, etc.) in memory, query operations can be performed directly in memory, avoiding disk I / O delays, thereby significantly improving query speed. Memory index is widely used in scenarios that require fast response and high concurrent queries. External index file is a file that stores index on disk, which is used to speed up data query. By maintaining an efficient index structure (such as B-tree, inverted index, etc.) on disk, query operations can quickly locate target data, reducing the time for scanning the entire text file. External index file is suitable for scenarios with large data volume and frequent queries. External index file can improve the efficiency of data query without occupying too much memory.

[0089] Assume that the optimizer generates execution plan 1 and execution plan 2. Execution plan 1 is a full table scan, which means that the entire users text file needs to be scanned, and then the records with age>38 are filtered. Execution plan 2 is an index scan, which uses the idx_age index to find records with age>38, reads the index page, finds the records that meet the conditions, and then reads the corresponding data page based on the records found by the index. Assume that the optimizer generates execution plan A and execution plan B. Execution plan A can use the memory index, and execution plan B can use the external index file.

[0090] In some embodiments of the present invention, step 102 may further include the following sub-steps:

[0091] Sub-step 31: determining the execution time and system resource consumption cost corresponding to each of the data processing plans;

[0092] The execution time refers to the total time required from the start of the data processing statement to the completion of all operations and the return of results, and the system resource consumption cost refers to the sum of various resources (such as CPU, memory, I / O, etc.) consumed by the system during the execution of the data processing statement. The consumption of these resources directly affects the performance and efficiency of the system.

[0093] After determining multiple data processing plans corresponding to the data processing requirements, it is necessary to determine the execution time and system resource consumption cost corresponding to each data processing plan.

[0094] Sub-step 32: determining the data processing costs respectively corresponding to the plurality of data processing plans according to the execution time respectively corresponding to each of the data processing plans and the system resource consumption cost respectively corresponding to each of the data processing plans;

[0095] The data processing cost includes the estimated execution time of the data processing plan and the system resource consumption cost.

[0096] In a specific implementation, the data processing cost may include the total time from the submission of a data query to the return of results, and may also include the memory usage, CPU occupancy, I / O resources, etc. consumed from the submission of a data query to the return of results.

[0097] For example, suppose the optimizer generates execution plan 1 and execution plan 2. Execution plan 1 is a full table scan, which means that the entire users text file needs to be scanned, and then the records with age>38 are filtered. Execution plan 2 is an index scan, which uses the idx_age index to find records with age>38, reads the index page, finds the records that meet the conditions, and then reads the corresponding data pages according to the records found by the index. Assume that the users table has 10,000 records, 100 records per page, and a total of 100 pages. The idx_age index covers 50% of the records. The cost of a full table scan includes: 1. I / O cost, reading 100 pages of data; 2. CPU cost: filtering 10,000 records. The cost of an index scan: 1. I / O cost: reading 50 pages of index data, reading 50 pages of table data (assuming that 50% of the records meet age>38); 2. CPU cost: filtering 5,000 records.

[0098] Specifically, in the above example, the I / O cost of the full table scan cost is to read 100 pages of data. Assuming that the I / O cost of each page is 1 unit, the I / O cost is 100 units. In the above example, the CPU cost of the full table scan cost is to filter 10,000 records and sort 5,000 records. Assuming that the CPU cost of filtering each record is 1 unit and the CPU cost of sorting each record is 1 unit, the CPU cost is: 10,000*1+5,000*1=15,000 units. In the above example, the memory cost of the full table scan cost is to sort 5,000 records. Assuming that the memory cost of sorting each record is 1 unit, the memory cost is: 5,000*1=5,000 units. Therefore, the total cost of the full table scan is I / O cost + CPU cost + memory cost: 100+15,000+5,000=20,100 units. In the above example, the I / O cost of the index scan cost is to read 50 pages of index data and 50 pages of table data. Assuming that the I / O cost of each page is 1 unit, the I / O cost is: 50+50=100 units; the CPU cost of the index scan cost in the above example is to filter 5000 records and sort 5000 records. Assuming that the CPU cost of filtering each record is 1 unit and the CPU cost of sorting each record is 1 unit, the CPU cost is: 5000*1+5000*1=10000 units; the memory cost of the index scan cost in the above example is to sort 5000 records. Assuming that the memory cost of sorting each record is 1 unit, the memory cost is: 5000*1=5000 units. Therefore, the total cost of the index scan is I / O cost + CPU cost + memory cost, 100+10000+5000=15100 units. That is, the total cost of the full table scan is 20100 units, and the total cost of the index scan is 15100 units.

[0099] For example, suppose the optimizer generates execution plan A and execution plan B. Execution plan A uses the memory index, and execution plan B uses the external index file. Execution plan A, a data processing plan using the memory index, first queries the memory index: finds the id list corresponding to age>38, and reads the corresponding row from the users.csv file based on the id list. The execution time corresponding to the query memory index is very fast, with a time complexity of O(1). Assuming that the users.csv file has N rows and the target number of rows is M, the time complexity of reading the file is O(N), but the actual number of target rows read is O(M). The system resource consumption cost includes: 1. Assuming the index size is K entries, each entry occupies C bytes, then the memory consumption is K*C bytes. CPU: Memory search and file reading operations consume less CPU resources. I / O: Read the users.csv file once, with fewer I / O operations. That is to say, the execution time in the data processing cost corresponding to the memory index is O(M), and it mainly depends on the target number of rows. The system resource consumption cost has higher memory consumption and fewer I / O operations. Execution plan B, data processing plan using external index file, read external index file: find the id list corresponding to age>38, and read the corresponding row from users.csv file according to the id list. The execution time includes the time to read the external index file and the time to read the target row of users.csv file. Assuming that the external index file has K rows, the time complexity of reading the file is O(K). Assuming that the users.csv file has N rows and the target number of rows is M, the time complexity of reading the file is O(N), but the actual number of target rows read is O(M). In the system resource consumption cost, since there is no need to maintain the memory index, the memory consumption is low, and the CPU resources consumed by file reading and search operations are low. The I / O consumption is to read the users_age_index.csv file once and read the users.csv file once, and the I / O operations are relatively more. Summary: Execution time: O(K+M), mainly depends on the size of the external index file and the target number of rows. System resource consumption cost: low memory consumption and more I / O operations.

[0100] In a preferred embodiment of the present invention, a machine learning algorithm may also be introduced to learn the user's data processing requirements, and automatically adjust the index structure and query plan to achieve more intelligent query optimization.

[0101] Step 103: According to the data processing cost, a target data processing plan is determined from multiple data processing plans, and based on the target data processing plan, data processing is performed on the target text file corresponding to the data processing requirement to obtain a data processing result.

[0102] First, it is necessary to evaluate the execution time and system resource consumption cost of each data processing plan in detail, which includes analyzing the consumption of resources such as CPU, memory, I / O, and the execution time complexity of each plan. Secondly, based on the evaluation results, select a plan that is optimal in terms of execution time and system resource consumption as the target data processing plan. In an optional embodiment of the present invention, the target data processing plan can be determined based on the specific application scenario and resource constraints, and finally the determined target data processing plan is used to perform data processing on the target text file, which includes reading index information, filtering data, and performing necessary calculations and operations to obtain the required data processing results. For data in cache and memory, in-memory computing technology can also be used to perform data processing on the target text file corresponding to the data processing requirements to obtain data processing results, such as using tools such as Apache Spark to increase the data processing speed.

[0103] In some embodiments of the present invention, step 103 may further include the following sub-steps:

[0104] Sub-step 41: determining a minimum data processing cost from the data processing costs corresponding to the plurality of data processing plans;

[0105] In a specific implementation, the target data processing plan can be determined based on the specific application scenario and resource constraints. For example, if the memory resources are sufficient and the I / O operations are few, the memory index plan can be selected; if the memory resources are limited and the I / O operations are many, the external index file plan can be selected. That is, according to the specific application scenario and the current resource constraints, the minimum data processing cost that meets the current application scenario and resource constraints is determined from the data processing costs corresponding to the multiple data processing plans.

[0106] In other embodiments of the present invention, a cost model can also be used to evaluate the efficiency of different query plans and select the optimal execution strategy. The cost model consists of the following key components: execution time, CPU consumption, memory consumption, and I / O consumption. Different query plans can be quantitatively evaluated through the cost model, and the optimal execution strategy can be selected. The specific steps may include: first, the resource consumption such as execution time, CPU consumption, memory consumption and I / O consumption are integrated into a cost function. For example, the cost function can be expressed as: cost = execution time + α*CPU consumption + β*memory consumption + γ*I / O consumption, where α, β, and γ are weight coefficients for adjusting the relative importance of different resource consumption. Secondly, for each query plan, its cost function value is calculated. The lower the cost function value, the higher the efficiency of the query plan. Finally, the cost function values ​​of different query plans are compared, and the query plan with the lowest cost is selected as the optimal execution strategy. For example, suppose there are two query plans: Plan A, the corresponding execution time is 100ms, CPU consumption is 50 units, memory consumption is 200MB, and I / O consumption is 100 units; Plan B, the corresponding execution time is 150ms, CPU consumption is 30 units, memory consumption is 100MB, and I / O consumption is 200 units. Assuming the weight coefficients α=1, β=0.01, γ=1, the cost function is: cost=execution time+α*CPU consumption+β*memory consumption+γ*I / O consumption. Calculate the cost of each plan: Plan A cost=100+1*50+0.01*200+1*100=252, Plan B cost=150+1*30+0.01*100+1*200=381, compare the costs: Plan A cost is 252, Plan B cost is 381; therefore, select Plan A as the optimal execution strategy. The cost model can be used to systematically evaluate the efficiency of different query plans and select the optimal execution strategy. The cost model comprehensively considers multiple factors such as execution time, CPU consumption, memory consumption, and I / O consumption, helping to maximize the efficiency and performance of data processing under limited resources. The cost model is used to deeply analyze and optimize queries, generate efficient query execution plans, reduce resource consumption, and improve query performance.

[0107] Sub-step 42: Determine the data processing plan corresponding to the minimum data processing cost as the target data processing plan.

[0108] After determining the minimum data processing cost that meets the current application scenario and resource constraints, the corresponding data processing plan can be determined and determined as the target data processing plan for data processing the target file corresponding to the data processing requirement to obtain the data processing result. In the above examples of execution plan 1 and execution plan 2, index scan can be selected as the optimal execution plan based on the data processing cost because it has a lower cost.

[0109] In some embodiments of the present invention, after "determining a target data processing plan from the multiple data processing plans" in step 103, the following sub-steps may also be included:

[0110] Sub-step 51: determining target data corresponding to the target data processing plan, and obtaining target data historical processing information for the target data;

[0111] After the target data plan is determined, the current index information may be updated or a new index field may be added to the current index information according to the target data corresponding to the target data plan and the historical processing information.

[0112] Sub-step 52: determining whether the target data processing plan satisfies the index adding strategy according to the target data historical processing information;

[0113] For example, after the target query plan is determined, it is found that there is no pre-established index on the field to be queried. At this time, it can be determined whether a new index is needed based on the user's historical processing information. Assume that the preset index addition strategy is to query the same field more than 10 times or the same field more than 100 times. Then after the target query plan is determined, it is determined based on the historical processing information of the target data that the same field is queried 100 times, and in this operation, it is determined that the target data processing plan meets the index addition operation.

[0114] Sub-step 53: when it is determined that the index adding strategy is satisfied, determining a third index field set based on the target data processing plan and the target data historical processing information;

[0115] When it is determined that the index addition strategy is met, it indicates that the corresponding field query frequency is high, and indexing the field can improve the subsequent query efficiency. Therefore, when it is determined that the index addition strategy is met, the third index field set is determined based on the target data processing plan and the target data historical processing information. It can be understood that the third index field set is a field set with a high query frequency, and the third index field set can be one or more.

[0116] Sub-step 54: Creating a third index for the target text file based on the third index field set.

[0117] After the third index field set is determined, a third index may be created for the target text file to improve the efficiency of this query and future queries.

[0118] In the specific implementation, it can be achieved by Figure 3 The code implementation of the text file processing method is implemented in this way. Figure 3A code implementation framework diagram provided for some embodiments of the present invention, wherein User is the end user of the system, GraphicalQueryBuilder, CommandLineInterface and API (Application ProgrammingInterface) belong to UserInteractionLayer, which are interfaces for user interaction with the system, SQLEngine and SQLOptimizer belong to SQLEngineLayer, which are responsible for parsing, optimizing and executing SQL queries, IndexBuilder and IndexesStatistics belong to IndexingAndOptimizationLayer, which are responsible for index creation, maintenance and storage of statistical information, and FileSystemAbstractionLayer belongs to DataStorageLayer, which provides a unified API to access files of different formats, such as ParquetFiles, CSVFiles and JSONFiles. As can be seen from the figure, the code implementation method of the present invention adopts a layered architecture design, and each layer is optimized for specific functions, which is convenient for separate updating and maintenance, thereby improving the scalability and maintainability of the system. Figure 4 As shown, it is a sequence diagram between program modules provided by some embodiments of the present invention. The user submits queries or commands through a graphical query builder, a command line interface or an API. These queries or commands are sent to the SQL engine (SQLEngineLayer). The SQL engine parses them and generates a preliminary query plan. The optimizer layer (SQL Optimizer) receives the query plan and uses indexes and statistics to optimize the query and generate a more efficient execution strategy. The index builder (IndexBuilder and IndexesStatistics) adds or updates indexes according to the optimizer's instructions and interacts with the index and statistics database to ensure that the index is up to date. The file system abstraction layer (DataStorageLayer) accesses the corresponding Parquet, CSV or JSON files according to the optimized query plan, performs data processing, and returns the data processing results to the user.

[0119] The embodiment of the present invention obtains the data processing requirements for the target text file, parses the data processing requirements, obtains the data processing statements corresponding to the data processing requirements, determines multiple data processing plans corresponding to the data processing statements, and determines the data processing costs corresponding to the multiple data processing plans respectively, determines the target data processing plan from the multiple data processing plans according to the data processing costs, and performs data processing on the target text file corresponding to the data processing requirements based on the target data processing plan to obtain the data processing result. By analyzing and optimizing the user's data processing requirements, an efficient data processing plan is used to access the text file, thereby reducing resource consumption and improving data processing performance.

[0120] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0121] Reference Figure 4 , shows a schematic diagram of the structure of a text file processing device provided by an embodiment of the present invention, which may specifically include the following modules:

[0122] The first acquisition module 401 is configured to acquire data processing requirements for a target text file, and parse the data processing requirements to obtain data processing statements corresponding to the data processing requirements;

[0123] A first determination module 402 is configured to determine a plurality of data processing plans corresponding to the data processing statement, and determine data processing costs respectively corresponding to the plurality of data processing plans;

[0124] The first processing module 403 is configured to determine a target data processing plan from the multiple data processing plans according to the data processing cost, and perform data processing on the target text file corresponding to the data processing requirement based on the target data processing plan to obtain a data processing result.

[0125] In an optional embodiment of the present invention, the device further comprises:

[0126] An index acquisition module, configured to acquire a text file and a first index field set for the text file;

[0127] The first index creation module is configured to create a first index for the text file based on the first index field set.

[0128] In an optional embodiment of the present invention, the device further comprises:

[0129] A first updating module is configured to obtain historical data processing information and current index information of the text file at a preset time interval;

[0130] A second updating module is configured to determine a second index field set according to the historical data processing information, wherein the second index field set is used to represent historical commonly used fields;

[0131] The third updating module is configured to update the target index information in the current index information that matches the second index field set.

[0132] In an optional embodiment of the present invention, the first determining module 402 includes:

[0133] A first determination submodule is configured to determine whether the data processing statement needs to be rewritten;

[0134] A first rewriting submodule is configured to rewrite the data processing statement to obtain a rewritten data processing statement when it is determined that the data processing statement needs to be rewritten;

[0135] A second determination submodule is configured to determine target text file information corresponding to the data processing statement and target index information of the target text file information;

[0136] The plan determination submodule is configured to determine a plurality of data processing plans corresponding to the data processing statement based on the rewritten data processing statement and the target index information.

[0137] In an optional embodiment of the present invention, the first determining module 402 includes:

[0138] A consumption determination submodule is configured to determine the execution time and system resource consumption cost corresponding to each of the data processing plans;

[0139] The cost determination submodule is configured to determine the data processing costs respectively corresponding to the multiple data processing plans according to the execution time respectively corresponding to each of the data processing plans and the system resource consumption cost respectively corresponding to each of the data processing plans.

[0140] The first processing module 403 includes:

[0141] A minimum cost determination submodule is configured to determine a minimum data processing cost from the data processing costs respectively corresponding to the plurality of data processing plans;

[0142] The target processing plan determination submodule is configured to determine the data processing plan corresponding to the minimum data processing cost as the target data processing plan.

[0143] In an optional embodiment of the present invention, the first processing module 403 further includes:

[0144] A third index initialization submodule is configured to determine target data corresponding to the target data processing plan and obtain target data historical processing information for the target data;

[0145] An index measurement determination submodule, configured to determine whether the target data processing plan satisfies an index adding strategy based on the target data historical processing information;

[0146] A third index determination submodule is configured to determine a third index field set based on the target data processing plan and the target data historical processing information when it is determined that the index adding strategy is satisfied;

[0147] The third index creation submodule is configured to create a third index for the target text file based on the third index field set.

[0148] In an optional embodiment of the present invention, the first acquisition module 401 further includes:

[0149] A graphical acquisition submodule is configured to acquire data processing requirements for a target text file through a graphical data processing interface;

[0150] Or, a command line acquisition submodule is configured to obtain data processing requirements for a target text file through a command line interface; wherein the graphical data processing interface includes a drag control; the drag control is used to drag data processing information corresponding to the data processing requirements.

[0151] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the above text file processing method is implemented.

[0152] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the text file processing method described above is implemented.

[0153] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0155] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0156] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0157] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0158] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0160] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0161] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0162] The above describes in detail the text file processing method and device, electronic device, and storage medium provided. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for processing a text file, characterized in that: The method comprises: Obtaining data processing requirements for a target text file, and parsing the data processing requirements to obtain data processing statements corresponding to the data processing requirements; Determine a plurality of data processing plans corresponding to the data processing statement, and determine data processing costs respectively corresponding to the plurality of data processing plans; According to the data processing cost, a target data processing plan is determined from the multiple data processing plans, and based on the target data processing plan, data processing is performed on the target text file corresponding to the data processing requirement to obtain a data processing result.

2. The method according to claim 1, characterized in that Before obtaining the data processing requirements for the target text file, the method further includes: Obtain a text file and a first index field set for the text file; A first index is created for the text file based on the first index field set.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining historical data processing information and current index information of the text file at preset time intervals; Determining a second index field set according to the historical data processing information, where the second index field set is used to represent historical commonly used fields; Update target index information in the current index information that matches the second index field set.

4. The method according to claim 1, characterized in that: The determining of a plurality of data processing plans corresponding to the data processing statements includes: Determining whether the data processing statement needs to be rewritten; When it is determined that the data processing statement needs to be rewritten, rewrite the data processing statement to obtain a rewritten data processing statement; Determining target text file information corresponding to the data processing statement and target index information of the target text file information; Based on the rewritten data processing statement and the target index information, multiple data processing plans corresponding to the data processing statement are determined.

5. The method according to claim 1, characterized in that The determining of the data processing costs respectively corresponding to the plurality of data processing plans includes: Determine the execution time and system resource consumption cost corresponding to each of the data processing plans; Determine the data processing costs respectively corresponding to the multiple data processing plans according to the execution time respectively corresponding to each of the data processing plans and the system resource consumption cost respectively corresponding to each of the data processing plans; Determining a target data processing plan from the multiple data processing plans based on the data processing cost includes: Determining a minimum data processing cost from the data processing costs respectively corresponding to the plurality of data processing plans; The data processing plan corresponding to the minimum data processing cost is determined as the target data processing plan.

6. The method according to claim 1, characterized in that After determining the target data processing plan from the multiple data processing plans, the method further comprises: Determine target data corresponding to the target data processing plan, and obtain target data historical processing information for the target data; Determining whether the target data processing plan satisfies the index adding strategy according to the target data historical processing information; In the case where it is determined that the index adding strategy is satisfied, determining a third index field set based on the target data processing plan and the target data historical processing information; Based on the third index field set, a third index is created for the target text file.

7. The method according to any one of claims 1 to 5, characterized in that: The obtaining of data processing requirements for the target text file includes: Obtain data processing requirements for target text files through a graphical data processing interface; Or, through a command line interface, obtain data processing requirements for the target text file; wherein the graphical data processing interface includes a drag control; the drag control is used to drag the data processing information corresponding to the data processing requirements.

8. A text file processing device, characterized in that: The device comprises: A first acquisition module is configured to acquire a data processing requirement for a target text file, and parse the data processing requirement to obtain a data processing statement corresponding to the data processing requirement; A first determination module is configured to determine a plurality of data processing plans corresponding to the data processing statement, and determine data processing costs respectively corresponding to the plurality of data processing plans; The first processing module is configured to determine a target data processing plan from the multiple data processing plans based on the data processing cost, and perform data processing on a target text file corresponding to the data processing requirement based on the target data processing plan to obtain a data processing result.

9. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the text file processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the text file processing method according to any one of claims 1 to 7 is implemented.

Images