Storage and calculation separated railway transportation income data analysis processing method and system

Abstract

The invention discloses a storage and calculation separated railway transportation income data analysis processing method and system, and belongs to the technical field of information processing. Converting the original railway transportation income data into a column type storage format file and outputting the file; executing a multi-thread concurrent reading and prefetching mechanism based on the output file; in the file data reading process, a smallpond rule engine is used for carrying out data quality verification; carrying out distributed storage and cold and hot hierarchical management on the data subjected to quality verification; and carrying out storage and calculation separation and elastic expansion based on the distributed storage and cold and hot hierarchical management. The method is used for solving the problems that an existing foreign cloud data warehouse technology is high in cost, poor in real-time performance and insufficient in adaptability; a foreign big data batch processing technology is complex in operation and maintenance and high in delay; a domestic traditional relational database is insufficient in expansibility and analysis capability; the construction and use cost of a domestic big data platform is high, and rule verification is not flexible.

Description

Technical Field

[0001] This invention belongs to the field of information processing technology, specifically relating to a method and system for analyzing and processing railway transportation revenue data with storage and computation separation. Background Technology

[0002] Currently, technologies for large-scale data processing and analysis have seen some development both domestically and internationally. Typical international solutions include cloud data warehouse products such as Google BigQuery, AWS Redshift, and Snowflake. These solutions rely on cloud computing platforms, enabling centralized storage of massive amounts of data and high-concurrency queries, making them suitable for enterprise-level data warehouses and complex OLAP analysis. However, their applications also have significant shortcomings: firstly, they depend on public cloud environments, and railway transportation revenue data involves strong data security and confidentiality requirements, making it difficult to meet the high-frequency, near-real-time processing requirements of transportation revenue data. Secondly, these systems have poor interface compatibility with existing railway information systems, making direct integration into existing business processes difficult.

[0003] In the field of big data processing abroad, technologies such as Apache Hive and Spark SQL are also widely used. Their characteristics include the ability to perform batch processing and offline analysis on large-scale datasets. However, these technologies are mainly geared towards offline tasks, lack real-time performance, and have large platform systems, complex deployment and maintenance, and high resource consumption. They are not suitable for business scenarios such as railway transportation revenue data that are continuous, high-frequency, and have high real-time requirements.

[0004] Domestically, traditional relational databases (such as Oracle, Sybase, and some domestically produced databases) have long been used in railway transportation revenue settlement and related processing systems, exhibiting advantages in transaction processing. However, these databases are primarily designed for storing structured data and performing transaction operations, lacking sufficient performance and scalability for batch analysis and large-scale historical data mining, making it difficult to support the rapid growth of transportation revenue data. Furthermore, data access from different sources is challenging, and data quality verification often relies on manual or decentralized implementation methods, lacking a unified, automated rule-driven mechanism.

[0005] In addition, some domestic big data platforms have attempted to introduce frameworks such as Hadoop and Hive for the storage and analysis of transportation revenue data. However, these platforms are complex to deploy, have high initial construction costs, require high system maintenance, and suffer from significant processing latency in batch processing mode, failing to meet the requirements for near real-time analysis of transportation revenue data. For the business side, the learning and usage threshold is high, and they lack flexibility in meeting the needs for customized rule verification for railway transportation revenue.

[0006] In summary, existing foreign cloud data warehouse technologies are characterized by high costs, poor real-time performance, and insufficient adaptability; foreign big data batch processing technologies are complex to operate and maintain and have high latency; domestic traditional relational databases lack scalability and analytical capabilities; and domestic big data platforms are costly to build and use and have inflexible rule validation. Summary of the Invention

[0007] This invention provides a storage-compute separation method and system for analyzing and processing railway transportation revenue data, which addresses the problems of high cost, poor real-time performance, and insufficient adaptability of existing foreign cloud data warehouse technologies; complex operation and maintenance and high latency of foreign big data batch processing technologies; insufficient scalability and analytical capabilities of domestic traditional relational databases; and high construction and use costs and inflexible rule verification of domestic big data platforms.

[0008] This invention is achieved through the following technical solution: A method for analyzing and processing railway transportation revenue data with separate storage and computation, the method comprising the following steps: Step 1: Convert the raw railway transportation revenue data into a columnar storage format file for output; Step 2: Execute a multi-threaded concurrent read and prefetch mechanism based on the output file of Step 1; Step 3: During the file data reading process in Step 2, use the Smallpond rule engine to perform data quality verification; Step 4: Utilize distributed storage and tiered hot / cold storage management for the data after quality verification in Step 3. Step 5: Based on the distributed storage and hot / cold tier management in Step 4, perform storage-compute separation and elastic scaling.

[0009] Furthermore, step 1 specifically involves using the Parquet format during the conversion process, combined with a block-based strategy and dictionary compression method; control parameters include block size and dictionary size, which will affect the conversion time. The approximate model is as follows:

[0010] in Indicates the size of the input file. Indicates I / O throughput bandwidth. This indicates the acceleration system resulting from compression and block optimization. Furthermore, the adoption of Parquet format, combined with a block strategy and dictionary compression method, specifically involves performing block strategy and dictionary compression sequentially, and then converting it into Parquet format.

[0011] Furthermore, step 2 specifically involves achieving multi-threaded concurrent reading through column pruning, predicate pushdown, and vectorized execution in sequence; the control parameters include the number of concurrent threads. Prefetch block size Cache hit rate Data read latency It can be approximated as: .

[0012] Furthermore, step 3 specifically involves introducing the Smallpond rule engine during the data reading process; employing structured representation and dynamically loading it at runtime; synchronously executing the verification of transportation revenue business rules during data loading; and controlling parameters including rule execution batch size and rule complexity coefficient; the verification model is defined as:

[0013] in Indicates the number of executions required for verification. Indicates the total number of records. Let be the rule complexity function.

[0014] Furthermore, step 4 specifically involves, at the storage level, using the 3FS distributed file system as the storage infrastructure, combined with a hot / cold tiering strategy and an automated backup mechanism. Hot data is stored on SSD nodes, while cold data is stored on HDD nodes or archive storage; control parameters include hot data window length and overall storage cost. With access performance It can be modeled as:

[0015] in This indicates the unit cost of thermal storage. This indicates the unit cost of cold storage. Indicates the total storage time range. and The range delays for hot and cold storage are respectively. Indicates the range ratio of thermal data; through reasonable adjustment This allows for a balance between storage costs and performance.

[0016] Furthermore, step 5 specifically involves controlling the number of computing nodes as a parameter. Task parallelism Number of storage nodes System processing capacity It can be modeled as:

[0017] in The storage access conflict coefficient reflects the I / O contention during concurrent access; it is adjusted... and Without increasing node pressure, it can significantly improve overall throughput.

[0018] A system for analyzing and processing railway transportation revenue data with a separation of storage and computation, the system using the aforementioned method for analyzing and processing railway transportation revenue data with a separation of storage and computation, the system comprising: Data conversion module: Converts raw railway transportation revenue data into a columnar storage format file for output; Data quality verification module: Based on the output file of the data conversion module, it performs multi-threaded concurrent reading and prefetching mechanism; The Smallpond rule engine is used to verify data quality during the file data reading process. Storage-compute separation processing module: This module uses distributed storage and tiered cold / hot data management to process the quality-verified data. Storage and computing are separated and elastically expanded based on distributed storage and hot / cold tiered management.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0020] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0021] The beneficial effects of this invention are: This invention utilizes columnar storage and online analytical databases (OLAP, such as DuckDB, ClickHouse, etc.) to achieve efficient conversion, fast reading, rule-based verification, and distributed management of large-scale railway transportation revenue data.

[0022] This invention breaks through the limitations of the traditional relational database's "coupling of computation and storage" by adopting a collaborative design of columnar storage and a lightweight OLAP engine, which allows data storage and computation to evolve independently, thereby effectively improving processing efficiency and system scalability.

[0023] The processing efficiency of this invention is greatly improved. In transportation revenue data files larger than 5GB, the conversion time is stabilized at around 30 seconds, and the verification and query response time can be controlled within 1 second.

[0024] This invention has good flexibility and scalability, and can increase computing resources as needed without affecting the stability of the storage layer to meet the processing needs during peak periods.

[0025] The execution of the business rules in this invention is more efficient and reliable, reducing the cost of manual review and secondary development.

[0026] This invention can provide near real-time query and analysis capabilities while ensuring the security and integrity of transportation revenue data, thus providing solid data support for railway transportation revenue management and decision-making. Attached Figure Description

[0027] Figure 1 This is the overall architecture diagram of the present invention.

[0028] Figure 2 This is a flowchart of the method of the present invention.

[0029] Figure 3 This is a schematic diagram of the rule-driven verification framework of the present invention. Detailed Implementation

[0030] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0031] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0032] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0033] The following is in conjunction with the appendix to this application specification. Figure 1-3 The technical solutions in the embodiments of this application are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0034] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0035] Implementation Method 1 This embodiment provides a method for analyzing and processing railway transportation revenue data with separate storage and computation, the method comprising the following steps: Step 1: Convert the raw railway transportation revenue data into a columnar storage format file for output; Step 2: Execute a multi-threaded concurrent read and prefetch mechanism based on the output file of Step 1; Step 3: During the file data reading process in Step 2, use the Smallpond rule engine to perform data quality verification; Step 4: Utilize distributed storage and tiered hot / cold storage management for the data after quality verification in Step 3. Step 5: Based on the distributed storage and hot / cold tier management in Step 4, perform storage-compute separation and elastic scaling; Furthermore, step 1 specifically involves efficiently converting the original railway transportation revenue data (usually a row-structured text file or CSV file) into a columnar storage format file. The conversion process uses Parquet format, combined with a block-based strategy and dictionary compression. Control parameters include block size (usually set to 64MB) and dictionary size (usually controlled at around 1MB to balance memory consumption). Conversion efficiency can be improved using built-in operators in the OLAP database, ensuring that single-file conversion events of 5GB or more are stably controlled within the range of 30-40 seconds. Mathematically, the conversion time can be... The approximate model is as follows:

[0036] in Indicates the size of the input file. Indicates I / O throughput bandwidth. This indicates the system acceleration brought about by compression and block optimization. 1).

[0037] Furthermore, the adoption of Parquet format, combined with a block strategy and dictionary compression method, specifically involves first performing block strategy and dictionary compression, and then converting it into Parquet format.

[0038] Specifically, chunking strategies and dictionary compression methods are existing approaches. These strategies and algorithms can be adjusted based on the specific data to shorten data conversion time. The purpose of data conversion is not only to standardize the format but also, and more importantly, to improve rule validation and file processing speed. Currently, a single data entry for railway transportation revenue can reach up to 5GB. Uploading this data to the server via a client and undergoing rule validation takes approximately 20 minutes, severely impacting user experience. Converting this data to Parquet format for columnar storage takes only about 30 seconds without data loss, and subsequent uploading and data validation are completed within seconds.

[0039] Furthermore, step 2 specifically involves improving performance by sequentially implementing column pruning (reading only the columns involved in the query), predicate pushdown (directly applying filtering conditions during reading), and vectorized execution (processing data in batch vector mode) to achieve multi-threaded concurrent reading. This reduces I / O input, reduces computation, and fully utilizes CPU parallel capabilities, thereby significantly improving the performance of big data queries. Control parameters include the number of concurrent threads. ( (usually adapted to the number of CPU cores) prefetch block size ( Recommended settings: 4MB-16MB; cache hit rate. ( (It needs to be kept above 80% to ensure performance); in the performance model, data read latency It can be approximated as:

[0040] in This indicates the actual amount of data read from the column. Indicates disk or storage bandwidth. This indicates the cache hit rate; with this optimization strategy, the reading and verification of 5-6GB files can be controlled within 1 second.

[0041] Furthermore, step 3 specifically involves introducing the Smallpond rule engine during the data reading process to implement business rule verification of transportation revenue data. This rule framework supports structured rule expression (JSON / YAML) and can be dynamically loaded at runtime. It enables synchronous execution of transportation revenue business rule verification during data loading, ensuring data quality assurance before entering the analysis process. The introduction of Smallpond significantly reduces manual intervention and redundant development costs, improving the universality and scalability of data verification. Control parameters include the rule execution batch size (…). The number of data rows to be validated in a single batch is determined (10k to 50k is recommended), and the rule complexity coefficient is also considered. (This improvement increases with the depth of expression nesting); the validation model can be defined as:

[0042] in Indicates the number of executions required for verification. Indicates the total number of records. Let be the rule complexity function; by optimizing batch processing and rule execution strategies, the complexity can be significantly reduced. This improves the overall verification speed.

[0043] Furthermore, step 4 specifically involves, at the storage level, the present invention adopts the 3FS distributed file system as the storage infrastructure, combined with a cold and hot tiering strategy and an automated backup mechanism. The 3FS architecture can improve storage throughput while maintaining data reliability, enabling historical and real-time data to coexist flexibly and meeting the performance requirements of transportation revenue data in a "high-frequency write - high-concurrency query" mode. Hot data (revenue data within the last 30 days) is stored on SSD nodes to ensure high-concurrency, low-latency access; cold data is stored on HDD nodes or archive storage to save costs. Control parameters include the hot data window length (…). (Typically 30 days, to ensure reliability); overall storage cost With access performance It can be modeled as:

[0044] in This indicates the unit cost of thermal storage. This indicates the unit cost of cold storage. Indicates the total storage time range. and The range delays for hot and cold storage are respectively. Indicates the range ratio of thermal data; through reasonable adjustment This allows for a balance between storage costs and performance.

[0045] Furthermore, step 5 specifically involves the present invention achieving elastic scaling of the system architecture through a "storage-compute separation" model. In this model, the storage layer (3FS file system) operates stably for a long period, responsible for providing highly available data access; the compute layer (OLAP engines such as DuckDB and ClickHouse) dynamically expands or shrinks according to task requirements, thereby achieving high availability and cost optimization of the system. Control parameters include the number of compute nodes. Task parallelism Number of storage nodes System processing capacity It can be modeled as:

[0046] in The storage access conflict coefficient reflects the I / O contention during concurrent access; it is adjusted... and Without increasing node pressure, it can significantly improve overall throughput.

[0047] The complete methodology for analyzing railway transportation revenue data can be abstracted into five stages: 1. Data conversion; 2. Concurrent reading; 3. Rule validation; 4. Distributed storage; 5. Storage-computation separation. These stages respectively achieve goals such as efficient format conversion, improved reading performance, controlled rule validation complexity, balancing cost and performance, and increased system throughput. The formulas for these five stages are not independent but rather progressive and mutually constraining, forming an overall performance delivery chain. → → → / → Data conversion efficiency affects read latency, read latency affects rule validation efficiency, rule validation load affects distributed storage design, and storage and access performance further determine the overall system throughput.

[0048] Implementation Method 2 This embodiment provides a railway transportation revenue data analysis and processing system with separate storage and computation. The system uses the railway transportation revenue data analysis and processing method with separate storage and computation as described in Embodiment 1. The system includes: Data conversion module: Converts raw railway transportation revenue data into a columnar storage format file for output; Data quality verification module: Based on the output file of the data conversion module, it performs multi-threaded concurrent reading and prefetching mechanism; The Smallpond rule engine is used to verify data quality during the file data reading process. Storage-compute separation processing module: This module uses distributed storage and tiered cold / hot data management to process the quality-verified data. Storage and computing are separated and elastically expanded based on distributed storage and hot / cold tiered management.

[0049] Implementation Method 3 This invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and processor are connected via a bus. Specifically, the processor implements any step in Embodiment 1 by running the computer program stored in the memory.

[0050] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0051] Memory may include read-only memory, flash memory, and random access memory, and provides instructions and data to the processor. Some or all of the memory may also include non-volatile random access memory.

[0052] It should be understood that if the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods described above can also be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0053] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0054] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0055] It should be noted that the methods and detailed examples provided in the above embodiments can be incorporated into the apparatus and devices provided in the embodiments for mutual reference, and will not be repeated here.

[0056] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0057] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0058] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A memory computation separation railway transportation revenue data analysis processing method, characterized in that, The method comprises the following steps: Step 1: converting original railway transportation income data into a columnar storage format file output; Step 2: performing multi-thread concurrent reading and prefetching mechanism based on the output file of step 1; Step 3: using smallpond rule engine for data quality checking during the file data reading of step 2; Step 4: using distributed storage and hot and cold layering management for the data after quality checking of step 3: Step 5: performing storage and calculation separation and elastic expansion based on the distributed storage and hot and cold layering management of step 4.

2. The method of claim 1, wherein, The step 1 is specifically, in the conversion process, adopting the Parquet format, and combining the block strategy and the dictionary compression method to realize; the control parameters include the block size and the dictionary size; the conversion time Approximate modeling is: wherein represents the input file size, represents the I / O throughput bandwidth, represents the speedup of the system due to compression and chunking optimization.

3. The method of claim 2, wherein, The Parquet format is used in combination with the block strategy and dictionary compression method, which specifically comprises sequentially performing the block strategy and dictionary compression, and then converting into the Parquet format.

4. The method of claim 1, wherein, The step 2 is specifically, through column clipping, predicate down push, vectorization execution to realize multi-thread concurrent reading; the control parameters include the number of concurrent threads , the pre-fetch block size , the cache hit rate , the data read delay , which can be approximately expressed as: 。 5. The method of claim 2, wherein, The step 3 specifically comprises introducing the smallpond rule engine during the data reading, using structured expression, and dynamically loading at runtime, synchronously executing the checking of the transportation income business rules during the data loading process, and the control parameters include rule execution batch size and rule complexity coefficient; the checking model is defined as: wherein denotes the number of executions required for verification, denotes the total number of records, is a rule complexity function.

6. The method of claim 5, wherein, The step 4 specifically comprises using 3FS distributed file system as the storage infrastructure, in combination with the hot and cold layering strategy and automatic backup mechanism at the storage level; Hot data is stored in SSD nodes, and cold data is stored in HDD nodes or archival storage; control parameters include hot data window length; overall storage cost With access performance Can be modeled as: wherein denotes the hot storage unit cost, denotes the cold storage unit cost, denotes the total storage time range, and are the range delays for hot and cold storage, respectively, denotes the hot data range proportion; by adjusting a balance between storage cost and performance can be achieved.

7. The method of claim 1, wherein, The step 5 is specifically, the control parameter includes the number of computing nodes , the task parallelism , the number of storage nodes ; the system processing capacity Can be modeled as: Wherein is a storage access conflict coefficient, reflecting the I / O competition in concurrent access; by adjusting and , without increasing the node pressure, the overall throughput can be significantly improved.

8. A memory compute separated railway transportation revenue data analysis processing system, characterized in that, The system uses the railway transportation income data analysis processing method of any one of claims 1-7, and the system comprises: a data conversion module: converting original railway transportation income data into a columnar storage format file output; a data quality checking module: performing multi-thread concurrent reading and prefetching mechanism based on the output file of the data conversion module; using smallpond rule engine for data quality checking during the file data reading; a storage and calculation separation processing module: using distributed storage and hot and cold layering management for the data after quality checking: performing storage and calculation separation and elastic expansion based on the distributed storage and hot and cold layering management.

9. A computer device, comprising: A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1-7.

Images