Data flow processing method and device and data flow system

Abstract

The invention provides a data flow processing method and device and a data flow system.The method comprises the steps that a parameter configuration table is obtained, and the parameter configuration table is a pre-designed table used for configuring a data extraction mode, a data processing mode and a data writing mode; according to the parameter configuration table, a virtual machine is adopted to extract data from a database, and source data is obtained; performing data processing on the extracted source data by adopting a virtual machine according to the parameter configuration table to obtain processed data; and writing the processed data into the target database by adopting the virtual machine according to the parameter configuration table. According to the scheme, the problem that the operation cost is relatively high as a physical machine cluster is used for data operation in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to a data flow processing method, a data flow processing device, a computer program product and a data flow processing system. BACKGROUND

[0002] With the rapid development of information technology, big data has been widely used in various industries. Storing and processing big data requires a large amount of storage resources and computing resources. How to complete the access and processing of big data on the basis of existing hardware resources and avoid the rapid growth of information system operation cost has become the focus of attention of small and medium-sized enterprises. Therefore, the big data operation scheme based on physical machine cluster has a relatively high operation cost, and a low-cost operation scheme is urgently needed. SUMMARY

[0003] The main purpose of the present application is to provide a data flow processing method, a data flow processing device, a computer program product and a data flow processing system to at least solve the problem of high operation cost of data operation using physical machine cluster in the prior art.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data flow processing method is provided, comprising: obtaining a parameter configuration table, wherein the parameter configuration table is a table designed in advance for configuring data extraction mode, data processing mode and data writing mode; extracting data from a database by using a virtual machine according to the parameter configuration table to obtain source data; processing the extracted source data by using the virtual machine according to the parameter configuration table to obtain processed data; and writing the processed data into a target database by using the virtual machine according to the parameter configuration table.

[0005] By applying the technical scheme of the present application, the pre-designed parameter configuration table is obtained to guide the extraction, processing and writing operations of data. The virtual machine is used to extract data (source data) from the database. In the virtual machine, the source data is processed according to the configuration table, and the processed data is written into the target database. Similarly, the virtual machine is used as a processing device in this process. Compared with the physical machine cluster, the virtual machine reduces the demand for hardware resources. Through the virtualization technology, the virtual machine can be run on fewer physical servers. The parameter configuration table enables different data processing requirements (such as data extraction mode, data processing mode and data writing mode) to be realized by simply changing the configuration table without major adjustment of the hardware architecture, thereby ensuring that the operation cost of the present application is low. BRIEF DESCRIPTION OF DRAWINGS

[0006] The drawings constituting a part of the specification illustrate the present application, the exemplary embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0007] Figure 1 A hardware structure block diagram of a mobile terminal showing a method of performing data flow processing provided in an embodiment of the present application is shown;

[0008] Figure 2 A flowchart showing a method of data flow processing provided in an embodiment of the present application is shown;

[0009] Figure 3 A data extraction flowchart of a database dedicated connection tool based on program packaging is shown;

[0010] Figure 4 A general flowchart of a lightweight big data extraction method based on concurrent queues and stream computing is shown;

[0011] Figure 5 A data table data extraction implementation flowchart is shown;

[0012] Figure 6 A structure block diagram of an apparatus for data flow processing provided in an embodiment of the present application is shown.

[0013] Among the above drawings, the following reference signs are included:

[0014] 102, processor; 104, memory; 106, transmission device; 108, input and output device. DETAILED DESCRIPTION

[0015] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0016] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0017] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0018] For ease of description, the following describes some nouns or terms related to the embodiments of the present application:

[0019] Physical machine: Bare Metal, refers to the operating system and application directly running on the physical server, without relying on the virtualization layer for resource isolation or abstraction. The physical machine directly uses the CPU, memory, hard disk and other hardware resources of the server, without the need for allocation through virtualization technology. This "bare machine" mode ensures the ultimate optimization of performance and efficient use of resources, but the cost is relatively high, and the server hardware and supporting facilities (such as computer room, power, etc.) need to be purchased or leased.

[0020] Virtual machine: Virtual Machine (VM) is a computer program or system that simulates the running environment of physical computer hardware through software, allowing multiple operating systems and application environments to run simultaneously on the same physical machine. By sharing physical hardware resources, server utilization is improved. By reducing the number of physical servers, system operating costs are reduced.

[0021] HA: High Availability, short for high availability cluster, refers to a solution that improves business continuity through dual or multi-machine cluster systems, usually including active nodes and standby nodes to ensure quick switching to maintain services in the event of node failure.

[0022] ETL: Extract-Transform-Load (Extract-Transform-Load) is the core link of data warehouse technology, which extracts data from multiple heterogeneous systems, cleans and converts, and loads it into the target database to achieve cross-system data integration and quality improvement.

[0023] The main problems existing in the prior art are: although asynchronous state snapshots, rule hot update side outputs, message buses and lake-warehouse integrated storage are used to realize data parsing and cleaning, window aggregation and dynamic multi-path shunting, the physical machine cluster software and hardware cost is high, and from the perspective of cost performance, this technology still has a lot of optimization space.

[0024] As described in the background section, existing technologies use physical machine clusters for data operations, which result in high operating costs. To address these issues, embodiments of this application provide a data transfer processing method, a data transfer processing apparatus, a computer program product, and a data transfer system.

[0025] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0026] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a data flow processing method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data flow processing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one instance, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0028] This embodiment provides a method for data transfer processing that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] Figure 2 This is a flowchart illustrating a data flow processing method according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0030] Step S201: Obtain the parameter configuration table, wherein the parameter configuration table is a pre-designed table for configuring the data extraction method, data processing method and data writing method;

[0031] Specifically, in this embodiment, a pre-designed parameter configuration table must first be obtained. This table is formulated by the system administrator or developer based on specific data processing requirements before implementing data flow processing. The table records processing parameters for different data tables, including the data extraction source address, source database type, target database address, target database type, and specific rules for data processing operations such as transcoding, field truncation, column reduction, and conditional filtering. In the above embodiment, the use of the parameter configuration table greatly improves the flexibility and configurability of data flow processing, avoiding poor adaptability caused by hard coding, and enabling the system to easily cope with the diverse needs of different data sources and target databases. This is because the parameter configuration table allows for dynamic adjustment of processing strategies before processing begins without modifying the source code, reducing maintenance costs and improving processing efficiency.

[0032] Step S202: According to the above parameter configuration table, use a virtual machine to extract data from the database to obtain the source data;

[0033] Specifically, based on the acquired parameter configuration table, the system will utilize a local program in the virtual machine's HA mode to read data from a specified database according to the source database connection address and source database type information in the table. This process involves communication with the database, and the virtual machine is responsible for performing the data extraction operation. In the above embodiment, using a virtual machine for data extraction significantly reduces hardware costs and maintenance complexity compared to traditional physical machine clusters, while ensuring the efficiency and accuracy of data extraction. This is because virtual machines can dynamically adjust resource usage, such as CPU and memory, avoiding the waste caused by the fixed resource allocation of physical clusters. Simultaneously, software-level optimizations of the virtual machine (such as concurrent queues and streaming computing) ensure data reading speed and system response time, achieving efficient and low-cost data extraction.

[0034] Step S203: According to the above parameter configuration table, the above virtual machine is used to process the extracted source data to obtain the processed data.

[0035] Specifically, after reading the source data, the virtual machine performs a series of processing operations based on the instructions in the parameter configuration table, such as transcoding, field truncation, column reduction, and conditional filtering. These processing operations are performed in memory using software operators in a streaming manner, without the need for intermediate files to be written to memory, thus achieving real-time data transformation and cleaning. In the above embodiment, the combination of streaming computing and software optimization strategies makes the data processing process highly efficient and resource-efficient. This is because streaming computing reduces I / O operations for data writing to memory, while the concurrent queue mechanism allows data processing tasks to be executed in parallel, avoiding the bottleneck of single-threaded processing. Therefore, it greatly improves data processing speed and throughput, reduces the demand for hardware resources, and improves cost-effectiveness.

[0036] Step S204: According to the above parameter configuration table, the above-processed data is written to the target database using the above virtual machine.

[0037] Specifically, the processed data is written to the specified target database by the virtual machine according to the connection information and type of the target database in the parameter configuration table. This write operation also follows the principles of streaming computing; data rows do not need to be stored in the middle and flow seamlessly directly from the processing stage to the write stage, ensuring the consistency and efficiency of data flow. In the above embodiment, the optimization of the data write stage also reduces operating costs and improves efficiency. This is because the streaming write mechanism avoids the additional overhead of data persistence and re-reading, while the dynamic resource scheduling of the virtual machine ensures the rapid completion of the write operation, maintaining high throughput even with large amounts of data, reducing the write time to the target database, thereby reducing the overall data processing latency and improving the real-time performance and cost efficiency of data flow.

[0038] This embodiment allows for the acquisition of a pre-designed parameter configuration table to guide data extraction, processing, and writing operations. A virtual machine is used to extract data (source data) from the database. On the virtual machine, necessary data processing is performed on the source data according to the configuration table, and the processed data is written to the target database. Similarly, a virtual machine is used as the processing device in this process. Compared to physical machine clusters, virtual machines reduce the demand for hardware resources. Through virtualization technology, virtual machines can run on fewer physical servers. The parameter configuration table allows different data processing requirements (such as data extraction methods, data processing methods, and data writing methods) to be achieved by simply changing the configuration table without major adjustments to the hardware architecture, thus ensuring low operating costs for this solution.

[0039] This embodiment provides a data flow processing method, the first step of which is to obtain a parameter configuration table. This parameter configuration table is a pre-designed table used to configure data extraction methods, data processing methods, and data writing methods. Based on the contents of the parameter configuration table, the virtual machine extracts data from the database to obtain the data source. This process is implemented through a "data table data extraction device," which, based on concurrent queuing and streaming computing, can efficiently extract data from various heterogeneous data sources (such as relational databases, column databases, real-time message queues, etc.).

[0040] After the data is extracted into the virtual machine, the next step is data processing. This step is entirely guided by the aforementioned parameter configuration table. The virtual machine processes the source data in real time according to the data processing rules set in the configuration table. Data processing may include, but is not limited to, data transcoding, field truncation, column reduction, and conditional filtering. All these operations follow the principle of streaming computing, meaning that data is processed continuously in the virtual machine's memory as a data stream, without needing to be written to intermediate files. By setting up "Data Table Data Extraction Implementation," streaming computing based on concurrent queues achieves high efficiency in data processing. Specifically, after extraction, the source data directly enters the virtual machine's memory and is processed in a streaming computing manner according to the rules set in the parameter configuration. This means that data processing tasks can be performed uninterruptedly in the form of a data row stream, greatly improving the real-time performance and throughput of data processing. In addition, by setting up the "Performance Monitoring and Optimization" module, the system can monitor the virtual machine's resource usage in real time, such as CPU and memory usage, and dynamically adjust the concurrent queue, thereby achieving efficient resource utilization in the virtual machine's HA mode. This mechanism ensures that during data processing, the virtual machine can flexibly adjust the number of queues according to the resource requirements of the current task, avoiding waste caused by over-allocation of resources, and also preventing processing delays caused by insufficient resources, thus ensuring the smoothness and high performance of data processing.

[0041] After data processing is complete, the next step is data writing. This process is still guided by the parameter configuration table mentioned above. Based on the target database connection information, target database type, and possible data writing rules (such as data type conversion and formatting) in the configuration table, the virtual machine directly writes the processed data rows to the target database using a streaming computation method. This data writing operation avoids the creation of intermediate files and is completed directly within the virtual machine, improving writing speed and reducing unnecessary I / O operations and storage space requirements. A "Performance Monitoring and Optimization" module is set up to monitor the performance of data table extraction, transformation, and loading, and dynamically adjust the number of concurrent queues. This ensures that during data writing, the virtual machine can monitor its own resource usage in real time, such as CPU and memory load, and adjust the number of concurrent queues accordingly to optimize writing performance. This mechanism ensures that the virtual machine can maintain high efficiency in data writing even under resource constraints, especially when facing large-scale data writing tasks. By dynamically adjusting the number of queues, it can balance resource usage, avoid performance bottlenecks caused by resource contention, and guarantee the stability and speed of data writing.

[0042] Specifically, such as Figure 3 As shown:

[0043] Database connection tools are based on program-packaged implementations. Currently, commonly used database import / export tools for small and medium-sized enterprises offer only basic functionality. Advanced functions such as transcoding, format processing, and content filtering require additional program calls, resulting in low throughput across the entire process. Furthermore, they are generally tied to specific operating systems or hardware, leading to poor versatility. For example, extracting data from a relational database to a non-relational database involves multiple independent processing steps: data reading, data transcoding, field truncation, column reduction, conditional filtering, and writing to the non-relational database.

[0044] Distributed database cluster data access framework. Large internet companies typically use distributed database cluster frameworks for data processing. The data processing engine within a distributed database ecosystem requires the deployment of at least 10 physical servers and the maintenance of persistent platform services such as application coordination, resulting in significant costs and high deployment and learning curves.

[0045] like Figure 4 As shown, this invention, based on a local program, constructs a lightweight big data extraction method and device through concurrent queues and streaming computing. It achieves high throughput performance in virtual machine HA mode, completing encoding / decoding, content filtering, input / output operations with minimal memory and CPU. It maximizes performance with minimal hardware resources, reducing deployment and maintenance pressure. It shields the differences between various data sources such as relational databases, non-relational databases, message queues, text files, and in-memory databases. It supports various processing scenarios such as data transcoding, field truncation, column reduction, and conditional filtering, achieving unified data extraction from heterogeneous data sources. Compared to traditional database connection tools using program wrappers, it significantly improves performance, is not limited by hardware or software deployment environments, and lowers the deployment and maintenance threshold compared to distributed databases, enabling big data processing scenarios at low cost.

[0046] The lightweight big data extraction method and apparatus based on concurrent queues and streaming computing are described in the following steps: (1) Set up a "Data Table Extraction Parameter Configuration" module to configure the upstream data table extraction, transformation and loading parameters. (2) Set up a "Data Table Data Extraction Implementation" apparatus to efficiently extract, transform and load heterogeneous big data based on concurrent queues and streaming computing. (3) Set up a "Performance Monitoring and Optimization" module to monitor the performance of data table extraction, transformation and loading, and dynamically optimize the number of concurrent queues.

[0047] To achieve efficient implementation of big data extraction from upstream source systems and reduce system operating costs for SMEs, this invention aims to research a lightweight big data extraction method and device based on concurrent queues and stream computing. Employing a virtual machine HA mode and relying on local programs, it achieves unified extraction and processing of data from heterogeneous data sources such as relational databases, column databases, and real-time message queues through stream computing. Compared to traditional database connection tools packaged in programs, its performance is significantly improved, comparable to distributed database physical machine clusters, and achieves big data extraction and processing in a low-cost manner.

[0048] The basic approach of this scheme is as follows: (1) Set up a "Data Table Extraction Parameter Configuration" module to configure the upstream data table extraction, transformation and loading parameters. (2) Set up a "Data Table Data Extraction Implementation" device to efficiently extract, transform and load heterogeneous big data based on concurrent queues and streaming computing. (3) Set up a "Performance Monitoring and Optimization" module to monitor the performance of data table extraction, transformation and loading, and dynamically optimize the number of concurrent queues.

[0049] The information collected is authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, and necessary confidentiality measures have been taken. The information does not violate public order and good morals, and corresponding operation portals are provided for users to choose to authorize or refuse. Users are also provided with corresponding operation portals to choose to agree to or refuse the automated decision-making results. If the user chooses to refuse, the process will proceed to the expert decision-making process to avoid relevant legal and public opinion risks.

[0050] In the specific implementation process, the parameter configuration table is obtained, including: obtaining multiple fields, including the source database connection address, source database type, whether to transcode, transcoding method, whether to truncate fields, truncation content, whether to reduce the number of columns, reduction content, whether to filter by conditions, filtering content, target database type, and target database connection address; obtaining the attribute values ​​of the above fields to obtain the above parameter configuration table.

[0051] In this solution, the creation and use of parameter configuration tables significantly improve the automation level of data processing, reduce manual intervention, and ensure the continuity and reliability of the data processing process. Through parameter configuration tables, the system can flexibly adapt to the characteristics of various data sources and target databases, meeting the data processing needs of different business scenarios without requiring additional hardware configuration.

[0052] By standardizing various data processing rules into a series of fields and their attribute values, the system can easily understand and execute these rules, thus automating and intelligentizing the data processing workflow. The fields and attribute values ​​in the parameter configuration table can be adjusted according to specific needs without modifying the source code, greatly improving the system's adaptability to different data sources and processing requirements. Because the parameter configuration table supports unified processing of heterogeneous data sources, it avoids the high costs of physical machine clusters and reduces maintenance and learning costs associated with frequent code modifications, enabling small and medium-sized enterprises to enjoy high-efficiency processing capabilities at a lower cost.

[0053] In implementing this method, obtaining the parameter configuration table becomes crucial. This step involves retrieving multiple fields and their respective attribute values ​​to construct the parameter configuration table. These fields include basic information such as source database connection address, source database type, target database type, and target database connection address, as well as key parameters used to define data processing rules, such as whether to transcode, transcoding method, whether to truncate fields, the content to be truncated, whether to reduce the number of columns, the content to be reduced in terms of the number of columns, whether to use conditional filtering, and the content to be filtered.

[0054] In some embodiments, according to the above parameter configuration table, a virtual machine is used to extract data from the database to obtain source data, including: extracting data from the corresponding database using the virtual machine according to the source database connection address and the source database type to obtain the source data, wherein different source database types have different calling methods; according to the above parameter configuration table, the virtual machine is used to process the extracted source data to obtain processed data, including at least one of the following: when the attribute value indicating whether to transcode is yes, the virtual machine is used to transcode the extracted source data according to the attribute value of the transcoding method to obtain the processed data; when the attribute value indicating whether to truncate a field is yes, the virtual machine is used to truncate the field content. The machine extracts the source data and performs data truncation processing to obtain the processed data. If the attribute value indicating whether to reduce the number of columns is yes, the machine performs data reduction processing on the extracted source data according to the column reduction content to obtain the processed data. If the attribute value indicating whether to use conditional filtering is yes, the machine performs data filtering processing on the extracted source data according to the conditional filtering content to obtain the processed data. According to the parameter configuration table, the machine writes the processed data into the target database, including: writing the processed data into the target database according to the target database type and the target database connection address; the writing method differs for different target database types.

[0055] This solution, by employing the steps described above, enables unified extraction and processing of data from heterogeneous data sources at a low cost, as well as efficient writing to the target database, significantly improving the flexibility, efficiency, and economy of data processing. Through streaming computing and dynamic optimization mechanisms, the latency of data processing and writing is reduced, achieving timely data processing, which is particularly important for real-time data analysis in the financial industry. Preprocessing steps such as data transcoding and field truncation ensure data quality and consistency, avoiding analytical errors caused by inconsistent data encoding or non-standard formats.

[0056] Specifically, the existence of parameter configuration tables eliminates hard-coded processing logic in the code, allowing system administrators to flexibly adjust data extraction, processing, and writing rules as needed without modifying the source code. This not only simplifies maintenance but also improves the system's adaptability to different data sources and target database types. The use of virtual machine HA mode and stream computing ensures high data processing efficiency even with limited resources, avoiding excessive reliance on hardware resources in traditional methods and reducing system operating costs. Simultaneously, concurrent queues and dynamic tuning mechanisms make the data processing process more stable and efficient, further enhancing the overall technical performance.

[0057] In this solution, the data extraction step is a crucial component. It involves intelligently invoking different data extraction methods based on the connection address and type of the source database to ensure efficiency and accuracy. First, the system administrator needs to input the source database connection information, including the connection address and database type, into the parameter configuration table. This forms the basis for the system to identify the database and select the appropriate extraction method. Different source database types require different extraction methods because they have different data storage structures and access interfaces. For example, for relational databases, the system uses SQL queries to efficiently extract data; for non-relational databases, such as MongoDB, the system may use its specific data retrieval methods; and for text files or message queues, data is read using appropriate readers. Once the virtual machine identifies the source database type, it will invoke a specific reader based on the configured connection address to efficiently extract data from the source database. If the source database is a relational database, the virtual machine will use SQL queries to read the data; if it is a non-relational database, it will use specific data access APIs to read the data. The extracted data will undergo corresponding preprocessing based on attribute values ​​in the parameter configuration table, such as whether to transcode, whether to truncate fields, whether to reduce the number of columns, and whether to filter based on conditions. For example, if the transcoding attribute is set to "Y", the virtual machine will perform encoding conversion according to the transcoding method attribute value. The processed data is then directly written to the target database using a streaming computing method, based on the target database type and connection address. This eliminates the need for creating and reading intermediate files, saving storage space and improving the overall efficiency of data processing.

[0058] When the attribute value for "whether to transcode" in the parameter configuration table is "Y", the system will perform data transcoding processing using the aforementioned virtual machine according to the transcoding method attribute value. This processing step ensures encoding compatibility between different systems, avoiding data parsing errors or data transmission problems caused by encoding mismatch. Through data transcoding processing, this invention ensures encoding consistency between different systems, improving the efficiency and accuracy of cross-system data transmission and processing.

[0059] When the attribute value of the "Whether to truncate" field in the parameter configuration table is "Y", it means that the source data needs to be truncated. Based on the attribute value of the truncated field, the virtual machine will truncate specific fields of the extracted source data to obtain processed data that better meets business requirements. This processing step ensures data refinement and standardization, avoids unnecessary data redundancy, and improves data processing efficiency. Data truncation removes irrelevant or redundant field information, ensuring data refinement and efficiency in subsequent processing and reducing the complexity of data processing.

[0060] When the "Reduce Column Count" attribute in the parameter configuration table above is set to "Y", it means that the source data needs to be reduced in the number of columns. Based on the attributes of the column reduction content, the virtual machine will remove unnecessary column information from the source data according to predefined rules or conditions, resulting in more concise processed data. Column reduction not only reduces data storage space but also improves data processing and transmission speed, making it an indispensable part of data preprocessing.

[0061] When the conditional filtering attribute in the above parameter configuration table is set to "Y", the virtual machine will filter out data rows that meet the conditions from the extracted source data based on the filtering criteria for subsequent processing. This process ensures that the processed data is more accurate and useful, avoids unnecessary data processing and storage, and improves the efficiency and economy of data flow.

[0062] After the data undergoes the aforementioned transcoding, field truncation, column reduction, or conditional filtering processes, the processed data will be efficiently written to the target database using a virtual machine based on the target database type and connection address information in the parameter configuration table. Since different target database types (such as relational databases, non-relational databases, message queues, text files, and in-memory databases) have different data writing mechanisms and format requirements, the virtual machine will adapt to these different data writing methods to ensure that the data can be correctly and efficiently loaded into the target database while meeting the target database's format and storage requirements. When the target database type is a relational database, the virtual machine will use SQL statements or similar interfaces to write the data to the corresponding database tables. Specific writing methods may include batch writing, single-row writing, etc., and the optimal writing method will be selected based on the performance characteristics of the target database and the dynamic adjustments of the aforementioned performance monitoring and optimization modules. For non-relational databases, such as NoSQL databases, the virtual machine will write the data to the corresponding collections or storage units through specific API interfaces. The writing process may involve data serialization and deserialization to conform to the storage format requirements of non-relational databases. If the target database is a message queue, the virtual machine will convert the processed data rows into message format and send them to the queue through the message queue's producer interface for subsequent consumers to subscribe to and process. For text files as the target database, the virtual machine will write the processed data rows to the file in a specified format (such as CSV or JSON), possibly using strategies such as file rolling and sharding to accommodate efficient storage of large amounts of data. When the target is an in-memory database, the virtual machine will directly load the data rows into memory, possibly through specific memory operation interfaces, ensuring fast data access and processing.

[0063] The "Data Table Extraction Parameter Configuration" module allows for parameterized configuration of upstream data table extraction, transformation, and loading. When extracting, transforming, and loading heterogeneous big data, this avoids hardcoding processing logic in the code. Instead, it pre-builds the program's execution framework, determining the specific processing flow by passing parameters at runtime, thus improving adaptability to different processing scenarios. The "Data Table Extraction Parameter Configuration Table" includes fields such as "Serial Number," "Data Table Name (English)," "Source Database Connection Address," "Source Database Type," "Target Database Connection Address," "Target Database Type," "Whether to Transcode," "Transcoding Method," "Whether to Truncate Fields," "Truncation Content," "Whether to Reduce Column Count," "Reduced Column Count Content," "Whether to Filter Conditions," and "Filter Condition Content." Table 1 shows the data table extraction parameter configuration table.

[0064] Table 1

[0065]

[0066] The values ​​for the "Source Database Type" and "Target Database Type" fields are as follows: "1" represents "Relational Database", "2" represents "Non-Relational Database", "3" represents "Message Queue", "4" represents "Text File", and "5" represents "In-Memory Database", etc. The values ​​for the "Transcoding", "Field Truncation", "Column Reduction", and "Conditional Filtering" fields are as follows: "Y" represents "Yes" and "N" represents "No". For fields with a value of "Y", corresponding processing rules need to be inserted into the relevant "Transcoding Method", "Field Truncation Content", "Column Reduction Content", and "Conditional Filtering Content" fields.

[0067] A "Data Table Data Extraction Implementation" device is set up, which efficiently extracts, transforms, and loads heterogeneous big data based on concurrent queues and streaming computing. This invention abstracts data from different upstream systems into a unified source data, abstracts common ETL tasks into a limited number of real-time streaming processing tasks, abstracts the processing logic in ETL (such as transcoding, filtering, and structured parsing) into operators, and unifies the input and output of source data / operators into data table row data, realizing a flexible combination of heterogeneous data sources and ETL processing. Starting with data extraction from "various source databases," each task is a streaming process based on row data. The processing results do not need to be written to a data file; they are directly transferred to the next task in the form of a data stream, and finally written to the target database.

[0068] like Figure 5As shown, according to the preset job schedule, a row of data in the "Data Table Extraction Parameter Configuration Table" is read, which is the configuration information of the source data table, and the data table data extraction operation is performed. The specific steps are as follows: ① Based on the "Source Database Connection Address" and "Source Database Type", the corresponding reader is called to read a row of data in the source data table. ② Based on the content of the "Whether to Transcode" and "Transcoding Method" fields, if "Whether to Transcode" is configured as "Y", the data in this row of the source data table is transcoded according to the "Transcoding Method". For operations where "Whether to Transcode" is configured as "N", this step is skipped. ③ Based on the content of the "Whether to Truncate Fields" and "Truncation Content" fields, if "Whether to Truncate Fields" is configured as "Y", the data in this row of the source data table is truncated according to the "Truncation Content". For operations where "Whether to Truncate Fields" is configured as "N", this step is skipped. ④ Based on the "Whether to Reduce Columns" and "Column Reduction Content" fields, if "Whether to Reduce Columns" is configured as "Y", perform column reduction on the current row of data in the source data table according to the "Column Reduction Content". If "Whether to Reduce Columns" is configured as "N", skip this step. ⑤ Based on the "Whether to Filter Conditions" and "Filter Condition Content" fields, if "Whether to Filter Conditions" is configured as "Y", perform condition filtering on the current row of data in the source data table according to the "Filter Condition Content". If "Whether to Filter Conditions" is configured as "N", skip this step. ⑥ Based on the "Target Database Connection Address" and "Target Database Type", call the corresponding writer to write a row of data to the target data table in the target database. Steps ① to ⑥ are pipelined operations, continuing until all rows of the source data table have been extracted, at which point the extraction job scheduling for that source data table ends.

[0069] Specific examples are as follows:

[0070] According to the preset job schedule, the data in the first row of the "Data Table Extraction Parameter Configuration Table" with serial number 1 is read, which is the source data table configuration information of Table1. Data extraction from Table1 is then performed, with the following steps: ① The "Source Database Connection Address" and "Source Database Type" are "SrcUrl1" and "1" (representing "Relational Database"), respectively. The "Relational Database Reader" is invoked to read the first row of data from the source data table "Table1" in the relational database. ② Based on the "Whether to Transcode" and "Transcoding Method" fields, with "Whether to Transcode" configured to "Y", the data in this row of the source data table is transcoded according to the "Transcoding Method". ③ Based on the "Whether to Truncate Fields" and "Truncation Content" fields, with "Whether to Truncate Fields" configured to "Y", the data in this row of the source data table is truncated according to the "Truncation Content". ④ Based on the "Whether to Reduce Column Count" and "Reduction Content" fields, with "Whether to Reduce Column Count" configured to "N", this step is skipped. ⑤ Based on the "Whether to Filter" and "Filter Content" fields, with "Whether to Filter" configured to "Y", the data in this row of the source data table is filtered according to the "Filter Content". ⑥ With "Target Database Connection Address" and "Target Database Type" set to "DesUrl1" and "2" respectively (representing "Non-Relational Database"), the "Non-Relational Database Writer" is invoked to write a row of data to the target data table "Table1" in the non-relational database. Steps ① to ⑥ are pipelined operations, continuing until all rows of the Table1 data table have been extracted, at which point the Table1 data table extraction job scheduling ends.

[0071] In the specific implementation process, the parameter configuration table also includes a sequence number. Each sequence number corresponds to a parameter for data extraction, data processing, and data writing. After obtaining the parameter configuration table, the method further includes: extracting the attribute value of the field with sequence number M in the parameter configuration table and generating a task, where M≥1, and the task is a data extraction, data processing, and data writing task; if the task corresponding to the field with sequence number M has been completed, extracting the attribute value of the field with sequence number M+1 and generating the next task; if the task corresponding to the field with sequence number N has been completed, determining that the job flow has ended, where N is the maximum value of the sequence number.

[0072] This solution automates the execution of data processing tasks, ensuring each task is completed sequentially and guaranteeing job continuity and integrity. Sequence numbering allows for easy adjustment of the data processing order or skipping of certain tasks, providing flexibility in the workflow. Automated task scheduling and continuous execution maximize the utilization efficiency of virtual machine resources and reduce idle time.

[0073] Because the parameter configuration table contains the sequence number of each task, the virtual machine can automatically identify the task to be executed and generate and execute tasks sequentially without manual intervention, thus achieving task automation. After completing a task (sequence number M), the virtual machine will immediately and automatically extract the next task (sequence number M+1) until all tasks (sequence number N) are completed, ensuring the continuity and integrity of the job. Data extraction, processing, and writing of each data table are all performed under unified job flow control. The automatic scheduling and continuous execution of tasks design allows for the full and efficient use of the virtual machine's CPU and memory resources, reducing waiting time between tasks and waste of system resources, thereby improving the overall data processing efficiency and the economy of resource utilization.

[0074] The sequence design of the parameter configuration table ensures the orderly execution of data extraction, data processing, and data writing tasks. At the start of each job, the virtual machine first reads the configuration information with sequence number M, and generates the corresponding data extraction, processing, and writing tasks based on this information. Once the task with sequence number M is completed, the virtual machine automatically extracts the configuration information with sequence number M+1 to generate the next task, until it reaches sequence number N, which is the last task in the parameter configuration table, at which point the job flow ends.

[0075] Suppose a retail company needs to extract customer purchase history, customer reviews, and customer information from multiple internal databases using a virtual machine, and then process and write the data to a target database. The parameter configuration table sets up tasks starting from number 1, which are executed sequentially for data extraction, processing, and writing. The virtual machine reads the configuration information for task number 1 in the parameter configuration table, which is the task of extracting customer purchase history. After completing task number 1 (e.g., extracting, filtering, and writing customer purchase history data to the target database), the virtual machine automatically extracts the configuration information for task number 2, which is the task of extracting customer review data. Subsequently, after completing task number 2, it continues to task number 3, which is the task of extracting basic customer information. When the virtual machine completes task number N in the parameter configuration table (i.e., the task of extracting basic customer information), the entire job workflow is considered complete, and the job is finished.

[0076] Assuming the parameter configuration table contains 5 tasks, numbered 1 to 5, corresponding to extracting customer purchase history, customer reviews, basic customer information, product inventory information, and sales forecast data, respectively. If the average data processing time for each task is 150 seconds, and the virtual machine immediately starts the next task after completing one, then the estimated completion time for the entire job is 750 to 1125 seconds, depending on the specific system load. This design ensures seamless connection and continuous execution of data processing tasks, improving the efficiency and accuracy of data processing.

[0077] In some embodiments, after obtaining the parameter configuration table, the method further includes: generating multiple concurrent queues when there are multiple parameter configuration tables, wherein the concurrent queues include at least one of the parameter configuration tables; and using the virtual machine to perform data extraction jobs, data processing jobs, and data writing jobs in parallel with the concurrent queues.

[0078] In this solution, by using concurrent queues and parallel job processing, the method of this invention can process multiple data sources or data tables simultaneously, significantly reducing the overall data processing time. Concurrent queues allow multiple tasks to execute simultaneously, fully utilizing the CPU and memory resources of the virtual machine and improving system resource utilization efficiency.

[0079] Because each concurrent queue contains a specific parameter configuration table, the virtual machine can launch multiple data processing flows simultaneously based on this configuration information. Each flow independently executes data extraction, processing, and writing jobs, thus achieving parallel data processing. Generating multiple concurrent queues within the virtual machine allows for dynamic adjustment of the number of queues based on the virtual machine's actual resource status (such as the number of CPU cores and available memory), ensuring optimal allocation of system resources, avoiding resource waste or bottlenecks, and improving efficiency. The concurrent queue mechanism is particularly suitable for processing data from various data sources, such as relational databases, column-oriented databases, and real-time message queues. It effectively addresses the complexity and diversity of heterogeneous data sources and simplifies the data processing flow.

[0080] After obtaining multiple parameter configuration tables, the method of this invention automatically generates multiple concurrent queues corresponding to their number, with each queue containing at least one item from a parameter configuration table. The virtual machine, through streaming computing technology, can process data extraction, data processing, and data writing jobs in parallel using concurrent queues. This parallel processing mechanism fully utilizes the resources of multi-core processors and virtual machines, significantly accelerating data processing, especially when dealing with large-scale datasets or multiple data sources, providing efficient batch data processing capabilities.

[0081] Suppose a social media company needs to extract user interaction data from multiple internal data sources (such as MySQL databases, MongoDB databases, Redis cache, and RabbitMQ message queues), perform data cleaning and transcoding, and finally write the processed data to a Hadoop data warehouse. Virtual machines generate multiple concurrent queues based on parameter configuration tables for the different data sources. Assume four concurrent queues are generated, each responsible for extracting data from a specific data source. The virtual machines simultaneously launch data processing jobs in the four concurrent queues, with each virtual machine instance in the queue independently performing data extraction, cleaning, transcoding, and loading. Assume each concurrent queue can process up to 500 records per second. During execution, the virtual machine's performance monitoring module monitors resource usage and dynamically adjusts the number of concurrent queues based on the actual load, thereby optimizing data processing efficiency and resource utilization.

[0082] If a social media platform needs to process a total of 2,000,000 interaction data points distributed across four data sources, with approximately 500,000 data points from each source, and in parallel processing mode, assuming each concurrent queue processes 500 records per second, considering the parallel execution efficiency of the concurrent queues, the entire data processing process is expected to be completed within 1666 to 3333 seconds. The exact time depends on other factors related to data processing, such as data complexity and system load.

[0083] In the specific implementation process, after generating multiple concurrent queues, the above method further includes: when there is only one virtual machine, obtaining the running information of the virtual machine, wherein the running information includes one or more of CPU utilization, memory utilization, network bandwidth, data processing rate, queue length, and queue waiting time; obtaining an adjustment model, wherein the adjustment model is one of SOM model, decision tree model, and LSTM model; forming a first training set by combining historical running information and corresponding first policy labels, and training the adjustment model with the first training set to obtain a first policy adjustment model, wherein the first policy label is the adjustment policy of the queue in the first training set; inputting the running information into the first policy adjustment model to obtain the first adjustment policy corresponding to the running information; adjusting the concurrent queue according to the first adjustment policy to obtain a first adjusted concurrent queue; and using the virtual machine to perform data extraction jobs, data processing jobs, and data writing jobs in parallel with the first adjusted concurrent queue.

[0084] In this solution, by monitoring the virtual machine's operating information in real time and dynamically adjusting the model, the method of this invention can optimize the scheduling of concurrent queues in real time, significantly improving data processing performance. The adjustment model can intelligently adjust queue parameters, such as queue length and waiting time, according to the current operating status, thereby maximizing resource utilization efficiency and avoiding resource waste.

[0085] By continuously monitoring virtual machine (VM) runtime information (such as CPU utilization and memory utilization), the system resource usage can be understood in a timely manner, providing a basis for subsequent dynamic adjustments. Historical runtime information and a first policy label are used as the first training set for model training, enabling the adjustment model to learn the optimal concurrent queue adjustment strategy under different runtime states. When real-time runtime information is input into the model, the resulting first adjustment strategy reflects the most suitable resource configuration and scheduling method. Based on the first adjustment strategy, the VM can dynamically adjust parameters such as the length and waiting time of the concurrent queue to adapt to the current runtime environment, achieving efficient resource utilization and optimized data processing jobs.

[0086] After generating multiple concurrent queues, this invention further enhances the system's performance and resource utilization efficiency when processing large-scale data by introducing real-time monitoring of virtual machine runtime information and a machine learning-based queue adjustment strategy. First, the system collects runtime information of the aforementioned virtual machines, including but not limited to CPU utilization, memory utilization, and network bandwidth. This information is crucial for assessing the current data processing load and bottlenecks. Then, the method of this invention obtains an adjustment model, which can be a SOM model, decision tree model, or LSTM model, depending on the specific scenario and requirements. Historical runtime information and corresponding first strategy labels are combined to form a first training set. The adjustment model is trained using machine learning, enabling it to predict the optimal queue adjustment strategy based on the runtime information. The trained first strategy adjustment model has the ability to intelligently adjust concurrent queue parameters (such as queue length and waiting time) according to the current runtime status. After obtaining the current runtime information, the system inputs this information into the pre-trained first strategy adjustment model to obtain the first adjustment strategy corresponding to the runtime information. Based on the first adjustment strategy, the system dynamically adjusts the parameters of the concurrent queues, such as increasing or decreasing the number of queues, adjusting queue length, or waiting time, to achieve optimal data processing efficiency and resource utilization. The adjusted concurrent queues can more accurately match current data processing needs, avoiding resource waste and improving data processing throughput. Suppose an online live streaming platform needs to analyze massive amounts of user behavior data in real time, including viewing history, bullet comments, and gift donations. This data comes from multiple different database tables and needs to be processed through concurrent queues. Due to the real-time and uncertain nature of user behavior data, the platform needs to adjust the number and parameters of concurrent queues for data processing at any time to ensure efficient resource utilization and timely data analysis. The virtual machine continuously monitors its running status, including CPU usage, memory usage, and data processing rate. Historical running information and a first policy label are used to form the first training set, training a decision tree model as the policy adjustment model. After real-time running information is input into the model, the first adjustment policy is obtained. According to the first adjustment policy, the virtual machine dynamically adjusts the parameters of the concurrent queues, such as increasing or decreasing the number of concurrent queues, adjusting the queue length and waiting time of each queue, to meet the processing needs of real-time user behavior data. Using the adjusted concurrent queues, the virtual machine processes various data extraction, data processing, and data writing jobs in parallel.

[0087] Suppose a financial institution needs to extract transaction data, account information, and market report data from multiple different data sources and process and analyze them in real time. Data extraction, processing, and writing operations are executed through concurrent queues. Because the data volume differs significantly between daytime trading peaks and nighttime market closures, the financial institution needs a method to automatically adjust queue parameters to cope with different data processing demands. During daytime trading peaks, the virtual machine's CPU utilization may reach 80%, memory utilization may be 70%, the data processing rate may be 1000 records per second, the queue length may reach 10000, and the queue waiting time may be 1 second. Using historical execution information and corresponding first strategy labels as the first training set, a decision tree model is trained to obtain the first strategy adjustment model. When the above execution information is input into the model, the first adjustment strategy is to increase the number of concurrent queues to 500 and adjust the queue waiting time to 0.5 seconds. Based on this first adjustment strategy, the financial institution dynamically adjusts the parameters of the concurrent queues to cope with the high data processing demands during the day. Using the first adjusted concurrent queue, the virtual machine efficiently performs data extraction, data processing, and data writing jobs in parallel.

[0088] Suppose a financial institution needs to process 200,000 records per hour during the day, while the demand drops to 20,000 records per hour during the night when the market is closed. By dynamically adjusting the concurrent queues, the financial institution can configure 500 concurrent queues during the day, each processing 400 records per second, while at night, this can be reduced to 100 concurrent queues, each processing 200 records per second. This dynamic adjustment mechanism significantly improves the resource utilization efficiency and data processing speed of the financial institution when handling different data volumes.

[0089] Suppose an e-commerce platform needs to analyze user shopping behavior, search history, and click data in real time. This data comes from multiple databases and real-time message queues. The processing needs for user behavior data differ significantly between day and night. The platform requires a method to intelligently adjust queue parameters based on real-time operational information. At night, the virtual machine's CPU utilization might be 40%, memory utilization 30%, data processing rate might be 500 records per second, queue length might reach 5000, and queue waiting time might be 0.8 seconds. Using historical operational information and corresponding first policy labels as the first training set, a SOM model is trained to obtain the first policy adjustment model. When the above operational information is input into the model, the first adjustment policy is to reduce the number of concurrent queues to 200 and adjust the queue waiting time to 0.3 seconds. Based on this first adjustment policy, the e-commerce platform dynamically adjusts the parameters of the concurrent queues to cope with the lower data processing demands at night. Using the adjusted concurrent queues, the virtual machine efficiently performs data extraction, data processing, and data writing jobs in parallel.

[0090] Suppose an e-commerce platform needs to process 500,000 records per hour during peak daytime hours, while the data processing demand drops to 50,000 records per hour at night. By dynamically adjusting concurrent queues, the e-commerce platform can configure 400 concurrent queues during the day, each processing 600 records per second, while at night this can be reduced to 200 concurrent queues, each processing 250 records per second. This method not only ensures data processing speed during peak daytime hours but also avoids excessive resource consumption at night, achieving efficient resource utilization.

[0091] A "Performance Monitoring and Optimization" module is set up to monitor the performance of data table extraction, transformation, and loading, and dynamically adjust the number of concurrent queues. A lightweight big data extraction method and device based on concurrent queues and streaming computing allows for simultaneous extraction of multiple source data tables, forming multiple concurrent queues. By monitoring the utilization of virtual machine CPU and memory resources in real time, the number of concurrent queues is dynamically adjusted, achieving efficient resource utilization in virtual machine HA mode. Traditional database connection tools using program-packaged software require multiple writes of intermediate files to complete transcoding, cleaning, and filtering operations, resulting in low overall efficiency, double the storage space requirements, and various database limitations. Existing open-source big data computing engines are based on distributed database ecosystems, requiring several high-configuration physical machines as hardware bases and the deployment of multiple supporting resident services such as application coordination.

[0092] The lightweight big data extraction method and device significantly improves performance compared to traditional database-specific connection tools that use program wrappers. It is comparable to the data extraction performance of distributed database physical machine clusters. It not only achieves unified extraction of data from heterogeneous data sources, but also achieves ultra-high efficiency comparable to physical machine clusters with low-cost virtual machine HA mode.

[0093] In some embodiments, after generating multiple concurrent queues, the method further includes: if there are multiple virtual machines, obtaining the number of concurrent queues to be processed; combining the running information and the number of concurrent queues into a dataset; combining the historical dataset and the corresponding second policy label into a second training set, training the adjustment model using the second training set to obtain a second policy adjustment model, wherein the second policy adjustment model is an adjustment policy for virtual machine resource allocation in the second training set; inputting the dataset into the second policy adjustment model to obtain a second adjustment policy corresponding to the dataset; adjusting the concurrent queues processed by the virtual machines according to the second adjustment policy to obtain a second adjusted concurrent queue; distributing the second adjusted concurrent queue to the corresponding virtual machines, and using the virtual machines to perform data extraction jobs, data processing jobs, and data writing jobs on the corresponding second adjusted concurrent queue.

[0094] In this solution, a second strategy adjustment model is used to intelligently schedule and optimize resources for concurrent queues in a multi-virtual machine environment. This method not only improves the overall throughput of data processing but also ensures balanced resource utilization and efficient execution of processing jobs. This approach is particularly suitable for scenarios requiring the processing of large-scale data, such as financial transaction data analysis and real-time monitoring of e-commerce operations.

[0095] Running information, including CPU utilization, memory utilization, and the number of concurrent queues to be processed, is acquired on each virtual machine, forming a multi-dimensional dataset. A second training set is created using historical datasets and second policy labels to train an LSTM model or decision tree model, which becomes the second policy adjustment model. This model can predict the optimal resource allocation strategy based on the running information of multiple virtual machines and the number of concurrent queues. Based on the second adjustment strategy, the system intelligently adjusts the number of concurrent queues processed by each virtual machine and the resource allocation ratio, making resource allocation more balanced, avoiding resource bottlenecks and waste, and improving data processing efficiency. The adjusted concurrent queues are distributed to the corresponding virtual machines, which process data extraction, data processing, and data writing jobs in parallel, achieving efficient processing of large-scale data.

[0096] When multiple virtual machines exist, this invention further proposes an intelligent scheduling and resource allocation mechanism for concurrent queues in a multi-virtual machine environment. First, the system obtains the number of concurrent queues to be processed to assess the total processing workload. Then, the virtual machine's runtime information (including CPU utilization, memory utilization, network bandwidth, etc.) and the number of concurrent queues are combined to form a dataset for model training and policy prediction.

[0097] Next, the system combines the historical dataset and the corresponding second policy labels into a second training set. This training set is used to train a second policy adjustment model, which focuses on adjusting virtual machine resource allocation strategies. Once the second policy adjustment model is obtained, the system inputs the current dataset into the model to obtain the corresponding second adjustment strategy. This strategy can include adjusting the number of concurrent queues, queue parameters, or resource allocation ratios for each virtual machine, to address the data processing challenges in a multi-virtual machine environment.

[0098] Suppose a telecom operator needs to analyze and process massive amounts of log data in real time, including user call records, network traffic, and device status. This data comes from multiple data sources and needs to be processed on multiple virtual machines. The processing requirements for log data fluctuate with changes in time, user activity, and network status. The system continuously monitors the operational information of each virtual machine and counts the number of concurrent queues awaiting processing. Historical operational information and a second policy label are combined to form a second training set, and a decision tree model is trained to become the second policy adjustment model. When the dataset consisting of real-time operational information and queue counts is input into the model, a second adjustment policy is obtained, such as adjusting the number of concurrent queues. Based on the second adjustment policy, the system dynamically adjusts the number of concurrent queues processed by each virtual machine and the resource allocation to cope with the constantly changing data processing requirements. The adjusted concurrent queues are distributed to each virtual machine, which efficiently extracts, processes, and writes log data in parallel.

[0099] Assuming that during peak hours, each virtual machine's CPU utilization reaches 70% and memory utilization reaches 60%, while the log data processing requirement is 100,000 records per minute, the system adjusts its model based on the second strategy prediction, suggesting that the concurrent queues for each virtual machine should be adjusted to 200 during peak hours to ensure data processing efficiency and balanced resource utilization.

[0100] Suppose an online education platform needs to analyze student learning behavior data, course evaluation data, and payment record data to optimize teaching content and strategies. This data comes from diverse sources, requires concurrent processing across multiple virtual machines, and its processing demands fluctuate with time, course activities, and user behavior. The platform continuously acquires runtime information for each virtual machine and counts the number of concurrent queues awaiting processing. Historical runtime information and a second policy label form a second training set, which is used to train an LSTM model as the second policy adjustment model. When the dataset consisting of real-time runtime information and queue counts is input into the model, a second adjustment policy is obtained, which may include dynamically adjusting the number of virtual machine processing queues. Based on the second adjustment policy, the system dynamically adjusts the number of concurrent queues and the resource allocation of virtual machines to ensure efficient execution of data processing jobs. The adjusted concurrent queues are then allocated to various virtual machines, which process data extraction, data processing, and data writing jobs in parallel.

[0101] During peak platform activity periods, the CPU and memory utilization of each virtual machine can reach 85% and 75%, respectively, while the processing requirements for learning behavior data, course evaluation data, and payment record data are 50,000 records per minute. Based on the predictions of the second strategy adjustment model, the concurrent queue for each virtual machine should be adjusted to 150 during peak activity periods to balance resource usage and data processing speed, ensuring the continuity and efficiency of the platform's analytical work.

[0102] The main advantages of this invention are as follows: Traditional methods use dedicated database connection tools and database import / export tools packaged in programs. These tools offer only basic functionality, requiring additional program calls for advanced functions such as transcoding, format processing, and content filtering. The throughput of the entire processing chain is low, and they are generally tied to specific operating systems or hardware, resulting in poor versatility. Large internet companies using distributed database ecosystems require at least 10 physical servers and the maintenance of persistent platform services such as application coordination, leading to huge costs and high deployment and learning curves. This invention implements a lightweight big data extraction method and device based on concurrent queues and stream computing. Employing a virtual machine HA mode and relying on local programs, it achieves unified extraction and processing of data from various heterogeneous data sources such as relational databases, column databases, and real-time message queues through stream computing. This achieves big data extraction and processing at low cost, and has high promotional value in big data processing scenarios for many small and medium-sized enterprises.

[0103] The main innovation of this invention is as follows: by setting up modules and devices such as "data table extraction parameter configuration, data table data extraction implementation, performance monitoring and optimization", a lightweight big data extraction method and device based on concurrent queues and streaming computing is realized. Based on a local program, using concurrent queues and streaming computing mode, it realizes a flexible combination of heterogeneous data sources and ETL processing, which greatly improves the batch data extraction capability of virtual machine HA mode. Compared with traditional database dedicated connection tools packaged by programs, the performance is greatly improved and can be compared with distributed database physical machine clusters. Big data extraction and processing are realized in a low-cost manner, which greatly reduces the operating cost of information systems.

[0104] This application also provides a data transfer processing apparatus. It should be noted that the data transfer processing apparatus of this application can be used to execute the data transfer processing method provided in this application. This apparatus is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0105] The following describes the data transfer and processing apparatus provided in the embodiments of this application.

[0106] Figure 6 This is a structural block diagram of a data flow processing apparatus according to an embodiment of this application. Figure 6 As shown, the device includes:

[0107] The first acquisition unit 10 is used to acquire a parameter configuration table, wherein the parameter configuration table is a pre-designed table for configuring data extraction method, data processing method and data writing method;

[0108] Extraction unit 20 is used to extract data from the database using a virtual machine according to the above parameter configuration table to obtain source data;

[0109] The first data processing unit 30 is used to process the extracted source data using the virtual machine according to the above parameter configuration table to obtain processed data.

[0110] The writing unit 40 is used to write the processed data into the target database using the virtual machine according to the above parameter configuration table.

[0111] This embodiment allows for the acquisition of a pre-designed parameter configuration table to guide data extraction, processing, and writing operations. A virtual machine is used to extract data (source data) from the database. On the virtual machine, necessary data processing is performed on the source data according to the configuration table, and the processed data is written to the target database. Similarly, a virtual machine is used as the processing device in this process. Compared to physical machine clusters, virtual machines reduce the demand for hardware resources. Through virtualization technology, virtual machines can run on fewer physical servers. The parameter configuration table allows different data processing requirements (such as data extraction methods, data processing methods, and data writing methods) to be achieved by simply changing the configuration table without major adjustments to the hardware architecture, thus ensuring low operating costs for this solution.

[0112] In the specific implementation process, the first acquisition unit includes a first acquisition module and a second acquisition module. The first acquisition module is used to acquire multiple fields, including the source database connection address, source database type, whether to transcode, transcoding method, whether to truncate fields, field truncation content, whether to reduce the number of columns, column reduction content, whether to filter by conditions, filter by conditions, target database type, and target database connection address. The second acquisition module is used to acquire the attribute values ​​of the above fields to obtain the above parameter configuration table.

[0113] In this solution, the creation and use of parameter configuration tables significantly improve the automation level of data processing, reduce manual intervention, and ensure the continuity and reliability of the data processing process. Through parameter configuration tables, the system can flexibly adapt to the characteristics of various data sources and target databases, meeting the data processing needs of different business scenarios without requiring additional hardware configuration.

[0114] In some embodiments, the extraction unit includes an extraction module for extracting data from the corresponding source database using the virtual machine based on the source database connection address and the source database type, to obtain the source data, wherein different source database types have different calling methods; the first data processing unit includes a transcoding module, a truncation module, a reduction module, and a filtering module. The transcoding module is used to perform data transcoding processing on the extracted source data using the virtual machine based on the attribute value of the transcoding method when the attribute value of whether to transcode is yes, to obtain the processed data; the truncation module is used to perform data transcoding processing on the extracted source data using the virtual machine based on the attribute value of whether to truncate a field when the attribute value of the field is yes, to obtain the processed data; The source data undergoes data truncation processing to obtain the processed data described above. The reduction processing module, when the attribute value indicating whether to reduce the number of columns is yes, uses the virtual machine to perform data reduction processing on the extracted source data according to the column reduction content, obtaining the processed data. The filtering processing module, when the attribute value indicating whether to conditionally filter is yes, uses the virtual machine to perform data filtering processing on the extracted source data according to the conditional filtering content, obtaining the processed data. The writing unit includes a writing module, which uses the virtual machine to write the processed data into the target database according to the target database type and the target database connection address. Different target database types have different writing methods.

[0115] This solution, by employing the steps described above, enables unified extraction and processing of data from heterogeneous data sources at a low cost, as well as efficient writing to the target database, significantly improving the flexibility, efficiency, and economy of data processing. Through streaming computing and dynamic optimization mechanisms, the latency of data processing and writing is reduced, achieving timely data processing, which is particularly important for real-time data analysis in the financial industry. Preprocessing steps such as data transcoding and field truncation ensure data quality and consistency, avoiding analytical errors caused by inconsistent data encoding or non-standard formats.

[0116] In the specific implementation process, the above-mentioned device further includes a first extraction unit, a second extraction unit, and a determination unit. The first extraction unit is used to extract the attribute value of the field with the sequence number M in the parameter configuration table, after obtaining the parameter configuration table, and to generate a task, where M≥1, and the task is a data extraction, data processing, and data writing task. The second extraction unit is used to extract the attribute value of the field with the sequence number M+1 when the task corresponding to the field with the sequence number M has been completed, and to generate the next task. The determination unit is used to determine the end of the job flow when the task corresponding to the field with the sequence number N has been completed, where N is the maximum value of the sequence number.

[0117] This solution automates the execution of data processing tasks, ensuring each task is completed sequentially and guaranteeing job continuity and integrity. Sequence numbering allows for easy adjustment of the data processing order or skipping of certain tasks, providing flexibility in the workflow. Automated task scheduling and continuous execution maximize the utilization efficiency of virtual machine resources and reduce idle time.

[0118] In some embodiments, the above-described apparatus further includes a generation unit and a second data processing unit. The generation unit is used to generate multiple concurrent queues after obtaining the parameter configuration table, provided that there are multiple parameter configuration tables. The concurrent queues include at least one of the parameter configuration tables. The second data processing unit is used to perform data extraction, data processing, and data writing operations in parallel using the virtual machine and the concurrent queues.

[0119] In this solution, by using concurrent queues and parallel job processing, the method of this invention can process multiple data sources or data tables simultaneously, significantly reducing the overall data processing time. Concurrent queues allow multiple tasks to execute simultaneously, fully utilizing the CPU and memory resources of the virtual machine and improving system resource utilization efficiency.

[0120] In specific implementation, the above-mentioned device further includes a second acquisition unit, a third acquisition unit, a first training unit, a first processing unit, a first adjustment unit, and a second processing unit. The second acquisition unit is used to acquire the running information of the virtual machine when there is only one virtual machine after generating multiple concurrent queues. The running information includes one or more of CPU utilization, memory utilization, network bandwidth, data processing rate, queue length, and queue waiting time. The third acquisition unit is used to acquire an adjustment model, which is one of the SOM model, decision tree model, and LSTM model. The first training unit is used to form a first training set by combining historical running information and corresponding first policy labels, and to train the adjustment model using the first training set to obtain a first policy adjustment model. The first policy label is the adjustment policy of the queue in the first training set. The first processing unit is used to input the running information into the first policy adjustment model to obtain the first adjustment policy corresponding to the running information. The first adjustment unit is used to adjust the concurrent queue according to the first adjustment policy to obtain a first adjusted concurrent queue. The second processing unit is used to use the virtual machine to perform data extraction, data processing, and data writing jobs in parallel with the first adjusted concurrent queue.

[0121] In this solution, by monitoring the virtual machine's operating information in real time and dynamically adjusting the model, the method of this invention can optimize the scheduling of concurrent queues in real time, significantly improving data processing performance. The adjustment model can intelligently adjust queue parameters, such as queue length and waiting time, according to the current operating status, thereby maximizing resource utilization efficiency and avoiding resource waste.

[0122] In some embodiments, the above-described apparatus further includes a fourth acquisition unit, a composition unit, a second training unit, a third processing unit, a second adjustment unit, and a third data processing unit. The fourth acquisition unit is used to acquire the number of concurrent queues to be processed when there are multiple virtual machines after generating multiple concurrent queues. The composition unit is used to compose the running information and the number of concurrent queues into a dataset. The second training unit is used to compose the historical dataset and the corresponding second policy label into a second training set, and use the second training set to train the adjustment model to obtain a second policy adjustment model, wherein the second policy adjustment model is an adjustment policy for virtual machine resource allocation in the second training set. The third processing unit is used to input the dataset into the second policy adjustment model to obtain a second adjustment policy corresponding to the dataset. The second adjustment unit is used to adjust the concurrent queues processed by the virtual machines according to the second adjustment policy to obtain the second adjusted concurrent queues. The third data processing unit is used to distribute the second adjusted concurrent queues to the corresponding virtual machines, and use the virtual machines to perform data extraction, data processing, and data writing operations on the corresponding second adjusted concurrent queues.

[0123] In this solution, a second strategy adjustment model is used to intelligently schedule and optimize resources for concurrent queues in a multi-virtual machine environment. This method not only improves the overall throughput of data processing but also ensures balanced resource utilization and efficient execution of processing jobs. This approach is particularly suitable for scenarios requiring the processing of large-scale data, such as financial transaction data analysis and real-time monitoring of e-commerce operations.

[0124] The aforementioned data transfer and processing apparatus includes a processor and a memory. The first acquisition unit, extraction unit, first data processing unit, and writing unit are all stored as program units in the memory, and the processor executes the program units stored in the memory to achieve the corresponding functions. All of the above modules are located in the same processor; alternatively, the modules may be located in different processors in any combination.

[0125] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the high operating costs associated with physical machine clusters used in existing data operations.

[0126] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0127] This invention provides a computer-readable storage medium including a stored program, wherein the program controls the device containing the computer-readable storage medium to perform the data transfer processing when it is running.

[0128] This invention provides a processor for running a program, wherein the program executes the data transfer processing method described above.

[0129] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements at least the method steps of data transfer processing. The device described herein may be a server, PC, PAD, mobile phone, etc.

[0130] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform a program that initializes a method step having at least a data flow processing step.

[0131] This invention provides a data transfer system, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for performing any of the above-described data transfer processes.

[0132] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0133] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0138] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0139] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0142] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for data flow processing, characterized in that, include: Obtain the parameter configuration table, wherein the parameter configuration table is a pre-designed table for configuring the data extraction method, data processing method and data writing method; Based on the parameter configuration table, a virtual machine is used to extract data from the database to obtain the source data; According to the parameter configuration table, the extracted source data is processed using the virtual machine to obtain processed data; According to the parameter configuration table, the virtual machine is used to write the processed data into the target database.

2. The method according to claim 1, characterized in that, Obtain the parameter configuration table, including: Obtain multiple fields, including source database connection address, source database type, whether to transcode, transcoding method, whether to truncate fields, truncate field content, whether to reduce column count, column count reduction content, whether to filter by conditions, filter by conditions content, target database type, and target database connection address. Obtain the attribute values ​​of the fields to obtain the parameter configuration table.

3. The method according to claim 2, characterized in that, Based on the parameter configuration table, a virtual machine is used to extract data from the database to obtain source data, including: Based on the source database connection address and the source database type, the virtual machine extracts data from the corresponding database to obtain the source data, wherein different source database types have different calling methods; According to the parameter configuration table, the extracted source data is processed using the virtual machine to obtain processed data, including at least one of the following: If the attribute value indicating whether transcoding is enabled is true, the extracted source data is transcoded using the virtual machine according to the attribute value of the transcoding method to obtain the processed data. If the attribute value representation of whether to truncate the field is yes, the virtual machine is used to process the extracted source data according to the truncation content of the field to obtain the processed data; If the attribute value indicating whether to reduce the number of columns is yes, the extracted source data is processed by the virtual machine according to the content of column reduction to obtain the processed data. If the attribute value of the conditional filtering is yes, the virtual machine is used to perform data filtering processing on the extracted source data according to the conditional filtering content to obtain the processed data. According to the parameter configuration table, the virtual machine is used to write the processed data into the target database, including: Based on the target database type and the target database connection address, the virtual machine is used to write the processed data into the target database. Different target database types have different writing methods.

4. The method according to claim 2, characterized in that, The parameter configuration table also includes serial numbers, each serial number corresponding to parameters for one data extraction, data processing, and data writing operation. After obtaining the parameter configuration table, the method further includes: Extract the attribute value of the field with the sequence number M from the parameter configuration table and generate a task, wherein M≥1, and the task is a data extraction, data processing and data writing task; If the task corresponding to the field with sequence number M has been completed, extract the attribute value of the field with sequence number M+1 and generate the next task. If the task corresponding to the field with sequence number N has been completed, the job flow is determined to be finished, where N is the maximum value of the sequence number.

5. The method according to any one of claims 1 to 4, characterized in that, After obtaining the parameter configuration table, the method further includes: When there are multiple parameter configuration tables, multiple concurrent queues are generated, wherein each concurrent queue includes at least one of the parameter configuration tables; The virtual machine is used to perform data extraction, data processing, and data writing jobs in parallel using the concurrent queue.

6. The method according to claim 5, characterized in that, After generating multiple concurrent queues, the method further includes: If there is one virtual machine, obtain the running information of the virtual machine, wherein the running information includes one or more of the following: CPU utilization, memory utilization, network bandwidth, data processing rate, queue length, and queue waiting time. Obtain the adjusted model, wherein the adjusted model is one of the SOM model, decision tree model, and LSTM model; The historical operation information and the corresponding first strategy label are combined to form a first training set. The adjustment model is trained using the first training set to obtain a first strategy adjustment model, wherein the first strategy label is the adjustment strategy of the queue in the first training set. The operation information is input into the first strategy adjustment model to obtain the first adjustment strategy corresponding to the operation information; The concurrent queue is adjusted according to the first adjustment strategy to obtain the first adjusted concurrent queue; The virtual machine is used to perform data extraction, data processing, and data writing jobs in parallel using the first adjusted concurrent queue.

7. The method according to claim 6, characterized in that, After generating multiple concurrent queues, the method further includes: If there are multiple virtual machines, obtain the number of concurrent queues to be processed; The runtime information and the number of concurrent queues are combined to form a dataset; The historical dataset and the corresponding second policy labels are combined to form a second training set. The adjustment model is trained using the second training set to obtain the second policy adjustment model, wherein the second policy adjustment model is the virtual machine resource allocation adjustment policy of the second training set. The dataset is input into the second policy adjustment model to obtain the second adjustment policy corresponding to the dataset; The concurrent queue processed by the virtual machine is adjusted according to the second adjustment strategy to obtain the second adjusted concurrent queue; The second adjusted concurrent queue is distributed to the corresponding virtual machine, and the virtual machine is used to perform data extraction, data processing and data writing jobs on the corresponding second adjusted concurrent queue.

8. A data flow processing apparatus, characterized in that, include: The first acquisition unit is used to acquire a parameter configuration table, wherein the parameter configuration table is a pre-designed table for configuring data extraction method, data processing method and data writing method; The extraction unit is used to extract data from the database using a virtual machine according to the parameter configuration table to obtain source data; The first data processing unit is used to process the extracted source data using the virtual machine according to the parameter configuration table to obtain processed data. The writing unit is used to write the processed data into the target database using the virtual machine according to the parameter configuration table.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the data flow processing method according to any one of claims 1 to 7.

10. A data transfer system, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for performing data transfer processing according to any one of claims 1 to 7.

Images