Data processing method and system and electronic equipment

Abstract

The invention discloses a data processing method and system and electronic equipment. Relates to the field of big data, and comprises the following steps: receiving business data sent by a client, processing the business data according to a line storage mode to obtain line storage data, and storing the line storage data in a first database; the row storage data is processed according to the column storage mode to obtain M column storage data, each column storage data is stored in a storage area unit corresponding to each column storage data in a second database, the second database is an extended database of the first database, and M is a positive integer. Through the data processing method and device, the problem that data processing efficiency and timeliness are low when data batch processing is adopted in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a data processing method, system, and electronic device. Background Art

[0002] In today's data-intensive business environment, with exponential data growth, traditional transaction processing systems face significant challenges. While these systems excel at handling frequent write operations and rapid transaction responses, their efficiency and responsiveness decline significantly when faced with complex analytical tasks involving large amounts of data. Consequently, enterprises often employ a data pipeline mechanism, building a separate data processing platform alongside the transaction processing system to batch integrate, cleanse, transform, and load data into an environment suitable for in-depth analysis and decision support. While this process improves data availability and analytical capabilities, it also presents significant drawbacks.

[0003] First, the entire process of extracting and transforming data from transaction processing systems to analytical platforms involves multiple stages, each requiring independent software tools and resources. This not only increases system complexity but also makes operations, maintenance, and management more difficult. Failure at any point in the data pipeline directly impacts data accuracy and the continuity of the analytical process.

[0004] Secondly, data pipelines are subject to significant latency. The time span from raw data generation to the finalization of usable analytical material can be tens of minutes or even hours. For business scenarios requiring immediate data feedback, such as real-time risk monitoring and dynamic pricing strategy adjustments, batch processing of data is inefficient, hindering data analysis efficiency.

[0005] Currently, no effective solution has been proposed to address the low data processing efficiency and timeliness issues of batch data processing used in related technologies. Summary of the Invention

[0006] The main purpose of this application is to provide a data processing method, system and electronic device to solve the problem of low data processing efficiency and timeliness in the related art using data batch processing.

[0007] To achieve the above-mentioned objectives, according to one aspect of the present application, a data processing method is provided. The method comprises: receiving business data sent by a client, processing the business data in a row-based manner to obtain row-based data, and storing the row-based data in a first database; processing the row-based data in a column-based manner to obtain M column-based data, and storing each column-based data in a storage area unit corresponding to each column-based data in a second database, wherein the second database is an extended database of the first database, and M is a positive integer.

[0008] Optionally, the business data is processed in a row storage manner to obtain row storage data, including: obtaining the first timestamp of the business data, and grouping the business data with the same first timestamp into a group to obtain N groups of business data, where N is a positive integer; arranging each group of business data according to the type of business data and the format requirements of the business table to obtain N row storage data, where each row storage data corresponds to a group of business data.

[0009] Optionally, the method also includes: upon receiving a data analysis instruction sent by the client, determining attribute information of the data to be analyzed according to the data analysis instruction, and obtaining the data to be analyzed from a second database according to the attribute information; processing the data to be analyzed according to the data analysis instruction, obtaining an analysis result, and sending the analysis result to the client.

[0010] Optionally, when receiving a data analysis instruction sent by the client, the method also includes: obtaining a second timestamp of the data analysis instruction; determining whether there is data with the second timestamp in the second database; when there is no data with the second timestamp in the second database, performing data synchronization operations on the first database and the second database until there is data with the second timestamp in the second database, and executing the step of obtaining the data to be analyzed from the second database according to the attribute information.

[0011] Optionally, processing the row-stored data in a column-stored manner to obtain M column-stored data includes: obtaining the data fields contained in the row-stored data to obtain M data fields; classifying each data in the row-stored data according to the data fields to obtain M groups of feature data; sorting the data in each group of feature data according to the third timestamp of each feature data to obtain M groups of sorted feature data, and determining each group of sorted feature data as a column-stored data to obtain M column-stored data.

[0012] Optionally, before receiving the business data sent by the client, the method also includes: generating a second database based on the first database, and reading the table structure in the first database; aggregating the existing data in the first database according to the data fields in the table structure to obtain P initial column storage data, where P is a positive integer; setting a storage area unit for each data field in the second database, and storing the P initial column storage data in the storage area unit of the corresponding data field.

[0013] Optionally, in the case that the first database is a distributed database, the method further includes: setting the second database as a target role, wherein the target role is used to indicate that the second database is only allowed to receive data from the first database.

[0014] To achieve the above-mentioned objectives, according to another aspect of the present application, a data processing system is provided. The system includes: a client for receiving business data input by a user and sending the business data to a first database for storage; a first database for storing the business data sent by the client; a row storage engine for converting the business data into row storage data in a storage format required by the first database before performing a storage operation on the first database; a column storage engine for receiving the row storage data sent by the row storage engine, converting the row storage data into column storage data, and storing the column storage data in a second database; and a second database for storing the column storage data and, when a business analysis operation is required, outputting the column storage data according to the analysis requirements of the analysis system.

[0015] To achieve the above-mentioned objectives, according to another aspect of the present application, a data processing device is provided. The device includes: a receiving unit, configured to receive business data sent by a client, process the business data in a row-based manner to obtain row-based data, and store the row-based data in a first database; and a first processing unit, configured to process the row-based data in a column-based manner to obtain M column-based data, and store each column-based data in a storage area unit corresponding to the respective column-based data in a second database, wherein the second database is an extended database of the first database, and M is a positive integer.

[0016] Optionally, the receiving unit includes: a first acquisition module, used to obtain the first timestamp of the business data, and group the business data with the same first timestamp into a group to obtain N groups of business data, where N is a positive integer; an arrangement module, used to arrange each group of business data according to the type of business data and the format requirements of the business table to obtain N rows of stored data, where each row of stored data corresponds to a group of business data.

[0017] Optionally, the device also includes: a determination unit, which is used to determine the attribute information of the data to be analyzed according to the data analysis instruction when receiving the data analysis instruction sent by the client, and obtain the data to be analyzed from the second database according to the attribute information; a second processing unit, which is used to process the data to be analyzed according to the data analysis instruction, obtain the analysis result, and send the analysis result to the client.

[0018] Optionally, when receiving a data analysis instruction sent by the client, the device also includes: an acquisition unit for acquiring a second timestamp of the data analysis instruction; a judgment unit for judging whether there is data with the second timestamp in the second database; an execution unit for synchronizing data between the first database and the second database when there is no data with the second timestamp in the second database, until there is data with the second timestamp in the second database, and executing the step of acquiring the data to be analyzed from the second database according to the attribute information.

[0019] Optionally, the first processing unit includes: a second acquisition module, used to acquire data fields contained in the row-stored data, and obtain M data fields; a classification module, used to classify each data in the row-stored data according to the data field, and obtain M groups of feature data; a determination module, used to sort the data in each group of feature data according to the third timestamp of each feature data, and obtain M groups of sorted feature data, and determine each group of sorted feature data as a column-stored data, and obtain M column-stored data.

[0020] Optionally, before receiving the business data sent by the client, the device also includes: a generation unit, used to generate a second database based on the first database, and read the table structure in the first database; an aggregation unit, used to aggregate the existing data in the first database according to the data fields in the table structure to obtain P initial column storage data, where P is a positive integer; a storage unit, used to set a storage area unit for each data field in the second database, and store the P initial column storage data in the storage area unit of the corresponding data field.

[0021] Optionally, in the case that the first database is a distributed database, the apparatus further comprises: a setting unit, configured to set the second database as a target role, wherein the target role is used to indicate that the second database is only allowed to receive data from the first database.

[0022] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, which includes a memory storing an executable program; and a processor for running the program, wherein the above-mentioned data processing method is executed when the program is running.

[0023] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer program product is provided, comprising computer instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0024] In an embodiment of the present application, a method is adopted in which business data sent by a client is received, and the business data is processed in a row storage manner to obtain row storage data, and the row storage data is stored in a first database; the row storage data is processed in a column storage manner to obtain M column storage data, and each column storage data is stored in a storage area unit corresponding to each column storage data in a second database. Through real-time data replication and conversion, the row storage data in the first database is converted into column storage data in real time, and then the business data is efficiently converted into a column storage format and synchronized to the second database in real time, so as to achieve the purpose of instantly updating the column storage copy and supporting real-time analysis and query. This process ensures the real-time update and consistency of the column storage data, thereby realizing low-latency transaction and analysis parallel processing, and improving the technical effect of business response speed, thereby solving the technical problem of low data processing efficiency and timeliness of data batch processing in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0026] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown;

[0027] Figure 2 is a schematic diagram of a data processing system provided according to Example 1 of the present application;

[0028] Figure 3 is a flow chart of the data processing method provided in Example 1 of the present application;

[0029] Figure 4 is a schematic diagram of a data processing device provided according to Example 2 of the present application;

[0030] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] It should be noted that the data processing method, system, and electronic device determined in the present disclosure can be used in the field of big data, and can also be used in any field other than the field of big data. The application field of the data processing method, system, and electronic device determined in the present disclosure is not limited.

[0035] It should be noted that the collected information, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) used in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize use or refuse use. If the user chooses to refuse, the expert decision-making process will be entered. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0036] The embodiments or examples of the present disclosure are not exhaustive, but are merely illustrations of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be arbitrarily combined; in addition, the various embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a certain embodiment or example can be arbitrarily combined with the optional methods or optional examples of other embodiments or examples.

[0037] Example 1

[0038] According to an embodiment of the present application, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal for implementing a data processing method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0040] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0042] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0043] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0044] In this embodiment, Figure 2 is a schematic diagram of a data processing system provided according to Example 1 of the present application, such as Figure 2 As shown, the aforementioned data processing method is performed using an optional data processing system as the execution body. The data processing system at least includes: a client 21, a first database 22, a row storage engine 23, a column storage engine 24, and a second database 25, wherein:

[0045] The client 21 is used to receive business data input by a user and send the business data to the first database 22 for storage.

[0046] The first database 22 is used to store the business data sent by the client 21 . Before storage, the business data is converted by the row storage engine 23 into row storage data in a storage format required by the first database 22 .

[0047] The column storage engine 24 is configured to receive the row storage data sent by the row storage engine 23 , convert the row storage data into column storage data, and store the column storage data in the second database 25 .

[0048] The second database 25 is used to store column-based data and output the column-based data according to the analysis requirements of the analysis system when a business analysis operation is required, thereby completing the analysis operation.

[0049] Under the above operating environment, this application provides Figure 3 The data processing method shown. Figure 3 is a flow chart of the data processing method provided in Example 1 of the present application, such as Figure 3 As shown, the method includes:

[0050] Step S301: receiving business data sent by a client, processing the business data in a row storage manner to obtain row storage data, and storing the row storage data in a first database.

[0051] It should be noted that the client refers to the source of business data, such as users' online transaction data. Business data refers to various information related to user transactions, such as transaction records, user behavior data, inventory changes, etc. Row storage is a form of data storage where each row represents a complete record, with all fields contiguously located. This format facilitates transaction processing and fast access to individual records. The primary database, primarily responsible for storing and processing transactional data, features high concurrent write and read capabilities to ensure business continuity and stability.

[0052] Specifically, the execution entity of this embodiment may be a business data processing system. The system begins receiving business data from the client and processes the data using a row-based storage method. That is, each piece of business data is treated as a new record, and all relevant fields are stored in the same row. For example, for a credit card transaction, the data includes information such as the transaction amount, timestamp, card number, and transaction type. The system organizes this information into a row of data and stores the data in the first database.

[0053] By processing and storing business data in a row-based format, the system efficiently manages each transaction or operation, providing fast response times and high concurrent write capabilities. As the storage center for row-based data, First Database ensures timely data updates and accurate transaction execution, meeting daily business needs.

[0054] In step S302, the row-stored data is processed in a column-stored manner to obtain M column-stored data, and each column-stored data is stored in a storage area unit corresponding to each column-stored data in a second database, wherein the second database is an extended database of the first database, and M is a positive integer.

[0055] It should be noted that the column storage method, that is, each type of value (field) is stored in the same column, is conducive to performing data analysis and statistical operations, that is, the data in a certain column can be directly operated without scanning the entire table. Column storage data is a data format processed by the column storage method, and data fields of the same type are stored centrally to facilitate data analysis and calculation. The storage area unit is a specific area within the second database divided for each group of column storage data, which is used to store and index data of the same type. The second database is an extension of the first database, mainly used for data analysis and processing, and has column storage capabilities and an efficient data query mechanism.

[0056] Specifically, the system further processes the row data stored in the row-based database using a column-based approach. This involves splitting each field in the row data into a set of "column" data, and then organizing this column-based data into multiple independent column-based data blocks. For example, from the row-based data of a credit card transaction, you can extract columns such as the transaction amount, timestamp, and card number. Each column contains the same information for all related transactions, forming M columns of column-based data.

[0057] The system then stores these M columns of data in separate storage areas within the second database. Each storage area corresponds to a data type or specific analysis dimension. For example, data in the "transaction amount" column is stored in the amount analysis area, while data in the "timestamp" column is stored in the timeline analysis area. This storage method physically categorizes and optimizes data based on analysis requirements, greatly improving the efficiency of subsequent data queries and analysis.

[0058] It should be noted that step S302 enables the second database to quickly respond to various complex analytical query requirements, such as range queries and aggregation operations, by organizing and storing data in columns, without having to traverse the entire database table, greatly reducing the time for data retrieval and processing. In addition, the setting of each storage area unit also provides a dedicated storage and retrieval mechanism for specific types of data, further optimizing the performance of data analysis. This conversion and storage process is the key to achieving low-latency transactions and parallel processing of analysis, ensuring the real-time and accuracy of data analysis. At the same time, due to the extended relationship between the second database and the first database, the second database and the first database can be regarded as a scheduling within the system, making the data transmission between the second database and the first database more efficient. In addition, when the first database receives business data, it will convert the business data in real time and send it to the second database, thereby ensuring the timeliness of the data in the second database.

[0059] The data processing method provided in the embodiment of the present application adopts a method of receiving business data sent by a client, and processing the business data in a row storage manner to obtain row storage data, and storing the row storage data in a first database; processing the row storage data in a column storage manner to obtain M column storage data, and storing each column storage data in a storage area unit corresponding to each column storage data in a second database. Through real-time data replication and conversion, the row storage data in the first database is converted into column storage data in real time, and then the business data is efficiently converted into a column storage format and synchronized to the second database in real time, achieving the purpose of instantly updating the column storage copy and supporting real-time analysis and query. This process ensures the real-time update and consistency of the column storage data, thereby realizing low-latency transaction and analysis parallel processing, and improving the technical effect of business response speed, thereby solving the technical problem of low data processing efficiency and timeliness of data batch processing in related technologies.

[0060] In order to ensure the accuracy of the row-stored data, optionally, in the data processing method provided in the embodiment of the present application, the business data is processed in a row-stored manner, and the row-stored data obtained includes: obtaining the first timestamp of the business data, and grouping the business data with the same first timestamp into a group to obtain N groups of business data, where N is a positive integer; arranging each group of business data according to the type of business data and the format requirements of the business table to obtain N row-stored data, where each row-stored data corresponds to a group of business data.

[0061] Specifically, when generating row-stored data, the system first extracts the business data's timestamp, known as the "first timestamp," from the received business data. Then, based on these timestamp values, the system groups all business data into several groups, each containing business data with the same first timestamp. For example, if the system receives multiple business data items at 10:00:00, and these multiple business data items correspond to the same business event, all business data items at that time will be grouped together, forming N groups of business data.

[0062] The system then processes each set of business data, arranging it into a row format based on the business data type and predefined business table format requirements. This includes operations such as data type conversion, filling in missing fields, and checking data integrity to ensure that each set of business data is correctly converted into a row. For example, for a set of transaction records, the system checks fields such as the transaction amount, transaction date, and transaction type. These fields are then integrated and arranged according to the row format requirements to form a complete row data record. Each row data entry corresponds to a set of business data, resulting in N complete rows of business data. Timestamp grouping and row data generation ensure that business data is efficiently organized and stored according to chronological order and row format.

[0063] This embodiment achieves orderly management and rapid conversion of large amounts of business data by processing the received business data into row storage, ensuring the efficiency of the entire link from data reception, processing to storage. Specifically, the timestamp grouping mechanism ensures that the system can distinguish data at different time points, avoids redundancy and errors in data processing, and lays a solid foundation for subsequent generation and storage of row storage data. At the same time, by converting each set of business data into row storage data, the system not only optimizes the physical storage structure of the data, but also improves the efficiency and response speed of transaction processing. For example, for a bank's credit card transaction system, it can quickly and accurately store transaction information as row storage data at the moment of receiving a transaction request, providing users with instant transaction confirmation services, while also providing a structured data source for subsequent transaction monitoring and data analysis. This approach greatly shortens the delay in data processing and improves the real-time availability of data.

[0064] In order to improve the efficiency of analyzing business data, optionally, in the data processing method provided in the embodiment of the present application, the method also includes: upon receiving a data analysis instruction sent by the client, determining attribute information of the data to be analyzed according to the data analysis instruction, and obtaining the data to be analyzed from the second database according to the attribute information; processing the data to be analyzed according to the data analysis instruction, obtaining an analysis result, and sending the analysis result to the client.

[0065] It should be noted that data analysis instructions are requests sent by the client to specify the data analysis tasks to be performed, including but not limited to statistics, queries, and predictive analysis. The attribute information of the data to be analyzed, i.e., the data feature descriptions determined by the data analysis instructions, such as data fields, time ranges, and data types, are used to guide data screening and extraction.

[0066] Specifically, after receiving a data analysis instruction from the client, the system first parses the instruction and extracts attribute information, such as the data field to be analyzed, the time range, and the analysis conditions. For example, if the instruction received requires statistics on the total amount of credit card transactions in the second quarter of 2023, the system will identify key attributes such as "credit card transactions" and "second quarter of 2023."

[0067] Furthermore, after obtaining the attribute information, the system directly extracts the required data for analysis from the second database based on the parsed attribute information, without having to obtain data from the first database. For example, the system will query the data stored in the "transaction amount" column and filter out records for "second quarter of 2023" to form the data set for analysis. This ensures that the data analysis instruction is accurately understood and efficiently extracts the data for analysis from the second database based on the instruction's attribute information. This precise data screening avoids unnecessary data processing and improves the efficiency and accuracy of data analysis.

[0068] Furthermore, after obtaining the data to be analyzed, the method described in the data analysis instruction can be used to process the data to be analyzed. Taking the "calculate the total credit card transaction amount in the second quarter of 2023" requirement in the sample instruction as an example, the system will sum up all the extracted "transaction amount" data to obtain the total transaction amount for that quarter. After obtaining the data analysis results, the system will process the analysis results to ensure that their format meets the client's receiving requirements, and then send the results back to the client. For example, in the above example, the system can return a report containing the total credit card transaction amount in the second quarter of 2023, thereby completing the user's data analysis needs.

[0069] This embodiment parses attribute information and obtains the data required for the analysis instruction from the second database based on the attribute information for analysis operations, completing the data analysis process from the receipt and parsing of the client data analysis instruction, to the accurate extraction and efficient analysis of the data in the second database, and finally generating the analysis results. It then quickly filters and calculates the required statistical values ​​from the data set, significantly improving the real-time nature of data processing and decision-making efficiency. It not only simplifies the interaction between the client and the database, reduces the delay of data analysis, but also enhances the analysis capability of the system, ensuring the speed and accuracy of the analysis process.

[0070] In order to ensure the accuracy of the data analysis results, optionally, in the data processing method provided in the embodiment of the present application, when a data analysis instruction sent by the client is received, the method also includes: obtaining a second timestamp of the data analysis instruction; determining whether there is data with the second timestamp in the second database; when there is no data with the second timestamp in the second database, synchronizing the first database with the second database until there is data with the second timestamp in the second database, and executing the step of obtaining the data to be analyzed from the second database according to the attribute information.

[0071] Specifically, when the system receives a data analysis instruction from a client, it first extracts the second timestamp from the instruction and uses it to determine the time period of the data to be analyzed. For example, if a client sends a command requesting analysis of all credit card transaction records between June 1 and June 30, 2023, that time period would be the second timestamp.

[0072] Furthermore, the system needs to determine whether data corresponding to the second timestamp already exists in the second database, thereby determining whether it needs to wait for the data in the first database to be synchronized to the second database before performing business analysis operations. For example, the system will check whether there is already column data for credit card transactions in June 2023 in the second database. By obtaining the timestamp of the data analysis instruction and checking the existence of the data, it can ensure that the required analysis data is accurately prepared before executing the data analysis. This avoids invalid analysis attempts due to missing data, ensures the timeliness and integrity of the data, and lays a solid foundation for subsequent data analysis.

[0073] Upon determining that the second database lacks data matching the second timestamp, the system initiates a data synchronization operation between the first and second databases. Specifically, the system reads row-level data changes from the first database since the last synchronization, such as newly added transaction records, and uses a real-time data replication and conversion engine to quickly convert these row-level data changes into column-level data, which is then synchronized to the second database. This ensures that the data in the second database meets the data requirements of the data analysis instructions. This real-time data synchronization mechanism ensures that even when business analysis needs are time-sensitive, the data used for analysis operations remains complete, effectively eliminating analysis errors caused by data delays and ensuring the accuracy and timeliness of analysis results.

[0074] This embodiment verifies and synchronizes the data in the second database, ensuring that when the client makes a data analysis request within a specific time range, the system can quickly determine the complete status of the data, and if data is missing from the second database, initiate data synchronization between the first database and the second database to ensure that the second database contains sufficient column-stored data that matches the instruction timestamp, thereby significantly improving the real-time response capability of data processing, reducing the delay of analysis results, and ensuring the integrity and timeliness of the data.

[0075] In order to ensure the accuracy of the column-stored data, optionally, in the data processing method provided in the embodiment of the present application, the row-stored data is processed in a column-stored manner to obtain M column-stored data, including: obtaining the data fields contained in the row-stored data to obtain M data fields; classifying each data in the row-stored data according to the data fields to obtain M groups of feature data; sorting the data in each group of feature data according to the third timestamp of each feature data to obtain M groups of sorted feature data, and determining each group of sorted feature data as a column-stored data to obtain M column-stored data.

[0076] It should be noted that the data fields may be various attributes or information units constituting bank deposit data, such as transaction amount, transaction time, card number, and the like.

[0077] When generating column-based data corresponding to row-based data, the system first identifies and extracts all data fields contained in the row-based data, such as "transaction amount," "transaction time," and "card number." Then, based on the characteristics of each data field, the system categorizes the data in the row-based data into multiple sets of feature data. For example, when processing credit card transaction records, the system categorizes all transaction amount values ​​into one set of feature data, all transaction time values ​​into another, and so on, until each of the M data fields corresponds to a set of feature data. This extraction of data fields and categorization of feature data ensures the orderliness and operability of the data, improving the efficiency and accuracy of data processing.

[0078] After obtaining multiple sets of feature data, the system further processes each set of feature data and sorts them according to the third timestamp in each feature data. Taking the "transaction time" feature data as an example, the system arranges all transaction time values ​​in chronological order to generate sorted feature data, thereby obtaining M sets of sorted feature data.

[0079] Finally, the system converts each set of sorted feature data into a columnar format, storing it in the database as columns. Each column represents the complete information of a data field, forming columnar data, thus completing the columnar data generation operation. For example, in a credit card transaction scenario, the system would store the sorted transaction time data as the time column and the transaction amount data as the amount column. This process continues until all M sets of feature data have been converted into columnar data. Through timestamp sorting and columnar storage conversion, the data structure is optimized to meet the needs of efficient data analysis.

[0080] This embodiment greatly enhances the system's ability to process and analyze big data by converting row-stored data to column-stored data, which not only simplifies data management but also significantly improves the real-time performance and efficiency of data analysis.

[0081] In order to ensure the consistency of data between the first database and the second database, optionally, in the data processing method provided in the embodiment of the present application, before receiving the business data sent by the client, the method also includes: generating a second database based on the first database, and reading the table structure in the first database; aggregating the existing data in the first database according to the data fields in the table structure to obtain P initial column storage data, where P is a positive integer; setting a storage area unit for each data field in the second database, and storing the P initial column storage data in the storage area unit of the corresponding data field.

[0082] It's important to note that the table structure defines the format of a table in a database, including field names, data types, and field order. Before receiving business data, the system must first generate an extension of the first database, also known as the second database. To do this, the system creates the second database based on the existing data and format of the first database. It first reads the table structure information from the first database, including the definitions and properties of all data fields, such as field names, data types, and the order in which they are arranged. For example, for credit card transaction data, the system reads the definitions of fields such as the transaction amount, transaction time, and card number.

[0083] After obtaining the table structure, the system aggregates the existing data in the first database based on the read table structure information. This means reorganizing the data by field to obtain P initial column-stored data. For example, taking credit card transaction data as an example, the system will aggregate all transaction amount data, transaction time data, etc., to form independent column-stored data sets. Thus, all transaction amount values ​​will be concentrated in one column, and all transaction time values ​​will be concentrated in another column, thus obtaining the initial column-stored data.

[0084] After obtaining multiple initial column-stored data, you need to assign specific storage areas to each data field in the second database. This means creating dedicated columns to store the corresponding field information. For example, you might assign a single storage area to the "Transaction Amount" data field to store the column-stored data for all transaction amounts. Subsequently, each of the P generated initial column-stored data is stored in the storage area of ​​the corresponding data field in the second database, resulting in a second database with the same data content as the first database.

[0085] This embodiment aggregates the existing data of the database column by column according to the table structure of the first database, obtains column-stored data, and stores the column-stored data in the second database, thereby completing the operation of synchronizing the existing data of the first database to the second database. This ensures that the consistency of data between the second database and the first database can be guaranteed when using the second database or adding new data to the second database.

[0086] Optionally, in the data processing method provided in an embodiment of the present application, when the first database is a distributed database, the method further includes: setting the second database as a target role, wherein the target role is used to indicate that the second database is only allowed to receive data from the first database.

[0087] Specifically, when the first database is a distributed database, in order to ensure the efficiency and consistency of data synchronization, the second database needs to be configured, that is, the second database is set as the target role. The responsibility of this role is limited to receiving data synchronization from the first database, and it does not participate in any data writing operations or distributed election processes.

[0088] For example, if the first database uses a distributed consensus algorithm based on the Paxos or Raft protocol, then during the configuration phase, the second database will be marked as a learner role, meaning that it only passively receives data and does not actively participate in data processing or decision-making logic, ensuring minimal network overhead for data synchronization.

[0089] By limiting the second database to the target role, efficient and secure data synchronization is ensured, avoiding data delays or consistency issues that may be caused when the second database participates in distributed elections or other high-load operations. At the same time, it simplifies the operation and maintenance complexity of the distributed system, allowing the second database to focus on data reception and storage, reducing unnecessary resource consumption.

[0090] It should be noted that when the first database is a stand-alone database, you can use, but are not limited to, binlog asynchronous replication to synchronize data to the asynchronous backup database of the first database. Then, the binlog is parsed and converted into column-based data and stored in the second database for big data analysis. This method does not affect the normal use of the transactional database, and the delay is often in seconds, thus ensuring the efficiency of data transmission.

[0091] In this embodiment, in a distributed environment, the timeliness of business processing and data analysis is enhanced by streamlining the functions of the second database.

[0092] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0093] Example 2

[0094] The present application also provides a data processing device. It should be noted that the data processing device of the present application can be used to execute the data processing method provided in the above embodiment. The data processing device provided in the present application is introduced below.

[0095] According to an embodiment of the present application, a device for implementing the above data processing method is also provided. Figure 4 is a schematic diagram of a data processing device provided according to Example 2 of the present application, such as Figure 4 As shown, the device includes:

[0096] The receiving unit 41 is configured to receive business data sent by a client, process the business data in a row storage manner, obtain row storage data, and store the row storage data in a first database.

[0097] The first processing unit 42 is used to process the row-stored data in a column-stored manner to obtain M column-stored data, and store each column-stored data in a storage area unit corresponding to each column-stored data in the second database, wherein the second database is an extended database of the first database, and M is a positive integer.

[0098] The data processing device provided by the embodiment of the present application receives the business data sent by the client through the receiving unit 41, and processes the business data in a row storage manner to obtain row storage data, and stores the row storage data in the first database; the first processing unit 42 processes the row storage data in a column storage manner to obtain M column storage data, and stores each column storage data in the storage area unit corresponding to each column storage data in the second database, wherein the second database is an extended database of the first database, and M is a positive integer. Through real-time data replication and conversion, the row storage data in the first database is converted into column storage data in real time, and then the business data is efficiently converted into a column storage format and synchronized to the second database in real time, achieving the purpose of instantly updating the column storage copy and supporting real-time analysis and query. This process ensures the real-time update and consistency of the column storage data, thereby achieving the technical effect of low-latency transaction and analysis parallel processing and improving business response speed, thereby solving the technical problem of low data processing efficiency and timeliness of data batch processing in related technologies.

[0099] Optionally, in the data processing device provided in the embodiment of the present application, the receiving unit 41 includes: a first acquisition module, used to obtain the first timestamp of the business data, and group the business data with the same first timestamp into a group to obtain N groups of business data, where N is a positive integer; an arrangement module, used to arrange each group of business data according to the type of business data and the format requirements of the business table to obtain N rows of stored data, where each row of stored data corresponds to a group of business data.

[0100] Optionally, in the data processing device provided in the embodiment of the present application, the device also includes: a determination unit, which is used to determine the attribute information of the data to be analyzed according to the data analysis instruction when receiving the data analysis instruction sent by the client, and obtain the data to be analyzed from the second database according to the attribute information; a second processing unit, which is used to process the data to be analyzed according to the data analysis instruction, obtain the analysis result, and send the analysis result to the client.

[0101] Optionally, in the data processing device provided in an embodiment of the present application, when a data analysis instruction is received from a client, the device further includes: an acquisition unit for acquiring a second timestamp of the data analysis instruction; a judgment unit for judging whether there is data with the second timestamp in the second database; and an execution unit for synchronizing data between the first database and the second database until data with the second timestamp exists in the second database, if there is no data with the second timestamp in the second database, and executing the step of acquiring the data to be analyzed from the second database according to the attribute information.

[0102] Optionally, in the data processing device provided in the embodiment of the present application, the first processing unit 42 includes: a second acquisition module, used to acquire the data fields contained in the row-stored data, and obtain M data fields; a classification module, used to classify each data in the row-stored data according to the data field, and obtain M groups of feature data; a determination module, used to sort the data in each group of feature data according to the third timestamp of each feature data, and obtain M groups of sorted feature data, and determine each group of sorted feature data as a column-stored data, and obtain M column-stored data.

[0103] Optionally, in the data processing device provided in an embodiment of the present application, before receiving the business data sent by the client, the device also includes: a generation unit, used to generate a second database based on the first database, and read the table structure in the first database; an aggregation unit, used to aggregate the existing data in the first database according to the data fields in the table structure to obtain P initial column storage data, where P is a positive integer; a storage unit, used to set a storage area unit for each data field in the second database, and store the P initial column storage data in the storage area unit of the corresponding data field.

[0104] Optionally, in the data processing device provided in an embodiment of the present application, when the first database is a distributed database, the device further includes: a setting unit for setting the second database as a target role, wherein the target role is used to characterize that the second database is only allowed to receive data from the first database.

[0105] It should be noted that the receiving unit 41 and the first processing unit 42 correspond to steps S301 to S302 in Example 1. The examples and application scenarios implemented by the receiving unit 41 and the first processing unit 42 are the same as those of the corresponding steps, but are not limited to the contents disclosed in Example 1. It should be noted that the modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The modules can also be part of a device and can be run in the computer terminal 10 provided in Example 1.

[0106] Example 3

[0107] An embodiment of the present application may provide an electronic device, Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 Only one is shown) processor 1002, memory 1004, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0108] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0109] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive business data sent by the client, and process the business data in a row storage manner to obtain row storage data, and store the row storage data in the first database; process the row storage data in a column storage manner to obtain M column storage data, and store each column storage data in the storage area unit corresponding to each column storage data in the second database, wherein the second database is an extended database of the first database, and M is a positive integer.

[0110] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: obtain the first timestamp of the business data, and group the business data with the same first timestamp into a group to obtain N groups of business data, where N is a positive integer; arrange each group of business data according to the type of business data and the format requirements of the business table to obtain N rows of stored data, where each row of stored data corresponds to a group of business data.

[0111] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: when receiving the data analysis instruction sent by the client, determine the attribute information of the data to be analyzed according to the data analysis instruction, and obtain the data to be analyzed from the second database according to the attribute information; process the data to be analyzed according to the data analysis instruction, obtain the analysis results, and send the analysis results to the client.

[0112] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: obtain the second timestamp of the data analysis instruction; determine whether there is data with the second timestamp in the second database; if there is no data with the second timestamp in the second database, synchronize the first database with the second database until the data with the second timestamp exists in the second database, and execute the step of obtaining the data to be analyzed from the second database according to the attribute information.

[0113] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: obtain the data fields contained in the row-stored data to obtain M data fields; classify the individual data in the row-stored data according to the data fields to obtain M groups of feature data; sort the data in each group of feature data according to the third timestamp of each feature data to obtain M groups of sorted feature data, and determine each group of sorted feature data as a column-stored data to obtain M column-stored data.

[0114] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: generate a second database based on the first database, and read the table structure in the first database; aggregate the existing data in the first database according to the data fields in the table structure to obtain P initial column storage data, where P is a positive integer; set a storage area unit for each data field in the second database, and store the P initial column storage data in the storage area unit of the corresponding data field.

[0115] The processor may also call the information and application stored in the memory through the transmission device to perform the following steps: setting the second database as a target role, wherein the target role is used to indicate that the second database is only allowed to receive data from the first database.

[0116] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 5 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 5 Different configurations shown.

[0117] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0118] Example 4

[0119] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in the first embodiment.

[0120] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0121] An embodiment of the present application also provides a computer program product, which is adapted to execute the steps of the data processing method when executed on a data processing device.

[0122] An embodiment of the present application further provides a computer-readable storage medium, which includes a stored executable program, wherein when the executable program runs, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned data processing method.

[0123] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0124] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0126] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0127] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0129] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Receive business data sent by the client, process the business data in a row storage manner to obtain row storage data, and store the row storage data in a first database; The row-stored data is processed in a column-stored manner to obtain M column-stored data, and each column-stored data is stored in a storage area unit corresponding to each column-stored data in a second database, wherein the second database is an extended database of the first database, and M is a positive integer.

2. The method according to claim 1, characterized in that The business data is processed according to the row storage mode, and the obtained row storage data includes: Obtaining a first timestamp of the service data, and grouping the service data having the same first timestamp into one group, to obtain N groups of service data, where N is a positive integer; Each group of business data is arranged according to the type of business data and the format requirements of the business table to obtain N rows of stored data, where each row of stored data corresponds to a group of business data.

3. The method according to claim 1, characterized in that The method further comprises: Upon receiving the data analysis instruction sent by the client, determining attribute information of the data to be analyzed according to the data analysis instruction, and acquiring the data to be analyzed from the second database according to the attribute information; The data to be analyzed is processed according to the data analysis instruction to obtain an analysis result, and the analysis result is sent to the client.

4. The method according to claim 3, characterized in that In the case of receiving the data analysis instruction sent by the client, the method further includes: Obtaining a second timestamp of the data analysis instruction; Determine whether the data at the second timestamp exists in the second database; If the data at the second timestamp does not exist in the second database, the first database and the second database are synchronized until the data at the second timestamp exists in the second database, and the step of obtaining the data to be analyzed from the second database according to the attribute information is executed.

5. The method according to claim 1, characterized in that The row-stored data is processed in a column-stored manner to obtain M column-stored data including: Obtaining data fields contained in the row-stored data to obtain M data fields; Classify each data in the row-stored data according to the data field to obtain M groups of feature data; The data in each set of feature data are sorted according to the third timestamp of each feature data to obtain M sets of sorted feature data, and each set of sorted feature data is determined as a column-stored data to obtain the M column-stored data.

6. The method according to claim 1, characterized in that Before receiving the service data sent by the client, the method further includes: Generate the second database according to the first database, and read the table structure in the first database; Aggregating the stock data in the first database according to the data fields in the table structure to obtain P initial column-stored data, where P is a positive integer; A storage area unit for each data field is set in the second database, and the P initial column data are stored in the storage area unit corresponding to the data field.

7. The method according to claim 1, characterized in that In the case where the first database is a distributed database, the method further includes: The second database is set as a target role, wherein the target role is used to indicate that the second database is only allowed to receive data from the first database.

8. A data processing system, characterized in that: include: The client is configured to receive business data input by a user and send the business data to the first database for storage; The first database is used to store the service data sent by the client; A row storage engine, configured to convert business data into row storage data in a storage format required by the first database before performing a storage operation on the first database; A column storage engine, configured to receive the row storage data sent by the row storage engine, convert the row storage data into column storage data, and store the column storage data in a second database; The second database is used to store the column-stored data and output the column-stored data according to the analysis requirements of the analysis system when a business analysis operation is required.

9. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory storing an executable program; A processor, configured to run the program, wherein the program, when running, executes the data processing method according to any one of claims 1 to 7.

Images