Data processing method, data query method, device and electronic equipment

Abstract

The application provides a data processing method, a data query method, a device and an electronic equipment, and relates to the technical field of big data processing. The method comprises the following steps: acquiring a data type of first real-time data to be processed; determining a target field to be updated and a row key for querying a database according to the first real-time data; in the case that the data type is a statistical type, querying batch data and second real-time data stored in a column family of the database according to the target field and the row key, so as to determine a target value corresponding to the target field; and updating the second real-time data according to the first real-time data and the target value. In the application, batch data and real-time data are stored in the database at the same time, so that the real-time data is not covered when the batch data is updated, and then the target value is determined from the batch data or the second real-time data, thereby improving the accuracy of the benchmark data for real-time data updating, and improving the accuracy of real-time data processing.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a data processing method, a data query method, an apparatus, and an electronic device. Background Technology

[0002] In the field of financial big data, user behavior data, transaction data, and other data are processed to provide reliable real-time results queries to support various scenarios that require real-time data feedback.

[0003] Data processing mainly includes batch data processing and real-time data processing. Batch data processing typically involves performing batch calculations on data and storing it in a database at a set time, such as midnight every day. Real-time data processing, on the other hand, processes the acquired data in real time and stores it in a database.

[0004] In related technologies, when high data accuracy is required, a combination of real-time data processing and batch data processing is used. However, when using both types of data processing, improving the accuracy of real-time data processing is a technical problem that needs to be solved. Summary of the Invention

[0005] This application provides a data processing method, a data query method, an apparatus, and an electronic device to solve the problem of low accuracy in real-time data processing in the prior art.

[0006] On the one hand, this application provides a data processing method, including:

[0007] Obtain the data type of the first real-time data to be processed;

[0008] Based on the first real-time data, determine the target field to be updated and the row key for querying the database;

[0009] When the data type is statistical, the target value corresponding to the target field is determined by querying the batch data and the second real-time data stored in the column family of the database based on the target field and the row key.

[0010] The second real-time data is updated based on the first real-time data and the target value.

[0011] On the other hand, this application provides a data query method, the method comprising:

[0012] Obtain the data query request sent by the terminal device;

[0013] Based on the query parameters carried in the data query request, the database is queried to obtain the first candidate query data in the batch data and the second candidate query data in the real-time data; the real-time data is updated using the aforementioned data processing method.

[0014] The target query data is determined based on the time of the first candidate query data and the time of the second candidate query data;

[0015] The target query data is sent to the terminal device.

[0016] On the other hand, this application provides a data processing apparatus, including:

[0017] The acquisition module is used to acquire the data type of the first real-time data to be processed;

[0018] The determination module is used to determine the target field to be updated and the row key for querying the database based on the first real-time data;

[0019] The query module is used to query the batch data and second real-time data stored in the column family of the database according to the target field and the row key when the data type is statistical type, so as to determine the target value corresponding to the target field;

[0020] An update module is used to update the second real-time data based on the first real-time data and the target value.

[0021] On the other hand, this application provides a data query device, the device comprising:

[0022] The acquisition module is used to acquire data query requests sent by the terminal device;

[0023] The query module is used to query the database according to the query parameters carried in the data query request to obtain the first candidate query data in the batch data and the second candidate query data in the real-time data; the real-time data is updated using the aforementioned data processing device.

[0024] The determination module is used to determine the target query data based on the time of the first candidate query data and the time of the second candidate query data;

[0025] The sending module is used to send the target query data to the terminal device.

[0026] On the other hand, embodiments of this application propose an electronic device, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a data processing method as described in one aspect, or a data query method as described in another aspect.

[0027] On the other hand, embodiments of this application propose a computer-readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the data processing method described in one aspect above, or to perform the data query method described in another aspect above.

[0028] On the other hand, embodiments of this application propose a computer program product, including a computer program that, when executed by a processor, implements the data processing method as described in one aspect above, or implements the data query method as described in another aspect above.

[0029] The data processing method, data query method, apparatus, and electronic device provided in this application acquire the data type of the first real-time data to be processed. Based on the first real-time data, the target field to be updated and the row key for querying the database are determined. If the data type is statistical, the batch data and the second real-time data stored in the column family of the database are queried based on the target field and the row key to determine the target value corresponding to the target field. The second real-time data is then updated based on the first real-time data and the target value. This application avoids overwriting real-time data during batch data updates by simultaneously storing batch data and real-time data in the database. The target value is then determined from either the batch data or the second real-time data, improving the accuracy of the baseline data for real-time data updates. Furthermore, the second real-time data is updated based on the target value used as the update baseline data and the first real-time data, thus improving the accuracy of real-time data processing. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0031] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0032] Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0033] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0034] Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0035] Figure 5 A flowchart illustrating a data query method provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0037] Figure 7 This is a schematic diagram of the structure of the data query device provided in the embodiments of this application;

[0038] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0039] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0041] In related technologies, batch data processing is used to correct real-time data in conjunction with real-time data processing, which to some extent solves the data deviation caused by the loss of real-time data collected the previous day. However, for some busy systems, real-time data is continuously generated and calculated, while batch data processing can only process data up to a certain time. This can lead to the problem of batch data processing results overwriting real-time data within a certain time period. For example, if the source system pushes the previous day's data to the target system at a certain time after midnight, and the target system triggers batch job scheduling and processes the data, completing the database update at time t1, then within the time window from midnight to t1, if user behavior data or transaction data is processed and stored in the database by the real-time stream processing engine, it may be overwritten by the batch data, meaning that real-time data within the time window from midnight to t1 is lost. This results in inaccurate real-time data for some users, which can only be corrected after the next batch job, reducing the accuracy of real-time data.

[0042] To address the aforementioned technical problems, this application proposes the following technical solution: Obtain the data type of the first real-time data to be processed; determine the target field to be updated and the row key for querying the database based on the first real-time data; if the data type is statistical, query the batch data and the second real-time data stored in the column family of the database based on the target field and the row key to determine the target value corresponding to the target field; update the second real-time data based on the first real-time data and the target value. By simultaneously storing batch data and real-time data in the database, batch data overwriting real-time data is avoided. Instead, the target value for updating is determined from either the batch data or the real-time data, and the second real-time data in the column family of the database is updated based on the target value and the first real-time data, thereby improving the accuracy of real-time data processing.

[0043] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0044] It should be noted that the acquisition, processing, storage, and application of data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0045] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0046] Step 101: Obtain the data type of the first real-time data to be processed.

[0047] In one implementation of this application, log data generated by the data source system is collected by the distributed log collection system Flume and written in real time to the high-throughput distributed publish-subscribe messaging system Kafka, or the source system is written asynchronously to the corresponding message set Topic in Kafka directly through the application programming interface (API) call. Then, the streaming data processing system Storm subscribes to the Topic from Kafka and consumes the first real-time data to be stored one by one in real time.

[0048] Log data includes behavioral logs, transaction logs, and database logs.

[0049] The data types of the first real-time data include statistical types and status types. Statistical types refer to data that accumulates over time, such as transaction data, click data, and browsing data, etc., which are not listed here. Status data is data that does not need to be accumulated, such as user mobile banking signing data.

[0050] Step 102: Based on the first real-time data, determine the target field to be updated and the row key for querying the database.

[0051] Real-time data is processed and stored in a database, such as HBase, a distributed, column-oriented open-source NoSQL database. In the database, data storage is based on tables, each containing a row key. Every row in the table must have a row key, which can be considered the primary key for queries. By using the row key, data from all columns of a row can be retrieved to determine the scope of the query. Specifically, the data retrieved using the row key is the data associated with that row key.

[0052] For example, if the first real-time data is a user's mobile banking contract data, then the corresponding row key is the user's mobile phone number. The data stored in the multiple columns associated with the mobile phone number are the contract status, contract date, contracting institution, and the time of the first real-time data, etc. By using the user's row key, i.e., the mobile phone number, you can query the relevant data associated with that user.

[0053] In this embodiment of the application, the target field to be updated can be determined by parsing the first real-time data. For example, if the first real-time data is mobile banking transaction data, then the target field to be updated can be determined as: phone_account.

[0054] Step 103: If the data type is statistical, query the batch data and second real-time data stored in the column family of the database according to the target field and row key to determine the target value corresponding to the target field.

[0055] Batch data can be updated at set intervals, such as starting updates 10 minutes after midnight every day. Real-time data, on the other hand, is updated in real-time. To prevent batch data updates from overwriting some real-time data, batch and real-time data are stored in different columns within a column family in the database. For example, both real-time and batch data for mobile banking are stored in column family F1 of the HBase database. Any number of columns can be created in column family F1 as needed, and each column has a corresponding column name.

[0056] In this embodiment, when the data type of the first real-time data is statistical, the data of each column corresponding to the row indicated by the row key can be located from the column family of the database based on the row key of the first real-time data. This includes stored batch data and second real-time data. Then, based on the target field to be updated, the time of the first data corresponding to the target field to be updated can be determined from the batch data, and the time of the second data corresponding to the target field to be updated can be determined from the second real-time data. By comparing the time of the first data and the time of the second data, a target value can be determined from either the first data or the second data. The target value is the most recent or latest data.

[0057] Among them, the row keys are usually identification information, such as mobile phone numbers, ID card numbers, etc.

[0058] In this process, batch data is processed and stored in the column families of the database in the following ways:

[0059] The source system unloads the incremental or full data files named with the data date from the previous day at the agreed time after midnight every day, and synchronizes the data files to the target system through file transfer protocols such as File Transfer Protocol (FTP). After receiving the batch data files, the target system performs batch calculations using Hive / Spark, processes the data according to preset rules, and synchronizes it to the column families of the database. For records with updated data, the batch data date is synchronized to the date of the currently processed data file.

[0060] Step 104: Update the second real-time data based on the first real-time data and the target value.

[0061] In this embodiment of the application, an update value is determined based on the first value and the target value of the target field in the first real-time data. That is, the update value is obtained by superimposing the first value on the target value. The update value is used to update the data of the target field in the second real-time data, and the time of the first real-time data is used to update the time corresponding to the data of the target field in the second real-time data.

[0062] For example, in mobile banking, the first real-time data is user A's transaction data on August 15, 2022, with one transaction. The row key is user A's mobile phone number, and the target field is the number of transactions. A query is performed in the database based on user A's mobile phone number and the number of transactions. Data corresponding to the target field is found in the batch data, indicating three transactions on August 14, 2022. Since the first real-time data also shows three transactions on August 14, 2022, the target value is determined to be three. However, the first real-time data only shows one transaction, so the determined update value is four. Therefore, the target field in the second real-time data is updated to four transactions, and August 15, 2022, is used as the date corresponding to the target field in the second real-time data.

[0063] In the data processing method of this application embodiment, the data type of the first real-time data to be processed is obtained. Based on the first real-time data, the target field to be updated and the row key for querying the database are determined. If the data type is statistical, the batch data and the second real-time data stored in the column family of the database are queried based on the target field and the row key to determine the target value corresponding to the target field. Based on the first real-time data and the target value, the second real-time data is updated. In this application, by storing batch data and real-time data in the database at the same time, the overwriting of real-time data when updating batch data is avoided. The target value is then determined from the batch data or the second real-time data, which improves the accuracy of the benchmark data for real-time data updates. Furthermore, based on the target value as the benchmark data for updates and the first real-time data, the second real-time data is updated, which improves the accuracy of real-time data processing.

[0064] Based on the above embodiments, Figure 2 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 2 As shown, the method includes:

[0065] Step 201: Obtain the data type of the first real-time data to be processed.

[0066] Step 202: Based on the first real-time data, determine the target field to be updated and the row key for querying the database.

[0067] Steps 201 and 202 can be explained in the foregoing embodiments, as the principle is the same, and will not be repeated here.

[0068] Step 203: If the data type is statistical, determine the service type corresponding to the first real-time data, and determine the target column family in the database based on the service type.

[0069] The service type refers to the way services are provided to users, including mobile banking services, digital RMB wallet services, and branch counter services.

[0070] In this embodiment of the application, when the data type of the first real-time data is determined to be statistical type, the target column family corresponding to the service type for storing the first real-time data is determined from multiple column families in the database according to the service type and the correspondence between the service type and the column family. The target column family is one of the multiple column families.

[0071] As an example, the column families for storing each service type can be pre-defined. For instance, the service types include mobile banking services and digital RMB wallet services. The data for mobile banking can be stored in column family F1, while the data for digital RMB wallets can be stored in column family F2.

[0072] Step 204: Based on the row key, query the time of the first data corresponding to the target field in the batch data stored in the target column family, and query the time of the second data corresponding to the target field in the second real-time data stored in the target column family.

[0073] The second real-time data is the real-time data that has already been processed and stored in the database. It is called the second real-time data in order to distinguish it from the first real-time data to be processed.

[0074] In this embodiment, one or more columns of data in a target column family can be queried based on the row key. Different columns store batch data and second real-time data respectively. Thus, based on the row key, the time of the first data corresponding to the target field in the batch data stored in the target column family and the time of the second data corresponding to the target field in the second real-time data stored in the target column family can be queried. As one implementation, the batch data is stored in a first designated column in the target column family, and the second real-time data is stored in a second designated column in the target column family. This realizes that the batch data and real-time data are stored in different storage locations in the database, avoiding data overwriting.

[0075] As an example, taking two users as an example, Table 1 below shows the batch data and second real-time data of the two users corresponding to the mobile banking terminal stored in the database.

[0076] Table 1

[0077]

[0078] DT indicates time. In this embodiment, the time granularity is based on date. In practical applications, a finer time granularity can also be used, but this application does not limit it. The first real-time data is the transaction data of user A on the mobile banking terminal. The time is August 21, 2022, the number of transactions is 1, the transaction data is of statistical type, and the row key is the user's mobile phone number, i.e., 137xxxx1072.

[0079] Therefore, based on the primary key 137xxxx1072, the batch data and the second real-time data in column family F1 can be searched. One implementation method is to determine the batch data and the second real-time data based on the first designated column allocated to the batch data and the second designated column allocated to the second real-time data in column family F1. Another implementation method distinguishes the batch data and the second real-time data using an identifier. Specifically, an "RT" identifier can be added to the second real-time data to indicate that it belongs to real-time data, i.e., the second real-time data. This allows the determination of whether the data stored in each column is batch data or second real-time data based on the data in the multiple columns included in column family F1.

[0080] Furthermore, the time of determining the first data from the found batch data, that is, the time of determining the number of transactions of user A in the matching data, and the time of determining the second data from the found second real-time data, that is, the time of determining the number of transactions of user A in the second real-time data.

[0081] It should be noted that the row key can be information used to uniquely identify a user, such as a user's mobile phone number, ID card number, or nickname. Different service types may use different row keys for queries.

[0082] Step 205: Compare the time of the first data and the time of the second data to determine the target value from the first data or the second data.

[0083] In this embodiment of the application, the time of the first data and the time of the second data are compared to determine the target value from the first data or the second data. The target data is the latest or most recent user data, and the latest user data is selected based on the data date, which ensures the accuracy of the real-time data determination.

[0084] In one scenario, if the time of the first data is earlier than the time of the second data, the second data is used as the target value. For example, if the time of the first data is August 20, 2022, and the time of the second data is August 21, 2022, then the second data is the closest in time and is used as the target value.

[0085] In the second scenario, if the time of the first data is later than the time of the second data, the first data is used as the target value. For example, if the time of the first data is August 21, 2022, and the time of the second data is August 20, 2022, then the time of the first data is closer, so the first data is used as the target value.

[0086] Step 206: Update the second real-time data based on the first real-time data and the target value.

[0087] Step 206 can be explained in the foregoing embodiments, as the principle is the same, and will not be repeated here.

[0088] In this embodiment, by storing batch data and second real-time data simultaneously in the database, the problems of data loss and duplicate consumption that exist with single real-time streaming data are avoided, the accuracy of the data is improved, and the timeliness of the data is improved for scenarios where batch data cannot meet the needs of real-time marketing and real-time recommendation. Furthermore, based on the time of the stored batch data and the time of the real-time data, the accuracy of the real-time data is ensured during the processing of the real-time data, thereby improving the accuracy of subsequent data queries.

[0089] Based on the above embodiments, Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 3 As shown, the method includes:

[0090] Step 301: Obtain the data type of the first real-time data to be processed.

[0091] Step 302: Based on the first real-time data, determine the target field to be updated and the row key for querying the database.

[0092] Step 303: If the data type is statistical, query the time of the first data corresponding to the target field in the batch data stored in the column family of the database, and query the time of the second data corresponding to the target field in the second real-time data stored in the column family of the database, based on the row key.

[0093] Step 304: Compare the time of the first data and the time of the second data.

[0094] Step 305: If the time of the first data is earlier than the time of the second data, then the second data is taken as the target value.

[0095] Step 306: If the time of the first data is later than the time of the second data, the first data is taken as the target value.

[0096] Step 307: Update the second real-time data based on the first real-time data and the target value.

[0097] Steps 301 to 307 can be explained in the foregoing embodiments, as the principle is the same, and will not be repeated here.

[0098] Step 308: If the data type is a status type, update the second real-time data based on the first real-time data.

[0099] Typically, status-type data does not need to be overlaid. Therefore, the second real-time data is updated based on the first real-time data. Specifically, the target column family in the database is determined based on the service type of the first real-time data. This target column family stores data of the same service type as the first real-time data. For example, if the service type of the first real-time data is mobile banking data, then the target column family stores data generated by mobile banking, such as transaction data and status data. Furthermore, the columns in the target column family used to store the second real-time data are determined based on the row key of the first real-time data. In one scenario, if the status data corresponding to the row key of the first real-time data is already stored in the database, the values ​​of each field in the already stored status data are updated based on the first real-time data. In another scenario, if there is no status data corresponding to the row key of the first real-time data, the first real-time data is parsed to determine the values ​​of each field corresponding to the status data, and these values ​​are stored in the target column family corresponding to the fields of the row key.

[0100] As an example, the first real-time data to be processed is the mobile banking contract data. Parsing the first real-time data yields the following data:

[0101] The mobile phone number registered for mobile banking;

[0102] The mobile banking subscription status is represented by A, where "1" indicates a normal subscription and "0" indicates a cancelled subscription.

[0103] The signing date for mobile banking is represented by B;

[0104] The institutions that sign up for mobile banking services are represented by C;

[0105] The data date of the first real-time data is represented by DT, which is the date the real-time data was generated.

[0106] The F1 column family contains at least one column, each used to store data for designated fields. This means some columns in the F1 column family are used to store fields corresponding to batch data, while others are used to store fields corresponding to real-time data. Each column uses the column family as a prefix. To facilitate differentiation, the fields corresponding to the contracted data in the batch data are stored in the first designated column of the F1 column family, denoted as F1:DT, F1:A, F1:B, and F1:C. The fields corresponding to the contracted data in the second real-time data are stored in the second designated column of the F1 column family, denoted as RT_DT, RT_A, RT_B, and RT_C. Each field is stored in the second designated column of F1.

[0107] For example, mobile user 130XXXX0001 activated mobile banking on August 27, 2022. The first real-time data generated is as follows:

[0108] Mobile phone number: 130XXXX0001;

[0109] A = 1 (normal contract signing);

[0110] B = 20220827;

[0111] C = 440xxx001;

[0112] DT = 20220827.

[0113] If the first real-time data mentioned above is consumed in real time, it will be updated in column family F1 of the database on August 27, 2022. The row key is the mobile phone number, and the corresponding columns are F1:RT_DT, F1:RT_A, F1:RT_B, and F1:RT_C. Based on the first real-time data of mobile phone user 130XXXX0001, the second real-time data will be updated to obtain the data corresponding to columns F1:RT_DT, F1:RT_A, F1:RT_B, and F1:RT_C on the right side of Table 2. At this point, the batch data update time has not yet arrived, so the batch data has not been updated. Normally, after midnight on August 28, 2022, all mobile banking transaction data for August 27, 2022 will be updated to column family F1 in the database. This includes the transaction data for mobile user 130XXXX0001. After processing, this data is stored in designated columns within column family F1, specifically F1:DT, F1:A, F1:B, and F1:C. See Table 2 below for details.

[0114] Table 2

[0115]

[0116] Here, A, B, and C are all different fields. Field A represents the mobile banking contract signing, field B represents the mobile banking contract signing date, and field C represents the mobile banking contract signing institution. The fields and column family identifiers are used to indicate a column in the column family.

[0117] Furthermore, in one scenario, if the user cancels their mobile banking account on the morning of August 28, 2022, and another piece of pending real-time data is obtained, then the first piece of real-time data would be:

[0118] Mobile phone number: 130XXXX0001;

[0119] Contract status: 0 (contract cancellation);

[0120] The first real-time data was generated on August 28, 2022.

[0121] Based on the first real-time data, the second real-time data in the database can be updated, that is, the value of F1:RT_DT in Table 2 is updated from 20220827 to 20220828, and F1:RT_A is set to 0.

[0122] In this embodiment, by storing batch data and second real-time data simultaneously in the database, the problems of data loss and duplicate consumption that exist with single real-time streaming data are avoided, the accuracy of the data is improved, and the timeliness of the data is improved for scenarios where batch data cannot meet the needs of real-time marketing and real-time recommendation. Furthermore, based on the time of the stored batch data and the time of the real-time data, the accuracy of the real-time data is ensured during the processing of the real-time data, thereby improving the accuracy of subsequent data queries.

[0123] Based on the above embodiments, Figure 4 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 4 As shown, after step 104, the following steps are also included:

[0124] Step 401: Obtain the first lifespan of the batch data and the second lifespan of the second real-time data.

[0125] In this embodiment, both batch data and second real-time data are stored in a database. Without a cleanup mechanism, this can lead to a large accumulation of historical real-time results data from inactive users, resulting in wasted storage resources and decreased query efficiency. Therefore, a first lifespan for batch data and a second lifespan for second real-time data can be set. The first lifespan and the second lifespan can be the same or different.

[0126] As an example, the database can be a NoSQL database like HBase, where column families can be configured with a time-to-live mechanism. This allows setting a first time-to-live for batch data and a second time-to-live for second-real-time data based on the time-to-live mechanism.

[0127] Step 402: If the first survival time of the batch data is longer than the set time, delete the matching data.

[0128] Step 403: If the second survival time of the second real-time data is longer than the set time, delete the second real-time data.

[0129] In this embodiment, data that has not been updated after exceeding the lifespan mechanism is automatically expired and cleaned up. Specifically, if the first lifespan of batch data is longer than a set time, matching data is deleted, and if the second lifespan of second real-time data is longer than a set time, second real-time data is deleted, thereby greatly reducing storage pressure and improving query efficiency.

[0130] As an example, the initial lifespan of batch data can be set to 7 days, meaning that batch jobs can be delayed for up to 7 days without data loss affecting queries. For real-time data, only user data generated within the last 7 days will be saved to avoid impacting data queries.

[0131] In this embodiment of the application, by setting a first survival time for batch data and a second survival time for second real-time data, batch data or second real-time data that exceeds the corresponding survival time is deleted, thereby greatly reducing storage pressure and improving query efficiency.

[0132] Based on the above embodiments, Figure 5 This is a flowchart illustrating a data query method provided in an embodiment of this application. Figure 5 As shown, it includes:

[0133] Step 501: Obtain the data query request sent by the terminal device.

[0134] The terminal device can be a smartphone, a PDA, a smart wearable device, etc., which will not be listed here or limited.

[0135] The execution entity in this embodiment is a processing node in a distributed system, used to provide query services.

[0136] In one implementation of this application, the processing node can also be understood as a data service module for providing services. It obtains the data query request sent by the terminal device by calling the interface provided by the data service module to query specified data. The data query request carries query parameters, which may be different in different scenarios. For example, if the terminal device wants to query the user's contract status data, the query parameters include the user's mobile phone number, service type, or contract method (mobile banking).

[0137] Step 502: Based on the query parameters carried in the data query request, query the database to obtain the first candidate query data in the batch data and the second candidate query data in the real-time data.

[0138] The real-time data is updated using the data processing method described in the previous embodiments. For details, please refer to the explanation in the previous embodiments. The principle is the same and will not be repeated here.

[0139] Step 503: Determine the target query data based on the time of the first candidate query data and the time of the second candidate query data.

[0140] In this embodiment, based on the query parameters carried in the data query request, the system queries the database for first candidate query data in the batch data corresponding to the query parameters and second candidate query data in the real-time data. Then, the time of the first candidate query data and the time of the second candidate query data are compared. If the time of the first candidate query data in the batch data is earlier than the time of the second candidate query data in the real-time data, it means that the time of the second candidate query data in the real-time data is closer to the current time, that is, the time of the second candidate query data is updated. In this case, the second candidate query data is used as the target query data. Otherwise, the first candidate query data is used as the target query data.

[0141] Step 504: Send the target query data to the terminal device.

[0142] The target query data is sent as the query result to the terminal device so that the user of the corresponding terminal device can obtain the latest real-time data, thereby improving the accuracy of data acquisition.

[0143] In the data query method of this application embodiment, both batch and real-time data dates are stored simultaneously to provide a unified data service. The correct latest user data is selected and returned based on the data date, which improves the accuracy of the queried data.

[0144] Based on the above embodiments, Figure 6 This is a schematic diagram of the structure of the data processing apparatus provided in an embodiment of this application. Figure 6 As shown, the six devices include:

[0145] The acquisition module 61 is used to acquire the data type of the first real-time data to be processed;

[0146] The determination module 62 is used to determine the target field to be updated and the row key for querying the database based on the first real-time data;

[0147] The query module 63 is used to query the batch data and the second real-time data stored in the column family of the database according to the target field and the row key when the data type is statistical type, so as to determine the target value corresponding to the target field;

[0148] The update module 64 is used to update the second real-time data based on the first real-time data and the target value.

[0149] Furthermore, as one implementation method, query module 63 is specifically used for:

[0150] Based on the row key, query the time of the first data corresponding to the target field in the batch data stored in the column family of the database, and query the time of the second data corresponding to the target field in the second real-time data stored in the column family of the database;

[0151] The time of the first data and the time of the second data are compared to determine the target value from the first data or the second data.

[0152] As one implementation method, the query module 63 is also specifically used to determine the service type corresponding to the first real-time data;

[0153] Based on the service type, determine the target column family in the database;

[0154] Based on the row key, query the time of the first data corresponding to the target field in the batch data stored in the target column family, and query the time of the second data corresponding to the target field in the second real-time data stored in the target column family.

[0155] As one implementation, the query module 63 is further used to: if the time of the first data is earlier than the time of the second data, use the second data as the target value;

[0156] If the time of the first data is later than the time of the second data, the first data is used as the target value.

[0157] As one implementation, the update module 64 is also used to update the second real-time data based on the first real-time data when the data type is a state type.

[0158] As one implementation, the update module 64 is also used to determine the update value based on the first value of the target field in the first real-time data and the target value;

[0159] The updated value is used to update the data of the target field in the second real-time data;

[0160] The time of the first real-time data is used to update the time corresponding to the target field in the second real-time data.

[0161] As one implementation, the device further includes:

[0162] The deletion module is specifically used to obtain the first survival time of the batch data and the second survival time of the second real-time data; if the first survival time of the batch data is greater than the set time, delete the matching data; if the first survival time of the second real-time data is greater than the set time, delete the second real-time data.

[0163] The data processing apparatus provided in this application embodiment can be used to execute the technical solution of the data processing method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0164] Based on the above embodiments, Figure 7 This is a schematic diagram of the structure of the data query device provided in an embodiment of this application. Figure 7 As shown, the 7 devices include:

[0165] The acquisition module 71 is used to acquire data query requests sent by the terminal device.

[0166] The query module 72 is used to query the database according to the query parameters carried in the data query request to obtain the first candidate query data in the batch data and the second candidate query data in the real-time data; the real-time data is updated by the method described in the foregoing method embodiments.

[0167] The determination module 73 is used to determine the target query data based on the time of the first candidate query data and the time of the second candidate query data.

[0168] The sending module 74 is used to send the target query data to the terminal device.

[0169] Furthermore, as one implementation, the determining module 73 is specifically used for:

[0170] Compare the time of the first candidate query data with the time of the second candidate query data;

[0171] If the time of the first candidate query data is earlier than the time of the second candidate query data, then the second candidate query data will be used as the target query data.

[0172] If the time of the first candidate query data is later than the time of the second candidate query data, then the first candidate query data will be used as the target query data.

[0173] The data query device provided in this application embodiment can be used to execute the technical solution of the data query method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0174] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0175] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device may include: transceiver 121, processor 122, and memory 123.

[0176] Processor 122 executes computer execution instructions stored in memory, causing processor 122 to perform the scheme in the above embodiments. Processor 122 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0177] The memory 123 is connected to the processor 122 via the system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0178] Transceiver 121 can be used to obtain the task to be run and its configuration information.

[0179] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0180] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0181] This application also provides a chip for executing instructions, which is used to execute the technical solutions of the methods described in the above embodiments.

[0182] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the methods described in the above embodiments.

[0183] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solution of the method in the above embodiments.

[0184] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0185] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized by, include: Obtain the data type of the first real-time data to be processed; Based on the first real-time data, determine the target field to be updated and the row key for querying the database; When the data type is statistical, the time of the first data corresponding to the target field in the batch data is queried according to the row key, and the time of the second data corresponding to the target field in the second real-time data is queried; wherein, the batch data is stored in a first designated column in the target column family, and the second real-time data is stored in a second designated column in the target column family, so that the batch data and the real-time data are stored in different storage locations in the database; The time of the first data is compared with the time of the second data. If the time of the first data is earlier than the time of the second data, the second data is used as the target value corresponding to the target field. If the time of the first data is later than the time of the second data, the first data will be used as the target value corresponding to the target field. The update value is determined based on the first value of the target field in the first real-time data and the target value; The updated value is used to update the data of the target field in the second real-time data; The time of the first real-time data is used to update the time corresponding to the target field in the second real-time data.

2. The method as described in claim 1, characterized in that, The step of querying the time of the first data corresponding to the target field in the batch data stored in the column family of the database according to the row key, and querying the time of the second data corresponding to the target field in the second real-time data stored in the column family of the database, includes: Determine the service type corresponding to the first real-time data; Based on the service type, determine the target column family in the database; Based on the row key, query the time of the first data corresponding to the target field in the batch data stored in the target column family, and query the time of the second data corresponding to the target field in the second real-time data stored in the target column family.

3. The method as described in claim 1, characterized in that, The method further includes: When the data type is a status type, the second real-time data is updated based on the first real-time data.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Obtain the first survival time of the batch data and the second survival time of the second real-time data; If the first lifespan of the batch data exceeds a set duration, the batch data will be deleted. If the first survival time of the second real-time data is greater than the set time, the second real-time data is deleted.

5. A data query method, characterized in that, The method includes: Obtain the data query request sent by the terminal device; Based on the query parameters carried in the data query request, the database is queried to obtain the first candidate query data in the batch data and the second candidate query data in the real-time data; the real-time data is updated using any of the data processing methods described in claims 1-4. The target query data is determined based on the time of the first candidate query data and the time of the second candidate query data; The target query data is sent to the terminal device.

6. The method as described in claim 5, characterized in that, The step of determining the target query data based on the time of the first candidate query data and the time of the second candidate query data includes: Compare the time of the first candidate query data with the time of the second candidate query data; If the time of the first candidate query data is earlier than the time of the second candidate query data, then the second candidate query data will be used as the target query data. If the time of the first candidate query data is later than the time of the second candidate query data, then the first candidate query data will be used as the target query data.

7. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the data type of the first real-time data to be processed; The determination module is used to determine the target field to be updated and the row key for querying the database based on the first real-time data; The query module is used to query the time of the first data corresponding to the target field in the batch data and the time of the second data corresponding to the target field in the second real-time data, based on the row key, when the data type is statistical. The time of the first data is compared with the time of the second data. If the time of the first data is earlier than the time of the second data, the second data is used as the target value corresponding to the target field. If the time of the first data is later than the time of the second data, the first data is used as the target value corresponding to the target field. The batch data is stored in a first set column in the target column family, and the second real-time data is stored in a second set column in the target column family, so that the batch data and the real-time data are stored in different storage locations in the database. The update module is configured to determine an update value based on the first value of the target field in the first real-time data and the target value; update the data of the target field in the second real-time data using the update value; and update the time corresponding to the data of the target field in the second real-time data using the time of the first real-time data.

8. The apparatus as claimed in claim 7, characterized in that, The query module is further used for: Determine the service type corresponding to the first real-time data; Based on the service type, determine the target column family in the database; Based on the row key, query the time of the first data corresponding to the target field in the batch data stored in the target column family, and query the time of the second data corresponding to the target field in the second real-time data stored in the target column family.

9. The apparatus as claimed in claim 7, characterized in that, The update module is further used for: When the data type is a status type, the second real-time data is updated based on the first real-time data.

10. The apparatus according to any one of claims 7-9, characterized in that, The device further includes: The deletion module is used to obtain the first survival time of the batch data and the second survival time of the second real-time data, and delete the batch data if the first survival time of the batch data is greater than a set time, and delete the second real-time data if the first survival time of the second real-time data is greater than the set time.

11. A data query device, characterized in that, The device includes: The acquisition module is used to acquire data query requests sent by the terminal device; The query module is used to query the database according to the query parameters carried in the data query request to obtain the first candidate query data in the batch data and the second candidate query data in the real-time data; the real-time data is updated using the data processing device according to any one of claims 7-10; The determination module is used to determine the target query data based on the time of the first candidate query data and the time of the second candidate query data; The sending module is used to send the target query data to the terminal device.

12. The apparatus as claimed in claim 11, characterized in that, The determining module is specifically used for: Compare the time of the first candidate query data with the time of the second candidate query data; If the time of the first candidate query data is earlier than the time of the second candidate query data, then the second candidate query data will be used as the target query data. If the time of the first candidate query data is later than the time of the second candidate query data, then the first candidate query data will be used as the target query data.

13. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-4 or 5-6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-4 or 5-6.

15. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-4 or 5-6.

Images