Data processing method and apparatus

Abstract

This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium; relating to databases and storage in cloud technology, the method includes: receiving incremental data and writing the incremental data into a data storage system; updating the column-level timestamps corresponding to all columns contained in the incremental data according to the timestamp of the incremental data; receiving a data query request; querying the data storage system according to the key name carried in the data query request to obtain data corresponding to the key name; comparing the timestamp of the data with the latest timestamp at the column level of the column containing the data, and returning the corresponding query result according to the comparison result. This application can improve the efficiency of data filtering in a data storage system.

Description

Technical Field

[0001] This application relates to the field of database technology in cloud technology, and more particularly to a data processing method, apparatus, electronic device and computer-readable storage medium. Background Technology

[0002] In the era of the Internet, especially the mobile Internet, data is being generated at an increasingly faster pace, and the performance requirements for data storage and processing (such as querying) are becoming increasingly demanding.

[0003] When updating a data storage system in batches based on newly generated data, related technologies provide data filtering solutions. This involves marking data to be deleted with a "Delete" flag to prevent it from being returned during queries. Based on this approach, the old and new batches of data first need to be joined using the row key of a row in the data table. This means linking the two sets of data together by the row key and marking any data present in the old batch but not in the new batch with a "Delete" flag.

[0004] However, since the Join operation is time-consuming, the server's efficiency in responding to subsequent data query requests will be affected by the time spent on the Join operation. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium that can improve the efficiency of data filtering, thereby ensuring the efficiency of responding to data query requests.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides a data processing method, including:

[0008] Receive incremental data and write the incremental data into the data storage system;

[0009] Based on the timestamp of the incremental data, update the column-level timestamps corresponding to all columns contained in the incremental data;

[0010] Receive data query requests;

[0011] The data storage system is queried according to the key name carried in the data query request to obtain the data corresponding to the key name;

[0012] The timestamp of the data is compared with the latest timestamp at the column level of the column containing the data, and the corresponding query results are returned based on the comparison result.

[0013] This application provides a data processing apparatus, including:

[0014] The receiving module is used to receive incremental data;

[0015] The writing module is used to write the incremental data into the data storage system;

[0016] The update module is used to update the column-level timestamps corresponding to all columns contained in the incremental data according to the timestamps of the incremental data.

[0017] The receiving module is also used to receive data query requests;

[0018] The query module is used to query the data storage system based on the key name carried in the data query request in order to obtain the data corresponding to the key name;

[0019] The comparison module is used to compare the timestamp of the data with the latest timestamp at the column level of the column containing the data, and return the corresponding query results based on the comparison results.

[0020] In the above scheme, the comparison module is further configured to return a query result that does not contain the data when the latest timestamp at the column level of the column containing the data is greater than the timestamp of the data; and to return a query result that contains the data when the latest timestamp at the column level of the column containing the data is less than or equal to the timestamp of the data.

[0021] In the above scheme, the update module is further configured to update the corresponding index file according to the storage address of the incremental data in the data storage system, and record all columns contained in the incremental data in the index file.

[0022] In the above scheme, the update module is further configured to read all columns contained in the incremental data from the index file when performing a loading operation on the index file, and update the column-level timestamps corresponding to all columns respectively.

[0023] In the above scheme, the update module is also used to perform the following operations during the lifetime of the same lock: perform a loading operation on the index file; read all columns contained in the incremental data from the index file, and update the column-level timestamps corresponding to all columns respectively.

[0024] In the above scheme, the update module is further configured to perform the following processing for any column among all columns included in the incremental data: update the column-level timestamp corresponding to any column to be the same as the timestamp of the incremental data.

[0025] In the above scheme, the writing module is further configured to write the incremental data to the corresponding address in the data storage system when the incremental data is provided by multiple data sources and the data corresponding to different columns in the data storage system is updated based on the incremental data.

[0026] In the above scheme, the writing module is further configured to write the incremental data to the corresponding address in the data storage system when the incremental data is provided by multiple data sources and the data corresponding to the same column in the data storage system is updated based on the incremental data; the device further includes an adding module, configured to add a unique corresponding tag for each key name for different key names in the data storage system; and configured to record a column-level timestamp corresponding to the tag in the index file according to the tag.

[0027] In the above scheme, the query module is further configured to query the corresponding storage address in the index file based on the key name, and query the data storage system based on the storage address to obtain the data corresponding to the key name.

[0028] This application provides an electronic device, including:

[0029] Memory, used to store executable instructions;

[0030] The processor, when executing executable instructions stored in the memory, implements the data processing method provided in the embodiments of this application.

[0031] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the data processing method provided in this application.

[0032] The embodiments of this application have the following beneficial effects:

[0033] Based on the timestamp of the incremental data, the column-level timestamps of all columns contained in the incremental data are updated. In this way, data filtering can be achieved by comparing the timestamp of the data with the latest timestamp at the column level of the column containing the data. Since the time for actual data deletion is saved, the efficiency of data filtering is improved, thereby ensuring the efficiency of responding to data query requests. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the architecture of the data processing system provided in the embodiments of this application;

[0035] Figure 2 This is a schematic diagram of the server structure provided in an embodiment of this application;

[0036] Figure 3 This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0037] Figure 4 This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0038] Figure 5 This is a schematic diagram illustrating the application of the data processing method provided in the embodiments of this application;

[0039] Figure 6 This is a diagram illustrating the query results provided by the relevant technologies;

[0040] Figure 7 This is a schematic diagram of the query results provided in an embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0044] When performing data filtering, related technologies typically mark data to be deleted with a "Delete" flag, preventing it from being returned during queries. Based on this approach, the old and new batches of data first need to be joined using RowKey, and then data present in the old batch but not in the new batch is marked with a "Delete" flag. However, the Join operation is time-consuming, significantly impacting the server's efficiency in responding to subsequent data query requests due to the time spent on the Join operation.

[0045] In view of this, embodiments of this application provide a data processing method, apparatus, electronic device, and computer-readable storage medium that can improve the efficiency of data filtering, thereby ensuring the efficiency of responding to data query requests.

[0046] The following describes an exemplary application of the electronic device using the application data processing method provided in the embodiments of this application. The electronic device using the application data processing method provided in the embodiments of this application can be implemented as various types of user terminals such as laptops and desktop computers, or as a server, such as a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The following will combine... Figure 1 This describes an exemplary application of an electronic device that implements a data processing method as a server.

[0047] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the data processing system 100 provided in this application embodiment. In order to improve the efficiency of data filtering and ensure the efficiency of responding to data query requests, the data processing system 100 includes: server 200, network 300, terminal 400 and data storage system 500, which will be described below.

[0048] Server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 400 and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0049] Server 200 is used to receive real-time incremental data and store it in data storage system 500. It also updates the column-level timestamps of all columns contained in the received incremental data based on the received incremental data's timestamp. Server 200 is also used to receive data query requests sent by terminal 400 via network 300, and to query data storage system 500 based on the key name carried in the query request to retrieve the data corresponding to the key name. Finally, server 200 compares the timestamp of the retrieved data with the latest column-level timestamp of the column containing the data, and returns the corresponding query result to terminal 400 based on the comparison result.

[0050] In some embodiments, server 200 can be a backend server for various Internet applications. That is, the real-time incremental data received by server 200 can be data generated during the operation of various Internet applications, such as user profile data in a recommendation system, including information such as the user's age, gender, occupation, and hobbies; or user operation record data, including the user's installed application list, the sequence of advertisements clicked by the user, and the keywords entered by the user in a search engine.

[0051] Network 300 is used to connect server 200 and terminal 400. Network 300 can be a wide area network, a local area network, or a combination of both.

[0052] The terminal 400 runs a client 410, which is used to send data query requests to the server 200 via the network 300, and to receive query results sent by the server 200 via the network 300.

[0053] The data storage system 500 is used to store the real-time incremental data received by the server 200, as well as the latest timestamps at the column level corresponding to all columns contained in the incremental data.

[0054] The following is about Figure 1 The structure of server 200 in the document will be explained. See [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2 The server 200 shown includes at least one processor 210, memory 240, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 230. It is understood that the bus system 230 is used to implement communication between these components. In addition to a data bus, the bus system 230 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 230.

[0055] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0056] The memory 240 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 240 may optionally include one or more storage devices physically located away from the processor 210.

[0057] The memory 240 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 240 described in this application embodiment is intended to include any suitable type of memory.

[0058] In some embodiments, memory 240 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0059] Operating system 241 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0060] The network communication module 242 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0061] In some embodiments, the data processing apparatus provided in this application can be implemented in software. Figure 2 A data processing device 243 stored in memory 240 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a receiving module 2431, a writing module 2432, an updating module 2433, a query module 2434, a comparison module 2435, and an adding module 2436. These modules are logically linked and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0062] In other embodiments, the data processing apparatus provided in this application can be implemented in hardware. As an example, the data processing apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0063] The electronic device provided in the embodiments of this application will be implemented as a server (e.g., Figure 1 The exemplary application shown in the example of server 200 illustrates the data processing method provided in the embodiments of this application.

[0064] In step S301, incremental data is received and written into the data storage system.

[0065] In some embodiments, when updating data, not all the original data in the data storage system is updated, but only a portion of the original data is updated. That is, it is an incremental update relative to all the original data stored in the data storage system. Therefore, after receiving incremental data (e.g., real-time incremental data), the server writes the received incremental data into the data storage system to update a portion of the original data in the data storage system.

[0066] For example, refer to Table 1, which is a schematic table of data stored in the data storage system at time T1 according to an embodiment of this application. As shown in Table 1, at time T1, the data stored in the data storage system includes V1-V6. Taking the determination of user characteristics as an example, when the real-time incremental data received by the server comes from the same data source, col1, col2, and col3 in Table 1 can correspond to the user's age, gender, and hobbies, respectively; key1 and key2 can correspond to the IDs of user A and user B, respectively.

[0067] Table 1. Schematic diagram of data stored in the data storage system at time T1

[0068]

[0069] Suppose that at time T2, the server receives incremental data, including V7-V8 and V11-V12. Then, the server writes the received incremental data into the data storage system to update the data in the data representation table at time T1, resulting in the data stored in the data representation table at time T2 as shown in Table 2. As shown in Table 2, the server updates V1 corresponding to key1 and col1 to V7, V2 corresponding to key1 and col2 to V8, V5 corresponding to key2 and col2 to V11, and V6 corresponding to key2 and col3 to V12; however, V3 corresponding to key1 and col3 and V4 corresponding to key2 and col1 are not updated in this instance.

[0070] Table 2. Schematic diagram of data stored in the data storage system at time T2.

[0071]

[0072] In some embodiments, after the server writes the received incremental data into the data storage system, it may also perform the following operations: update the corresponding index file according to the storage address of the incremental data in the data storage system, and record all columns contained in the incremental data in the index file.

[0073] For example, to improve data retrieval efficiency, the server can read the original data stored in the data storage system and generate a corresponding index file based on the storage address of the original data. Therefore, after writing incremental data into the data storage system, the server can also update the index file based on the storage address of the incremental data in the data storage system. Furthermore, the server can additionally record all columns contained in the incremental data in the index file. Taking Table 2 as an example, the real-time incremental data V7-V8 and V11-V12 received by the server at time T2 contains columns col1, col2, and col3. Therefore, the server also additionally records col1, col2, and col3 in the index file.

[0074] In other embodiments, the real-time incremental data received by the server may be provided by multiple data sources. For example, to obtain user profiles, the real-time incremental data received by the server may be provided by different social media platforms. Taking Table 1 as an example, key1 and key2 in Table 1 may correspond to the accounts of the same user (e.g., user A) on two different social media platforms. Here, key1 may correspond to user A's account on Weibo, and key2 may correspond to user A's account on WeChat.

[0075] For example, when the server receives incremental data provided by multiple data sources, and updates the data corresponding to different columns in the data storage system based on this incremental data (i.e., multiple data sources updating the same set of keys in the data storage system, and the columns updated by these data sources do not overlap), incremental data can be written to the corresponding address in the data storage system. In this case, since the columns updated by the incremental data provided by multiple data sources do not overlap, there will be no situation where loading data that is already ready causes queries to fail to return results.

[0076] For example, when the server receives incremental data provided by multiple data sources, and updates data corresponding to at least one identical column in the data storage system based on this incremental data (i.e., updating the same column in the data storage system from multiple data sources, but updating different batches of keys), incremental data is written to the corresponding address in the data storage system. In this case, because the incremental data provided by multiple data sources updates the same column in the data storage system, loading late-ready data will cause queries on early-ready data to fail, as the column-level timestamp has been updated to the same timestamp as the late-ready data. To avoid situations where data readiness times differ but the same column is updated, the server can also perform the following operations: for different key names in the data storage system, add a unique corresponding tag for each key name, and record the column-level timestamp corresponding to the tag in the index file. That is, the server can add a unique corresponding tag for each batch of keys, and also maintain a set of column-level timestamps for each type of tag.

[0077] In step S302, the column-level timestamps corresponding to all columns contained in the incremental data are updated according to the timestamps of the incremental data.

[0078] In some embodiments, the server updates the column-level timestamps corresponding to all columns in the incremental data according to the timestamps of the incremental data. This can be achieved by: when performing a load operation on the index file, reading all columns in the currently received incremental data from the index file, and updating the column-level timestamps corresponding to all columns according to the timestamps of the incremental data.

[0079] For example, using Tables 1 and 2 above as examples, at time T1, the column-level timestamps corresponding to col1, col2, and col3 in Table 1 are all T1. At time T2, the server receives incremental data including V7-V8 and V11-V12, containing columns col1, col2, and col3, and records col1, col2, and col3 in the index file. Subsequently, when the server calls the query process to load the index file, it reads all the columns contained in the current index from the index file and updates the column-level timestamps of these columns. That is, the server updates the timestamps of V7-V8 and V11-V12 to T2 based on the incremental data, and also updates the column-level timestamps corresponding to col1, col2, and col3 to T2.

[0080] In other embodiments, to ensure that data can be filtered correctly, the server may also perform index file loading and column-level timestamp update operations in the following ways: During the lifetime of the same lock, perform the following operations: perform a loading operation on the index file; read all columns contained in the incremental data from the index file, and update the column-level timestamps corresponding to each column respectively.

[0081] For example, when the server loads the index file during the query process, it can perform both the index file loading operation and the column-level timestamp update operation using the same lock. This ensures that the column-level timestamp update and the index file loading operation are performed within the same lock, guaranteeing that the timestamp update and the index file loading time are consistent, thus ensuring that the data is filtered correctly. This is because if the column-level timestamp update operation occurs before the index file loading operation, data will be filtered incorrectly; conversely, if the timestamp update operation occurs after the index file loading operation, data filtering will fail.

[0082] In some embodiments, the server updates the column-level timestamps corresponding to all columns contained in the incremental data in the following way: for any column among all columns contained in the incremental data, perform the following processing: update the column-level timestamp corresponding to any column to be the same as the timestamp of the incremental data.

[0083] For example, using Table 2 above as an example, assuming that the incremental data received by the server at time T3 contains columns col1 and col2, the server updates the column-level timestamps corresponding to col1 and col2 to T3 according to the timestamp T3 of the currently received incremental data, while the column-level timestamp corresponding to col3 remains T2.

[0084] In step S303, a data query request is received.

[0085] In some embodiments, when a user (e.g., an operator) needs to obtain data, they can send a data query request to the server through a terminal, so that the server can respond upon receiving the data query request and return the corresponding query results to the terminal.

[0086] In step S304, the data storage system is queried according to the key name carried in the data query request to obtain the data corresponding to the key name.

[0087] In some embodiments, a user may send a data query request via a terminal that includes a key name, so that the server can query the data storage system based on the key name in the received data query request, thereby obtaining the data corresponding to the key name.

[0088] For example, the server queries the data storage system based on the key name carried in the data query request to obtain the data corresponding to the key name. This can be achieved in the following way: query the corresponding storage address in the index file based on the key name, and then query the data storage system based on the queried storage address to obtain the data corresponding to the key name in the data storage system.

[0089] In step S305, the timestamp of the data is compared with the latest timestamp at the column level of the column containing the data, and the corresponding query results are returned based on the comparison result.

[0090] In some embodiments, Figure 3 The illustrated step S305 can be achieved through Figure 4 The steps S3051 to S3053 shown are implemented, and will be combined with Figure 4 The steps shown are explained.

[0091] In step S3051, it is determined whether the latest timestamp at the column level of the column containing the data is greater than the timestamp of the data. If yes, step S3052 is executed; otherwise, step S3053 is executed.

[0092] In step S3052, the query results that do not contain data are returned.

[0093] In step S3053, the query results containing the data are returned.

[0094] In some embodiments, after the server retrieves the data corresponding to the key name in the data storage system based on the key name carried in the data query request, it may also perform the following operations: determine whether the latest timestamp at the column level of the column containing the retrieved data is greater than the timestamp of the data. If the latest timestamp at the column level of the column containing the data is greater than the timestamp of the data, it indicates that the retrieved data is not the latest data, and the server returns a query result that does not contain the data to the terminal; if the latest timestamp at the column level of the column containing the data is less than or equal to the timestamp of the data, it indicates that the retrieved data is the latest data, and the server returns a query result that contains the data to the terminal.

[0095] To illustrate, let's continue using Table 2 as an example. Assume the user's data query request sent via the terminal carries the key name `key1`. The server queries the data storage system based on the received key name `key1` to retrieve the data corresponding to `key1`, which are V7, V8, and V3. Next, the server compares the timestamps of V7, V8, and V3 with the latest timestamps at the column level of their respective columns. Specifically, the server compares the timestamp of V7 with the latest timestamp at the column level of `col1`, the timestamp of V8 with the latest timestamp at the column level of `col2`, and the timestamp of V3 with the latest timestamp at the column level of `col3`. Since the timestamp of V7 is T2, which is the same as the latest timestamp T2 at the column level of `col1`, and the timestamp of V8 is T2, which is the same as the latest timestamp T2 at the column level of `col2`, and the timestamp of V3 is T1, which is less than the latest timestamp T2 at the column level of `col3`, the server only returns data V7 and V8 to the terminal, and not data V3.

[0096] This application embodiment targets incremental data written in batches to a data storage system. Based on the timestamp of the incremental data, the column-level timestamps corresponding to all columns contained in the incremental data are updated. When a data query request is subsequently received, the timestamp of the data corresponding to the key name carried in the data query request can be directly compared with the latest timestamp at the column level of the column containing the data. The corresponding query result is returned based on the comparison result. In this way, the purpose of data filtering can be achieved by comparing the timestamp of the data with the timestamp at the column level of the column containing the data, thereby improving the efficiency of data filtering.

[0097] In other embodiments, the data processing method provided in this application can also be implemented in conjunction with blockchain technology.

[0098] Blockchain refers to an encrypted, chain-like storage structure of transactions formed by blocks. It is a shared database in which the data or information stored is characterized by being unforgeable, traceable, and collectively maintained.

[0099] For example, see Figure 5 , Figure 5 This is an application diagram of the data processing method provided in the embodiments of this application, including a blockchain network 600 (consensus nodes 610-1 to 610-3 are shown as an example), an authentication center 700, and business entities 800 / 900, which will be described below.

[0100] The types of blockchain networks are flexible and diverse; for example, they can be any type of public chain, private chain, or consortium chain. Taking a public chain as an example, any electronic device of a business entity (e.g., Figure 1 Both the server 200 and the terminal 400 in the blockchain can access the blockchain network 600 and become client nodes without authorization. Taking the consortium blockchain as an example, after obtaining authorization, the electronic devices under the business entity can access the blockchain network 600 and become client nodes.

[0101] As an example, when blockchain network 600 is a consortium blockchain, business entities 800 / 900 register with certification authority 700 to obtain their respective digital certificates. The digital certificates include the business entity's public key and the digital signature signed by certification authority 700 with the public key and identity information of business entities 800 / 900. This digital signature is attached to the transaction (e.g., for incremental data to be uploaded to the blockchain, or for data query requests) along with the business entity's digital signature for the transaction and sent to blockchain network 600. Blockchain network 600 then retrieves the digital certificate and digital signature from the transaction to verify the reliability of the transaction (i.e., whether it has been tampered with) and the identity information of the business entity that sent the message. Blockchain network 600 will verify the identity, such as whether it has the authority to initiate the transaction.

[0102] In some embodiments, client nodes may act only as observers of the blockchain network 600, providing support for business entities to initiate transactions. For the functions of the consensus nodes 610 of the blockchain network 600, such as sorting, consensus services, and ledger functions, the client nodes may implement them by default or selectively (e.g., depending on the specific business needs of the business entity). This allows for the maximum migration of the business entity's data and business processing logic to the blockchain network 600, achieving trustworthiness and traceability of data and business processing through the blockchain network 600.

[0103] Consensus nodes in blockchain network 600 receive data from different business entities (e.g.) Figure 4 The business entity (800 / 900) submits transactions, executes transactions to update or query the ledger, and various intermediate or final results of the transaction execution can be returned to the business entity's client node for display.

[0104] The following example illustrates an exemplary application of blockchain networks, using the server uploading received incremental data to a blockchain network for storage as an example. (See also...) Figure 5 , Figure 5 Client node 810 in the middle can correspond to Figure 1 Server 200 in the middle.

[0105] First, the logic for uploading incremental data to the blockchain is set up on client node 810. For example, when real-time incremental data is received, client node 810 sends the received incremental data to blockchain network 600 and generates a corresponding transaction. The transaction includes: the smart contract that needs to be called to upload the incremental data to the blockchain and update the column-level timestamps of all columns contained in the incremental data according to the timestamp of the incremental data, as well as the parameters passed to the smart contract; the transaction also includes the digital certificate of client node 810, the signed digital signature, and broadcasts the transaction to consensus node 610 in blockchain network 600.

[0106] Then, when consensus node 610 in blockchain network 600 receives a transaction, it verifies the digital certificate and digital signature carried in the transaction. If the verification is successful, it confirms whether business entity 800 has the authority to conduct the transaction based on the identity of the business entity 800 carried in the transaction. Any verification error in the digital signature or the authority verification will cause the transaction to fail. After successful verification, consensus node 610 signs its own digital signature (for example, by encrypting the transaction digest using the private key of node 610-1) and continues to broadcast it in blockchain network 600.

[0107] Finally, after receiving the successfully verified transaction, consensus node 610 in blockchain network 600 populates the transaction into a new block and broadcasts it. When broadcasting a new block, consensus node 610 in blockchain network 600 verifies the new block, for example, verifying whether the digital signatures of the transactions in the new block are valid. If the verification is successful, the new block is appended to the end of its stored blockchain, and the state database is updated according to the transaction results. The transactions in the new block are then executed: for transactions that submit incremental data, key-value pairs including the incremental data are added to the state database; for transactions that submit update operations, the smart contract is invoked to update the column-level timestamps corresponding to all columns contained in the incremental data according to the timestamps of the incremental data.

[0108] Let's take the example of a terminal sending a data query request to a blockchain network to illustrate an exemplary application of a blockchain network. See [link to relevant documentation]. Figure 5 , Figure 5 Client node 910 in the middle can correspond to Figure 1 Terminal 400 in the middle.

[0109] In some embodiments, the types of data that client node 910 can query in blockchain network 600 can be determined by consensus node 610 through restricting the transaction permissions that the client phase of the business entity can initiate. When client node 910 has the permission to initiate a data query, it can generate a transaction for querying data and submit it to blockchain network 600. The data query request carries a key name so that consensus node 610 can execute the transaction to query the data corresponding to the key name from the state database. Then, blockchain network 600 calls a smart contract to compare the timestamp of the queried data with the latest timestamp at the column level of the data's column, and returns the corresponding query result to client node 910 based on the comparison result.

[0110] The following continues to describe an exemplary structure of the data processing apparatus 243 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the data processing device 243 of the memory 240 may include: a receiving module 2431, a writing module 2432, an updating module 2433, a query module 2434, a comparison module 2435, and an adding module 2436.

[0111] The receiving module 2431 is used to receive incremental data; the writing module 2432 is used to write the incremental data into the data storage system; the updating module 2433 is used to update the column-level timestamps corresponding to all columns contained in the incremental data according to the timestamp of the incremental data; the receiving module 2431 is also used to receive data query requests; the query module 2434 is used to query the data storage system according to the key name carried in the data query request to obtain the data corresponding to the key name; and the comparison module 2435 is used to compare the timestamp of the data with the latest timestamp at the column level of the column where the data is located, and return the corresponding query results according to the comparison results.

[0112] In some embodiments, the comparison module 2435 is further configured to return a query result that does not contain data when the latest timestamp at the column level of the column containing the data is greater than the timestamp of the data; and to return a query result that contains data when the latest timestamp at the column level of the column containing the data is less than or equal to the timestamp of the data.

[0113] In some embodiments, the update module 2433 is further configured to update the corresponding index file according to the storage address of the incremental data in the data storage system, and record all columns contained in the incremental data in the index file.

[0114] In some embodiments, the update module 2433 is further configured to read all columns contained in the incremental data from the index file when a load operation is performed on the index file, and update the column-level timestamps corresponding to each column respectively.

[0115] In some embodiments, the update module 2433 is further configured to perform the following operations during the lifetime of the same lock: perform a load operation on the index file; read all columns contained in the incremental data from the index file, and update the column-level timestamps corresponding to each column.

[0116] In some embodiments, the update module 2433 is further configured to perform the following processing for any column among all columns contained in the incremental data: update the column-level timestamp corresponding to any column to be the same as the timestamp of the incremental data.

[0117] In some embodiments, the writing module 2432 is further configured to write incremental data to the corresponding address in the data storage system when incremental data is provided by multiple data sources and the data corresponding to different columns in the data storage system is updated based on the incremental data.

[0118] In some embodiments, the writing module 2432 is further configured to write incremental data to the corresponding address in the data storage system when incremental data is provided by multiple data sources and the data corresponding to the same column in the data storage system is updated based on the incremental data; the data processing device 243 further includes an adding module 2436, configured to add a unique corresponding label for each key name for different key names in the data storage system; and configured to record a column-level timestamp corresponding to the label in the index file according to the label.

[0119] In some embodiments, the query module 2434 is further configured to query the corresponding storage address in the index file based on the key name, and query the data storage system based on the storage address to obtain the data corresponding to the key name.

[0120] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For any technical details not covered in the data processing apparatus provided in this application embodiment, please refer to... Figure 3-4 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0121] The following example, using an information recommendation scenario, illustrates an exemplary application of the embodiments of this application in a real-world application scenario.

[0122] Example, Figure 1 Server 200 in the system can be the backend server of the recommendation system, which is used to periodically update the profile data of registered users, and based on the latest profile data of registered users, retrieve information that matches the latest profile data of registered users from the information to be recommended, and then sort and push the information.

[0123] For example, user profile data can be published on a daily basis, with multiple data sources waiting up to 24 hours for publication (new data arriving after the previous batch has just started needs to wait 24 hours before being published). Data from all data sources in each batch, regardless of whether it has been updated or not, is merged together to create a full index. Publishing the index is a complete replacement for the entire table; the new batch index can directly replace the old batch index without needing to consider data deletion.

[0124] Ideally, user profile data should be indexed and published immediately upon availability. This "on-demand" data delivery model is called on-demand publishing. Under on-demand publishing, each source data source typically updates individual columns of all keys, i.e., incremental updates relative to the entire table. During incremental updates, data present in older batches but not in newer batches needs to be considered when deleting it.

[0125] For example, see Figure 6 When returning query results, incremental updates may also return older batches of user profile data. (In this case, because the user profile data received by the server includes some historical user profile data, the information recalled by the server in the recommended information does not perfectly match the latest user profile data, resulting in poor accuracy of subsequent information recommendations.) The older batch of user profile data refers to... Figure 6 The data in the table includes V3, V4 (data from time T1) and V8, V12 (data from time T2). This is because the current time has been updated to T3, meaning the data at time T3 corresponds to the latest version of the user profile data. However, some cells were empty at time T3 (i.e., not all user profile data was updated), indicating that this data does not exist in the latest version. In other words, this data needs to be deleted when returning the query results. However, incremental updates do not include this information unless each empty cell is marked with a "Delete" flag to indicate that it is an empty cell and no data is returned. However, marking all empty cells is a very time-consuming operation that severely impacts data filtering efficiency.

[0126] In view of this, embodiments of this application provide a data processing method that, for incremental data written in batches to a data storage system, achieves filtering and deletion by comparing the timestamp of the data with a global timestamp (i.e., the column-level timestamp of the column containing the data), thereby improving the efficiency of the deletion operation. The data processing method provided in this application embodiment will be described in detail below.

[0127] For example, the data is timestamped, and the timestamps of data in the same batch are all the same, representing the batch's readiness time. The on-demand deployment scheme requires additional recording of all columns included in the batch of data in the index file's meta tag. Here, meta tag refers to metadata information at the index file level; existing metadata information includes the index file's size, name, and total number of data rows, etc. This embodiment of the application also additionally records all columns included in the batch of data, as well as the column-level timestamp for each column, in the index file's meta tag, and maintains this information by calling the query process. When the server calls the query process to load the index file, it first reads the columns included in the current index from the index file's meta tag and updates the column-level timestamps of these columns. Thus, when the server receives a subsequent data query request, it compares the timestamp of the data (i.e., the data corresponding to the query request) with the column-level timestamp of the column containing the data. If the column-level timestamp is greater than the data's timestamp, it means the data was not included in the most recent update and is removed from the query results. If the column-level timestamp is less than or equal to the data's timestamp, it means the data was included in the most recent update and is returned. This ensures that the server receives only the latest user profile data, allowing it to retrieve information matching the latest user profile data from the recommended information, and then sort and recommend the retrieved information, improving the accuracy of information recommendations. Furthermore, by comparing the data's timestamp with the column-level timestamp, data filtering is achieved, thus improving the efficiency of data filtering.

[0128] In other embodiments, when loading the index file, the update of the column-level timestamps and the loading of the index file can occur within the same lock. This ensures that the update time of the column-level timestamps is consistent with the loading time of the index file, guaranteeing that data can be filtered correctly. This is because if the update of the column-level timestamps occurs earlier than the loading of the index file, data may be filtered incorrectly; if the update of the column-level timestamps occurs later than the loading of the index file, data filtering will fail.

[0129] In some embodiments, a single data table can contain multiple types of keys, such as QQ, WUID, IMEI, and IDFA. Data corresponding to different keys may update the same columns in the table, but they may be provided by different data sources, meaning the readiness times of the data corresponding to different keys are different. For example, QQ and WUID may both update the age and gender columns, but the data for QQ and WUID are ready separately. In this case, loading later-ready data will cause earlier-ready data to fail to be returned during queries because the column-level timestamps have been updated to a later time. To avoid situations where data has different readiness times but updates the same columns, the server can distinguish them by assigning a unique tag to each key. If multiple data sources update the same batch of keys, the columns updated by these data sources must be non-overlapping. If multiple data sources update the same columns in the table but update different batches of keys, the server adds a unique tag to each batch of keys, and the query process also maintains a set of column-level timestamps for each tag.

[0130] For example, such as Figure 7 As shown, when updating the same column in a data table based on multiple data sources, but updating keys in different batches, a unique tag is added to each batch of keys. That is, a unique tag `Tag1` is added to `key1`, and a unique tag `Tag2` is added to `key2`. In other words, in actual use, keys in the same batch are categorized by their type, so the tag is recorded as the key type. Thus, when returning query results, because each tag maintains a corresponding set of column-level timestamps, the server compares the data's timestamp with the timestamp of the tag's corresponding column, rather than comparing it with the timestamp of the entire column. This avoids the situation where early-ready data cannot be returned during the query due to loading later-ready data.

[0131] This application embodiment targets incremental data written in batches to a data storage system. By comparing the timestamp of the data with a global timestamp, data filtering can be achieved without relying on the data from the old batch, using only the data from the new batch, plus some additional information (i.e., the timestamp of the data and the column-level timestamp of the column where the data is located), thus greatly improving the efficiency of data filtering.

[0132] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described in this application.

[0133] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the data processing method provided in this application, for example... Figure 3 or Figure 4 The data processing method is shown.

[0134] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0135] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0136] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0137] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0138] In summary, the embodiments of this application have the following beneficial effects:

[0139] For incremental data written in batches to the data storage system, the timestamps at the column level of all columns contained in the incremental data are updated based on the timestamp of the incremental data. When a data query request is received subsequently, the timestamp of the data corresponding to the key name carried in the data query request can be directly compared with the latest timestamp at the column level of the data column, and the corresponding query results can be returned based on the comparison result. In this way, the purpose of data filtering can be achieved by comparing the timestamp of the data with the timestamp at the column level of the data column, thus improving the efficiency of data filtering.

[0140] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Receive incremental data and write the incremental data into the data storage system; Based on the storage address of the incremental data in the data storage system, update the corresponding index file and record all columns contained in the incremental data in the metadata information of the index file; When a load operation is performed on the index file, all columns contained in the incremental data are read from the metadata information of the index file, and the column-level timestamps corresponding to all columns are updated according to the timestamps of the incremental data. Receive data query requests; The data storage system is queried according to the key name carried in the data query request to obtain the data corresponding to the key name; Compare the timestamp of the data with the latest timestamp at the column level of the column containing the data; If the latest timestamp at the column level of the column containing the data is greater than the timestamp of the data, return query results that do not contain the data.

2. The method according to claim 1, characterized in that, After comparing the timestamp of the data with the latest timestamp at the column level of the column containing the data, the method further includes: When the latest timestamp at the column level of the column containing the data is less than or equal to the timestamp of the data, the query result containing the data is returned.

3. The method according to claim 1, characterized in that, When performing a load operation on the index file, the step of reading all columns contained in the incremental data from the metadata information of the index file, and updating the column-level timestamps corresponding to each column according to the timestamps of the incremental data, includes: Perform the following operations during the lifetime of the same lock: Perform a loading operation on the index file; Read all columns contained in the incremental data from the index file, and update the column-level timestamps corresponding to each column according to the timestamps of the incremental data.

4. The method according to claim 1, characterized in that, The step of updating the column-level timestamps corresponding to all the columns includes: For any column among all the columns contained in the incremental data, perform the following processing: Update the column-level timestamp corresponding to any of the columns to be the same as the timestamp of the incremental data.

5. The method according to claim 1, characterized in that, The step of writing the incremental data into the data storage system includes: When the incremental data is provided by multiple data sources and the data corresponding to different columns in the data storage system is updated based on the incremental data, the incremental data is written to the corresponding address in the data storage system.

6. The method according to claim 1, characterized in that, The step of writing the incremental data into the data storage system includes: When the incremental data is provided by multiple data sources respectively, and the data corresponding to the same column in the data storage system is updated based on the incremental data, the incremental data is written to the corresponding address in the data storage system. After writing the incremental data, the method further includes: For each key name in the data storage system, a unique corresponding label is added; Based on the label, record the column-level timestamp corresponding to the label in the index file.

7. The method according to any one of claims 1 to 6, characterized in that, The step of obtaining the data corresponding to the key name includes: Based on the key name, query the corresponding storage address in the index file, and then query the data storage system based on the storage address to obtain the data corresponding to the key name.

8. A data processing apparatus, characterized in that, The device includes: The receiving module is used to receive incremental data; The writing module is used to write the incremental data into the data storage system; The update module is used to update the corresponding index file according to the storage address of the incremental data in the data storage system, and record all columns contained in the incremental data in the metadata information of the index file; The update module is further configured to, when performing a loading operation on the index file, read all columns contained in the incremental data from the metadata information of the index file, and update the column-level timestamps corresponding to all columns according to the timestamps of the incremental data; The receiving module is also used to receive data query requests; The query module is used to query the data storage system based on the key name carried in the data query request in order to obtain the data corresponding to the key name; The comparison module is used to compare the timestamp of the data with the latest timestamp at the column level of the column containing the data; when the latest timestamp at the column level of the column containing the data is greater than the timestamp of the data, the query result that does not contain the data is returned.

Images