Data warehousing method, device, system, apparatus, and computer medium

Abstract

The application discloses a data warehousing method, device, system, equipment and computer medium, which can be applied to various scenes such as cloud technology, file storage, block chain and data processing. The method comprises the following steps: after detecting a target database or self-restarting, acquiring the latest first checkpoint information, wherein the first checkpoint information comprises at least one of the following: a first snapshot identifier of the first checkpoint information, a first snapshot timestamp of the first checkpoint information, and a first operator identifier of an operator used for submitting data to the target database; determining a first submission state of first historical data corresponding to the first checkpoint information according to the first checkpoint information; if the first submission state does not indicate that the first historical data has been successfully submitted to the target database, deleting residual data of the first historical data in the target database; acquiring the first historical data from a first message queue device; and storing the first historical data to the target database, thereby improving the data consistency of the data stored to the database.

Description

Technical Field

[0001] This application relates to data storage technology, specifically to a data entry method, apparatus, system, device, and computer medium. Background Technology

[0002] Among related technologies, ClickHouse brings convenience to data analysis due to its superior query performance. However, when building a digital warehouse based on a combination of Flink stream computing and storage methods such as ClickHouse and Druid, data loss and duplicate writing may occur after data is written to ClickHouse through Flink, resulting in poor data consistency. Summary of the Invention

[0003] This application provides a data entry method, apparatus, system, device, and computer medium that can improve the data consistency of data stored in the database.

[0004] On one hand, embodiments of this application provide a data import method, the method comprising:

[0005] When a restart of the target database or itself is detected, the most recent first checkpoint information is obtained. The first checkpoint information includes at least one of the following: the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database.

[0006] Determine the first submission status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information.

[0007] If the first submission status does not indicate that the first historical data has been successfully submitted to the target database, then delete the residual data of the first historical data in the target database;

[0008] The first historical data is obtained from the first message queue device;

[0009] The first historical data is stored in the target database.

[0010] On the other hand, embodiments of this application provide a data processing system, including:

[0011] The second message queue device is used to receive pending data uploaded by terminal devices of multiple objects;

[0012] A statistical device is used to perform statistics on the data to be processed and obtain statistical results of the data to be processed within a preset time period.

[0013] A first message queue device is used to receive the statistical results;

[0014] An inbound buffer device is used to consume the statistical results and store them in a target database. It is also used to, upon detecting a restart of the target database or itself, obtain the most recent first checkpoint information, which includes at least one of the following: a first snapshot identifier of the first checkpoint information, a first snapshot timestamp of the first checkpoint information, and a first operator identifier for an operator used to submit data to the target database. Based on the first checkpoint information, it determines a first submission status of the first historical data corresponding to the first checkpoint information. If the first submission status does not indicate that the first historical data has been successfully submitted to the target database, it deletes any residual data of the first historical data in the target database. It also obtains the first historical data from a first message queue device and stores the first historical data in the target database.

[0015] The target database is used to store the statistical results;

[0016] The statistical results refer to the first historical data in the aforementioned data entry method.

[0017] On the other hand, embodiments of this application provide a data input device, the device comprising:

[0018] The acquisition unit is used to acquire the most recent first checkpoint information when the target database or itself is detected to be restarted. The first checkpoint information includes at least one of the following: the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database.

[0019] The determining unit is configured to determine the first submission status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information.

[0020] The deletion unit is used to delete residual data of the first historical data in the target database if the submission status does not indicate that the first historical data has been successfully submitted to the target database.

[0021] The acquisition unit is further configured to acquire the first historical data from the first message queue device;

[0022] A storage unit is used to store the first historical data into the target database.

[0023] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted for loading by a processor to perform the steps in the data entry method as described in any of the above embodiments.

[0024] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory, the memory storing a computer program, the processor executing the steps in the data entry method as described in any of the above embodiments by calling the computer program stored in the memory.

[0025] On the other hand, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the data entry method as described in any of the above embodiments.

[0026] This application embodiment obtains the most recent first checkpoint information after detecting a target database restart or its own restart. The first checkpoint information includes at least one of the following: a first snapshot identifier of the first checkpoint information, a first snapshot timestamp of the first checkpoint information, and a first operator identifier of an operator used to submit data to the target database. Based on the first checkpoint information, a first submission status of the first historical data corresponding to the first checkpoint information is determined. If the first submission status does not indicate that the first historical data has been successfully submitted to the target database, residual data of the first historical data in the target database is deleted. The first historical data is obtained from a first message queue device. This scheme of storing the first historical data in the target database can detect whether the data has been successfully stored in the target database after detecting a target database restart or its own restart. If the relevant data has not been successfully stored in the target database, it is re-stored, avoiding data loss and duplicate data writing, and improving the data consistency of the data stored in the database. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the structure of the data processing system provided in the embodiments of this application.

[0029] Figure 2a This is a flowchart illustrating the data entry method provided in an embodiment of this application.

[0030] Figure 2b This is a schematic diagram illustrating the data processing process of the inbound buffer device provided in this application embodiment after detecting the target database or restarting itself.

[0031] Figure 2cThis is a schematic diagram illustrating the data processing process of the warehousing buffer device provided in this application embodiment during the pre-submission and submission stages.

[0032] Figure 2d This is a schematic diagram illustrating a data loss scenario provided in an embodiment of this application.

[0033] Figure 2e This is a schematic diagram illustrating data loss after using this solution, provided as an embodiment of this application.

[0034] Figure 2f This is a schematic diagram illustrating the situation of duplicate data writing when counting clicks per minute on a platform, as provided in an embodiment of this application.

[0035] Figure 2g This is a schematic diagram illustrating the effect of using this solution in the embodiments of this application, where the problem of repeated writing no longer occurs.

[0036] Figure 3 This is a schematic diagram of the data entry device provided in an embodiment of this application.

[0037] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] The embodiments of this application can be applied to various scenarios such as cloud technology, file storage, blockchain, and data processing.

[0040] This application provides a data entry method, apparatus, system, device, and computer medium. Specifically, the data entry method of this application can be executed by a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, smart voice interaction device, smart home appliance, wearable smart device, aircraft, smart vehicle terminal, etc. The terminal can also include a client, which can be a video client, browser client, or instant messaging client, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0041] First, some of the nouns or terms that appear in the description of the embodiments of this application are explained as follows:

[0042] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0043] Database: Simply put, it can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, capable of being shared by multiple users, with minimal redundancy, and independent of application programs.

[0044] A blockchain system can be a distributed system formed by clients and multiple nodes (any form of computing device connected to the network, such as servers and user terminals) connected through network communication. The nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). In a distributed system, any machine, such as a server or terminal, can join and become a node. A node includes a hardware layer, a middleware layer, an operating system layer, and an application layer.

[0045] Apache Flink is an open-source computing platform for distributed data stream processing and batch data processing, providing support for both stream processing and batch processing applications.

[0046] TDBank collects data in real time from the business data source, performs preprocessing and distributed message caching, and then distributes it to the backend offline and online processing systems according to message subscription.

[0047] ClickHouse is a column-oriented database management system (DBMS) with an MPP (Online Analytical Processing) architecture for Online Analytical Processing (OLAP). It can generate analytical data reports in real time using SQL (Structured Query Language).

[0048] Apache Druid is a real-time analytical database designed for fast querying and analysis of large datasets (“OLAP” queries). Druid is most commonly used as a database to support applications that require real-time ingestion, high query performance, and high stability.

[0049] Idempotence is a mathematical and computer science concept, commonly found in abstract algebra. In programming, an idempotent operation is characterized by the fact that its effect is the same regardless of how many times it is executed.

[0050] Data consistency is crucial when multiple users attempt to access a database simultaneously, and their transactions use the same data. Four scenarios may occur: lost updates, undetermined correlations, inconsistent analysis, and phantom reads. Data consistency is a key aspect of data quality assessment. Generally, consistency refers to whether the information attributes of the same data subject are identical across different datasets, and whether entities and attributes conform to consistency constraints.

[0051] Transaction: A transaction is the smallest indivisible unit of work; usually, a transaction corresponds to a complete business transaction (such as a bank account transfer transaction, which is the smallest unit of work). A complete business transaction requires a batch of DML (insert, update, delete) statements to complete together.

[0052] Pre-commit: This involves performing actions before code submission, such as code packaging and code checks; these are called hooks. You can think of it as executing a function (callback) before the commit. If this function executes successfully, the commit can proceed; otherwise, it prevents the commit from happening.

[0053] Checkpoint: This is an internal event that, when activated, triggers the database write process (DBWR) to write dirty data blocks from the data buffer cache to the data files. In database systems, writing to the log file and writing to the data file are the two most I / O-intensive operations. Writing to the data file is a distributed write operation, while writing to the log file is a sequential write operation. Therefore, to ensure database performance, databases typically ensure that all log files are written to the log file before a commit is completed, while dirty data blocks are stored in the buffer cache and written to the data files in batches periodically.

[0054] The sink operator writes the final result of the computation to a specific location. The sink in Flink is one of the three main logical structures (source, transform, sink), and its function is to output the processed data from Flink to an external system.

[0055] PrepareStatement is a sub-interface of Statement that can pre-compile SQL (Structured Query Language) statements. The pre-compiled statement is stored in a PrepareStatement object, which can then be used to execute the SQL statement efficiently multiple times. Furthermore, PrepareStatement allows the use of placeholders to write SQL statements.

[0056] Database shards are an example of a shared-nothing architecture. This means that shards are autonomous: they do not share any identical data or server resources. However, in some cases, it makes sense to replicate certain tables to each shard as reference tables. For example, suppose an application's database relies on a fixed conversion rate for weight measurements. By replicating a table containing the necessary conversion rate data to each shard, it helps ensure that all the data required for queries is stored in each shard.

[0057] The mapState helper function is syntactic sugar for state, which helps generate computed properties. Syntactic sugar refers to a certain syntax added to a computer language.

[0058] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application. The data processing system includes: a second message queue device 10, a statistics device 11, a first message queue device 12, an ingestion buffer device 13, and a target database 14, wherein:

[0059] The second message queue device 10 is used to receive pending data uploaded by terminal devices of multiple objects;

[0060] The object can be a user, and the second message queue device 10 can be a device used to store TDbank message queues.

[0061] Optionally, the second message queue device 10 can also be a device for storing Kafka message queues.

[0062] Statistical device 11 is used to perform statistics on the data to be processed and obtain the statistical results of the data to be processed within a preset time period;

[0063] Specifically, the statistical device 11 can be a device used to perform Flink stream computation.

[0064] Optionally, the aforementioned data to be processed can be any one or more of the following:

[0065] The object being viewed and the time it was viewed;

[0066] The purchased object and the purchase time of the purchased object, and the corresponding order generation time;

[0067] The registered object and its registration time;

[0068] The subject of the exposure and the exposure time of the subject of the exposure.

[0069] The aforementioned statistical analysis of the data to be processed includes any one or more of the following: determining the number of times multiple preset objects to be viewed are viewed within a preset time period, determining the number of times multiple preset objects to be purchased are purchased and the number of orders generated within a preset time period, determining the number of times multiple preset objects to be registered are registered within a preset time period, and determining the number of times multiple preset objects to be exposed are exposed within a preset time period.

[0070] Accordingly, the statistical results include one or more of the following: the number of times multiple preset objects are viewed within a preset time period, the number of times multiple preset objects are purchased and the number of orders generated within a preset time period, the number of times multiple preset objects are registered within a preset time period, and the number of times multiple preset objects are exposed within a preset time period.

[0071] Optionally, the object being viewed may be a webpage, the object being purchased may be a product, the object being registered may be a website or application, and the object being exposed may be news.

[0072] Optionally, the aforementioned data to be processed and statistical results may also be other information, such as data related to games or advertisements, and this application does not limit this.

[0073] The first message queue device 12 is used to receive the statistical results;

[0074] The first message queue device 12 can be a device used to store TDbank message queues or Kafka message queues.

[0075] The inbound buffer device 13 is used to consume the statistical results and store (including pre-commit and commit) the statistical results into the target database. It is also used to, upon detecting a restart of the target database or itself, obtain the most recent first checkpoint information, the first checkpoint information including at least one of the following: a first snapshot identifier of the first checkpoint information, a first snapshot timestamp of the first checkpoint information, and a first operator identifier of the operator used to submit data to the target database; determine the first commit status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information; if the first commit status does not indicate that the first historical data has been successfully submitted to the target database, delete the residual data of the first historical data in the target database; obtain the first historical data from the first message queue device; and store the first historical data into the target database.

[0076] The target database 14 can be a database that does not support transactions, such as ClickHouse, or it can be a database that does support transactions; this application does not limit this.

[0077] The target database 14 is used to store the statistical results.

[0078] After the aforementioned statistical device 11 performs statistics on the aforementioned data to be processed and uploads it to the first message queue device 12, the aforementioned database buffer device 13 can consume the statistical results. After consuming the aforementioned statistical results, the aforementioned database buffer device 13 writes them into the target database. This scheme can decouple the calculation and storage process for the data to be processed, so that even if the target database fails, it will not affect the calculation process for the data to be processed, and the corresponding statistical results will not be lost. It can effectively avoid the impact on the stability of the target database caused by the concurrent control of multiple programs.

[0079] Optionally, the concurrency of the aforementioned inbound buffer device 13 can be the same as the number of partitions in the first message queue device 12.

[0080] Optionally, after storing the aforementioned statistical results to the target database and successfully storing them, the aforementioned storage buffer device 13 can generate location information to indicate the next batch of statistical results in the first message queue device 12. Specifically, the location information can be offset information to ensure that all statistical results before the aforementioned location information is generated have been successfully stored.

[0081] Optionally, the first checkpoint information may further include first historical data recently consumed from the first message queue device; wherein the first historical data includes the aforementioned statistical results, and the first message queue device 12 and the second message queue device 10 are used to store message queues.

[0082] The solution in this application can increase the consistency guarantee when storing data in the database by adding a first message queue device 12 and an ingestion buffer device 13 to the data processing system in related technologies, thereby avoiding duplicate data storage in the database, avoiding data loss and duplicate data writing, and preventing abnormal Flink stream data computation due to database failure.

[0083] Optionally, the system further includes an observation device, which is used to acquire the user's viewing instruction and retrieve and display the first historical data from the target database according to the viewing instruction.

[0084] Specifically, the aforementioned observation equipment may be equipped with a display interface, through which users can trigger the aforementioned viewing commands.

[0085] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0086] This application provides a data import method. Figure 2aThis is a flowchart illustrating a data import method provided in an embodiment of this application. The method can be executed by the aforementioned import buffer device 13, and may include the following steps S201-S205:

[0087] S201. When the target database 14 or itself (i.e., the inbound buffer device 13) is detected to be restarted, the most recent first checkpoint information is obtained. The first checkpoint information includes at least one of the following: the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database 14.

[0088] The most recent first checkpoint information is the first snapshot information closest to the current time. That is, the first snapshot information includes: the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database 14.

[0089] The target database 14 or the inbound buffer device 13 can be automatically restarted or restarted after receiving a restart instruction from relevant personnel after going offline due to equipment failure, network failure, or software failure.

[0090] Optionally, the aforementioned inbound buffer device 13 can be a Flink inbound buffer device 13, which can take snapshots of local data according to a preset period to obtain corresponding snapshot information.

[0091] The operator used to submit data to the target database 14 can be a sink operator.

[0092] S202. Determine the first submission status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information.

[0093] Optionally, in the aforementioned S202, determining the first submission status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information includes S2021-S2022:

[0094] S2021. Perform an idempotency operation on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database 14 to obtain a first idempotency result.

[0095] Optionally, in the aforementioned S2021, performing an idempotent operation on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database 14 to obtain a first idempotent result includes: performing a hash operation on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database 14 to obtain a first hash result; and using the first hash result as the first idempotent result.

[0096] S2022. Determine whether a first mapping relationship corresponding to the first idempotent result is stored locally, wherein the first mapping relationship stores the first submission status of the first historical data; if so, obtain the first submission status from the first mapping relationship.

[0097] If the first mapping relationship corresponding to the first idempotent result is not stored locally, it means that the first submission status of the first historical data has not been stored locally, and the pre-submission process of the first historical data has not been completed. In this case, the second historical data is directly obtained from the first message queue device 12 and stored in the target database 14. The second historical data and the first historical data are from the same batch of data.

[0098] If the first mapping relationship corresponding to the first idempotent result is not stored locally, in some optional embodiments of this application, obtaining the second historical data from the first message queue device 12 includes: obtaining data receiving snapshot information corresponding to the first checkpoint information, wherein the data receiving snapshot information includes the second historical data received by the storage buffer device 13 from the first message queue device 12.

[0099] Optionally, in some other optional embodiments of this application, obtaining the second historical data from the first message queue device 12 includes: obtaining data receiving snapshot information corresponding to the first checkpoint information, wherein the data receiving snapshot information includes offset information of the second historical data received by the storage buffer device 13 from the first message queue device 12, the offset information being used to indicate the position information of the second historical data in the first message queue device 12; and obtaining the second historical data from the first message queue device 12 according to the offset information.

[0100] S203. If the first submission status does not indicate that the first historical data has been successfully submitted to the target database 14, then delete the residual data of the first historical data in the target database 14.

[0101] S204. Obtain the first historical data from the first message queue device 12;

[0102] In some optional embodiments of this application, obtaining the first historical data from the first message queue device 12 includes: obtaining data receiving snapshot information corresponding to the first checkpoint information, wherein the data receiving snapshot information includes the first historical data received by the inbound buffer device 13 from the first message queue device 12.

[0103] Optionally, in some other optional embodiments of this application, obtaining the first historical data from the first message queue device 12 includes: obtaining data receiving snapshot information corresponding to the first checkpoint information, wherein the data receiving snapshot information includes offset information of the first historical data received by the storage buffer device 13 from the first message queue device 12, wherein the offset information is used to indicate the position information of the first historical data in the first message queue device 12; and obtaining the first historical data from the first message queue device 12 according to the offset information.

[0104] Optionally, after obtaining the first historical data from the first message queue device 12, the first historical data is stored in the target database; after obtaining the second historical data from the first message queue device 12, the second historical data is stored in the target database.

[0105] The method of storing the second historical data in the target database is similar to that of storing the first historical data in the target database.

[0106] S205. Store the first historical data in the target database 14.

[0107] In some optional embodiments of this application, Figure 2bThis is a schematic diagram illustrating the data processing process of the inbound buffer device provided in this application embodiment after detecting a restart of the target database 14 or itself. After detecting a restart of the target database 14 or itself, the inbound buffer device 13 obtains the most recent first checkpoint information, which includes at least one of the following: a first snapshot identifier of the first checkpoint information, a first snapshot timestamp of the first checkpoint information, and a first operator identifier of the operator used to submit data to the target database 14. The device then calculates a first idempotent result after hashing these three information. It determines whether the aforementioned first idempotent result (or the first mapping relationship corresponding to the first idempotent result) is stored locally. If so, it obtains the first submission status of the first historical data corresponding to the first idempotent result and performs the operation of deleting or resubmitting the first historical data according to the first submission status. If not, it obtains the second historical data from the first message queue device 12 and stores the second historical data in the target database 14.

[0108] In some optional embodiments of this application, in the aforementioned S205, storing the first historical data in the target database 14 includes:

[0109] Execute the pre-submission process for the first historical data;

[0110] A snapshot is taken of the first historical data to obtain second checkpoint information, including a second snapshot identifier of the second checkpoint information, a second snapshot timestamp of the second checkpoint information, and a second operator identifier of the operator used to submit data to the target database 14.

[0111] Execute the submission process for the first historical data;

[0112] Optionally, the aforementioned second checkpoint information may also include the first historical data.

[0113] In some optional embodiments of this application, the pre-submission process of the first historical data includes: initializing the second submission state of the first historical data to a submission preparation state.

[0114] The pre-submission process for the first historical data also includes:

[0115] The SQL PreparedStatement is initialized, specifically including setting the table name information and field name information in the target database 14 used to store the aforementioned first historical data, and may also include initializing the field positions in the field name information.

[0116] The process of submitting the first historical data includes: submitting the first historical data to the target database 14.

[0117] Specifically, the aforementioned inbound buffer device 13 can perform two stages of submission, which may include a pre-submission stage and a submission stage. The inbound buffer device 13 performs the aforementioned pre-submission process in the pre-submission stage and the aforementioned submission process in the submission stage.

[0118] Optionally, the pre-submission stage further includes: performing an idempotency operation on the second snapshot identifier of the second checkpoint information, the second snapshot timestamp of the second checkpoint information, and the second operator identifier of the operator used to submit data to the target database 14 to obtain a second idempotency result.

[0119] Optionally, performing an idempotency operation on the second snapshot identifier of the second checkpoint information, the second snapshot timestamp of the second checkpoint information, and the second operator identifier of the operator used to submit data to the target database 14 to obtain a second idempotency result includes: performing a hash operation on the second snapshot identifier of the second checkpoint information, the second snapshot timestamp of the second checkpoint information, and the second operator identifier of the operator used to submit data to the target database 14 to obtain a second hash result; and using the second hash result as the second idempotency result.

[0120] During the pre-submission phase, the aforementioned inbound buffer device 13 is also used to store the second idempotent result and the second submission status of the first historical data through MapState.

[0121] In some optional embodiments of this application, the data processing procedures of the inbound buffer device 13 during the pre-submission and submission phases can be found in [reference needed]. Figure 2c As shown, in the pre-commit phase, the inbound buffer device 13 is specifically used to: determine the second idempotent result, store the second commit state through MapState, and initialize the second commit state to a ready-to-commit state. In the commit phase, the inbound buffer device 13 is used to: commit the first historical data, and determine the second commit state based on the corresponding commit result. If the second commit state is a ready-to-commit state or a failed commit state, then the residual data of the first historical data in the target database 14 is first deleted, and the first historical data is resubmitted. The process ends after confirming that the first historical data has been successfully committed, and if it is confirmed that the first historical data has not been successfully committed, it returns to resubmit the first historical data.

[0122] Specifically, after submitting the first historical data, if no feedback is received from the target database 14 within a preset time period, the second submission status of the first historical data will not be changed. If a prompt message indicating submission failure is received from the target database 14 within the preset time period, the second submission status of the first historical data will be changed to the submission failure status. If a prompt message indicating submission success is received from the target database 14 within the preset time period, the second submission status of the first historical data will be changed to the submission success status.

[0123] Optionally, before executing the submission process of the first historical data, the inbound buffer device 13 further includes:

[0124] Obtain the second idempotency result;

[0125] Obtain the first fragment identifier corresponding to each data item in the first historical data;

[0126] Obtain the second submission status;

[0127] A second mapping relationship is created based on the second idempotency result, the first fragment identifier, and the second commit status.

[0128] Optionally, one side of the aforementioned second mapping relationship is the second idempotent result, and the other side is the first fragment identifier and the second submission status. The first fragment identifier is the second target location information, and the second target location information is the location information in the target database.

[0129] The aforementioned submission of the first historical data includes: submitting the first historical data and the second target location information to the target database, so that the target database stores the first historical data based on the second target location information.

[0130] Accordingly, the first mapping relationship also stores the first target location information corresponding to the first historical data. The first target location information is the location information in the target database. Before obtaining the first historical data from the first message queue device, the method further includes: submitting the first historical data and the first target location information to the target database, so that the target database stores the first historical data based on the first target location information.

[0131] Optionally, the aforementioned second mapping relationship is a key-value pair, where the key of the key-value pair is the first shard identifier and the second submission status, and the value of the key-value pair is the second idempotent result.

[0132] The second submission state in the aforementioned second mapping relationship can adaptively change based on the submission result during the submission process of the first historical data.

[0133] In some optional embodiments of this application, the aforementioned first historical data includes a preset number of sub-data items, each sub-data item includes data content and the acquisition time of the data content, each sub-data item corresponds to a sub-data fragment identifier, and the first fragment identifier contained in the key of the aforementioned key-value pair is the multiple sub-data fragment identifiers corresponding to the multiple sub-data items contained in the first historical data.

[0134] Optionally, the aforementioned first fragment identifier can be calculated in the following way:

[0135] For each piece of sub-data, obtain the time field contained in the sub-data;

[0136] Obtain the number of shards contained in the target database 14;

[0137] The sub-data fragment identifier corresponding to the sub-data is determined based on the time field and the number of fragments;

[0138] Use the fragment identifiers of the multiple sub-data items corresponding to the multiple sub-data items as the first fragment identifier.

[0139] In some optional embodiments of this application, determining the sub-data fragment identifier corresponding to the sub-data based on the time field and the fragment number includes:

[0140] Calculate the modulus between the time field and the number of slices;

[0141] The module is used as the identifier for the sub-data fragment.

[0142] In some other optional embodiments of this application, the above method further includes:

[0143] The first historical data is classified to obtain multiple data groups of different types. Each group of first historical data corresponds to one type. Each group of first historical data may include one or more sub-data items. Among them, the multiple types may include at least two of the following: games, advertisements, news, etc.

[0144] Furthermore, in some other optional embodiments of this application, the aforementioned first fragment identifier can be calculated in the following manner:

[0145] For each piece of sub-data, obtain the time field contained in the sub-data and the type corresponding to the data group to which the sub-data belongs;

[0146] Obtain the number of shards contained in the target database 14;

[0147] The sub-data fragment identifier corresponding to the sub-data is determined based on the time field, the type, and the number of fragments;

[0148] Use the fragment identifiers of the multiple sub-data items corresponding to the multiple sub-data items as the first fragment identifier.

[0149] In some optional embodiments of this application, determining the sub-data fragment identifier corresponding to the sub-data based on the time field, the type, and the number of fragments includes:

[0150] Calculate the modulus between the time field and the number of slices;

[0151] The corresponding shard group identifier is determined according to the module. The multiple shards contained in the target database 14 can be divided into multiple shard groups. Each shard group corresponds to a shard group identifier, and each shard group identifier corresponds to a module.

[0152] Get the numerical value corresponding to the type;

[0153] Based on the numerical value and the preset correspondence, determine the sub-data fragment identifier corresponding to the sub-data in the fragment group corresponding to the fragment group identifier.

[0154] Optionally, for a shard in a shard group, a shard may store only one type of data. This scheme can improve the viewing speed of this type of data for relevant personnel on the observation equipment side.

[0155] Optionally, the aforementioned first idempotency result and the first mapping relationship can be stored using MapState.

[0156] Optionally, regarding the creation of the aforementioned first mapping relationship, the method further includes:

[0157] Obtain the first idempotency result; obtain the second fragment identifier corresponding to each data item in the first historical data; obtain the first submission status;

[0158] A first mapping relationship is created based on the first idempotency result, the second fragment identifier, and the first submission status, wherein the second fragment identifier is the aforementioned first target location information.

[0159] Optionally, one side of the aforementioned first mapping relationship is the first idempotent result, and the other side is the second fragment identifier and the first submission status.

[0160] Optionally, the aforementioned first mapping relationship is a key-value pair, where the key of the key-value pair is the second shard identifier and the first submission status, and the value of the key-value pair is the first idempotent result.

[0161] The first submission state in the aforementioned first mapping relationship can adaptively change according to the submission result during the submission process of the first historical data.

[0162] In some optional embodiments of this application, the aforementioned first historical data includes a preset number of sub-data items, each sub-data item includes data content and the acquisition time of the data content, each sub-data item corresponds to a sub-data fragment identifier, and the first fragment identifier contained in the key of the aforementioned key-value pair is the multiple sub-data fragment identifiers corresponding to the multiple sub-data items contained in the first historical data.

[0163] In some optional embodiments of the application, the method further includes:

[0164] When the first submission status is either submission preparation status or submission failure status, it is determined that the submission status does not indicate that the first historical data has been successfully submitted to the target database 14.

[0165] Optionally, the first submission status can be represented by a status identifier, which can be an Arabic numeral.

[0166] Optionally, when the status flag is 0, it indicates that the submission status is in the submission preparation state; when the status flag is 1, it indicates that the submission status is in the submission failure state; and when the status flag is 2, it indicates that the submission status is in the submission success state.

[0167] Optionally, the aforementioned deletion of residual data of the first historical data in the target database 14 includes: querying whether residual data of the first historical data is stored in the target database 14; if so, deleting the residual data, wherein the residual data is data contained in the first historical data. If not, obtaining the first historical data from the first message queue device; and storing the first historical data in the target database. The specific methods for obtaining the first historical data from the first message queue device and storing the first historical data in the target database are described above and will not be repeated here.

[0168] The step of submitting the first historical data to the target database 14 includes: submitting the first historical data and the first target location information to the target database 14, so that the target database 14 stores the first historical data based on the first target location information.

[0169] Optionally, submitting the first historical data to the target database 14 can be performed during the submission phase.

[0170] Optionally, the method further includes:

[0171] Obtain the value of the flag bit used to indicate whether the residual data was successfully deleted;

[0172] If the value of the flag indicates that the residual data has been successfully deleted, then the first historical data is retrieved from the first message queue device 12.

[0173] Specifically, the aforementioned flag bit used to indicate whether the residual data has been successfully deleted can be the corresponding value bit of is_done. When the flag bit is 1, it indicates that the residual data has been successfully deleted, and when the flag bit is 0, it indicates that the residual data has not been successfully deleted.

[0174] In some other optional embodiments of this application, in order to correct data for a specific time period, the above method further includes the following S01-S06:

[0175] S01. Obtain the start time information sent by the management device;

[0176] S02. Obtain the third checkpoint information corresponding to the start time information. The third checkpoint information includes at least one of the following: the third snapshot identifier of the third checkpoint information, the third snapshot timestamp of the third checkpoint information, and the third operator identifier of the operator used to submit data to the target database 14.

[0177] S03. Determine the third submission status of the third historical data corresponding to the third checkpoint information based on the third checkpoint information.

[0178] S04. If the third submission status does not indicate that the third historical data has been successfully submitted to the target database 14, then delete the residual data of the third historical data in the target database 14.

[0179] S05. Obtain the third historical data from the first message queue device 12;

[0180] S06. The third historical data is stored in the target database 14.

[0181] Specifically, the specific implementation methods of S01-S06 can be found in the foregoing content, and will not be repeated here.

[0182] Optionally, when the above method detects that the current time has reached a preset time, it stops storing data to the target database 14, where the preset time is the time when the start time information sent by the management device is obtained.

[0183] Optionally, in addition to receiving data from relevant personnel through the management device to correct the time interval between the current time and the aforementioned start time information, the solution of this application can also enable relevant personnel to correct data within a preset time period through the management device. Accordingly, the aforementioned acquisition of the start time information sent by the management device also includes: acquisition of the start time information and end time information sent by the management device.

[0184] Optionally, the above method further includes sending the start time information and end time information, or the data corrected from the start time to the current time, to the management device for display by the management device.

[0185] Optionally, the above method further includes: clearing the data stored through MapState according to a preset period, wherein the preset period can be 24 hours.

[0186] The solution presented in this application provides a low-cost and efficient way to achieve data consistency when writing to massive storage components that do not support transactions at the Flink level. It avoids any unnecessary additional development or custom development for specific storage components. Instead, it offers a method for consistent writing to general-purpose, non-transactional storage components at the Flink level, making it more lightweight and convenient to use. It solves the data consistency problem in Flink when writing to storage that does not support transactions.

[0187] This application embodiment obtains the most recent first checkpoint information after detecting a restart of the target database 14 or itself. The first checkpoint information includes at least one of the following: a first snapshot identifier of the first checkpoint information, a first snapshot timestamp of the first checkpoint information, and a first operator identifier of an operator used to submit data to the target database 14. Based on the first checkpoint information, a first submission status of the first historical data corresponding to the first checkpoint information is determined. If the first submission status does not indicate that the first historical data has been successfully submitted to the target database 14, residual data of the first historical data in the target database 14 is deleted. The first historical data is obtained from the first message queue device 12. This scheme of storing the first historical data in the target database 14 can detect whether the data has been successfully stored in the target database 14 after detecting a restart of the target database 14 or itself. If the relevant data has not been successfully stored in the target database 14, it is re-stored, avoiding data loss and duplicate data writing, and improving the data consistency of the data stored in the database.

[0188] Figure 2d This is a diagram illustrating a data loss scenario. The data statistics window is 1 minute (60 seconds). If the timestamp interval in the gray box is greater than 1 minute, it indicates that data loss has occurred, resulting in incorrect timestamps during data entry.

[0189] Figure 2e This diagram illustrates a data loss scenario after using this solution. As shown, after implementing this solution, no further data loss will occur. The time interval is continuous (1 minute).

[0190] Figure 2fThis is a diagram illustrating the situation of duplicate data writing when counting clicks per minute on a platform, as provided in the embodiments of this application. At the same time, two identical data entries with click_num (click count) of 2 appeared for the iOS platform, which are abnormal data. Figure 2g This solution ensures that data will no longer be written repeatedly.

[0191] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0192] To facilitate better implementation of the data import method of this application, this application also provides a data import device. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of the structure of a data entry device provided in an embodiment of this application. The data entry device 30 may include:

[0193] The acquisition unit 31 is used to acquire the most recent first checkpoint information when the target database 14 or itself is detected to restart. The first checkpoint information includes at least one of the following: the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database 14.

[0194] Determining unit 32 is used to determine the first submission status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information.

[0195] The deletion unit 33 is used to delete the residual data of the first historical data in the target database 14 if the submission status does not indicate that the first historical data has been successfully submitted to the target database 14.

[0196] The acquisition unit 31 is further configured to acquire the first historical data from the first message queue device 12;

[0197] Submission unit 34 is used to submit the first historical data to the target database 14.

[0198] Optionally, when the aforementioned device is used to determine the first submission status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information, it is specifically used for:

[0199] An idempotency operation is performed on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database 14 to obtain a first idempotency result.

[0200] Determine whether a first mapping relationship corresponding to the first idempotent result is stored locally, wherein the first mapping relationship stores the first submission status of the first historical data;

[0201] If so, then obtain the first submission status from the first mapping relationship.

[0202] Optionally, the device is further configured to: if the first mapping relationship corresponding to the first idempotent result is not stored locally, directly obtain the second historical data from the first message queue device and store the second historical data in the target database; wherein the second historical data and the first historical data are the same batch of data.

[0203] Optionally, when the device is used to obtain the first historical data from the first message queue device, it is specifically used for:

[0204] Obtain data reception snapshot information corresponding to the first checkpoint information. The data reception snapshot information includes offset information of the first historical data received from the first message queue device. The offset information is used to indicate the position information of the first historical data in the first message queue device.

[0205] The first historical data is obtained from the first message queue device based on the offset information.

[0206] Optionally, the first mapping relationship also stores the first target location information corresponding to the first historical data. The first target location information is the location information in the target database 14. The device is further configured to: submit the first historical data and the first target location information to the target database 14, so that the target database 14 stores the first historical data based on the first target location information.

[0207] Optionally, the device is further configured to: when the first submission status is a submission preparation status or a submission failure status, determine that the first submission status does not indicate that the first historical data has been successfully submitted to the target database 14.

[0208] Optionally, when the device is used to delete residual data of the first historical data in the target database 14, it is specifically used to: query whether the target database 14 stores residual data of the first historical data; if so, delete the residual data, wherein the residual data is data contained in the first historical data.

[0209] Optionally, the device is further used for:

[0210] Obtain the value of the flag bit used to indicate whether the residual data was successfully deleted;

[0211] If the value of the flag indicates that the residual data has been successfully deleted, then the first historical data is retrieved from the first message queue device 12.

[0212] Optionally, when the device performs an idempotency operation on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database 14 to obtain a first idempotency result, it is specifically used for:

[0213] A hash operation is performed on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database 14 to obtain the first hash result;

[0214] The first hash result is used as the first idempotent result.

[0215] Each unit in the aforementioned data entry device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can invoke and execute the operations corresponding to each unit.

[0216] Optionally, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0217] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be... Figure 1 The inbound buffer device 13 is shown. (As shown in the image.) Figure 4 As shown, the computer device 400 may include: a communication interface 401, a memory 402, a processor 403, and a communication bus 404. The communication interface 401, memory 402, and processor 403 communicate with each other via the communication bus 404. The communication interface 401 is used for data communication between the computer device 400 and external devices. The memory 402 can be used to store software programs and modules, and the processor 403 runs the software programs and modules stored in the memory 402, such as the software programs for the corresponding operations in the foregoing method embodiments.

[0218] Optionally, the processor 403 can invoke software programs and modules stored in the memory 402 to perform the following operations:

[0219] When a restart of the target database 14 or itself is detected, the most recent first checkpoint information is obtained. The first checkpoint information includes at least one of the following: the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database 14.

[0220] Determine the first submission status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information.

[0221] If the first submission status does not indicate that the first historical data has been successfully submitted to the target database 14, then delete the residual data of the first historical data in the target database 14;

[0222] The first historical data is obtained from the first message queue device 12;

[0223] Submit the first historical data to the target database 14.

[0224] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the methods described in the embodiments of this application; for brevity, these will not be elaborated further here.

[0225] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described in the embodiments of this application. For brevity, these details will not be elaborated further here.

[0226] This application also provides a computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described in the embodiments of this application. For the sake of brevity, these details will not be elaborated further here.

[0227] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0228] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0229] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0230] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0231] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0232] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0233] In addition, the functional units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0234] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0235] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data import method, characterized in that, The method includes: When a restart of the target database or itself is detected, the most recent first checkpoint information is obtained. The first checkpoint information includes at least one of the following: the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database. Determining the first submission status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information includes: performing an idempotent operation on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database to obtain a first idempotent result; determining whether a first mapping relationship corresponding to the first idempotent result is stored locally, wherein the first submission status of the first historical data is stored in the first mapping relationship; if so, obtaining the first submission status from the first mapping relationship. If the first submission status does not indicate that the first historical data has been successfully submitted to the target database, then delete the residual data of the first historical data in the target database; Obtain the value of the flag bit used to indicate whether the residual data was successfully deleted; If the value of the flag indicates that the residual data has been successfully deleted, then the first historical data is retrieved from the first message queue device. The first historical data is stored in the target database through a pre-submission phase and a submission phase. In the pre-submission phase, the second submission state of the first historical data is initialized to a ready-to-submit state. A snapshot of the first historical data is taken to obtain second checkpoint information, and an idempotency operation is performed on the information in the second checkpoint information to obtain a second idempotency result. The second idempotency result and the second submission state of the first historical data are stored together. In the submission phase, the first historical data is submitted to the target database, and the second submission state is determined according to the corresponding submission result. If the second submission state is a ready-to-submit state or a submission failure state, the residual data of the first historical data in the target database is deleted first, and the first historical data is resubmitted.

2. The method according to claim 1, characterized in that, If the first mapping relationship corresponding to the first idempotent result is not stored locally, the second historical data is directly obtained from the first message queue device and stored in the target database; wherein the second historical data and the first historical data are from the same batch of data.

3. The method according to claim 1, characterized in that, Obtaining the first historical data from the first message queue device includes: Obtain data reception snapshot information corresponding to the first checkpoint information. The data reception snapshot information includes offset information of the first historical data received from the first message queue device. The offset information is used to indicate the position information of the first historical data in the first message queue device. The first historical data is obtained from the first message queue device based on the offset information.

4. The method according to claim 1, characterized in that, The first mapping relationship also stores the first target location information corresponding to the first historical data. The first target location information is the location information in the target database. The method further includes: submitting the first historical data and the first target location information to the target database, so that the target database stores the first historical data based on the first target location information.

5. The method according to claim 1, characterized in that, An idempotency operation is performed on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database to obtain a first idempotency result, including: A hash operation is performed on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database to obtain a first hash result; The first hash result is used as the first idempotent result.

6. The method according to claim 1, characterized in that, The method further includes: When the first submission status is either submission preparation status or submission failure status, it is determined that the first submission status does not indicate that the first historical data has been successfully submitted to the target database.

7. The method according to claim 1, characterized in that, Deleting residual data of the first historical data in the target database includes: querying whether the target database stores residual data of the first historical data; if so, deleting the residual data, wherein the residual data is data contained in the first historical data.

8. A data processing system, characterized in that, include: The second message queue device is used to receive pending data uploaded by terminal devices of multiple objects; A statistical device is used to perform statistics on the data to be processed and obtain statistical results of the data to be processed within a preset time period. A first message queue device is used to receive the statistical results; An inbound buffer device is used to consume the statistical results and store the statistical results in the target database. It is also used to obtain the most recent first checkpoint information after detecting that the target database or itself has restarted. The first checkpoint information includes at least one of the following: a first snapshot identifier of the first checkpoint information, a first snapshot timestamp of the first checkpoint information, and a first operator identifier of the operator used to submit data to the target database. Determining the first submission status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information includes: performing an idempotent operation on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database to obtain a first idempotent result; determining whether a first mapping relationship corresponding to the first idempotent result is stored locally, wherein the first mapping relationship stores the first submission status of the first historical data, and if so, obtaining the first submission status from the first mapping relationship; if the first submission status does not indicate that the first historical data has been successfully submitted to the target database, deleting the residual data of the first historical data in the target database; obtaining the value of a flag bit used to indicate whether the residual data has been successfully deleted; if the value of the flag bit indicates that the residual data has been successfully deleted... If the residual data has been successfully deleted, the system triggers the retrieval of the first historical data from the first message queue device. The first historical data is then stored in the target database through a pre-submission phase and a submission phase. In the pre-submission phase, the second submission state of the first historical data is initialized to a ready-to-submit state. A snapshot of the first historical data is taken to obtain second checkpoint information, and an idempotency operation is performed on the information in the second checkpoint information to obtain a second idempotency result. The second idempotency result is stored along with the second submission state of the first historical data. In the submission phase, the first historical data is submitted to the target database, and the second submission state is determined according to the corresponding submission result. If the second submission state is a ready-to-submit state or a submission failure state, the residual data of the first historical data in the target database is first deleted, and the first historical data is resubmitted. The target database is used to store the statistical results; The statistical results include the first historical data as described in any one of claims 1-7.

9. The system according to claim 8, characterized in that, The system also includes observation equipment. The observation device is used to obtain the user's viewing instructions, and to retrieve and display the first historical data from the target database according to the viewing instructions.

10. A data entry device, characterized in that, The device includes: The acquisition unit is used to acquire the most recent first checkpoint information when the target database or itself is detected to be restarted. The first checkpoint information includes at least one of the following: the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database. The determining unit is configured to determine the first submission status of the first historical data corresponding to the first checkpoint information based on the first checkpoint information, including: performing an idempotent operation on the first snapshot identifier of the first checkpoint information, the first snapshot timestamp of the first checkpoint information, and the first operator identifier of the operator used to submit data to the target database to obtain a first idempotent result; determining whether a first mapping relationship corresponding to the first idempotent result is stored locally, wherein the first mapping relationship stores the first submission status of the first historical data, and if so, obtaining the first submission status from the first mapping relationship; The deletion unit is used to delete residual data of the first historical data in the target database if the submission status does not indicate that the first historical data has been successfully submitted to the target database. The acquisition unit is further configured to acquire the value of a flag bit indicating whether the residual data has been successfully deleted; if the value of the flag bit indicates that the residual data has been successfully deleted, then the acquisition of the first historical data from the first message queue device is triggered. A storage unit is used to store the first historical data into the target database through a pre-submission phase and a submission phase. In the pre-submission phase, the second submission state of the first historical data is initialized to a ready-to-submit state. A snapshot of the first historical data is taken to obtain second checkpoint information, and an idempotency operation is performed on the information in the second checkpoint information to obtain a second idempotency result. The second idempotency result and the second submission state of the first historical data are stored together. In the submission phase, the first historical data is submitted to the target database, and the second submission state is determined according to the corresponding submission result. If the second submission state is a ready-to-submit state or a submission failure state, the residual data of the first historical data in the target database is deleted first, and the first historical data is resubmitted.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the steps of the method as described in any one of claims 1-7.

12. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executes the steps of the method according to any one of claims 1-7 by calling the computer program stored in the memory.

13. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

Images