Data extraction method and apparatus, electronic device, and computer readable medium

Abstract

Embodiments of the present disclosure disclose a data extraction method, device, electronic equipment and computer readable medium. A specific implementation of the method comprises: detecting, by using a daemon process, whether a target data set extraction task is completed; in response to detecting that the target data set extraction task is not completed, determining whether a target timestamp changes within a first time length, wherein the target timestamp is a timestamp of starting the target data set extraction task or querying a target data set from a database during execution of the target data set extraction task; in response to determining that the target timestamp does not change within the first time length, executing the target data set extraction task again and updating the target timestamp. The implementation can quickly and efficiently realize extraction of the target data set.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, and in particular, to a data extraction method and device, electronic equipment and computer readable medium. BACKGROUND

[0002] At present, when a company performs a certain business, it often needs to obtain static inventory data at a certain time to complete the above business. Therefore, the stability and accuracy of the inventory snapshot are crucial to each company. For the acquisition of inventory snapshots, the commonly used way is to configure a task to collect data from the database at a fixed time to obtain inventory snapshots.

[0003] However, when the above method is used to obtain inventory snapshots, the following technical problems often exist:

[0004] When the system is down or the database connection times out, the inventory snapshot cannot be accurately generated. Even if the inventory snapshot is generated later, it still has the problem of time and effort. SUMMARY

[0005] The summary part of the present disclosure is used to introduce the concepts in a brief form, which will be described in detail in the specific embodiments part. The summary part of the present disclosure is not intended to identify the key features or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of the present disclosure propose a data extraction method, device, electronic equipment and computer readable medium to solve the technical problems mentioned in the background part.

[0007] In a first aspect, some embodiments of the present disclosure provide a data extraction method, comprising: detecting, by a daemon process, whether a target data set extraction task is completed; in response to detecting that the target data set extraction task is not completed, determining whether a target timestamp changes within a first time length, wherein the target timestamp is a timestamp of starting the target data set extraction task or querying a target data set from a database during execution of the target data set extraction task; in response to determining that the target timestamp does not change within the first time length, executing the target data set extraction task again, and updating the target timestamp.

[0008] Optionally, the re-executing the target data set extraction task includes: determining the first target database as a query data source of the target data set to query the target data set; in response to determining that the number of times of query failure of the first target data set corresponding to the first target database reaches a first number and the task version information of each time of target data set query is same, determining the second target database as the query data source of the target data set to query the target data set; and in response to determining that the number of times of query failure of the second target data set corresponding to the second target database is less than a second number and the task version information of each time of target data set query is same, extracting the target data set from the second target database and sending the target data set to the target message queue.

[0009] Optionally, the method further includes: adding primary key information to each data in the target data set to obtain an added data set; determining a data amount of the added data set as a first data amount and determining a data amount of the target data set in the target message queue as a second data amount; and in response to the first data amount being equal to the second data amount, performing data verification on the added data set.

[0010] Optionally, the method further includes: in response to determining that the number of times of query failure of the first target data set is less than the first number and the task version information of each time of target data set query is same, continuing to query the target data set from the first target database.

[0011] Optionally, the method further includes: in response to determining that the task version information of each time of target data set query corresponding to the first target database is not same or the task version information of each time of target data set query corresponding to the second target database is not same, ending the re-executed target data set extraction task.

[0012] Optionally, the data verification on the added data set includes: based on the data set in the first target database or the data set in the second target database, performing data verification on the added data set by using a preset data verification method.

[0013] Optionally, the method further includes: for each time of querying the target data set from the first target database or each time of querying the target data set from the second target database, re-updating the updated target timestamp.

[0014] Optionally, the determining whether the first target database fails to query the data set includes: in response to determining that a query duration of the first target database querying the target data set is greater than a second duration, generating information representing that the first target database fails to query the target data set.

[0015] Optionally, whether the second target database fails to query the data set is determined by: in response to determining that a query duration of the second target database querying the target data set is greater than a third duration, generating information indicating that the second target database fails to query the target data set.

[0016] In a second aspect, some embodiments of the present disclosure provide a data extraction apparatus, comprising: a detection unit configured to detect, by using a daemon process, whether a target data set extraction task is completed; a determination unit configured to, in response to detecting that the target data set extraction task is not completed, determine whether a target timestamp changes within a first duration, wherein the target timestamp is a timestamp of querying a target data set from a database during execution of the target data set extraction task or a timestamp of starting the target data set extraction task; and an execution updating unit configured to, in response to determining that the target timestamp does not change within the first duration, execute the target data set extraction task again and update the target timestamp.

[0017] Optionally, the apparatus further comprises: in response to detecting that the target data set extraction task is executed again, updating task version information.

[0018] Optionally, the execution updating unit is configured to: determine a first target database as a query data source of the target data set to query the target data set; in response to determining that a number of times of querying the target data set corresponding to the first target database fails to reach a first number and that task version information of each time of querying the target data set is same, determine a second target database as the query data source of the target data set to query the target data set; and in response to determining that a number of times of querying the target data set corresponding to the second target database fails to be less than a second number and that the task version information of each time of querying the target data set is same, extract the target data set from the second target database and send the target data set to a target message queue.

[0019] Optionally, the apparatus further comprises: adding primary key information to each data in the target data set to obtain an added data set; determining a data amount of the added data set as a first data amount and determining a data amount of the target data set in the target message queue as a second data amount; and in response to the first data amount being equal to the second data amount, performing data verification on the added data set.

[0020] Optionally, the apparatus further comprises: in response to determining that the number of times of querying the target data set corresponding to the first target database fails to be less than the first number and that the task version information of each time of querying the target data set is same, continuing to query the target data set from the first target database.

[0021] Optionally, the apparatus further comprises: in response to determining that the number of query failures of the second target data set reaches the second number or the added data set fails the data check, performing a target operation corresponding to the alarm information.

[0022] Optionally, the apparatus further comprises: in response to determining that the task version information of each target data set query corresponding to the first target database is different or the task version information of each target data set query corresponding to the second target database is different, ending the re-executed target data set extraction task.

[0023] Optionally, the apparatus further comprises: writing the added data set to the first target database and / or the second target database.

[0024] Optionally, the apparatus further comprises: based on the data set in the first target database or the data set in the second target database, performing data check on the added data set by using a preset data check method.

[0025] Optionally, the apparatus further comprises: for each time of querying the target data set from the first target database or each time of querying the target data set from the second target database, re-updating the updated target timestamp.

[0026] In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0027] In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method described in any implementation manner of the first aspect.

[0028] The various embodiments of this disclosure have the following beneficial effects: the data extraction methods of some embodiments of this disclosure can quickly and efficiently extract the target dataset. Specifically, when the system crashes or the database connection times out, an accurate inventory snapshot cannot be generated. Even if an inventory snapshot is generated subsequently, it is time-consuming and labor-intensive. Based on this, the data extraction methods of some embodiments of this disclosure can first utilize a daemon process to detect whether the target dataset extraction task is completed. Here, by having a daemon thread constantly detect whether the target dataset extraction task is completed, the supervision of the target timestamp can be improved when the target dataset extraction task is not completed. Then, in response to detecting that the target dataset extraction task is not completed, it is determined whether the target timestamp has changed within a first time period. The target timestamp is the timestamp when the target dataset extraction task is started or when the target dataset is queried from the database during the execution of the target dataset extraction task. It should be noted that by determining whether the target timestamp has changed within a first time period, various problems that may occur during the target dataset extraction task, such as system crashes, database connection failures, etc., can be effectively avoided. Thus, the efficiency of data extraction is greatly improved. Finally, in response to the confirmation that the target timestamp remained unchanged within the first time period, the target dataset extraction task is executed again, and the target timestamp is updated. Here, the fact that the target timestamp remained unchanged within the first time period indicates that some problems occurred during the target dataset extraction task, causing it to stagnate or progress slowly. Therefore, the target dataset extraction task needs to be executed again to further extract the dataset, thus improving the efficiency of dataset extraction. In addition, updating the target timestamp allows the daemon process to monitor whether any problems occurred during the subsequent execution of the re-executed target dataset extraction task, ensuring efficient dataset extraction. Attached Figure Description

[0029] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0030] Figure 1 This is a schematic diagram illustrating an application scenario of a data extraction method according to some embodiments of the present disclosure;

[0031] Figure 2 These are flowcharts of some embodiments of the data extraction method according to this disclosure;

[0032] Figure 3is a flowchart of another embodiment of the data extraction method according to the present disclosure;

[0033] Figure 4 is a schematic diagram of a target timestamp changing in some embodiments of the data extraction method according to the present disclosure;

[0034] Figure 5 is a structural schematic diagram of some embodiments of the data extraction apparatus according to the present disclosure;

[0035] Figure 6 is a structural schematic diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided so as to more completely and comprehensively understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure.

[0037] It should also be noted that, for ease of description, only parts related to the present application are shown in the drawings. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0038] It should be noted that the terms "first", "second", and the like mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0039] It should be noted that the terms "one", "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0040] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0041] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0042] Figure 1 is a schematic diagram of one application scenario of the data extraction method according to some embodiments of the present disclosure.

[0043] In Figure 1In the application scenario, electronic device 101 can first use daemon process 102 to detect whether the target dataset extraction task 103 has been completed. Then, in response to detecting that the target dataset extraction task 103 has not been completed, it determines whether the target timestamp 104 has changed within a first duration 105. The target timestamp 104 is the timestamp at which the target dataset extraction task 103 starts or during the execution of the target dataset extraction task 103, and the target dataset is queried from the database. Finally, in response to determining that the target timestamp 104 has not changed within the first duration 105, the target dataset extraction task 103 is executed again, and the target timestamp 104 is updated to obtain an updated target timestamp 106.

[0044] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0045] It should be understood that Figure 1 The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.

[0046] Continue to refer to Figure 2 The flowchart 200 illustrates some embodiments of a data extraction method according to the present disclosure. The data extraction method includes the following steps:

[0047] Step 201: Use the daemon process to check whether the target dataset extraction task has been completed.

[0048] In some embodiments, the execution entity of the above data extraction method (e.g. Figure 1 The electronic device 101 shown can utilize a daemon process to continuously monitor whether the target dataset extraction task is complete. This daemon process can be a thread providing general services in the background during program execution, such as a garbage collection process. The target dataset extraction task can be an extraction task targeting a specific target dataset. The target dataset extraction task can correspond to obtaining an inventory snapshot. The target dataset can be extracted from various databases.

[0049] Optionally, in response to the daemon detecting the completion of the target dataset extraction task, the task status corresponding to the target dataset extraction task can be updated to "task completed".

[0050] Optionally, when the task state corresponding to the target data set extraction task is "task completed", the execution subject can restart the completion of the target data set extraction task after a certain period of time to periodically obtain the inventory snapshot.

[0051] In step 202, in response to detecting that the target data set extraction task is not completed, it is determined whether the target timestamp changes within a first time length.

[0052] In some embodiments, in response to detecting that the target data set extraction task is not completed, the execution subject can determine whether the target timestamp changes within a first time length. The target timestamp is the timestamp of querying the target data set from the database when the target data set extraction task is started or during the execution of the target data set extraction task. The first time length can be pre-set. For example, 30s. The database can be various databases for querying the target data set. For example, MySQL (relational database management system). The target data set extraction task can include querying the target data set and extracting the target data set.

[0053] As an example, in response to the daemon detecting that the target data set extraction task is not completed, the daemon can determine whether the target timestamp changes within a first time length.

[0054] In step 203, in response to determining that the target timestamp does not change within the first time length, the target data set extraction task is executed again, and the target timestamp is updated.

[0055] In some embodiments, in response to determining that the target timestamp does not change within the first time length, the execution subject can execute the target data set extraction task again, and update the target timestamp.

[0056] It should be noted that executing the target data set extraction task again does not make the target data set extraction task that has not ended end, but restarts the target data set extraction task that has not ended. Instead, the target data set extraction task is executed again in the process of continuing the target data set extraction task that has not ended.

[0057] Here, the target timestamp is updated for the subsequent daemon to supervise the execution of the target data set extraction task executed again.

[0058] Optionally, in response to detecting the re-execution of the target data set extraction task, the execution subject can update the task version information. The task version information can represent the execution version information corresponding to the current target data extraction task. The updated task version information can be the task version information of the re-executed target data set extraction task. As an example, the task version information can be an identification value. For example, the task version information can be "1.3.4".

[0059] As an example, in response to detecting the re-execution of the target data set extraction task, the execution subject can update the task version information from "1.3.4" to "1.3.5".

[0060] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: The data extraction method of some embodiments of the present disclosure can quickly and efficiently extract the target data set. Specifically, when the system is down or the database connection times out, the inventory snapshot cannot be accurately generated. Even if the inventory snapshot is generated later, there is a problem of time and effort. Based on this, the data extraction method of some embodiments of the present disclosure can first use a daemon to detect whether the target data set extraction task is completed. Here, the daemon thread detects whether the target data set extraction task is completed at all times, which can improve the subsequent supervision of the target timestamp when the target data set extraction task is not completed. Then, in response to detecting that the target data set extraction task is not completed, it is determined whether the target timestamp changes within a first time period. The target timestamp is the timestamp of querying the target data set from the database when the target data set extraction task is started or during the execution of the target data set extraction task. It should be noted that by determining whether the target timestamp changes within the first time period, various problems that can occur during the target data set extraction task can be effectively avoided, such as system downtime, database connection problems, etc. Thus, the efficiency of data extraction is greatly improved. Finally, in response to determining that the target timestamp does not change within the first time period, the target data set extraction task is re-executed, and the target timestamp is updated. Here, the target timestamp not changing within the first time period can indicate that some problems have occurred during the target data set extraction task, resulting in no progress or slow progress of the target data set extraction task. Therefore, the target data set extraction task needs to be re-executed to further extract the data set. This further improves the data set extraction efficiency. In addition, the target timestamp is updated to subsequently supervise whether the re-executed target data set extraction task has some problems during execution through the daemon, thereby ensuring efficient extraction of the data set.

[0061] Further reference is made to Figure 3, shows a flow 300 of another embodiment of the data extraction method according to the present disclosure. The data extraction method comprises the following steps:

[0062] Step 301, using a daemon process, detecting whether a target data set extraction task is completed.

[0063] Step 302, in response to detecting that the target data set extraction task is not completed, determining whether a target timestamp changes within a first time length.

[0064] Step 303, in response to determining that the target timestamp does not change within the first time length, executing the target data set extraction task again, and updating the target timestamp.

[0065] In some embodiments, the specific implementation of steps 301-303 and the technical effects brought by them can refer to Figure 2 Steps 201-203 in the corresponding embodiment, which will not be repeated here.

[0066] Step 304, determining a first target database as a query data source of the target data set to query the target data set.

[0067] In some embodiments, the execution subject (for example Figure 1 The electronic device 101 shown) can determine the first target database as the query data source of the target data set to query the target data set.

[0068] As an example, the execution subject can determine a MySQL database as the query data source of the target data set to query the target data set.

[0069] Step 305, in response to determining that the number of query failures of the first target database corresponding to the first target data set reaches a first number, and the task version information of each target data set query is the same, determining a second target database as the query data source of the target data set to query the target data set.

[0070] In some embodiments, in response to determining that the number of query failures of the first target database corresponding to the first target data set reaches a first number, and the task version information of each target data set query is the same, the execution subject can determine the second target database as the query data source of the target data set to query the target data set. Wherein, the first number can be pre-set. For example, 3 times. Wherein, the each target data set query is a query that fails to query the target data set. The first target data query failure number can be the number of times of querying the target data set that fails to query the target data set.

[0071] It should be noted that the first target database corresponding to the first target data set query failure number reaches the first number, and each time the first target database query target data set fails, it is determined whether the task version information is the same, that is, whether the task version information has been transformed. For the process of querying the target data set in the first target database, there may be system downtime or database connection timeout, etc. Long time leads to target timestamp not updated. Further, the daemon thread will forcibly execute the target data set extraction task again. At this time, the re-start of the target data set extraction task will update the task version information. Leading to the first target database in the process of querying the target data set, the task version information is different. In this regard, the first target database needs to determine the task version information of each target data set query in the process of querying the target data set to ensure that no other problems occur in the first target database query process.

[0072] In some optional implementations of some embodiments, whether the first target database queries the data set fails is determined by the following steps:

[0073] In response to determining that the query duration of the first target database querying the target data set is greater than the second duration, the first target database can generate information representing the failure of the first target database querying the target data set. Wherein the second duration can be pre-set. For example, the second duration can be 10 seconds.

[0074] Step 306, in response to determining that the second target database corresponding to the second target data set query failure number is less than the second number, and the task version information of each target data set query is the same, the target data set is extracted from the second target database, and the target data set is sent to the target message queue.

[0075] In some embodiments, in response to determining that the number of times of query failure of the second target database corresponding to the second target data set is less than the second number, and the task version information of each time of query of the target data set is same, the execution subject can extract the target data set from the second target database, and send the target data set to the target message queue. The second number can be preset, for example, 3. The second target database can be an ES (Elasticsearch) database for storing data. The ES database can be a non-relational database. The target message queue can be a message middleware (JMQ). Here, the target data set is sent to the target message queue for use by the upstream data. The each time of query of the target data set includes a query of the target data set failure and a query of the target data set success. The number of times of query failure of the second target database can be the number of times of query failure of the second target database for the target data set.

[0076] It should be noted that for the number of times of query failure of the second target database corresponding to the second target data set is less than the second number, after each time of query failure of the second target database for the target data set, it is determined whether the task version information is same, i.e., whether the task version information is changed. During the process of querying the target data set by the second target database, there can be a system downtime or a database connection timeout, etc. Long time causes the target timestamp not to be updated. Further, the daemon thread will forcibly execute the target data set extraction task again. At this time, the re-starting of the target data set extraction task will update the task version information again. This causes the task version information to be different during the process of querying the target data set by the second target database. For this, the second target database needs to determine that the task version information of each time of query of the target data set is same during the process of querying the target data set.

[0077] In some optional implementations of some embodiments, the step further includes:

[0078] Firstly, a primary key information is added to each data in the target data set to obtain an added data set. The primary key information can be index information. The primary key information can include, but is not limited to, at least one of the following: department code, article code, warehouse code, article level, inventory type, inventory state, data time.

[0079] Secondly, the execution subject can determine the data amount of the added data set as a first data amount, and determine the data amount of the target data set in the target message queue as a second data amount.

[0080] Thirdly, in response to the first data amount being equal to the second data amount, the execution subject performs data verification on the added data set.

[0081] In some optional implementations of some embodiments, the above steps further include: in response to determining that the number of times of the first target database query failure is less than the first number and the task version information of each target database query is the same, the execution subject can continue to query the target dataset from the first target database. Wherein, the each target database query can be a query that fails in the first target database.

[0082] It should be noted that for the first target database corresponding to the first target dataset query failure number is less than the first number, each time the first target database query target dataset fails, it will be determined whether the task version information is the same, that is, whether the task version information has changed. For the process of querying the target dataset in the first target database, there may be system downtime or database connection timeout, etc. Long time leads to target timestamp not updated. Further, the daemon thread will forcibly execute the target dataset extraction task again. At this time, the re-launching of the target dataset extraction task will update the task version information again. Leading to the first target database in the process of querying the target dataset, the task version information is different. For this, the first target database needs to determine the task version information of each target dataset query in the process of querying the target dataset.

[0083] Optionally, the above steps further include: in response to determining that the number of times of the second target database query failure reaches the second number or the added dataset fails the data check, the execution subject can perform a target operation corresponding to the alarm information. Wherein, the target operation can be an operation performed for the alarm information. For example, alarm information sending operation, alarm information display operation, etc. As an example, the alarm information can be UMP alarm information.

[0084] In some optional implementations of some embodiments, the above steps further include: in response to determining that the task version information of each target dataset query corresponding to the first target database is not the same or the task version information of each target dataset query corresponding to the second target database is not the same, the execution subject can end the re-executed target dataset extraction task.

[0085] It should be noted that the task version information of each target data set query corresponding to the first target database is different, or the task version information of each target data set query corresponding to the second target database is different, which can represent that the first target database or the second target database may have a system downtime or a database connection timeout during the query of the target data set. Long time target timestamp is not updated. Further, the daemon thread will forcibly execute the target data set extraction task again. Therefore, the above execution subject can end the execution of the corresponding target data set extraction task.

[0086] Optionally, the above steps further include that the above execution subject can write the above added data set to the above first target database and / or the above second target database.

[0087] Optionally, the above data verification on the above added data set can include the following steps:

[0088] Based on the data set in the first target database or the data set in the second target database, the execution subject can use a preset data verification method to verify the data of the added data set.

[0089] As an example, the execution subject can first determine the data amount of the added data set as a third data amount. Then, in response to determining that the query data source of the target data set is the first target database, it is determined whether the number of data sets in the first target database is between third data amount x (1-2%) and third data amount x (1+2%). Further, in response to determining that the number of data sets in the first target database is between third data amount x (1-2%) and third data amount x (1+2%), it is determined that the data verification of the added data set is passed.

[0090] As an example, the execution subject can first determine the data amount of the added data set as a third data amount. Then, in response to determining that the query data source of the target data set is the second target database, it is determined whether the number of data sets in the second target database is between third data amount x (1-2%) and third data amount x (1+2%). Further, in response to determining that the number of data sets in the second target database is between third data amount x (1-2%) and third data amount x (1+2%), it is determined that the data verification of the added data set is passed.

[0091] As a further example, the execution subject can first determine the data amount of the added data set as a third data amount. Then, in response to determining that the query data source of the target data set is the first target database, the execution subject determines whether the number of the first target database is between the third data amount x (1-2%) and the third data amount x (1+2%). Further, in response to determining that the number of the first target database is between the third data amount x (1-2%) and the third data amount x (1+2%), the execution subject determines that the added data set passes the data check.

[0092] As a further example, the execution subject can first determine the inventory amount corresponding to the added data set as a first inventory amount. Then, the execution subject determines the inventory amount of the first target database as a second inventory amount. Further, in response to determining that the query data source of the target data set is the first target database, the execution subject determines whether the second inventory amount is between the first inventory amount x (1-2%) and the first inventory amount x (1+2%). Finally, in response to determining whether the second inventory amount is between the first inventory amount x (1-2%) and the first inventory amount x (1+2%), the execution subject determines that the added data set passes the data check.

[0093] Similarly, the execution subject can first determine the inventory amount corresponding to the added data set as a first inventory amount. Then, the execution subject determines the inventory amount of the second target database as a third inventory amount. Further, in response to determining that the query data source of the target data set is the second target database, the execution subject determines whether the third inventory amount is between the first inventory amount x (1-2%) and the first inventory amount x (1+2%). Finally, in response to determining whether the third inventory amount is between the first inventory amount x (1-2%) and the first inventory amount x (1+2%), the execution subject determines that the added data set passes the data check.

[0094] In some optional implementations of some embodiments, the method further includes that, for each time the first target database queries the target data set or each time the second target database queries the target data set, the execution subject can update the updated target timestamp again.

[0095] As an example, as Figure 4As shown, before the first target database 401 queries the target data set 402, the target timestamp can be the timestamp 403. The timestamp 403 can be: "1625629554". After the first target database 401 fails to query the target data set 402 for the first time, the target timestamp is updated from the timestamp 403 to the timestamp 404. The timestamp 404 can be: "1625629614". After the first target database 401 fails to query the target data set 402 for the second time, the target timestamp is updated from the timestamp 404 to the timestamp 405. The timestamp 405 can be: "1625629674". After the first target database 401 succeeds to query the target data set 402 for the third time, the target timestamp is updated from the timestamp 405 to the timestamp 406. The timestamp 406 can be: "1625629734".

[0096] In some optional implementations of some embodiments, whether the second target database fails to query the data set is determined by the following steps:

[0097] In response to determining that the query duration of the second target database querying the target data set is greater than a third duration, the second target database can generate information indicating that the second target database fails to query the target data set. The third duration can be pre-set, for example, 10 seconds.

[0098] From Figure 3 It can be seen that, compared with the description of some embodiments corresponding to Figure 2 , the flow 300 of the data extraction method in some embodiments corresponding to Figure 3 emphasizes the specific steps of target data set querying and target data set extraction. Therefore, the scheme described in these embodiments can more accurately and efficiently achieve the extraction of the target data set by using the first target database and the second target database to query multiple times in sequence.

[0099] Further referring to Figure 5 , as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a data extraction device. These device embodiments correspond to the method embodiments shown in Figure 2 , and the device can be specifically applied to various electronic devices.

[0100] As Figure 5As shown, a data extraction apparatus 500 includes a detection unit 501, a determination unit 502, and an execution updating unit 503. The detection unit 501 is configured to detect, by using a daemon process, whether a target data set extraction task is completed. The determination unit 502 is configured to, in response to detecting that the target data set extraction task is not completed, determine whether a target timestamp changes within a first time length, wherein the target timestamp is a timestamp of querying a target data set from a database during execution of the target data set extraction task or a timestamp of starting the target data set extraction task. The execution updating unit 503 is configured to, in response to determining that the target timestamp does not change within the first time length, execute the target data set extraction task again and update the target timestamp.

[0101] In some optional implementations of some embodiments, the execution updating unit 503 can be further configured to: determine a first target database as a query data source of the target data set to query the target data set; in response to determining that a number of query failures of the first target database corresponding to the target data set reaches a first number and task version information of each time of querying the target data set is same, determine a second target database as the query data source of the target data set to query the target data set; and in response to determining that a number of query failures of the second target database corresponding to the target data set is less than a second number and the task version information of each time of querying the target data set is same, extract the target data set from the second target database and send the target data set to a target message queue.

[0102] In some optional implementations of some embodiments, the apparatus 500 further includes an adding unit, a data amount determining unit, and a data checking unit (not shown in the figure). The adding unit can be configured to add primary key information to each data in the target data set to obtain an added data set. The data amount determining unit can be configured to determine a data amount of the added data set as a first data amount and determine a data amount of the target data set in the target message queue as a second data amount. The data checking unit can be configured to, in response to the first data amount being equal to the second data amount, perform data checking on the added data set.

[0103] In some optional implementations of some embodiments, the apparatus 500 further includes a querying unit (not shown in the figure). The querying unit can be configured to, in response to determining that the number of query failures of the first target database corresponding to the target data set is less than the first number and the task version information of each time of querying the target data set is same, continue to query the target data set from the first target database.

[0104] In some optional implementations of some embodiments, the apparatus 500 further comprises an ending unit (not shown in the figure). The ending unit is configured to end the target data set extraction task being executed again, in response to determining that the task version information of each target data set query corresponding to the first target database is different or the task version information of each target data set query corresponding to the second target database is different.

[0105] In some optional implementations of some embodiments, the data checking unit is further configured to check the added data set by using a preset data checking method based on the data set in the first target database or the data set in the second target database.

[0106] In some optional implementations of some embodiments, the apparatus 500 further comprises a timestamp updating unit (not shown in the figure). The timestamp updating unit is configured to update the target timestamp again after updating, for each time of querying the target data set from the first target database or each time of querying the target data set from the second target database.

[0107] It can be understood that the units described in the apparatus 500 correspond to the respective steps in the method described with reference to Figure 2 The operations, features and advantages described above for the method also apply to the apparatus 500 and the units included therein, and will not be described here again.

[0108] Reference is made below to Figure 6 which shows a structural schematic diagram of an electronic device (for example, the electronic device 101 in Figure 1 ) suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0109] As shown in Figure 6 , the electronic device 600 can include a processing apparatus (for example, a central processor, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage apparatus 608. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing apparatus 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0110] In general, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 608 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 609. The communication devices 609 can allow the electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it is understood that all of the shown devices are not required to be implemented or present. More or fewer devices can alternatively be implemented or present. Figure 6 Each block shown in the flowcharts can represent a device or multiple devices as needed.

[0111] In particular, processes described above with reference to the flowcharts can be implemented as a computer software program according to some embodiments of the present disclosure. For example, some embodiments of the present disclosure include a computer program product including a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In some such embodiments, the computer program can be downloaded and installed from a network through the communication devices 609, or installed from the storage devices 608, or installed from the ROM 602. When the computer program is executed by the processing devices 601, the above-described functions defined in the methods of some embodiments of the present disclosure are performed.

[0112] Note that the computer-readable medium or media used to provide the computer program sequence to the computer system can be embedded in a computer program product, which comprises all the respective features, which are provided with the computer program sequence, and which are enumerated above. It is understood that the computer-readable medium or media described herein are included in the computer program product, or are a component of the computer program product. In some embodiments of the disclosure, the computer-readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the disclosure, a computer-readable storage medium can be any tangible medium that contains, or stores a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the disclosure, a computer-readable signal medium can include a computer-readable storage medium in baseband or propagated as a carrier wave in a propagated data signal, which contains a computer-readable program code. Such a propagated signal can take a wide variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0113] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0114] The computer readable medium can be included in the electronic device; or can exist separately from the electronic device. The computer readable medium bears one or more programs, which, when executed by the electronic device, cause the electronic device to: detect, by using a daemon, whether a target data set extraction task is completed; in response to detecting that the target data set extraction task is not completed, determine whether a target timestamp changes within a first time length, wherein the target timestamp is a timestamp of querying a target data set from a database at a time of starting the target data set extraction task or during execution of the target data set extraction task; in response to determining that the target timestamp does not change within the first time length, execute the target data set extraction task again, and update the target timestamp.

[0115] Computer program code for carrying out operations of some embodiments of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0116] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the opposite order, depending on the functionality involved. It is also noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0117] The units described in some embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. The described units can also be arranged in a processor, for example, can be described as: a processor comprising a detection unit, a determination unit and an execution update unit. Among them, the names of these units do not constitute a limitation on the units themselves in some cases, for example, the detection unit can also be described as: a unit that detects whether the target data set extraction task is completed by using a daemon.

[0118] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, non-limiting example types of hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0119] The above description is merely some preferred embodiments of the present disclosure and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with each other to form a technical solution with similar functions to the technical features disclosed in the embodiments of the present disclosure (but not limited to).

Claims

1. A data extraction method, comprising: detecting, by a daemon, whether a target data set extraction task is completed; in response to detecting that the target data set extraction task is not completed, determining whether a target timestamp changes within a first time length, wherein the target timestamp is a timestamp of starting the target data set extraction task or querying a target data set from a database during execution of the target data set extraction task; in response to determining that the target timestamp does not change within the first time length, executing the target data set extraction task again during continuous execution of the target data set extraction task that is not completed, and updating the target timestamp, wherein task version information of each target data set extraction task is different, and the target timestamp and the task version information are used to determine whether an exception occurs during execution of the corresponding target data set extraction task.

2. The method of claim 1, wherein, the executing the target data set extraction task again comprises: determining a first target database as a query data source of the target data set to query the target data set; in response to determining that a number of query failures of a first target data set corresponding to the first target database reaches a first number, and the task version information of each target data set query is the same, determining a second target database as the query data source of the target data set to query the target data set; in response to determining that a number of query failures of a second target data set corresponding to the second target database is less than a second number, and the task version information of each target data set query is the same, extracting the target data set from the second target database, and sending the target data set to a target message queue.

3. The method of claim 2, wherein, The method further comprises: adding primary key information to each data in the target data set to obtain an added data set; determining a data amount of the added data set as a first data amount, and determining a data amount of the target data set in the target message queue as a second data amount; in response to the first data amount being equal to the second data amount, performing data verification on the added data set.

4. The method of claim 3, wherein, The method further comprises: in response to determining that the number of query failures of the first target data set is less than the first number and the task version information of each target data set query is the same, continuing to query the target data set from the first target database.

5. The method of claim 2, wherein, The method further comprises: in response to determining that the task version information of each target data set query corresponding to the first target database is different or the task version information of each target data set query corresponding to the second target database is different, ending the target data set extraction task executed again.

6. The method of claim 3, wherein, The performing data verification on the added data set comprises: performing data verification on the added data set by using a preset data verification method based on the data set in the first target database or the data set in the second target database.

7. The method of claim 2, wherein, The method further comprises: updating the updated target timestamp again for each query of the target data set from the first target database or each query of the target data set from the second target database.

8. The method of claim 2, wherein, whether the first target database fails to query the data set is determined by the following steps: in response to determining that the first target database queries the target data set for a duration greater than a second duration, information is generated indicating that the first target database fails to query the target data set.

9. The method of claim 2, wherein, whether the second target database fails to query the data set is determined by the following steps: in response to determining that the second target database queries the target data set for a duration greater than a third duration, information is generated indicating that the second target database fails to query the target data set.

10. A data extraction apparatus comprising: a detection unit configured to detect, using a daemon process, whether a target data set extraction task is complete; a determination unit configured to, in response to detecting that the target data set extraction task is not complete, determine whether a target timestamp changes within a first duration, wherein the target timestamp is a timestamp of querying a target data set from a database during execution of the target data set extraction task or a timestamp of starting the target data set extraction task; an execution update unit configured to, in response to determining that the target timestamp does not change within the first duration, execute the target data set extraction task again during continuation of the target data set extraction task that is not complete, and update the target timestamp, wherein task version information of each target data set extraction task is different, and the timestamp and the task version information are used to determine whether an abnormal query occurs during execution of the corresponding target data set extraction task.

11. An electronic device comprising: one or more processors; a memory device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-9.

12. A computer readable medium having stored thereon a computer program, wherein, the program is executed by the processor to implement the method of any one of claims 1-9. the program is executed by the processor to implement the method of any one of claims 1-9.

Images