Incremental data acquisition method, device, electronic device and storage medium
By creating a data filtering structure in the incremental database of the hospital business system, obtaining the location coded value based on the data identification, and judging the index identification to determine the incremental data, the problem of low efficiency in obtaining the full data is solved, and fast and accurate incremental data is achieved Backup.
Patent Information
- Application Number
- CN202411342918.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-25
AI Technical Summary
In databases such as hospital business systems, due to improper timestamps or primary key segmentation, incremental data acquisition cannot be effectively performed, resulting in full data acquisition, inefficiency and waste of resources.
By creating a data filter structure in an incremental database, such as a Bloom filter, obtaining the position coded value based on the data identification and judging the index identification, we determine the incremental data to avoid obtaining the full data.
Improves data backup speed, reduces resource usage, and ensures accuracy in incremental data acquisition without timestamps or timestamps are not available.
Smart Images

Figure CN118861025B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to an incremental data acquisition method, device, electronic device and storage medium. Background Art
[0002] When the data in the business system is updated, it is necessary to incrementally update the newly added business data to ensure that the data can be accurately backed up. In the prior art, incremental data is generally acquired based on the timestamp of the data. However, in some application scenarios, such as hospital business systems, due to the uneven design of the business system database of the hospital business system, some systems may have difficulties in incremental data acquisition based on timestamps or difficulty in splitting data based on non-integer primary keys, which makes it impossible to acquire incremental data based on timestamps, and only full data can be acquired. This acquisition method will result in repeated acquisition of data, slow data synchronization, and occupy more resources. Summary of the invention
[0003] In view of the above technical problems, the technical solution adopted by the present invention is:
[0004] According to a first aspect of the present invention, a method for acquiring incremental data is provided, the method comprising the following steps:
[0005] S100, at the current calculation time, obtain any data table D from the current business database g The timestamp field TF g , if TF g If the set is empty or the acquired timestamp field is an invalid field, execute S200; the value of g is 1 to J, where J is the number of data tables in the current business database.
[0006] S200, determine whether there is a corresponding table D in the current incremental database g If the data filtering structure corresponding to the identifier exists, execute S400; if not, execute S300; wherein the data filtering structure includes Q index tuples, and the rth index tuple includes (L r , P r ), L r is the position identifier corresponding to the rth index tuple, P r is the index identifier corresponding to the rth index tuple, and the value of r ranges from 1 to Q; the initial values of the index identifiers in the data filtering structure are all the first index values.
[0007] S300, in the current incremental database, data table D g Create a corresponding data filtering structure and execute S400.
[0008] S400, based on data sheet Dg The data identifier corresponding to the i-th data in the , obtain the position encoding value set H corresponding to the i-th data g i ={h g i1 ,h g i2 ,……,h g ij ,……,h g in}, execute S500; wherein, h g ij is the position encoding value of the i-th data obtained based on the j-th position encoding function, i ranges from 1 to m, and m is D g The number of data in , j ranges from 1 to n, and n is the number of position encoding functions.
[0009] S500, traverse H g i , for h g ij , if data table D g Corresponding data filtering structure FS g Zhong and h g ij The corresponding position identifier and the corresponding index identifier are the first index value, and h g ij The corresponding index identifier is stored in the corresponding current index identifier record set R g ij In the corresponding data filter structure FS g The corresponding index identifier is set to the second index value, and S600 is executed; if the data table D g Corresponding current data filtering structure FS g Zhong and h g ij The corresponding position identifier and the corresponding index identifier are the second index value, and h g ij The corresponding index identifier is stored in the corresponding current index identifier record set R g ij , execute S600; R g ij The initial value of is the empty set.
[0010] S600, if R g ij All index identifiers in are the second index values, and the i-th data is not used as incremental data. If R g ijThere is at least one index identifier as the first index value, the i-th data is used as incremental data and the i-th data is obtained from the current business database; execute S700.
[0011] S700, set i=i+1, if i≤m, execute S400, otherwise, execute S100. According to a second aspect of the present invention, there is provided an incremental data acquisition device, comprising:
[0012] The data acquisition module is used to obtain the timestamp field of any data table from the current business database at the current calculation moment.
[0013] The judgment module is used to determine whether the timestamp field of the acquired data table is an empty set or an invalid field. If so, it is determined whether there is a data filtering structure corresponding to the identifier of the data table in the current incremental database. If so, the position code of any data in the data table is obtained. If not, a corresponding data filtering structure is created for the data table in the current incremental data database.
[0014] The position code acquisition module is used to acquire the position code of any data in the data table.
[0015] A judgment module is used to store the index identifier corresponding to the position code in a corresponding current index identifier record set when it is determined that the index identifier corresponding to the position code value of any data in the data filtering structure corresponding to the data table is a first index value, and set the index identifier corresponding to the position code in the corresponding current data filtering structure to a second index value; and when it is determined that the index identifier corresponding to the position code value of any data in the data filtering structure corresponding to the data table is a second index value, store the index identifier corresponding to the position code in the corresponding current index identifier record set; the initial value of the current index identifier record set is an empty set.
[0016] An incremental data determination module is used to not treat the data as incremental data when it is determined that all index identifiers corresponding to the current index identifier record set corresponding to any data in the data table are second index values, and to treat any data as incremental data when it is determined that there is at least one index identifier as the first index value in the current index identifier record set corresponding to any data in the data table.
[0017] An incremental data acquisition module is used to acquire the incremental data determined by the incremental data determination module from the current business database. According to a third aspect of the present invention, there is provided an electronic device, comprising a processor and a memory; the processor is used to execute the method described in the first aspect of the present invention by calling a program or instruction stored in the memory. According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the first aspect of the present invention.
[0018] The present invention has at least the following beneficial effects:
[0019] The technical solution provided by the embodiment of the present invention can filter the data based on the position code obtained by the data identifier to obtain incremental data when the data has no timestamp or the timestamp is unavailable, instead of obtaining the full data, thereby improving the data backup speed and reducing data backup resources.
[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0022] Figure 1 A flowchart of a method for acquiring incremental data provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.
[0025] It should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.
[0026] An embodiment of the present invention provides a method for acquiring incremental data, the method comprising: Figure 1 The following steps are shown:
[0027] S100, at the current calculation time, obtain any data table D from the current business database g The timestamp field TF g , if TF g If the set is empty or the acquired timestamp field is an invalid field, execute S200; the value of g is 1 to J, where J is the number of data tables in the current business database.
[0028] In an embodiment of the present invention, the processor that performs incremental data acquisition acquires enhanced data from the business database according to a set time period. The set time period can be determined based on actual conditions. In an exemplary embodiment, the set time period can be one day. The processor performs incremental data acquisition operations at each calculation moment.
[0029] In an exemplary embodiment of the present invention, the business database may be a hospital business database. In the embodiment of the present invention, an invalid field is an unusable field, for example, a timestamp field that inaccurately describes business data, that is, the timestamp field is not updated when the data changes.
[0030] S200, determine whether there is a match with data table D in the current incremental data database. g If the data filtering structure corresponding to the identifier exists, execute S400; if not, execute S300.
[0031] In an embodiment of the present invention, the identifier of the data table may be a unique identifier of the data table. The data filtering structure may include Q index tuples, wherein the rth index tuple includes (L r , P r ), L r is the position identifier corresponding to the rth index tuple, P r is the index identifier corresponding to the rth index tuple, and the value of r ranges from 1 to Q; the initial values of the index identifiers in the data filtering structure are all the first index values.
[0032] In an embodiment of the present invention, the index tuple may be a binary array, the position identifier may be represented by a number, the index identifier may be represented by 0 or 1, wherein the first index value may be represented by 0 and the second index value may be represented by 1. In an embodiment of the present invention, a specific implementation of the data filtering structure may be a Bloom filter.
[0033] S300, in the current incremental database, data table D g Create a corresponding data filtering structure and execute S400.
[0034] In the embodiment of the present invention, when creating a data filter structure for each data table, the length of the created data filter structure is related to the data volume and average data growth rate of the corresponding data table.
[0035] Furthermore, S300 may specifically include:
[0036] S301, obtain data table D g The number of data in q g .
[0037] In the embodiment of the present invention, q can be obtained based on the SQL query statement. g For example, it can be obtained by querying the statement "select count(1) from ${dbName}.${tbName}".
[0038] S302, based on q g , Data Table D g The corresponding average data growth rate c g And the preset time T, get the data table D g The length of the corresponding data filter structure LN g Among them, LN g =Roundup(q g ×c g ×T), Roundup ( ) means rounding up.
[0039] In the embodiment of the present invention, c g Based on data table D g For example, a historical time period is set, such as a data table D within half a year or a year. g The historical data of the set time period is used to obtain the average data growth rate as c g , which can be obtained through SQL statements. For example, it can be obtained through the following statement:
[0040] “Select
[0041] Avg((cnt2-cnt1) / cnt1) as rate
[0042] from (
[0044] select
[0045] ${dt},
[0046] cnt as cnt1,
[0047] lead(cnt)over(order by ${dt} asc) as cnt2
[0048] from (
[0050] select
[0051] ${dt},
[0052] count(1) as cnt
[0053] from ${dbName}.${tbName}
[0054] group by
[0055] ${dt}
[0056] )aa
[0057] )bb;".
[0058] In the embodiment of the present invention, the preset time length T can be set based on actual needs, as long as it can ensure that the space capacity of the data filtering structure is large enough, for example, it can be 365 days.
[0059] In another embodiment of the present invention, if c cannot be obtained g , LN can be set g =q g ×b. b is a preset value. In an exemplary embodiment, b=10.
[0060] Those skilled in the art know that the index identifiers in the created data filtering structure are all first index values.
[0061] S400, based on data sheet D g The data identifier corresponding to the i-th data in the , obtain the position encoding value set H corresponding to the i-th data g i ={h g i1 ,h g i2 ,……,h g ij ,……,hg in}, execute S500; wherein, h g ij is the position encoding value of the i-th data obtained based on the j-th position encoding function, i ranges from 1 to m, and m is the value of the data table D g The number of data in , j ranges from 1 to n, and n is the number of position encoding functions.
[0062] In an embodiment of the present invention, the position encoding function may be a random mapping function, specifically an unbiased hash function. In an embodiment of the present invention, the data filtering structure may have two characteristic parameters, one is the number of random mapping functions, and the other is the fault tolerance rate. The fault tolerance rate is used to characterize the error rate that the data filtering structure can allow. In an embodiment of the present invention, the number of position encoding functions is negatively correlated with the fault tolerance rate of the corresponding data filtering structure, that is, the lower the fault tolerance rate, the more the number of position encoding functions, and the specific functional relationship between the two can be set based on actual needs. In general, the lower the fault tolerance rate, the larger the length of the Bloom filter (the larger the storage space occupied), and the more hash functions need to be calculated. Assuming 3 Bloom filters, the lower the fault tolerance rate, the more likely it is that 10 will be needed to locate. In the actual production process, it can be adjusted according to the importance of the business data. For example, if the business data is transaction data, it is very important data, and the data accuracy should be guaranteed as much as possible, and the error rate should be extremely low. For example, if the business data is log data, the data volume will be very large, the data value is relatively low, and the error rate can be relatively increased.
[0063] In the embodiment of the present invention, the data identifier corresponding to each piece of data may be the primary key field of the corresponding data table, and the primary key field has the following unique attributes: the value of the primary key is unique, and there will not and cannot be duplicate values; non-nullability: the value of the primary key cannot be NULL (null value). It is known to those skilled in the art that any method of using a position encoding function to positionally encode the primary key value of data to obtain a corresponding position encoding value falls within the protection scope of the present invention.
[0064] S500, traverse H g i , for h g ij , if data table D g Corresponding data filtering structure FS g Zhong and h g ij The corresponding position identifier and the corresponding index identifier are the first index value, and h g ij The corresponding index identifier is stored in the corresponding current index identifier record set R g ij In the corresponding data filter structure FS gThe corresponding index identifier is set to the second index value, and S600 is executed; if the data table D g Corresponding data filtering structure FS g Zhong and h g ij The corresponding position identifier and the corresponding index identifier are the second index value, and h g ij The corresponding index identifier is stored in the corresponding current index identifier record set R g ij , execute S600; R g ij The initial value of is the empty set.
[0065] S600, if R g ij All index identifiers in the R are the second index value, indicating that the i-th data may have been backed up. In this case, the i-th data is not used as incremental data. g ij If there is at least one index identifier as the first index value, indicating that the i-th data has not been backed up, the i-th data is used as incremental data and the i-th data is obtained from the current business database; execute S700.
[0066] In the embodiment of the present invention, the incremental data may be the data added in the data table. In addition, the data obtained from the business database may be stored at a specified location.
[0067] It is known to those skilled in the art that in FS g When the state is the initial state, that is, the newly created state, the data table D g The index identifiers corresponding to all the data in may all be the first index values, that is, when the data filtering structure is used for the first time, a full data backup will be performed on the data table.
[0068] S700, set i=i+1, if i≤m, execute S500, otherwise, execute S100.
[0069] The incremental data acquisition method provided by the embodiment of the present invention can obtain the corresponding position coding value based on the data identifier of the data for data without a timestamp, and then use the data filtering structure to determine whether the index identifiers corresponding to the position coding values are all second identifiers. If so, they are not used as incremental data. Otherwise, they are used as incremental data. This can avoid performing a full data backup every time, thereby improving the data backup speed and reducing data backup resources.
[0070] Since the position encoding function may cause duplicate encoded position encoding values, that is, the hash values calculated for multiple elements are the same, then their array indexes obtained after modulo array length are also the same, which may lead to misjudgment and make the acquired incremental data inaccurate.
[0071] In order to make the acquired incremental data as accurate as possible, the method provided by the embodiment of the present invention may further include the following steps:
[0072] S800, judge FS g Whether the running time of the program reaches the preset time, if so, execute S900.
[0073] In the embodiment of the present invention, FS g The running time refers to FS g The time interval from the creation to the current time. The preset time can be set based on actual needs, for example, it can be 30~90 days.
[0074] S900, Get FS g The corresponding fault tolerance number ft g =dv g ×k g ;dv g For FS g The corresponding incremental data volume, k g For FS g The corresponding fault tolerance.
[0075] In the embodiment of the present invention, dv g Based on FS g The total amount of incremental data obtained can be obtained based on SQL statement queries.
[0076] In the embodiment of the present invention, k g With data sheet D g The data level is negatively correlated, that is, if the data table D g The higher the data level, the smaller the corresponding fault tolerance. g The data level of is determined based on the corresponding data volume and is positively correlated with the corresponding data volume. That is, if the corresponding data volume is larger, it means that D g In an exemplary embodiment, if the amount of data in the data table is less than 100,000, that is, in the interval [0, 10 5 ), the corresponding data level can be the first level S1. If the data volume of the data table is 100,000, that is, in the interval [10 5 , 10 6 ), the corresponding data level can be the second level S2. If the data volume of the data table is 100,000, that is, in the interval [10 5 , 106 ), the corresponding data level can be the second level S2. If the data volume of the data table is in the range of [10 6 , 10 7 ), the corresponding data level can be the third level S3. If the data volume of the data table is in the tens of millions, that is, in the interval [10 7 , 10 8 ), the corresponding data level can be the fourth level S4. If the data volume of the data table is in the billions, that is, in the interval [10 9 , +∞), the corresponding data level may be the fifth level S5. Among them, the importance corresponding to S1 to S5 increases in sequence.
[0077] In the embodiment of the present invention, the specific setting of the fault tolerance rate can be set based on actual needs. In an illustrative embodiment, the setting of the fault tolerance rate can be set with reference to the following Table 1:
[0078] Table 1
[0079]
[0080] S1000, if ft g ≥ft0 g , delete FS g , the data table D g All data in is taken as incremental data and extracted from the current business database; ft0 g For data table D g The corresponding fault tolerance threshold.
[0081] In the embodiment of the present invention, ft0 g With data sheet D g The data level is negatively correlated, that is, if the data table D g The higher the data level, the smaller the corresponding fault tolerance threshold. The fault tolerance data threshold can be determined based on actual conditions. In an illustrative embodiment, it can be set in the manner shown in Table 2 below:
[0082] Table 2
[0083]
[0084] In this embodiment, the fault tolerance number of the data filtering structure is calculated according to the preset time, and if the obtained fault tolerance number is greater than the corresponding fault tolerance number threshold, a full data backup is performed, so that the backed-up data can be ensured to be as accurate as possible. In addition, since the full data backup is performed after the preset time, compared with performing the full data backup every time, the resources occupied by the data backup can still be reduced.
[0085] Furthermore, in the embodiment of the present invention, S100 further includes the following steps:
[0086] If TF g Valid fields, based on data table D g The data identifier and timestamp fields corresponding to each piece of data form the data table D g The corresponding current data reference table is based on the data table D g Corresponding to the current data reference table and the previous data reference table, obtain data table D g Incremental data in .
[0087] That is, if data table D g If the timestamp field of is a valid field, the added data can be obtained based on the data identifier and the timestamp field. The specific acquisition method can adopt the existing technology, for example, it can be obtained based on the incremental data extraction solution disclosed in CN 102521225 A.
[0088] Based on the same inventive concept, an embodiment of the present invention provides an incremental data acquisition device, comprising:
[0089] The data acquisition module is used to obtain the timestamp field of any data table from the current business database at the current calculation moment.
[0090] A judgment module is used to judge whether the timestamp field of the acquired data table is an empty set or an invalid field. If so, it is judged whether there is a data filtering structure corresponding to the identifier of the data table in the current incremental data database. If so, the position code of any data in the data table is obtained. If not, a corresponding data filtering structure is created for the data table in the current incremental data database, and the position code of any data in the data table is obtained.
[0091] In an embodiment of the present invention, the data filtering structure may include Q index tuples, wherein the rth index tuple includes (L r , P r ), L r is the position identifier corresponding to the rth index tuple, P r is the index identifier corresponding to the rth index tuple, and the value of r ranges from 1 to Q; the initial values of the index identifiers in the data filtering structure are all the first index values.
[0092] In an embodiment of the present invention, the index tuple may be a binary array, the position identifier may be represented by a number, the index identifier may be represented by 0 or 1, wherein the first index value may be represented by 0 and the second index value may be represented by 1. In an embodiment of the present invention, a specific implementation of the data filtering structure may be a Bloom filter.
[0093] In the embodiment of the present invention, when creating a data filter structure for each data table, the length of the created data filter structure is related to the data volume and average data growth rate of the corresponding data table. In the embodiment of the present invention, the judgment module specifically creates a data filter structure for the data table through the following operations:
[0094] S10, obtain the number of data in the data table q g .
[0095] In the embodiment of the present invention, q can be obtained based on the SQL query statement. g For example, it can be obtained by querying the statement "select count(1) from ${dbName}.${tbName}".
[0096] S20, based on q g , the average data growth rate c corresponding to the data table g And the preset time length T, obtain the length LN of the data filter structure corresponding to the data table g Among them, LN g =Roundup(q g ×c g ×T), Roundup ( ) means rounding up.
[0097] In the embodiment of the present invention, c g It can be obtained based on the historical data of the data table. For example, a historical time period is set, such as the historical data of the data table within half a year or a year, and the average data growth rate within the set time period is obtained as c g , which can be obtained through SQL statements.
[0098] In the embodiment of the present invention, the preset time length T can be set based on actual needs, as long as it can ensure that the space capacity of the data filtering structure is large enough, for example, it can be 365 days.
[0099] In another embodiment of the present invention, if c cannot be obtained g , LN can be set g =q g ×b. b is a preset value. In an exemplary embodiment, b=10.
[0100] Those skilled in the art know that the index identifiers in the created data filtering structure are all first index values.
[0101] A position code acquisition module is used to obtain the position code of any data in any data table.
[0102] In an embodiment of the present invention, the position code acquisition module can use multiple position code functions to obtain the position code value. The position code function can be a random mapping function, specifically an unbiased hash function. In an embodiment of the present invention, the data filtering structure can have two characteristic parameters, one is the number of random mapping functions, and the other is the fault tolerance rate. The fault tolerance rate is used to characterize the error rate that the data filtering structure can allow. In an embodiment of the present invention, the number of position code functions is negatively correlated with the fault tolerance rate of the corresponding data filtering structure, that is, the lower the fault tolerance rate, the more the number of position code functions, and the specific functional relationship between the two can be set based on actual needs. In general, the lower the fault tolerance rate, the larger the length of the Bloom filter (the larger the storage space occupied), and the more hash functions need to be calculated. Assuming 3 Bloom filters, the lower the fault tolerance rate, the more likely it is that 10 will be needed to locate. In the actual production process, it can be adjusted according to the importance of the business data. For example, if the business data is transaction data, it is very important data, and the data accuracy should be guaranteed as much as possible, and the error rate should be extremely low. For example, if the business data is log data, the data volume will be very large, the data value is relatively low, and the error rate can be relatively increased.
[0103] In the embodiment of the present invention, the data identifier corresponding to each piece of data may be the primary key field of the corresponding data table, and the primary key field has the following unique attributes: the value of the primary key is unique, and there will not and cannot be duplicate values; non-nullability: the value of the primary key cannot be NULL (null value). It is known to those skilled in the art that any method of using a position encoding function to positionally encode the primary key value of data to obtain a corresponding position encoding value falls within the protection scope of the present invention.
[0104] A judgment module is used to store the index identifier corresponding to the position code in a corresponding current index identifier record set when it is determined that the index identifier corresponding to the position code value of any data in the data filtering structure corresponding to the data table is a first index value, and set the index identifier corresponding to the position code in the corresponding current data filtering structure to a second index value; and when it is determined that the index identifier corresponding to the position code value of any data in the data filtering structure corresponding to the data table is a second index value, store the index identifier corresponding to the position code in the corresponding current index identifier record set; the initial value of the current index identifier record set is an empty set.
[0105] An incremental data determination module is used to not treat the data as incremental data when it is determined that all index identifiers corresponding to the current index identifier record set corresponding to any data in the data table are second index values, and to treat any data as incremental data when it is determined that there is at least one index identifier as the first index value in the current index identifier record set corresponding to any data in the data table.
[0106] The incremental data acquisition module is used to acquire the incremental data determined by the incremental data determination module from the current business database. In the embodiment of the present invention, the data acquired from the business database can be stored in a specified location.
[0107] Furthermore, the incremental data acquisition device provided by the embodiment of the present invention further includes:
[0108] The full data acquisition module is used to determine whether the fault tolerance number corresponding to the data filtering structure reaches the corresponding preset fault tolerance number threshold. If so, the data filtering structure is deleted and the corresponding data table is acquired as the full data.
[0109] The device can be used to perform Figure 1 Therefore, for the functions that can be realized by each functional module of the device, please refer to Figure 1 The description of the illustrated embodiment is not repeated in detail.
[0110] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.
[0111] The embodiment of the present invention further provides a non-transitory computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.
[0112] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for acquiring incremental data, characterized in that: The method comprises the following steps: S100, at the current calculation time, obtain any data table D from the current business database g The timestamp field TF g , if TF g If the set is empty or the acquired timestamp field is an invalid field, execute S200; the value of g ranges from 1 to J, where J is the number of data tables in the current business database; S200, determine whether there is a corresponding table D in the current incremental database g If the data filtering structure corresponding to the identifier exists, execute S400; if not, execute S300; wherein the data filtering structure includes Q index tuples, and the rth index tuple includes (L r , P r ), L r is the position identifier corresponding to the rth index tuple, P r is the index identifier corresponding to the rth index tuple, and the value of r ranges from 1 to Q; the initial values of the index identifiers in the data filtering structure are all the first index values; S300, in the current incremental database, data table D g Create a corresponding data filtering structure and execute S400; S400, based on data sheet D g The data identifier corresponding to the i-th data in the , obtain the position encoding value set H corresponding to the i-th data g i ={h g i1 ,h g i2 ,……,h g ij ,……,h g in }, execute S500; wherein, h g ij is the position encoding value of the i-th data obtained based on the j-th position encoding function, i ranges from 1 to m, and m is D g The number of data in, j ranges from 1 to n, and n is the number of position encoding functions; S500, traverse H g i , for h g ij , if data table D g Corresponding data filtering structure FS g Zhong and h g ij The corresponding position identifier and the corresponding index identifier are the first index value, and h g ij The corresponding index identifier is stored in the corresponding current index identifier record set R g ij In the corresponding data filter structure FS g The corresponding index identifier is set to the second index value, and S600 is executed; if the data table D g Corresponding current data filtering structure FS g Zhong and h g ij The corresponding position identifier and the corresponding index identifier are the second index value, and h g ij The corresponding index identifier is stored in the corresponding current index identifier record set R g ij , execute S600; R g ij The initial value of is an empty set; S600, if R g ij All index identifiers in are the second index values, and the i-th data is not used as incremental data. If R g ij There is at least one index identifier as the first index value, the i-th data is used as incremental data and the i-th data is obtained from the current business database; execute S700; S700, set i=i+1, if i≤m, execute S400, otherwise, execute S100.
2. The method according to claim 1, characterized in that The method further comprises the steps of: S800, judge FS g Whether the running time of reaches the preset time, if yes, execute S900; S900, Get FS g The corresponding fault tolerance number ft g =dv g ×k g ;dv g For FS g The corresponding incremental data volume, k g For FS g The corresponding fault tolerance rate; S1000, if ft g ≥ft0 g , delete FS g , D g All data in is taken as incremental data, and D is obtained from the current business database g All data in ;ft0 g For data table D g The corresponding fault tolerance threshold.
3. The method according to claim 2, characterized in that The number of the position encoding functions is negatively correlated with the error tolerance of the corresponding data filtering structure.
4. The method according to claim 2, characterized in that: ft0 g With D g The data level is negatively correlated.
5. The method according to claim 1, characterized in that: The S100 also includes: If TF g is a valid field, based on D g The data identification and timestamp fields corresponding to each piece of data form D g The corresponding current data reference table is based on D g The corresponding current data reference table and the previous data reference table are used to obtain D g Incremental data in .
6. The method according to claim 1, characterized in that S300 specifically includes: S301, obtain data table D g The number of data in q g ; S302, based on q g , Data Table D g The corresponding average data growth rate c g And preset time T, get data table D g The length of the corresponding data filter structure LN g Among them, LN g =Roundup(q g ×c g ×T), Roundup ( ) means rounding up.
7. An incremental data acquisition device, characterized in that: include: The data acquisition module is used to obtain the timestamp field of any data table from the current business database at the current calculation time; A judgment module is used to judge whether the timestamp field of the acquired data table is an empty set or an invalid field. If so, it is judged whether there is a data filtering structure corresponding to the identifier of the data table in the current incremental database. If so, it is used to obtain the position code of any data in the data table. If not, it is used to create a corresponding data filtering structure for the data table in the current incremental database. A position code acquisition module is used to acquire the position code of any data in the data table; a judgment module, for, when determining that the index identifier corresponding to the position identifier corresponding to the position code value of any data in the data filtering structure corresponding to the data table is the first index value, storing the index identifier corresponding to the position code in the corresponding current index identifier record set, and setting the index identifier corresponding to the position code in the corresponding current data filtering structure to the second index value; And when it is determined that the index identifier corresponding to the position identifier corresponding to the position code value of any data in the data filtering structure corresponding to the data table is the second index value, the index identifier corresponding to the position code is stored in the corresponding current index identifier record set; the initial value of the current index identifier record set is an empty set; An incremental data determination module, configured to, when it is determined that all index identifiers corresponding to a current index identifier record set corresponding to any data corresponding to the data table are the second index value, not use the data as incremental data, and when it is determined that at least one index identifier in a current index identifier record set corresponding to any data corresponding to the data table is the first index value, use the any data as incremental data; The incremental data acquisition module is used to acquire the incremental data determined by the incremental data determination module from the current business database.
8. The device according to claim 7, characterized in that Also includes: The full data acquisition module is used to determine whether the fault tolerance number corresponding to the data filtering structure reaches the corresponding preset fault tolerance number threshold. If so, the data filtering structure is deleted and the corresponding data table is acquired as the full data.
9. An electronic device, characterized in that: including a processor and a memory; The processor is used to execute the steps of the method according to any one of claims 1 to 6 by calling the program or instruction stored in the memory.
10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the method according to any one of claims 1 to 6.
