A real-time and non-real-time data migration method and device for aerospace discrete manufacturing

Through real-time or non-real-time data migration methods for discrete manufacturing of aerospace, the latest key attributes and incremental data of the data source are determined, the incremental data is judged using the Bloom filter, and the migration is carried out through a unified data synchronization framework, which solves the complexity of data interoperability The problem of efficient connection and unified migration between data sources is achieved, and data migration efficiency is improved.

CN118331945BActive Publication Date: 2024-11-15BEIJING XINGHANG MECHANICAL ELECTRICAL EQUIP CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410502718.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-25
Publication Date
2024-11-15
Estimated Expiration
2044-04-25

AI Technical Summary

Technical Problem

In the prior art, data interoperability and system integration are complex, resulting in data barriers, limiting business innovation and process optimization. How to improve the liquidity between data has become an important topic.

Method used

Provide a real-time or non-real-time data migration method for discrete manufacturing of aerospace. By determining the latest key attributes of the data source every preset period, determining incremental data, and using the Bloom filter to determine whether the data object is incremental data. Migrate incremental data to the target address through a unified data synchronization framework.

Benefits of technology

It effectively improves data migration efficiency, reduces the range of incremental data search, greatly improves data migration efficiency, and realizes efficient connections and unified data migration between data sources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118331945B_ABST
    Figure CN118331945B_ABST
Patent Text Reader

Abstract

The present application discloses a real-time or non-real-time data migration method for aerospace discrete manufacturing, the method comprising: for any data source, determining the latest key attributes of the data table of the data source at every preset period; wherein the latest key attributes are used to characterize the data changes of the data source within the most recent preset period; determining the incremental data corresponding to the latest key attributes; and migrating the incremental data from the data source to the target address.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a real-time and non-real-time data migration method and device for aerospace discrete manufacturing. Background Art

[0002] In existing technologies, the diversity of technical architectures has complicated data interoperability and system integration, and the existence of data barriers has further restricted business innovation and process optimization.

[0003] In view of this, how to improve the fluidity of data has become an important issue. Summary of the invention

[0004] The present application provides a real-time or non-real-time data migration method and device for aerospace discrete manufacturing, which is used to solve the technical problem of how to more efficiently circulate data for different data sources.

[0005] In order to solve the above technical problems, this application provides the following technical solutions:

[0006] The present application provides a real-time or non-real-time data migration method for aerospace discrete manufacturing, the method comprising:

[0007] For any data source, determine the latest key attribute of the data table of the data source at every preset period; wherein the latest key attribute is used to characterize the data change of the data source in the most recent preset period;

[0008] Determining incremental data corresponding to the latest key attribute;

[0009] The incremental data is migrated from the data source to the target address.

[0010] Optionally, the latest key attributes of the data table of the data source are determined to include:

[0011] Determine a first temporary table and a second temporary table of the data source; wherein the first temporary table and the second temporary table are copies of a data table of the data source at a first moment and a second moment, respectively, and the first moment and the second moment are respectively a start moment and an end moment of a most recent preset period;

[0012] Perform a circular comparison operation; wherein the first circular comparison operation includes: comparing the first temporary table with the second temporary table, and recording the difference between the first temporary table and the second temporary table;

[0013] The non-first cycle comparison operation includes: removing a corresponding non-primary attribute from the first temporary table and the second temporary table compared in the last cycle comparison operation, forming a new first temporary table and a new temporary table, comparing the newly formed first temporary table and the second temporary table, and recording the difference between the newly formed first temporary table and the second temporary table;

[0014] When the new first temporary table and the second temporary table formed by a certain loop comparison operation do not contain non-primary attributes, the next loop comparison operation is not performed;

[0015] According to the differences between the first temporary table and the second temporary table recorded in each cyclic comparison operation, the latest key attribute of the data table of the data source is determined.

[0016] Optionally, determining the latest key attribute of the data table of the data source according to the difference between the first temporary table and the second temporary table recorded in each cyclic comparison operation includes:

[0017] If the difference between the newly formed first temporary table and the second temporary table meets the preset condition after removing a corresponding non-primary attribute, then the non-primary attribute belongs to the latest key attribute of the data table of the data source.

[0018] Optionally, determining the incremental data corresponding to the latest key attribute includes:

[0019] For any data object corresponding to any latest key attribute, determine whether the data object already exists at the target address;

[0020] If not, the data object is determined as incremental data corresponding to the latest key attribute.

[0021] Optionally, the method further includes:

[0022] Determine the number of Bloom filters and the bit vector size of each Bloom filter and the hash function corresponding to each Bloom filter;

[0023] For any data object, if the data object needs to be migrated to a target address, each hash function is used to calculate the target position of the data object in the corresponding Bloom filter, and the target position is set to a preset value.

[0024] Optionally, the incremental data includes data objects;

[0025] Determining the incremental data corresponding to the latest key attribute includes:

[0026] For any data object corresponding to any latest key attribute, use each hash function to calculate the expected position of the data object in the corresponding Bloom filter;

[0027] According to the value of the expected position, it is determined whether the data object is incremental data.

[0028] Optionally, judging whether the data object is incremental data according to the value of the expected position includes:

[0029] If the values ​​of the expected positions are all preset values, then the data object is not incremental data.

[0030] Optionally, the method further includes:

[0031] Connect various data sources through a unified data synchronization framework to migrate incremental data from each data source to the target address.

[0032] Optionally, the data synchronization framework is constructed using a Framework+plugin architecture;

[0033] Migrating the incremental data from the data source to the target address includes:

[0034] Collect incremental data through the data collection module and send the collected incremental data to the Framework;

[0035] The data writing module requests data from the Framework and writes the requested data to the target address.

[0036] The present application also provides a real-time or non-real-time data migration device for aerospace discrete manufacturing, the device comprising:

[0037] The key attribute module is used to determine the latest key attribute of the data table of any data source at every preset period; wherein the latest key attribute is used to characterize the data change of the data source in the latest preset period;

[0038] An incremental data module, used to determine the incremental data corresponding to the latest key attribute;

[0039] The data synchronization module is used to migrate the incremental data from the data source to the target address.

[0040] At least one of the above technical solutions adopted in this application can achieve the following beneficial effects:

[0041] By connecting various data sources, uniformly determining incremental data and performing data migration, data migration efficiency can be effectively improved. By determining key attributes, determining incremental data corresponding to key data, and performing data migration, the incremental data search range can be effectively reduced, greatly improving data migration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in this application or the prior art, the drawings required for use in this application or the prior art description are briefly described below. Obviously, the following only describes the drawings required for use in some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0043] Figure 1 It is a flowchart of the real-time or non-real-time data migration method for aerospace discrete manufacturing in the first embodiment of this specification.

[0044] Figure 2 It is a schematic diagram of the key attribute determination process in the first embodiment of this specification.

[0045] Figure 3 This is a schematic diagram of Hash algorithm space consumption.

[0046] Figure 4 This is a schematic diagram of the space consumption of the Bloom Filter algorithm.

[0047] Figure 5 This is a schematic diagram comparing different data flow frameworks.

[0048] Figure 6 It is a schematic diagram of the data migration process in the first embodiment of this specification.

[0049] Figure 7 It is a timing diagram of the DataE job life cycle.

[0050] Figure 8 It is an application schematic diagram of the first embodiment of this specification.

[0051] Fig. 9 It is a structural diagram of a real-time or non-real-time data migration device for aerospace discrete manufacturing in the second embodiment of this specification. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments involved in the specific implementation methods are only part of the embodiments of this application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the specific implementation methods without making creative work should belong to the scope of protection of this application.

[0053] The first embodiment of this specification (hereinafter referred to as "embodiment one") provides a real-time or non-real-time data migration method for aerospace discrete manufacturing. The execution subject of embodiment one can be a terminal (including but not limited to mobile phones, computers, pads, wearable devices, televisions) or a server or an operating system or an application or a data migration platform or a data migration system, etc., that is, the execution subject can be various and can be set, used or changed as needed. In addition, there can also be a third-party application to assist the execution subject in executing embodiment one. For example, the data migration method in embodiment one can be executed by a server, and an application corresponding to the server can be installed on the terminal (held by the user), and data can be transmitted between the terminal or application and the server, and data can be collected or input or output or processed through the terminal or application, thereby assisting the server in executing the data migration method in embodiment one.

[0054] like Figure 1 As shown, the real-time or non-real-time data migration method for aerospace discrete manufacturing provided in the first embodiment includes:

[0055] S101: For any data source, determine the latest key attribute of the data table of the data source at every preset period; wherein the latest key attribute is used to characterize the data change of the data source in the latest preset period;

[0056] First, in actual situations, there may be various data sources, including but not limited to "Mysql", "Oracle", "DB2", "Sybase", "ShenTong", "Hive", "Hbase", "ElasticSearch", "MongoDB" and other structured, semi-structured and unstructured data sources or databases and corresponding "JDBC Driver", "ODBC Driver", "Phoneix", "DB2 Driver", "Syb Driver" and other driver interfaces and data engines. Various data sources can store and manage various real-time and non-real-time data.

[0057] The execution subject of the first embodiment may adapt or access various data sources to obtain data from the accessed data sources. For example, the various data sources may be adapted or accessed in the following manners:

[0058] Data source connection configuration:

[0059] 1. Determine the data source information

[0060] Clarify the basic information of each data source, including but not limited to database type (such as MySQL, Oracle, SQLServer, etc.), host address, port number, database name, user name or password, etc.

[0061] 2. Choose the right connection technology

[0062] According to the data source type, use the corresponding JDBC registration middleware driver for connection.

[0063] 3. Configure the connection string

[0064] The connection string is the core part of the data source connection configuration, which tells the execution subject of the first embodiment (or the corresponding application installed on the execution subject of the first embodiment) how to find and connect to the data source. The format of the connection string varies depending on the database type, but generally includes information such as the database type, host address, port number, database name, etc. Different connection strings can be automatically matched and generated according to the selected database type.

[0065] 4. Configure the data source in the execution subject of the first embodiment (or the corresponding application installed on the execution subject of the first embodiment)

[0066] The data source may be configured in a suitable manner so that the execution subject of the first embodiment (or the corresponding application installed on the execution subject of the first embodiment) can use the connection string mentioned above to connect to the data source. For example, the data source may be configured by setting relevant parameters in a configuration file (such as application.properties, application.yml, or a configuration file of a database connection pool).

[0067] Test connectivity:

[0068] In the corresponding application, you can write simple test code to try to connect to the data source and perform some basic operations (such as using select 1; for testing). If the operation is successful and a normal query result is returned, it means that the connection configuration is correct; if it cannot be returned normally, an error will be reported, indicating that the connection to the data source is wrong. You can recheck the correctness of the relevant configuration information and driver files.

[0069] Through the above method, connection configuration and connectivity testing of various data sources (including various heterogeneous multi-source databases) can be achieved.

[0070] In the first embodiment, the upstream and downstream relationships of the data tables in each data source and the flow of data can also be traced back, wherein the upstream field refers to the field that affects the value of the current field, and the downstream field is the field affected by the current field.

[0071] In order to trace the upstream and downstream relationships of data tables and the flow of data in the data extraction task, the following methods can be used:

[0072] Query metadata: For different data sources, use the metadata management tools or query statements of the data source to obtain the definition, field information, constraint relationships, etc. of the data table. This information can help you understand the relationship between data tables.

[0073] Analyze the relevant information of the configured data source: Through the relevant information of the configured data source, you can quickly understand the conversion and flow process of data from different data sources from the source system to the target data table.

[0074] In the first embodiment, for any data source, the latest key attribute of the data table of the data source can be determined every preset period; wherein the latest key attribute is used to characterize the data change of the data source in the latest preset period.

[0075] like Figure 2 As shown, for any data source (hereinafter referred to as "data source S"), determining the latest key attributes of the data table of the data source S may include:

[0076] S1011: Determine a first temporary table and a second temporary table of a data source S; wherein the first temporary table and the second temporary table are copies of a data table of the data source S at a first moment and a second moment, respectively, the first moment being the start moment of a most recent preset period, and the second moment being the end moment of the most recent preset period;

[0077] Among them, for any preset cycle, the start time of the preset cycle is taken as the first time, and the end time is taken as the second time. When the next preset cycle comes, the end time of the previous preset cycle becomes the start time of the next preset cycle, that is, the second time of the previous preset cycle becomes the first time of the next preset cycle, and the end time of the next preset cycle is taken as the second time. In other words, after each preset cycle, the start time and end time of the most recent preset cycle are taken as the first time and the second time, respectively.

[0078] Looking only at the most recent preset cycle, the copy of the data table (source temporary table) of data source S at the first moment of the most recent preset cycle can be used as the first temporary table; at the second moment of the most recent preset cycle, the copy of the data table (target temporary table) of data source S can be used as the second temporary table, thereby determining the first temporary table and the second temporary table of data source S.

[0079] Hereinafter, the first temporary table is referred to as table A, and the second temporary table is referred to as table B1.

[0080] S1013: Execute a circular comparison operation; wherein the first circular comparison operation includes: comparing (comparing all tables, the same below) the first temporary table and the second temporary table, and recording the difference between the first temporary table and the second temporary table; the non-first circular comparison operation includes: removing a corresponding non-primary attribute from the first temporary table and the second temporary table compared in the last circular comparison operation, respectively, to form a new first temporary table and a new temporary table, comparing the newly formed first temporary table and the second temporary table, and recording the difference between the newly formed first temporary table and the second temporary table;

[0081] Specifically, the first loop comparison operation is to compare Table A and Table B1, and record the differences between Table A and Table B1. The table differences in Example 1 include but are not limited to row-level differences (which rows exist in Table A but not in Table B1, or vice versa) and / or column-level differences (which attributes have inconsistent values ​​in the two tables) and / or attribute differences. Among them, Table C1 can be generated to record the differences between Table A and Table B1.

[0082] The second loop comparison operation includes: removing a corresponding non-primary attribute from Table A and Table B1, that is, removing a non-primary attribute from Table A and removing the corresponding non-primary attribute from Table B1. After removing the non-primary attribute from Table A, a new first temporary table is formed (that is, the first temporary table is updated by removing the non-primary attribute), which is recorded as Table A1; after removing the non-primary attribute from Table B1, a new second temporary table is formed (that is, the second temporary table is updated by removing the non-primary attribute), which is recorded as Table B11. Compare the newly formed Table A1 and Table B11, and record the differences between the two. Among them, Table C11 can be generated to record the differences between Table A1 and Table B11.

[0083] The third cyclic comparison operation includes: removing a corresponding non-primary attribute from Table A1 and Table B11, that is, removing a non-primary attribute from Table A1 and removing the corresponding non-primary attribute from Table B11. After removing the non-primary attribute from Table A1, a new first temporary table is formed, which is recorded as Table A2; after removing the non-primary attribute from Table B11, a new second temporary table is formed, which is recorded as Table B12. Compare the newly formed Table A2 and Table B12, and record the differences between the two. Among them, Table C12 can be generated to record the differences between Table A2 and Table B12.

[0084] And so on. It can be seen that the non-first loop comparison operation includes: removing a corresponding non-primary attribute from the first temporary table and the second temporary table compared in the last loop comparison operation, forming a new first temporary table and a new temporary table, comparing the newly formed first temporary table and the second temporary table, and recording the difference between the newly formed first temporary table and the second temporary table;

[0085] S1015: When the new first temporary table and the second temporary table formed by a certain circular comparison operation do not contain non-primary attributes, the circular comparison operation is no longer performed;

[0086] As the loop comparison operation is continuously performed, the non-primary attributes in the first temporary table and the second temporary table are continuously removed, until the new first temporary table Ai and the second temporary table B1i formed by a certain (for example, the i+1th) loop comparison operation do not contain non-primary attributes (indicating that all non-primary attributes have been removed once), and the next loop comparison operation is no longer performed. Of course, the differences between the first temporary table Ai and the second temporary table B1i are also compared and recorded, and a table C1i can be formed to record the differences between the first temporary table Ai and the second temporary table B1i.

[0087] In the first embodiment, the latest key attribute of the data table of the data source can be determined according to the difference between the first temporary table and the second temporary table recorded in each cyclic comparison operation. The latest key attribute is the key attribute formed in the latest preset period or the key attribute corresponding to the latest preset period.

[0088] Among them, determining the latest key attribute of the data table of the data source according to the difference between the first temporary table and the second temporary table recorded in each cyclic comparison operation may include: if the difference between the newly formed first temporary table and the second temporary table meets the preset condition after removing a corresponding non-primary attribute, then the non-primary attribute belongs to the latest key attribute of the data table of the data source. Specifically, starting from the second cyclic comparison operation, if the difference between the newly formed first temporary table and the second temporary table meets the preset condition after removing a corresponding non-primary attribute in a certain cyclic comparison operation (for example, the difference between the newly formed first temporary table and the second temporary table reaches the preset condition), then the non-primary attribute belongs to the latest key attribute of the data table of the data source, that is, the key attribute formed in the most recent preset period or the key attribute corresponding to the most recent preset period. Similarly, if the difference between the newly formed first temporary table and the second temporary table meets the preset condition after removing a corresponding non-primary attribute in another cyclic comparison operation, then the non-primary attribute belongs to the latest key attribute of the data table of the data source, that is, the key attribute formed in the most recent preset period or the key attribute corresponding to the most recent preset period.

[0089] It can be seen that the non-primary attributes removed in each cyclic comparison operation may be determined as the latest key attributes. In addition, the key attributes formed in the latest preset period or the key attributes corresponding to the latest preset period can be formed into a set D.

[0090] It can be seen from the above that, through each cyclic comparison operation, a first temporary table sequence A to Ai, a second temporary table sequence B1 to B1i, and a table sequence C1 to C1i for recording differences are formed.

[0091] The specific implementation of the above process can be as follows:

[0092] Input: source temporary table A, target temporary table B;

[0093] Output: set D of key attributes;

[0094] method:

[0095] begin

[0096]

[0097] A and B are compared in their entirety to obtain the change table C;

[0098] do{remove a non-primary attribute X from A that has not been removed i Get an A i ;

[0099] Remove a non-primary attribute X from B that has not been removed before i Get B i ;

[0100] A i With B i Compare the entire table to obtain the change table C i ;

[0101] f(C i ! =C)

[0102] D←D∪X i ;}

[0103] until (except the main attribute, all attributes of A and B have been removed once);

[0104] D←D∪{all main attributes of A and B};

[0105] return D;

[0106] end

[0107] In the first embodiment, the key attribute can be defined as a field that changes within a preset period. As can be seen from the above, the latest key attribute will cause a difference between the first temporary table and the second temporary table that meets the preset conditions, so that the latest key attribute can characterize the data change of the data source S within the most recent preset period.

[0108] When the next preset period comes, the first temporary table and the second temporary table can be determined again as above, and the first temporary table of the next preset period is the second temporary table of the previous preset period, and the above-mentioned cyclic comparison operation is continued to determine the latest key attributes of the data table of the data source, that is, the key attributes formed in the next preset period or the key attributes corresponding to the next preset period.

[0109] Of course, the key attributes formed in different preset periods or the key attributes corresponding to different preset periods may be different.

[0110] In addition, in the above content, i can be regarded as an index variable used to indicate the number of loops, or more specifically, it is an identifier used to track or distinguish the copy of the temporary table currently being processed. Moreover, i is not fixed. In the actual algorithm implementation, i may be an integer variable used to increment in the loop and build the name or reference of the temporary table when needed. In pseudocode or high-level algorithm descriptions, representations such as Table Ai or Table B1i can be used to simplify the description, while in the actual programming language implementation, arrays, lists or other data structures may be used to store and manage references to these temporary tables.

[0111] S103: Determine the incremental data corresponding to the latest key attribute;

[0112] As described above, the latest key attribute can characterize the data changes of the data source S within the most recent preset period, so the data corresponding to the latest key attribute can be determined, and the incremental data of the data source S within the most recent preset period, that is, the incremental data corresponding to the latest key attribute, can be determined therefrom.

[0113] In the first embodiment, data may be referred to as a data object, and a data object may be a piece of data, a group of data, or a unit of data, which is not specifically limited in the first embodiment. Determining the incremental data corresponding to the latest key attribute may include: for any data object corresponding to any latest key attribute, determining whether the data object already exists at the target address; if not, determining the data object as the incremental data corresponding to the latest key attribute.

[0114] The following describes how to determine whether the data object already exists at the target address:

[0115] Reference Figure 3 and Figure 4 Generally speaking, a Bloom Filter consists of k independent hash functions h1,h2,...h k (value range is [0,1,...m]∈N) and a bit vector. When initialized, all elements of the bit vector are 0. When a new data needs to be inserted, k hash functions are used to hash the set S of n data objects = {s1,s2,...sn}(m>n) a data object calculates an address sequence [h1(s1),h2(s2),...h k (s1)], and then set the position value of the address sequence corresponding to the bit vector to 1, then the bit vector is said to be loaded into s1.

[0116] When querying whether a data object exists in a set, the bits of the bit vector corresponding to the address sequence of the data object are checked first. If they are all 1, the data object belongs to the set, otherwise it does not belong to the set. However, even if a uniform hash function is used and multiple different hash functions are used, address conflicts are inevitable. Therefore, the Bloom Filter algorithm may set the same bit in the bit vector to 1 multiple times, so misjudgment may occur when performing data queries.

[0117] From the above analysis, we can see that when the Bloom Filter algorithm is used to deduplicate database table records, it is not necessary to store the actual information of each record in the database table, only 0 or 1 needs to be stored. Therefore, each record only needs one bit of space. Each time data is extracted, it will first determine whether the record belongs to the set. At this time, the record will be re-hashed multiple times. When the corresponding positions obtained after the hash operation are all 1, it is determined that the record already exists in the set, and the data record will no longer be extracted. Otherwise, the record will be extracted as new data into the target database table, and the set information will be updated.

[0118] Compared with traditional hash storage methods, the biggest advantage of the Bloom Filter algorithm is naturally its space efficiency. In addition, since the Bloom Filter algorithm does not need to deal with collisions, the time to insert or search for an element in a collection is relatively constant, which is equal to the time required to calculate the hash function. In addition to considering time consumption and space consumption, other issues related to the algorithm application environment must also be considered, such as robustness and search accuracy. For the Bloom Filter algorithm, the error rate of the algorithm and the size of the bit array must also be considered.

[0119] 1) Error rate estimation

[0120] Suppose the bit array has m bits in total, the number of elements in the set S is n, and k hash functions are used. Then, after mapping all elements in the set to the bit array, the probability that a bit in the array is still 0 is:

[0121]

[0122] Now let's check whether an element is in the set. The k positions required to indicate whether an element is in the set are set to "1" according to the above method, but this method may cause the algorithm to mistakenly believe that an element that is not in the set is detected as being in the set. The error rate P is determined by the following formula:

[0123]

[0124] Find the limit of equation (2), when When , the error rate P in the above formula reaches the minimum value:

[0125] P min =2 -k ≈0.6185 m / n (3);

[0126] From the above analysis, it can be seen that the error probability P is related to three factors: the number of elements in the set n, the length of the bit array m, and the number of hash functions k.

[0127] 2) Bit array length design

[0128] For a given error rate P, the bit array length m is:

[0129]

[0130] Then, the number of hash functions k is:

[0131]

[0132] When the error rate P reaches the minimum value,

[0133]

[0134] From this we can deduce:

[0135]

[0136] From formula (6) and formula (7), we can get:

[0137]

[0138] The above formula is exactly the probability that a bit is not set to 1 when using k hash functions to insert n elements. Therefore, in practical applications, half of the bit array is kept empty to ensure the minimum error rate.

[0139] According to practical experience, when the number of hash functions is optimal, if the error rate is required not to exceed ε, m needs to be at least 1.44 times the minimum value.

[0140] When the error rate P does not meet the set indicator, it is necessary to recalculate and update the bit array again.

[0141] In view of the above analysis, Example 1 specifically proposes a K-group combination Bloom Filter. The basic idea of ​​the K-group combination Bloom Filter is as follows:

[0142] Two bit vectors are used to represent the Hash set, one of which is N bits long V1, and the other is N / k (k is the number of groups, i.e. the number of Bloom filters) long V2. The d Hash functions are divided into two parts d1 and d2, d1 is used for V1 mapping, and d2 is used for V2 mapping. When inserting an element, d1 maps the element to V1 and sets 1 at the corresponding position of V1, and d2 maps the element to V2 and sets 1 at the corresponding position of V2. When determining an element, check whether the relevant positions of V1 and V2 are 1 after being mapped by d1 and d2 respectively. If all are 1, the element is considered to belong to the set.

[0143] The following is a formal description of the K-group combined Bloom Filter algorithm:

[0144]

[0145] Among them, V(i,j) represents the jth bit in the i-th bit vector. The K-group combination Bloom filter is actually composed of multiple basic BloomFilters, and each BloomFilter participates in the overall operation as part of the K-group combination. The error discrimination rate of the entire K-group combination Bloom Filter can be obtained:

[0146]

[0147] It can be concluded that the error discrimination rate of the K-point combined Bloom Filter is lower than the error discrimination rate of the BloomFilter analyzed above.

[0148] Specifically, the K-combination Bloom filter is composed of two or more basic Bloom filters, each of which works independently but together forms a whole. When each element (including data objects, the same below) is inserted, it will be assigned to different Bloom filters according to certain rules (such as hash functions). When querying, the corresponding Bloom filter will also be queried according to the same rules.

[0149] The actual application process of K-type combined Bloom filter can be described as follows:

[0150] In the first embodiment, the number of Bloom filters can be determined and multiple Bloom filters can be set, and the bit vector size of each Bloom filter and the hash function corresponding to each Bloom filter can be determined. The number of Bloom filters can be determined according to the data scale and error discrimination rate requirements.

[0151] For any element, a hash function is used to calculate its position in each Bloom filter (ie, the position in each Bloom filter is calculated by the hash function corresponding to the Bloom filter), and the bit at the corresponding position is set to 1.

[0152] For the element to be queried, the hash function is also used to calculate its position in each Bloom filter, and check whether the bits at these positions are all 1. If the corresponding bits in all Bloom filters are 1, it is considered that the element may exist (there is still a possibility of false positive); otherwise, it is considered that the element must not exist.

[0153] In the first embodiment, when the data source S is initially accessed, all or part of the data of the data source S may be migrated or synchronized to the target address. Furthermore, for any data object, if the data object needs to be migrated to the target address, as described above, each hash function is used to calculate the target position of the data object in the corresponding Bloom filter, and the target position is set to a preset value (the first value may be 1).

[0154] In this way, each time the latest key attribute is determined, determining the incremental data corresponding to the latest key attribute may include: for any data object corresponding to any latest key attribute, using each hash function as described above to respectively calculate the expected position of the data object in the corresponding Bloom filter (the meaning of the expected position is the same as the target position described above); judging whether the data object is incremental data according to the value of the expected position. Specifically, judging whether the data object is incremental data according to the value of the expected position may include: if all expected positions are preset values, it is considered that the data object already exists at the target address, and the data object is not incremental data; otherwise, it is considered that the data object does not exist at the target address, and the data object is determined to be the incremental data (from the data source S) corresponding to or generated by the latest key attribute.

[0155] Through the above content, the extraction of incremental data from the data source S is achieved.

[0156] S105: Migrate the incremental data from the data source to the target address.

[0157] After the incremental data of the data source S is determined, the incremental data can be migrated or synchronized from the data source S to a target address (eg, a target database).

[0158] In the first embodiment, various data sources can be connected through a unified integrated data synchronization framework (referred to as "DataE", which is also equivalent to a storage framework), and complex mesh synchronization links (such as Figure 5 The left half of the diagram shows a star-shaped data link (centered on Data). Figure 5 As shown in the right half, DataE is responsible for connecting various data sources as an intermediate transmission carrier to migrate incremental data from each data source to the target address. When a new data source needs to be connected, it only needs to be connected to DataE, and it can be seamlessly connected with the existing data source.

[0159] like Figure 6 As shown in the figure, DataE, as an offline data synchronization framework, can be built using the Framework+plugin architecture. The data source reading and writing are abstracted into Reader / Writer plug-ins and incorporated into the entire data extraction framework, including:

[0160] As a data acquisition module, Reader is responsible for collecting data from the data source and sending the data to the Framework.

[0161] As a data writing module, Writer is responsible for continuously requesting data from Framework and writing the data to the target address.

[0162] Framework is used to connect Reader and Writer, serving as a data transmission channel between the two, and handles core technical issues such as buffering, flow control, concurrency, and data conversion.

[0163] The establishment process of DataE may include:

[0164] Architecture design: Design the Framework+plugin architecture and abstract the data source reading and writing into a plug-in form for easy expansion and maintenance.

[0165] Development and testing: Implement the interface specifications of the Reader and Writer plug-ins, develop the corresponding data source reading and writing logic, and conduct exhaustive testing to ensure the correctness and stability of the functions.

[0166] Optimization and iteration: Based on actual application scenarios and feedback, the framework is optimized for performance and functional iteration to continuously improve its processing efficiency and stability.

[0167] DataE Architecture:

[0168] Reader plug-in: responsible for reading data from various heterogeneous data sources, converting it into a unified format or structure, and then sending it to the Framework.

[0169] Writer plug-in: Responsible for receiving data transmitted by the Framework and performing necessary data conversion and writing operations according to the requirements of the target address, such as the target database or storage system.

[0170] Framework: As the core part, it connects Reader and Writer plug-ins and handles core technical issues such as data transmission, buffering, flow control, concurrency control, and data conversion. It ensures reliable data transmission and efficient processing.

[0171] Based on the above description, migrating the incremental data from the data source to the target address may include:

[0172] Collect incremental data through the data collection module and send the collected incremental data to the Framework;

[0173] The data writing module requests data from the Framework and writes the requested data to the target address.

[0174] The following is a specific example to illustrate the incremental data writing or synchronization process:

[0175] like Figure 7 As shown in the figure, DataE completes a single data synchronization job, called a Job. After DataE receives a Job, it will start a process to complete the entire job synchronization process. The DataE Job module is the central management node of a single job, and is responsible for data extraction, subtask segmentation (converting a single job calculation into multiple sub-Tasks), TaskGroup management and other functions.

[0176] After DataE Job is started, it will be divided into multiple small Tasks (subtasks) according to different source-side segmentation strategies for concurrent execution. After splitting into multiple Tasks, DataE Job will call the Scheduler module to recombine the split Tasks and assemble them into TaskGroups according to the configured concurrent data volume. Each TaskGroup is responsible for completing all assigned Tasks with a certain concurrency. The default setting for the concurrency of a single task group is 5. Each Task is started by TaskGroup. After the Task is started, the Reader→Channel→Writer thread will be started to complete the task.

[0177] Functional Description of Job Interface and Task Interface

[0178] In data synchronization or processing tasks, Job and Task are two very important interfaces, which represent global jobs and specific task execution respectively. The following will explain the steps in these two interfaces in detail:

[0179] Job Interface

[0180] 1.init: initialization work

[0181] The Job object initializes the work, and the test can get the configuration related to this plugin through super.getPluginJobConf(). The read plugin gets the Reader part of the configuration, and the write plugin gets the Writer part.

[0182] When a new Job object is created, the init method is called. This method is mainly used to perform some initialization operations, such as loading and parsing the configuration related to the current plugin. In the init method, you can get the configuration information related to the current plugin through super.getPluginJobConf(). For a read plugin, it usually gets the configuration of the Reader part; for a write plugin, it gets the configuration of the Writer part.

[0183] 2.prepare: global preparation work, such as odpswriter clearing the target table.

[0184] The prepare method is called before the Job object executes specific tasks to perform some global preparation work. For example, if it is an ODPS (Open Data Processing Service) Writer, it may clear the target table in this method to ensure that the data synchronization is clean.

[0185] 3.split: split task

[0186] Split Task. The parameter adviceNumber is the number of splits recommended by the framework, which is generally the concurrency configured at runtime. The value returned is the configuration list of Task.

[0187] The split method is used to split a job into multiple tasks according to the adviceNumber recommended by the framework. This number of splits is usually the concurrency configured at runtime. Each Task is responsible for processing a portion of the data, thereby speeding up data synchronization or processing in parallel. This method returns a list of Task configurations, each corresponding to a Task.

[0188] 4.post: global post-processing, such as the Rename operation after MysqlWriter synchronizes the shadow table.

[0189] The post method is called after all tasks are executed to perform some global post-processing. For example, in MysqlWriter, it may be used to perform a Rename operation after synchronizing the shadow table to ensure that the final state of the data is correct.

[0190] 5.destroy: destroy the job (destruction of the Job object itself)

[0191] The destroy method is used to release the resources occupied by the Job object and complete the destruction of the Job object.

[0192] For example:

[0193]

[0194]

[0195]

[0196] Task Interface

[0197] 1.init: initialization work

[0198] Initialize the Task object. At this point, you can use super.getPluginJobConf() to get the configuration related to this Task.

[0199] When a new Task object is created, the init method is called. Similar to the init of Job, it is mainly used to load and parse the configuration related to the current Task. The configuration here is one of the configuration lists returned by the split method of Job.

[0200] 2.prepare: local preparation

[0201] The prepare method is called before the Task starts to execute a specific task. It is used to perform some local preparation work, such as establishing a database connection.

[0202] 3.startRead: read data

[0203] The startRead method is used to read data from the data source and write the data to the RecordSender. The RecordSender will write the data to the cache queue connecting the Reader and the Writer, waiting for the subsequent WriterTask to process.

[0204] 4.startWrite: Read data from RecordReceiver and write it to the target data source. The data in RecordReceiver comes from the cache queue between Reader and Writer.

[0205] 5.post: local post-processing

[0206] The post method is called after the Task finishes reading or writing data. It is used to perform some local post-processing work, such as closing the database connection.

[0207] 6.destroy: Destruction work (destruction work of the Task object itself)

[0208] The destroy method is used to release the resources occupied by the Task object and complete the destruction of the Task object.

[0209] Through the collaborative work of the Job and Task interfaces, data synchronization or processing tasks can be completed efficiently. The Job is responsible for global scheduling and splitting, while the Task is responsible for specific reading and writing operations.

[0210] Embodiment 1 can achieve the following beneficial effects:

[0211] In the first embodiment, by connecting various data sources, uniformly determining incremental data and performing data migration, the data migration efficiency can be effectively improved. By determining key attributes, determining incremental data corresponding to key data, and performing data migration, the incremental data search range can be effectively reduced, greatly improving the data migration or synchronization effect and efficiency, and facilitating the subsequent processing efficiency after data migration or synchronization.

[0212] An important issue in incremental data extraction is the degree of dependency of incremental data on attributes. Implementation Example 1 is based on the full table comparison method. Through the iterative method of cyclic comparison, the comparison range is gradually reduced, so as to accurately and quickly determine which non-primary attribute changes are the key factors that cause the difference between the first temporary table and the second temporary table, and accurately determine the key attributes. On the basis of determining the key attributes, it is only necessary to determine the incremental data based on the key attributes, which can improve the efficiency and accuracy of determining the incremental data, significantly reduce the number of fields during incremental extraction, and improve data processing performance.

[0213] In the first embodiment, the incremental data is determined by the K-group combined Bloom Filter algorithm, aiming to reduce the error discrimination rate (i.e., the false positive rate) and optimize the space utilization efficiency, and has the following characteristics:

[0214] By introducing multiple Bloom filters and reasonably distributing elements, even if one Bloom filter incorrectly indicates that an element exists (false positive), other Bloom filters can correctly point out that the element does not exist, thereby correcting the error. This can significantly reduce the false positive rate, that is, reduce the number of cases where non-existent elements are misjudged as existing, and reduce the false positive rate.

[0215] By distributing the data into multiple smaller Bloom filters, we can reduce space usage and optimize space efficiency while maintaining a low error rate.

[0216] Since the size of each Bloom filter is relatively small, the amount of computation required for querying is also reduced accordingly, thereby improving query efficiency.

[0217] The structure of the group-based Bloom filter allows the number of groups and the configuration of the Bloom filter to be adjusted according to actual conditions to adapt to data processing scenarios of different scales and requirements, and has high flexibility and scalability.

[0218] It can be seen that the K-group combined Bloom Filter algorithm provided in Example 1 can efficiently and quickly determine whether a data object exists at a target address by increasing redundant information while maintaining space efficiency, effectively reducing the error discrimination rate, thereby improving the accuracy and efficiency of incremental data determination. Through the structure and working principle of the algorithm, it performs well in reducing the error discrimination rate, optimizing space efficiency, and improving query efficiency, providing an efficient and reliable solution for related data processing fields.

[0219] The K-point combination Bloom filter provided in Example 1 can be applied to scenarios where it is necessary to efficiently and quickly determine whether an element exists in a set, but at the same time it is hoped to minimize misjudgments, such as cache penetration protection, big data deduplication, URL deduplication in web crawlers, etc., and has universal applicability.

[0220] In the first embodiment, the data synchronization framework can realize the integrated real-time storage of heterogeneous multi-source structured, semi-structured and unstructured discrete information or discrete data feature incremental data, effectively improve the efficiency of data migration or synchronization, and has the following characteristics:

[0221] Flexibility: Through plug-in design, the DataE framework can easily support a variety of heterogeneous data sources and can easily expand support for new data sources.

[0222] Efficiency: The Framework handles core technical issues such as buffering and flow control, ensuring efficient data transmission and processing and reducing resource waste.

[0223] Seamless connection: By abstracting data source reading and writing into plug-ins, DataE can achieve seamless connection between new data sources and existing data sources, simplifying the complexity of data synchronization.

[0224] Scalability: The modular design of the framework allows its functions to be expanded and optimized according to needs to adapt to changing business scenarios.

[0225] In the data extraction task in Example 1, by analyzing the upstream and downstream relationships and data flow of the data tables, it is possible to determine which data tables are upstream tables that provide original data or intermediate processing results; and which data tables are downstream tables that receive and use the data from the upstream tables, which is conducive to accurately and comprehensively determining the data corresponding to or generated by key attributes.

[0226] The application scenarios of the first embodiment are wide-ranging. For example, in the scenario of aerospace discrete manufacturing, the data generated by the data source may be, for example, real-time and / or non-real-time data generated by aerospace discrete manufacturing, and the incremental data may be, for example, discrete data feature incremental data, thereby being able to efficiently process discrete data in heterogeneous multi-source databases, realize real-time migration and incremental extraction of data, provide powerful data support for aerospace discrete manufacturing, and promote continuous innovation and improvement of business, such as Figure 8 As shown, Figure 8 The unit divisions and names in the diagram are examples only.

[0227] Embodiment 1 can support the development of various business applications, and open up the connection between multi-source heterogeneous data to achieve unified management and efficient use of data.

[0228] like Fig. 9 As shown, the second embodiment of this specification provides a real-time or non-real-time data migration device for aerospace discrete manufacturing corresponding to the method described in the first embodiment, and the device includes:

[0229] The key attribute module 202 is used to determine the latest key attribute of the data table of any data source at every preset period; wherein the latest key attribute is used to characterize the data change of the data source in the latest preset period;

[0230] An incremental data module 204 is used to determine the incremental data corresponding to the latest key attribute;

[0231] The data synchronization module 206 is used to migrate the incremental data from the data source to the target address.

[0232] Optionally, the latest key attributes of the data table of the data source are determined to include:

[0233] Determine a first temporary table and a second temporary table of the data source; wherein the first temporary table and the second temporary table are copies of a data table of the data source at a first moment and a second moment, respectively, and the first moment and the second moment are respectively a start moment and an end moment of a most recent preset period;

[0234] Perform a circular comparison operation; wherein the first circular comparison operation includes: comparing the first temporary table with the second temporary table, and recording the difference between the first temporary table and the second temporary table;

[0235] The non-first cycle comparison operation includes: removing a corresponding non-primary attribute from the first temporary table and the second temporary table compared in the last cycle comparison operation, forming a new first temporary table and a new temporary table, comparing the newly formed first temporary table and the second temporary table, and recording the difference between the newly formed first temporary table and the second temporary table;

[0236] When the new first temporary table and the second temporary table formed by a certain loop comparison operation do not contain non-primary attributes, the next loop comparison operation is not performed;

[0237] According to the differences between the first temporary table and the second temporary table recorded in each cyclic comparison operation, the latest key attribute of the data table of the data source is determined.

[0238] Optionally, determining the latest key attribute of the data table of the data source according to the difference between the first temporary table and the second temporary table recorded in each cyclic comparison operation includes:

[0239] If the difference between the newly formed first temporary table and the second temporary table meets the preset condition after removing a corresponding non-primary attribute, then the non-primary attribute belongs to the latest key attribute of the data table of the data source.

[0240] Optionally, determining the incremental data corresponding to the latest key attribute includes:

[0241] For any data object corresponding to any latest key attribute, determine whether the data object already exists at the target address;

[0242] If not, the data object is determined as incremental data corresponding to the latest key attribute.

[0243] Optionally, the incremental data module 204 is further used to determine the number of Bloom filters and the bit vector size of each Bloom filter and the hash function corresponding to each Bloom filter;

[0244] For any data object, if the data object needs to be migrated to a target address, each hash function is used to calculate the target position of the data object in the corresponding Bloom filter, and the target position is set to a preset value.

[0245] Optionally, the incremental data includes data objects;

[0246] Determining the incremental data corresponding to the latest key attribute includes:

[0247] For any data object corresponding to any latest key attribute, use each hash function to calculate the expected position of the data object in the corresponding Bloom filter;

[0248] According to the value of the expected position, it is determined whether the data object is incremental data.

[0249] Optionally, judging whether the data object is incremental data according to the value of the expected position includes:

[0250] If the values ​​of the expected positions are all preset values, then the data object is not incremental data.

[0251] Optionally, the data synchronization module 206 is further configured to connect various data sources through a unified data synchronization framework so as to migrate incremental data from various data sources to a target address.

[0252] Optionally, the data synchronization framework is constructed using a Framework+plugin architecture;

[0253] Migrating the incremental data from the data source to the target address includes:

[0254] Collect incremental data through the data collection module and send the collected incremental data to the Framework;

[0255] The data writing module requests data from the Framework and writes the requested data to the target address.

[0256] The above embodiments can be used in combination. The above description is only for this application and is not intended to limit this application. For those skilled in the art, this application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application should be included in the scope of the claims of this application.

Claims

1. A real-time and non-real-time data migration method for aerospace discrete manufacturing, characterized in that: The method comprises: for any data source, determining the latest key attribute of the data table of the data source at every preset period; wherein the latest key attribute is used to characterize the data change of the data source in the latest preset period; determining the incremental data corresponding to the latest key attribute; and migrating the incremental data from the data source to the target address; Determining the latest key attribute of the data table of the data source includes: determining the first temporary table and the second temporary table of the data source; wherein the first temporary table and the second temporary table are copies of the data table of the data source at the first moment and the second moment, respectively, and the first moment and the second moment are the start moment and the end moment of the most recent preset period, respectively; performing a cyclic comparison operation; wherein the first cyclic comparison operation includes: comparing the first temporary table and the second temporary table, and recording the difference between the first temporary table and the second temporary table; the non-first cyclic comparison operation includes: removing a corresponding non-primary attribute from the first temporary table and the second temporary table compared in the last cyclic comparison operation, respectively, to form a new first temporary table and a second temporary table, comparing the newly formed first temporary table and the second temporary table, and recording the difference between the newly formed first temporary table and the second temporary table; when the new first temporary table and the second temporary table formed by a certain cyclic comparison operation do not contain the non-primary attribute, the next cyclic comparison operation is no longer performed; according to the difference between the first temporary table and the second temporary table recorded in each cyclic comparison operation, determining the latest key attribute of the data table of the data source; According to the difference between the first temporary table and the second temporary table recorded in each cyclic comparison operation, the latest key attribute of the data table of the data source is determined, including: if after removing a corresponding non-primary attribute, the difference between the newly formed first temporary table and the second temporary table meets the preset condition, then the non-primary attribute belongs to the latest key attribute of the data table of the data source.

2. The method according to claim 1, characterized in that Determining the incremental data corresponding to the latest key attribute includes: for any data object corresponding to any latest key attribute, determining whether the data object already exists at the target address; if not, determining the data object as the incremental data corresponding to the latest key attribute.

3. The method according to claim 1, characterized in that The method also includes: determining the number of Bloom filters and the bit vector size of each Bloom filter and the hash function corresponding to each Bloom filter; for any data object, if the data object needs to be migrated to the target address, using each hash function to calculate the target position of the data object in the corresponding Bloom filter, and setting the target position to a preset value.

4. The method according to claim 3, characterized in that The incremental data includes data objects; determining the incremental data corresponding to the latest key attribute includes: for any data object corresponding to any latest key attribute, using each hash function to calculate the expected position of the data object in the corresponding Bloom filter; and judging whether the data object is incremental data based on the value of the expected position.

5. The method according to claim 4, characterized in that Judging whether the data object is incremental data according to the value of the expected position includes: if the values ​​of the expected positions are all preset values, then the data object is not incremental data.

6. The method according to claim 1, characterized in that The method further includes: connecting each data source through a unified data synchronization framework so as to migrate incremental data from each data source to a target address.

7. The method according to claim 6, characterized in that The data synchronization framework is constructed using the Framework+plugin architecture; migrating the incremental data from the data source to the target address includes: collecting incremental data through a data collection module and sending the collected incremental data to the Framework; requesting data from the Framework through a data writing module and writing the requested data to the target address.

8. A real-time or non-real-time data migration device for aerospace discrete manufacturing, characterized in that: The device comprises: a key attribute module, used for determining the latest key attribute of a data table of any data source at every preset period; wherein the latest key attribute is used to characterize the data change of the data source in the latest preset period; an incremental data module, used for determining the incremental data corresponding to the latest key attribute; and a data synchronization module, used for migrating the incremental data from the data source to the target address; Determining the latest key attribute of the data table of the data source includes: determining the first temporary table and the second temporary table of the data source; wherein the first temporary table and the second temporary table are copies of the data table of the data source at the first moment and the second moment, respectively, and the first moment and the second moment are the start moment and the end moment of the most recent preset period, respectively; performing a cyclic comparison operation; wherein the first cyclic comparison operation includes: comparing the first temporary table and the second temporary table, and recording the difference between the first temporary table and the second temporary table; the non-first cyclic comparison operation includes: removing a corresponding non-primary attribute from the first temporary table and the second temporary table compared in the last cyclic comparison operation, respectively, to form a new first temporary table and a second temporary table, comparing the newly formed first temporary table and the second temporary table, and recording the difference between the newly formed first temporary table and the second temporary table; when the new first temporary table and the second temporary table formed by a certain cyclic comparison operation do not contain the non-primary attribute, the next cyclic comparison operation is no longer performed; according to the difference between the first temporary table and the second temporary table recorded in each cyclic comparison operation, determining the latest key attribute of the data table of the data source; According to the difference between the first temporary table and the second temporary table recorded in each cyclic comparison operation, the latest key attribute of the data table of the data source is determined, including: if after removing a corresponding non-primary attribute, the difference between the newly formed first temporary table and the second temporary table meets the preset condition, then the non-primary attribute belongs to the latest key attribute of the data table of the data source.