Data fusion processing method, device, equipment and storage medium

Abstract

The embodiment of the present invention discloses a data fusion processing method, device, equipment and storage medium, the method comprising: partitioning the newly added data according to a preset partitioning rule to obtain at least one target partition, wherein the number of target partitions is the same as the number of historical partitions; obtaining a set data output form, wherein the data output form includes an incremental data output form and a full data output form; according to the data output form, performing fusion processing on the data in each target partition to obtain and output the processed result data. Compared to the prior art in which the data is output in full each time, the technical solution provided by the embodiment of the present invention divides the data output into full data output and multiple incremental data outputs, and utilizes different data fusion processing methods for different data output forms to obtain and output the processed result data, thereby improving the data fusion processing efficiency and reducing resource consumption.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a data fusion processing method, apparatus, device, and storage medium. Background Art

[0002] With the development of emerging technologies such as the Internet, the Internet of Things, and social networks, a large amount of access data will be generated. In order to more intuitively describe and analyze the objects to which the data belongs, it is necessary to fuse the data of the same object and then perform statistical analysis. For example, as the system's online operation time extends, the amount of collected data continues to accumulate, resulting in a continuous increase in the amount of data processed and stored. As the amount of data grows, the processing time of the original scheduled collision fusion function module to load and group and merge the full amount of historical data and newly collected and extracted data becomes longer and longer, and the resource consumption becomes more and more and the stability becomes worse and worse. In addition, the increase in the amount of stored data also leads to a decrease in the storage performance of the newly added data. Therefore, a data collision fusion method is needed to solve the problems of low stability, high resource consumption, and slow speed when merging large amounts of data. At the same time, the fusion results are refined and classified to reduce the amount of data in some storage media, such as index storage media. Summary of the Invention

[0003] The embodiments of the present invention provide a data fusion processing method, apparatus, device and storage medium. By dividing data output into full data output and multiple incremental data outputs, different data fusion processing methods are used for different data output forms to obtain and output processed result data, thereby improving data processing efficiency and reducing resource consumption.

[0004] In a first aspect, an embodiment of the present invention provides a data fusion processing method, including:

[0005] Partition the newly added data according to the preset partitioning rules to obtain at least one target partition, where the number of target partitions is the same as the number of historical partitions;

[0006] Obtaining a set data output format, wherein the data output format includes an incremental data output format and a full data output format;

[0007] According to the data output form, the data in each target partition is fused and processed to obtain the processed result data and output it.

[0008] In a second aspect, an embodiment of the present invention further provides a data fusion processing device, comprising:

[0009] A partition processing module is used to partition the newly added data according to a preset partitioning rule to obtain at least one target partition, wherein the number of target partitions is the same as the number of historical partitions;

[0010] An output form acquisition module, used to acquire a set data output form, wherein the data output form includes an incremental data output form and a full data output form;

[0011] The fusion processing module is used to perform fusion processing on the data in each target partition according to the data output form, obtain the processed result data and output it.

[0012] In a third aspect, an embodiment of the present invention further provides a computer device, comprising: a memory and one or more processors;

[0013] The memory is used to store one or more programs;

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the data fusion processing method as described in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform the data fusion processing method as described in the first aspect.

[0016] The embodiment of the present invention discloses a data fusion processing method, device, equipment and storage medium, the method comprising: partitioning the newly added data according to a preset partitioning rule to obtain at least one target partition, wherein the number of target partitions is the same as the number of historical partitions; obtaining a set data output form, wherein the data output form includes an incremental data output form and a full data output form; according to the data output form, performing fusion processing on the data in each target partition to obtain and output the processed result data. Compared to the prior art in which the data is output in full each time, the technical solution provided by the embodiment of the present invention divides the data output into full data output and multiple incremental data outputs, and utilizes different data fusion processing methods for different data output forms to obtain and output the processed result data, thereby improving the data fusion processing efficiency and reducing resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a data fusion processing method provided in the first embodiment of the present invention;

[0018] Figure 2 A schematic diagram of the data fusion processing process provided by an embodiment of the present invention;

[0019] Figure 3 A structural block diagram of a data fusion processing device provided in the second embodiment of the present invention;

[0020] Figure 4This is a structural block diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0022] Example 1

[0023] Figure 1 This is a flow chart of a data fusion processing method provided in the first embodiment of the present invention. This embodiment is applicable to situations where data collision fusion is performed periodically during system operation. The method can be performed by a data fusion processing device, which can be composed of hardware and / or software and can generally be integrated into a computer device. The method specifically includes the following steps:

[0024] S110: Partition the newly added data according to a preset partitioning rule to obtain at least one target partition.

[0025] The number of target partitions is the same as the number of historical partitions. The number of partitions can be planned and set based on the storage size of the historical data and the resource size. When partitioning data, ensure that the data is distributed as evenly as possible across the partitions. In this embodiment, the preset partitioning rule can be a hash partitioning rule, such as a node remainder partitioning rule. If hash partitioning causes severe data skew, consider using range partitioning or custom partitioning methods.

[0026] In this embodiment, the partition label can be expressed as part-i, and each partition can be labeled with part-1, part-2, ..., part-N. For example, the history partition can be expressed as history-full-part-i, and each history incremental partition can be expressed as history-modify1-part-i, history-modify2-part-i, ..., etc.

[0027] Specifically, for the newly added data read, the newly added data is placed into target partitions according to the preset partitioning rules to obtain one or more target partitions.

[0028] S120: Obtain the set data output format.

[0029] The data output formats include incremental data output and full data output. It is clear that incremental data output refers to the output of newly added data, where the newly added data may be an update to historical data or may be unrelated to historical data. Full data output refers to the output of all recorded data.

[0030] In existing technologies, data fusion processing often involves fusing each newly added data with the full data set and then outputting the fused full data set. This results in excessive data storage and prolonged output processing time. This embodiment utilizes a method of outputting the full data set in addition to multiple incremental data outputs for data output.

[0031] For example, the user can set a full data output every ten days, and incremental data output every other day. It is clear that the interval between full data output and incremental data output can be set according to user needs.

[0032] Specifically, the data output format set by the user this time is obtained from the data fusion processing device to be used for subsequently determining the data fusion processing method.

[0033] S130 , performing fusion processing on the data in each target partition according to the data output format, obtaining the processed result data and outputting it.

[0034] Specifically, after obtaining the data output format, select the data fusion processing method corresponding to the data output format. If the data output format is incremental, select the data fusion processing method corresponding to the incremental data output format to fuse the data in each target partition; if the data output format is full, select the data fusion processing method corresponding to the full data output format to fuse the data in each target partition. The processed result data is then obtained and output to the user.

[0035] The embodiment of the present invention discloses a data fusion processing method, device, equipment and storage medium, the method comprising: partitioning the newly added data according to a preset partitioning rule to obtain at least one target partition, wherein the number of target partitions is the same as the number of historical partitions; obtaining a set data output form, wherein the data output form includes an incremental data output form and a full data output form; according to the data output form, performing fusion processing on the data in each target partition to obtain and output the processed result data. Compared to the prior art in which the data is output in full each time, the technical solution provided by the embodiment of the present invention divides the data output into full data output and multiple incremental data outputs, and utilizes different data fusion processing methods for different data output forms to obtain and output the processed result data, thereby improving the data fusion processing efficiency and reducing resource consumption.

[0036] As an optional embodiment of the embodiment of the present invention, on the basis of the above embodiment, the step of partitioning the newly added data according to the preset partitioning rule to obtain at least one target partition can be specifically described as follows:

[0037] S111. Read the newly added data sequence from the distributed file system.

[0038] Specifically, use the distributed computing framework to read the current newly added data sequence from the distributed file system. For example, each piece of data read can be represented as row.

[0039] Exemplarily, this time, the personnel information data is used as an example for illustration. The personnel information data includes the following fields: personnel number, name, mobile phone number, and discovery times. The distributed file system takes the Hadoop Distributed File System (HDFS) as an example. The current newly added data is placed in / data / batch20210908 of HDFS. Assume that the current newly read newly added data includes the following content:

[0040] Person1, null value, 131xxxx1111, 1;

[0041] Person2, null value, null value, 1;

[0042] Person3, Zhang San, 156xxxx6666, 1.

[0043] S112. Process the newly added data sequence according to the preset processing rule to generate a set of key-value pair data, where the key-value pair data includes key-value pair identifiers.

[0044] Specifically, for each piece of newly added data, the Message-Digest Algorithm (MD5) can be calculated according to the data merging dimension field to generate the key-value pair identifier. For example, the key-value pair identifier can be represented as key. Represent key as the key-value pair data identifier and row as the key-value pair data content to generate <key, row> key-value pair data. Processing each piece of newly added data according to the preset processing rule will generate a key-value pair data. Multiple pieces of newly added data in the newly added data sequence can generate multiple key-value pair data, thus forming a set of key-value pair data.

[0045] Exemplarily, continue to describe with the example in the above step S111. Taking the personnel number field in the dataset as the data fusion and merging dimension, the key-value pair data generated by data processing is:

[0046] <md5-Person1, (Person1, null value, 131xxxx 1111, 1)>;

[0047] <md5-Person2,(Person2, null, null, 1)>;

[0048] <md5-Person3,(Person3, Zhang San, 156xxxx 6666, 1)>.

[0049] S113. Partition the key-value pair data according to the hash partitioning rule to obtain at least one initial partition.

[0050] Specifically, performing hash partitioning on the key-value pair data can be partitioning by taking the remainder with the node, and putting the key-value pair data into the initial partitions respectively to obtain one or more initial partitions.

[0051] Exemplarily, for the <key,row> key-value pair data generated in step S112, use hash partitioning to partition the data. The specific partitioning rule can be as follows: partitionId = key.hashCode % N, where partitionId represents the partition identifier to which the key-value pair data corresponding to the key should be assigned, key.hashCode represents the hash value of the key in the data, and N represents the number of partitions. Among them, the number of partitions should be consistent with the historical number of partitions.

[0052] Continuing the example described in step S112 above, perform hash partitioning on the data in step S112. Assume the number of partitions is 2. Table 1 shows the result of partitioning the key-value pair data, as shown in Table 1:

[0053] Table 1

[0054]

[0055] S114. In each initial partition, sort the corresponding key-value pair data according to each key-value pair identifier included, and determine the sorted initial partition as the target partition.

[0056] Specifically, after partitioning, sort the key-value pair data in each partition according to each key-value pair identifier included, ensure that the data in each partition is arranged in ascending order of the key-value pair identifier, and determine the sorted initial partition as the target partition.

[0057] As an optional embodiment of the embodiment of the present invention, based on the above embodiment, according to the data output form, the steps of performing fusion processing on the data in each target partition, obtaining the processed result data and outputting can be specifically described as:

[0058] S131. For each target partition, perform merging processing on the included key-value pair data to obtain at least one new key-value pair data.

[0059] It is clear that for each target partition, the key-value pair data therein may have the same corresponding key-value pair identifiers. In this case, the key-value pair data with the same key-value pair identifiers can be merged and processed. The processing method can be to merge the data business fields, etc., to form a new key-value pair data.

[0060] For example, the new data is an ordered iterator<key,row> , which can be recorded as IteratorNew. Traversing IteratorNew to get the current data is recorded as<key1,rowNew> , traverse IteratorNew to obtain all data with key = key1 and perform data business field fusion as<key1,rowNew> .

[0061] S132. Obtain a historical partition whose partition number is the same as the target partition.

[0062] Specifically, according to the partition tag part-i, read the historical full data partition history-full-part-i and all historical incremental data partitions history-modify1-part-i, history-modify2-part-i, etc., traverse all historical data in reverse order, and for each key group, take the first value of the group as the latest historical data corresponding to the key, and output the data structure Iterator<key,row> .

[0063] For example, the system's full historical data is stored in / data / history / history-full-part-1 and data / history / history-full-part-2 of HDFS. Table 2 shows the partitioning of the full historical data:

[0064] Table 2

[0065] history-full-part-1 data history-full-part-2 data <md5-Person1,(Person1, Wang San, null, 2)> <md5-Person2,(Person2, test, null value, 1)>

[0066] The system's historical new data is stored in / data / history / history-modify1-part-1 and data / history / history-modify1-part-2 of HDFS. Table 3 shows the historical new data partitions:

[0067] Table 3

[0068] history-modify1-part-1 data history-modify1-part-2 data <md5-Person1,(Person1, Wang San, 156xxxx1111, 3)> <md5-Person4,(Person4, Li Si, null, 1)>

[0069] Read the historical data in reverse order, first read history-modify1* and then read history-full* to merge the historical results. Table 4 shows the historical partitions after the historical full data and historical incremental data are merged:

[0070] Table 4

[0071]

[0072] Among them, the business merging rules can be: data with certain values ​​and empty values ​​are merged to output the certain value, the two certain values ​​are merged into a concatenated string separated by ";", and the number of times historical new data and historical full data are discovered are merged into the sum of the two numbers.

[0073] S133. Based on the key-value pair identifiers of each key-value pair data in the historical partition and the target partition, combined with the data output form, data fusion processing is performed on the data in the historical partition and the target partition.

[0074] It is clear that the key-value pair identifiers of each key-value pair data in the historical partition and the key-value pair identifiers of each key-value pair data in the target partition may have a greater than, equal to, or less than relationship.

[0075] Specifically, different data fusion processing methods are used to fuse the data in the historical partition and the target partition, depending on the relationship between the key-value pair identifiers of each key-value pair in the historical partition and the key-value pair identifiers of each key-value pair in the target partition, as well as the output format of the read data, such as incremental data output and full data output. For example, if the data output is incremental, the iterator IteratorNew that traverses the newly added data is the primary iterator, and the fusion is complete after traversing IteratorNew. If the data output is full, both IteratorHis and IteratorNew iterators must be traversed.

[0076] S134: Obtain and output the result data corresponding to each target partition.

[0077] Specifically, the result data corresponding to each target partition is obtained and output to the user.

[0078] In this embodiment, if the output form of the read data is the new data output form, the iterator IteratorNew that traverses the new data is mainly used, and the data fusion processing is completed after the traversal of IteratorNew is completed.

[0079] As an optional embodiment of the present invention, based on the above embodiment, the steps of performing data fusion processing on the data in the historical partition and the target partition based on the key-value pair identifiers of the key-value pairs in the historical partition and the target partition and in combination with the data output format can be expressed as follows:

[0080] S1331. Use the first key-value pair identifier in the historical partition and the target partition as the first key-value pair identifier and the second key-value pair identifier, respectively.

[0081] For example, the first key-value pair identifier of the historical partition is used as the first key-value pair identifier, which is recorded as<key1,rowHis> ; The first key-value pair identifier of the target partition is used as the second key-value pair identifier, recorded as<key2,rowNew> .

[0082] S1332: If the data output format is an incremental data output format, compare the first key-value pair identifier and the second key-value pair identifier.

[0083] Specifically, the first key-value pair identifier and the second key-value pair identifier are represented in the form of strings, and the size of the strings can be compared, that is, the relationship between the first key-value pair identifier and the second key-value pair identifier can be greater than, equal to, or less than. For example, if the first key-value pair identifier is represented as key1 and the second key-value pair identifier is represented as key2, the comparison result can be expressed as: key1>key2, key1=key2, or key1 <key2。

[0084] S1333. If the first key-value pair identifier is greater than the second key-value pair identifier, determine that the key-value pair data corresponding to the second key-value pair identifier is the result data, use the next key-value pair identifier of the second key-value pair identifier as the new second key-value pair identifier, and return to continue the comparison operation of the key-value pair identifiers until the second key-value pair identifier is the last key-value pair identifier.

[0085] In this embodiment, it is clear that the key-value pair data in the historical partition and the target partition are arranged from small to large according to the key-value pair identifier. If the first key-value pair identifier is greater than the second key-value pair identifier, it indicates that the current newly added data has no corresponding historical data, and the newly added data can be obtained as result data, that is, the key-value pair data corresponding to the second key-value pair identifier is the result data.

[0086] The target partition is traversed continuously, and the next key-value pair identifier after the second key-value pair identifier is used as the new second key-value pair identifier. The process returns to step S1332 to continue comparing key-value pair identifiers until the second key-value pair identifier becomes the last key-value pair identifier. When the second key-value pair identifier becomes the last key-value pair identifier, the key-value pair data in the target partition has been traversed, and the data fusion process ends.

[0087] S1334. If the first key-value pair identifier is equal to the second key-value pair identifier, the key-value pair data corresponding to the first key-value pair identifier and the key-value pair data corresponding to the second key-value pair identifier are merged, and the merged data is determined to be the result data. The next key-value pair identifier of the first key-value pair identifier is used as the new first key-value pair identifier, and the next key-value pair identifier of the second key-value pair identifier is used as the new second key-value pair identifier. The comparison operation of the key-value pair identifiers is returned to continue until the second key-value pair identifier is the last key-value pair identifier.

[0088] In this embodiment, it is clear that the key-value pair data in the historical partition and the target partition are arranged from small to large according to the key-value identifier. If the first key-value pair identifier is equal to the second key-value pair identifier, it indicates that the current newly added data has corresponding historical data. The key-value pair data corresponding to the first key-value pair identifier and the key-value pair data corresponding to the second key-value pair identifier are merged based on the business integration of the fields, and the updated data after the merge is the result data.

[0089] Exemplarily, the business merging rule may be: data with a certain value and a null value are merged to output a certain value, two certain values ​​are merged into a concatenated string separated by ";", and the number of times new data and historical data are found is merged into the sum of the two numbers.

[0090] Continue traversing the target partition, using the next key-value pair identifier after the first key-value pair identifier as the new first key-value pair identifier, and the next key-value pair identifier after the second key-value pair identifier as the new second key-value pair identifier, and return to step S1332 to continue comparing key-value pair identifiers until the second key-value pair identifier becomes the last key-value pair identifier. When the second key-value pair identifier becomes the last key-value pair identifier, it indicates that the key-value pair data in the target partition has been traversed, and the data fusion process ends.

[0091] S1335: If the first key-value pair identifier is smaller than the second key-value pair identifier, use the next key-value pair identifier of the first key-value pair identifier as the new first key-value pair identifier, and return to continue the comparison operation of the key-value pair identifiers.

[0092] In this embodiment, it is clear that the key-value pair data in the historical partition and the target partition are arranged from small to large according to the key-value identifier. If the first key-value pair identifier is smaller than the second key-value pair identifier, it indicates that the current newly added data has corresponding historical data, but the key-value pair data corresponding to the current first key-value pair identifier is not the historical data corresponding to the current newly added data. The next key-value pair identifier of the first key-value pair identifier is used as the new first key-value pair identifier, and the process returns to step S1332 to continue the key-value pair identifier comparison operation, that is, the next key-value pair identifier of the first key-value pair identifier is compared with the second key-value pair identifier.

[0093] It is clear that the comparison between the new first key-value pair identifier and the second key-value pair identifier may also produce three possible comparison results: the new first key-value pair identifier is greater than, equal to, or less than the second key-value pair identifier. If the new first key-value pair identifier is greater than the second key-value pair identifier, step S1333 is executed; if the new first key-value pair identifier is equal to the second key-value pair identifier, step S1334 is executed; if the new first key-value pair identifier is less than the second key-value pair identifier, step S1335 is continued, and the next key-value pair identifier of the new first key-value pair identifier is used as the new first key-value pair identifier, and subsequent operations are continued until the first key-value pair identifier is greater than or equal to the second key-value pair identifier.

[0094] In this embodiment, if the read data output format is the full data output format, it is necessary to traverse the iterator IteratorHis of the historical data and the iterator IteratorNew of the newly added data.

[0095] As an optional embodiment of the present invention, based on the above embodiment, the steps of performing data fusion processing on the data in the historical partition and the target partition based on the key-value pair identifiers of the key-value pairs in the historical partition and the target partition and in combination with the data output format can be specifically expressed as follows:

[0096] S1331′: Use the first key-value pair identifier in the historical partition and the target partition as the first key-value pair identifier and the second key-value pair identifier respectively.

[0097] For example, the first key-value pair identifier of the historical partition is used as the first key-value pair identifier, which is recorded as<key1,rowHis> ; The first key-value pair identifier of the target partition is used as the second key-value pair identifier, recorded as<key2,rowNew> .

[0098] S1332′: If the data output format is the full data output format, compare the first key-value pair identifier and the second key-value pair identifier.

[0099] Specifically, the first key-value pair identifier and the second key-value pair identifier are represented in the form of strings, and the size of the strings can be compared, that is, the relationship between the first key-value pair identifier and the second key-value pair identifier can be greater than, equal to, or less than. For example, if the first key-value pair identifier is represented as key1 and the second key-value pair identifier is represented as key2, the comparison result can be expressed as: key1>key2, key1=key2, or key1 <key2。

[0100] S1333'. If the first key-value pair identifier is greater than the second key-value pair identifier, determine that the key-value pair data corresponding to the second key-value pair identifier is the result data, use the next key-value pair identifier of the second key-value pair identifier as the new second key-value pair identifier, and return to continue the comparison operation of the key-value pair identifiers until the first key-value pair identifier is the last key-value pair identifier and the second key-value pair identifier is the last key-value pair identifier.

[0101] In this embodiment, it is clear that the key-value pair data in the historical partition and the target partition are arranged in ascending order according to the key-value identifier. If the first key-value pair identifier is greater than the second key-value pair identifier, it indicates that the current newly added data has no historical data, and the newly added data can be obtained as the result data, that is, the key-value pair data corresponding to the second key-value pair identifier is the result data.

[0102] The target partition is traversed continuously, and the next key-value pair identifier after the second key-value pair identifier is used as the new second key-value pair identifier. The process returns to step S1332' to continue comparing key-value pair identifiers until both the first key-value pair identifier and the second key-value pair identifier are the last key-value pair identifiers. When both the first key-value pair identifier and the second key-value pair identifier are the last key-value pair identifiers, the key-value pair data in the historical partition and the target partition have been traversed, and the data fusion process is complete.

[0103] S1334'. If the first key-value pair identifier is equal to the second key-value pair identifier, the key-value pair data corresponding to the first key-value pair identifier and the key-value pair data corresponding to the second key-value pair identifier are merged, and the merged data is determined to be the result data. The next key-value pair identifier of the first key-value pair identifier is used as the new first key-value pair identifier, and the next key-value pair identifier of the second key-value pair identifier is used as the new second key-value pair identifier. The comparison operation of the key-value pair identifiers is returned to continue until the first key-value pair identifier is the last key-value pair identifier and the second key-value pair identifier is the last key-value pair identifier.

[0104] In this embodiment, it is clear that the key-value pair data in the historical partition and the target partition are arranged from small to large according to the key-value identifier. If the first key-value pair identifier is equal to the second key-value pair identifier, it indicates that the current newly added data has corresponding historical data. The key-value pair data corresponding to the first key-value pair identifier and the key-value pair data corresponding to the second key-value pair identifier are merged based on the business integration of the fields, and the updated data after the merge is the result data.

[0105] Exemplarily, the business merging rule may be: data with a certain value and a null value are merged to output a certain value, two certain values ​​are merged into a concatenated string separated by ";", and the number of times new data and historical data are found is merged into the sum of the two numbers.

[0106] Continue traversing the target partition, using the next key-value pair identifier of the first key-value pair identifier as the new first key-value pair identifier, and the next key-value pair identifier of the second key-value pair identifier as the new second key-value pair identifier, and return to step S1332' to continue the key-value pair identifier comparison operation until both the first key-value pair identifier and the second key-value pair identifier are the last key-value pair identifiers. When both the first key-value pair identifier and the second key-value pair identifier are the last key-value pair identifiers, it indicates that the key-value pair data in the historical partition and the target partition have been traversed, and the data fusion process is complete.

[0107] S1335'. ​​If the first key-value pair identifier is smaller than the second key-value pair identifier, determine that the data corresponding to the first key-value pair identifier is the result data, use the next key-value pair identifier of the first key-value pair identifier as the new first key-value pair identifier, and return to continue the comparison operation of the key-value pair identifiers until the first key-value pair identifier is the last key-value pair identifier and the second key-value pair identifier is the last key-value pair identifier.

[0108] In this embodiment, it is clear that the key-value pair data in the historical partition and the target partition are arranged in ascending order according to the key-value identifier. If the first key-value pair identifier is smaller than the second key-value pair identifier, it indicates that there is no corresponding new data in the current historical data. The historical data can be obtained as the result data, that is, the key-value pair data corresponding to the first key-value pair identifier is the result data. The historical partition is continued to be traversed, and the key-value pair identifier next to the first key-value pair identifier is used as the new first key-value pair identifier. The execution returns to step S1332' to continue the key-value pair identifier comparison operation.

[0109] It is clear that the comparison result of the new first key-value pair identifier and the second key-value pair identifier may also produce three situations: the new first key-value pair identifier is greater than, equal to, or less than the second key-value pair identifier. If the new first key-value pair identifier is greater than the second key-value pair identifier, execute step S1333'; if the new first key-value pair identifier is equal to the second key-value pair identifier, execute step S1334'; if the new first key-value pair identifier is less than the second key-value pair identifier, continue to execute step S1335'. ​​Until the first key-value pair identifier is the last key-value pair identifier and the second key-value pair identifier is the last key-value pair identifier. When the first key-value pair identifier and the second key-value pair identifier are both the last key-value pair identifiers, it indicates that the key-value pair data in the historical partition and the target partition have been traversed, and the data fusion processing is completed.

[0110] Optionally, based on the key-value pair identifiers of the key-value pairs in the historical partition and the target partition, combined with the data output format, data fusion processing is performed on the data in the historical partition and the target partition, including:

[0111] If the data output mode is full data output and there is no new data in the target partition, the historical data in the corresponding historical partition is obtained as the result data.

[0112] Specifically, if the data output format is full data output, if the new data in the current partition is empty, it means that there is no corresponding new data in the current historical partition. In this case, the historical data in the corresponding historical partition can be obtained as the result data. For example, you can directly copy the historical file data output without loading and reading historical data, reducing data reading.

[0113] As an optional embodiment of the embodiment of the present invention, based on the above embodiment, after performing fusion processing on the data in each target partition according to the data output form, obtaining the processed result data and outputting it, the following is further included:

[0114] For each target partition, the result data is stored in the distributed file system respectively, and the index repository and the content repository are updated according to the result data, wherein the index repository stores the result data of the index update, and the content repository stores the result data of the content update.

[0115] Specifically, the result data for each partition obtained from the above steps, whether full or incremental, is stored on the distributed file system HDFS by partition tag. The output file name ends with the partition tag to facilitate the next data collision and fusion. New data (New) and index update data (Mod-index) are also stored in the index repository, reducing the index storage data. New data (New) and content update data (Mod-content) are stored in the content repository.

[0116] For example, the data is output to a new HDFS directory, / data / history0908 / , with the output files ending in "part-1" and "part-2" for easy use in the next collision and fusion. Newly added data and updated data are stored in the ES index library and HBase columnar storage library, respectively.

[0117] For a clearer presentation, the data fusion processing methods for the newly added data output form and the full data output form are respectively illustrated by examples. Table 5 shows the data contained in the target partition and the historical partition; Table 6 shows the result data obtained if the incremental data output form is used this time; Table 7 shows the result data obtained if the full data output form is used this time.

[0118] Table 5

[0119]

[0120] Table 6

[0121]

[0122] Table 7

[0123]

[0124] Calculated using mobile phone number as the index field:

[0125] (Person3, Zhang San, 156xxxx6666, 1) is the newly added data;

[0126] (Person1, Wang San, 131xxxx1111; 156xxxx1111, 6) is the updated data, and the mobile phone number field is updated, which is the index update data;

[0127] (Person2, test, null value, 2) is updated data, and the index field is not updated;

[0128] (Person4, Li Si, null value, 1) is historical data.

[0129] In order to more clearly describe the data fusion processing process provided by the embodiment of the present invention, Figure 2 A schematic diagram of the data fusion processing process provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown:

[0130] During the data fusion processing, the required processing objects are all based on the distributed computing framework to obtain new data and historical data from the distributed file system. The data fusion processing process needs to go through: reading and processing the new data; partitioning processing to obtain the target partition and sorting; obtaining the historical partition; merging the data in the target partition and the historical partition; and finally obtaining the result data; the result data are all stored on the distributed file system according to the partition tag, and are stored in the index database and content database respectively according to whether the result data is an index update or a content update.

[0131] As an alternative to the above-mentioned embodiment, the data fusion processing steps for the newly added data output format and the data fusion processing steps for the full data output format are specified. Data fusion processing is performed using a full data plus multiple increments method, which improves data processing efficiency and reduces resource consumption. Simultaneously, the result data is stored separately in a distributed file system, and the index repository and content repository are updated based on the result data, facilitating the next data fusion process and reducing the amount of indexed data to be stored.

[0132] Example 2

[0133] Figure 3 This is a structural block diagram of a data fusion processing device provided in the second embodiment of the present invention, such as Figure 3 As shown, the device includes: a partition processing module 21, an output form acquisition module 22 and a fusion processing module 23.

[0134] A partition processing module 21 is used to partition the newly added data according to a preset partitioning rule to obtain at least one target partition, wherein the number of target partitions is the same as the number of historical partitions;

[0135] The output form acquisition module 22 is used to acquire a set data output form, wherein the data output form includes an incremental data output form and a full data output form;

[0136] The fusion processing module 23 is used to perform fusion processing on the data in each target partition according to the data output form, obtain the processed result data and output it.

[0137] Optionally, the partition processing module 21 includes:

[0138] A new data sequence reading unit is added to read the new data sequence from the distributed file system;

[0139] A key-value pair data generating unit, configured to process the newly added data sequence according to a preset processing rule to generate a set of key-value pair data, wherein the key-value pair data includes a key-value pair identifier;

[0140] An initial partition obtaining unit, configured to partition the data of each key-value pair according to a hash partitioning rule to obtain at least one initial partition;

[0141] The target partition acquisition unit is used to sort the corresponding key-value pair data in each initial partition according to the key-value pair identifiers included therein, and determine the sorted initial partition as the target partition.

[0142] Optionally, the fusion processing module 23 includes:

[0143] A new key-value pair data acquisition unit is used to merge the key-value pair data contained in each target partition to obtain at least one new key-value pair data;

[0144] A historical partition acquisition unit, used to acquire a historical partition having the same partition number as the target partition;

[0145] A data fusion processing unit is used to perform data fusion processing on the data in the historical partition and the target partition based on the key-value pair identifiers of the key-value pairs in the historical partition and the target partition in combination with the data output form;

[0146] The result data acquisition unit is used to obtain the result data corresponding to each target partition and output it.

[0147] Optionally, the data fusion processing unit may be specifically used for:

[0148] The first key-value pair identifier in the historical partition and the target partition is used as the first key-value pair identifier and the second key-value pair identifier respectively;

[0149] If the data output format is an incremental data output format, comparing the first key-value pair identifier and the second key-value pair identifier;

[0150] If the first key-value pair identifier is greater than the second key-value pair identifier, the key-value pair data corresponding to the second key-value pair identifier is determined to be the result data, the key-value pair identifier next to the second key-value pair identifier is used as the new second key-value pair identifier, and the comparison operation of the key-value pair identifiers is continued until the second key-value pair identifier becomes the last key-value pair identifier;

[0151] If the first key-value pair identifier is equal to the second key-value pair identifier, the key-value pair data corresponding to the first key-value pair identifier and the key-value pair data corresponding to the second key-value pair identifier are merged, the merged data is determined to be the result data, the next key-value pair identifier of the first key-value pair identifier is used as the new first key-value pair identifier, the next key-value pair identifier of the second key-value pair identifier is used as the new second key-value pair identifier, and the comparison operation of the key-value pair identifiers is continued until the second key-value pair identifier is the last key-value pair identifier;

[0152] If the first key-value pair identifier is smaller than the second key-value pair identifier, the next key-value pair identifier of the first key-value pair identifier is used as the new first key-value pair identifier, and the operation of comparing the key-value pair identifiers is continued.

[0153] Optionally, the data fusion processing unit may be specifically used for:

[0154] The first key-value pair identifier in the historical partition and the target partition is used as the first key-value pair identifier and the second key-value pair identifier respectively;

[0155] If the data output format is the full data output format, compare the first key-value pair identifier and the second key-value pair identifier;

[0156] If the first key-value pair identifier is greater than the second key-value pair identifier, the key-value pair data corresponding to the second key-value pair identifier is determined to be the result data, the key-value pair identifier next to the second key-value pair identifier is used as the new second key-value pair identifier, and the comparison operation of the key-value pair identifiers is continued until the first key-value pair identifier is the last key-value pair identifier and the second key-value pair identifier is the last key-value pair identifier;

[0157] If the first key-value pair identifier is equal to the second key-value pair identifier, the key-value pair data corresponding to the first key-value pair identifier and the key-value pair data corresponding to the second key-value pair identifier are merged, the merged data is determined to be the result data, the next key-value pair identifier of the first key-value pair identifier is used as the new first key-value pair identifier, the next key-value pair identifier of the second key-value pair identifier is used as the new second key-value pair identifier, and the comparison operation of the key-value pair identifiers is continued until the first key-value pair identifier is the last key-value pair identifier and the second key-value pair identifier is the last key-value pair identifier;

[0158] If the first key-value pair identifier is less than the second key-value pair identifier, the data corresponding to the first key-value pair identifier is determined to be the result data, and the next key-value pair identifier of the first key-value pair identifier is used as the new first key-value pair identifier, and the comparison operation of the key-value pair identifiers is returned to continue until the first key-value pair identifier is the last key-value pair identifier and the second key-value pair identifier is the last key-value pair identifier.

[0159] Optionally, the data fusion processing unit may be specifically used for:

[0160] If the data output mode is full data output and there is no new data in the target partition, the historical data in the corresponding historical partition is obtained as the result data.

[0161] Optionally, the device further includes:

[0162] The storage module is used to store the result data of each target partition in the distributed file system respectively, and update the index repository and content repository according to the result data, wherein the index repository stores the result data of index update, and the content repository stores the result data of content update.

[0163] The above device can execute the data fusion processing method provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not fully described in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0164] Example 3

[0165] Figure 4 This is a structural block diagram of a computer device provided in Example 3 of the present invention, such as Figure 4 As shown, the computer device includes a processor 31, a memory 32, an input device 33 and an output device 34; the number of processors 31 in the computer device can be one or more. Figure 4 In the figure, a processor 31 is taken as an example; the processor 31, memory 32, input device 33 and output device 34 in the computer device can be connected by a bus or other means. Figure 4 The bus connection is taken as an example.

[0166] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and modules, such as the modules corresponding to the data fusion processing method in the embodiments of the present invention (for example, the partition processing module 21, the output form acquisition module 22, and the fusion processing module 23 in the data fusion processing device). The processor 31 executes the software programs, instructions, and modules stored in the memory 32 to execute various functional applications and data processing of the computer device, thereby implementing the above-mentioned data fusion processing method.

[0167] The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal. Furthermore, the memory 32 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory 32 may further include memory remotely located relative to the processor 31, and such remote memory may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0168] The input device 33 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer device. The output device 34 may include a display device such as a display screen.

[0169] Example 4

[0170] A fourth embodiment of the present invention further provides a storage medium containing computer-executable instructions. When the computer-executable instructions are executed by a computer processor, the computer-executable instructions are used to perform a data fusion processing method. The method includes:

[0171] Partition the newly added data according to the preset partitioning rules to obtain at least one target partition, where the number of target partitions is the same as the number of historical partitions;

[0172] Obtain the set data output format, where the data output format includes incremental data output format and full data output format;

[0173] According to the data output form, the data in each target partition is fused and processed to obtain the processed result data and output it.

[0174] Of course, the computer executable instructions of the storage medium provided by the embodiment of the present invention are not limited to the operations of the method described above, but can also execute related operations in the data fusion processing method provided by any embodiment of the present invention.

[0175] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0176] It is worth noting that in the embodiment of the above-mentioned data fusion processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0177] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data fusion processing method, characterized in that: include: Partition the newly added data according to the preset partitioning rules to obtain at least one target partition, where the number of target partitions is the same as the number of historical partitions; Obtaining a set data output format, wherein the data output format includes an incremental data output format and a full data output format; According to the data output form, the data in each target partition is fused and processed to obtain the processed result data and output it; Partitioning the newly added data according to the preset partitioning rules to obtain at least one target partition includes: Reading a newly added data sequence from a distributed file system; wherein, the newly added data sequence is read from the distributed file system by using a distributed computing framework; Processing the newly added data sequence according to a preset processing rule to generate a set of key-value pair data, wherein the key-value pair data includes a key-value pair identifier; Partitioning each of the key-value pairs of data according to a hash partitioning rule to obtain at least one initial partition; In each initial partition, the corresponding key-value pair data is sorted according to the key-value pair identifiers included therein, and the sorted initial partition is determined as the target partition; According to the data output form, the data in each target partition is fused and processed to obtain the processed result data and output it, including: For each target partition, merge the key-value pairs contained therein to obtain at least one new key-value pair. Obtain the historical partition whose partition number is the same as the target partition; Based on the key-value pair identifiers of each key-value pair data in the historical partition and the target partition, combined with the data output form, data fusion processing is performed on the data in the historical partition and the target partition; Obtain and output the result data corresponding to each target partition.

2. The method according to claim 1, characterized in that The key-value pair identifiers of the key-value pair data in the historical partition and the target partition, combined with the data output form, performing data fusion processing on the data in the historical partition and the target partition, include: The first key-value pair identifier in the historical partition and the target partition is used as the first key-value pair identifier and the second key-value pair identifier respectively; If the data output format is an incremental data output format, comparing the first key-value pair identifier and the second key-value pair identifier; If the first key-value pair identifier is greater than the second key-value pair identifier, determining that the key-value pair data corresponding to the second key-value pair identifier is the result data, taking the key-value pair identifier next to the second key-value pair identifier as the new second key-value pair identifier, and returning to continue the key-value pair identifier comparison operation until the second key-value pair identifier becomes the last key-value pair identifier; If the first key-value pair identifier is equal to the second key-value pair identifier, the key-value pair data corresponding to the first key-value pair identifier and the key-value pair data corresponding to the second key-value pair identifier are merged, the merged data is determined to be the result data, the next key-value pair identifier of the first key-value pair identifier is used as the new first key-value pair identifier, the next key-value pair identifier of the second key-value pair identifier is used as the new second key-value pair identifier, and the comparison operation of the key-value pair identifiers is continued until the second key-value pair identifier becomes the last key-value pair identifier; If the first key-value pair identifier is smaller than the second key-value pair identifier, the key-value pair identifier next to the first key-value pair identifier is used as the new first key-value pair identifier, and the process returns to continue the comparison operation of the key-value pair identifiers.

3. The method according to claim 1, characterized in that The key-value pair identifiers of the key-value pair data in the historical partition and the target partition, combined with the data output form, performing data fusion processing on the data in the historical partition and the target partition, include: The first key-value pair identifier in the historical partition and the target partition is used as the first key-value pair identifier and the second key-value pair identifier respectively; If the data output format is a full data output format, comparing the first key-value pair identifier and the second key-value pair identifier; If the first key-value pair identifier is greater than the second key-value pair identifier, determining that the key-value pair data corresponding to the second key-value pair identifier is the result data, taking the next key-value pair identifier of the second key-value pair identifier as the new second key-value pair identifier, and returning to continue the key-value pair identifier comparison operation until the first key-value pair identifier is the last key-value pair identifier and the second key-value pair identifier is the last key-value pair identifier; If the first key-value pair identifier is equal to the second key-value pair identifier, the key-value pair data corresponding to the first key-value pair identifier and the key-value pair data corresponding to the second key-value pair identifier are merged, the merged data is determined to be the result data, the next key-value pair identifier of the first key-value pair identifier is used as the new first key-value pair identifier, the next key-value pair identifier of the second key-value pair identifier is used as the new second key-value pair identifier, and the comparison operation of the key-value pair identifiers is continued until the first key-value pair identifier is the last key-value pair identifier and the second key-value pair identifier is the last key-value pair identifier; If the first key-value pair identifier is less than the second key-value pair identifier, the data corresponding to the first key-value pair identifier is determined to be the result data, and the next key-value pair identifier of the first key-value pair identifier is used as the new first key-value pair identifier, and the comparison operation of the key-value pair identifiers is returned to continue until the first key-value pair identifier is the last key-value pair identifier and the second key-value pair identifier is the last key-value pair identifier.

4. The method according to claim 1, wherein The key-value pair identifiers of the key-value pair data in the historical partition and the target partition, combined with the data output form, performing data fusion processing on the data in the historical partition and the target partition, include: If the data output format is a full data output format and there is no new data in the target partition, the historical data in the corresponding historical partition is obtained as the result data.

5. The method according to claim 1, wherein After performing fusion processing on the data in each target partition according to the data output form, obtaining the processed result data and outputting it, the method further includes: For each target partition, the result data is stored in a distributed file system respectively, and an index repository and a content repository are updated according to the result data, wherein the index repository stores the result data of index update, and the content repository stores the result data of content update.

6. A data fusion processing device, characterized in that: include: A partition processing module is used to partition the newly added data according to a preset partitioning rule to obtain at least one target partition, wherein the number of target partitions is the same as the number of historical partitions; An output form acquisition module, used to acquire a set data output form, wherein the data output form includes an incremental data output form and a full data output form; A fusion processing module is used to perform fusion processing on the data in each target partition according to the data output form, obtain the processed result data and output it; The partition processing module includes: a newly added data sequence reading unit, a key-value pair data generation unit, an initial partition acquisition unit and a target partition acquisition unit; The newly added data sequence reading unit is used to read the newly added data sequence from the distributed file system; wherein the newly added data sequence is read from the distributed file system by using a distributed computing framework; The key-value pair data generating unit is configured to process the newly added data sequence according to a preset processing rule to generate a set of key-value pair data, wherein the key-value pair data includes a key-value pair identifier; The initial partition obtaining unit is used to partition the data of each key value pair according to the hash partitioning rule to obtain at least one initial partition; The target partition acquisition unit is configured to sort the corresponding key-value pair data in each initial partition according to the key-value pair identifiers included therein, and determine the sorted initial partition as the target partition; The fusion processing module includes: a new key-value pair data acquisition unit, a historical partition acquisition unit, a data fusion processing unit and a result data acquisition unit; The new key-value pair data acquisition unit is used to merge the key-value pair data contained in each target partition to obtain at least one new key-value pair data; The historical partition acquisition unit is used to acquire the historical partition with the same partition number as the target partition; The data fusion processing unit is used to perform data fusion processing on the data in the historical partition and the target partition based on the key-value pair identifiers of the key-value pairs in the historical partition and the target partition in combination with the data output form; The result data acquisition unit is used to obtain and output the result data corresponding to each target partition.

7. A computer device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data fusion processing method according to any one of claims 1 to 5.

8. A storage medium containing computer-executable instructions, characterized in that: When executed by a computer processor, the computer executable instructions are used to perform the data fusion processing method according to any one of claims 1 to 5.

Images