Data processing method and device

Abstract

The present application provides a method and device for data processing, including: based on an association condition, determining a first key-value pair group from a first data set, the first key-value pair of the first key-value group including the content of a first key field in the first data set and first data corresponding to the first key field in the first data set, the first data having a first tag, and the association condition including that the content of the first key field is equal to the content of a second key field in a second data set; based on the association condition, determining a second key-value pair group from a second data set, the second key-value pair in the second key-value pair group including the content of the second key field and second data corresponding to the second key field in the second data set, the second data having a second tag; merging the key-value pairs that meet the association condition in the first key-value pair group and the second key-value pair group to obtain a third key-value pair group; and obtaining an association result from the third key-value pair group based on the association method, the first tag, and the second tag.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method and device for data processing. Background Art

[0002] In a database, when processing data in multiple data tables, the data in multiple data tables can be associated according to the association conditions. However, in a big data environment, the amount of data in a data table can reach tens of billions or even hundreds of billions, and it is difficult to perform association calculations on multiple data sets including massive data.

[0003] In the prior art, slice broadcast comparison is used to implement association calculation on multiple data sets including massive data. This method will occupy a large amount of processor resources and take a long time to complete the association calculation. Summary of the invention

[0004] The present application provides a data processing method and device, which are conducive to improving the efficiency of associative calculation on a data table containing massive data and saving processor resources.

[0005] In a first aspect, a method for data processing is provided, the method comprising: determining a first key-value pair group from a first data set based on an association condition, the first key-value pair of the first key-value group comprising the content of a first key field in the first data set and first data corresponding to the first key field in the first data set, the first data having a first tag, and the association condition comprising that the content of the first key field is equal to the content of a second key field in a second data set; determining a second key-value pair group from a second data set based on the association condition, the second key-value pair in the second key-value pair group comprising the content of the second key field and second data corresponding to the second key field in the second data set, the second data having a second tag; merging the key-value pairs in the first key-value pair group and the second key-value pair group that meet the association condition to obtain a third key-value pair group, the third key-value pair in the third key-value pair group comprising at least one of the following: the content of the first key field, the first data and the second data, the content of the first key field and the first data, or the content of the second key field and the second data; obtaining an association result from the third key-value pair group based on the association method, the first tag and the second tag.

[0006] The method provided in the embodiment of the present application adds tags to the data in the key-value pairs to facilitate the distinction between data in different data sets, which is beneficial to improving the efficiency of associative calculations on data tables containing massive data, saving processor resources, and improving the versatility of the algorithm.

[0007] In combination with the first aspect, in certain implementations of the first aspect, in the third key-value pair, a first separator is included between the first data and the second data, and a second separator is included between different data in the first data and the second data.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the association condition also includes: the content of the third key field in the first data set is equal to the content of the fourth key field in the second data set; the first key-value pair also includes the content of the third key field, and the content of the third key field is connected to the content of the first key field through a third separator; the second key-value pair also includes the content of the fourth key field, and the content of the fourth key field is connected to the content of the second key field through the third separator.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the association condition also includes: the content of the fifth key field in the second data set is equal to the content of the sixth key field in the third data set; the method also includes: determining a fourth key-value pair group from the association result, the key-value pairs in the fourth key-value pair group include the content of the fifth key field and the third data corresponding to the fifth key field in the association result, and the third data has a third tag; determining a fifth key-value pair group from the third data set, the key-value pairs in the fifth key-value pair group include the content of the sixth key field and the fourth data corresponding to the sixth key field in the fifth data set, and the fourth data has a fourth tag; merging the key-value pairs in the fourth key-value pair group and the fifth key-value pair group that meet the association condition to obtain a sixth key-value pair group, the sixth key-value pair group including the content of the fifth key field, the third data and the fourth data, the content of the fifth key field and the third data, or the content of the sixth key field and the fourth data; based on the association method, the third tag and the fourth tag, obtaining an association result from the sixth key-value pair group.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the data included in the first data set and / or the second data set is data filtered according to the filtering condition.

[0011] In combination with the first aspect, in some implementations of the first aspect, the association mode is left association, right association, inner association, or full association.

[0012] In combination with the first aspect, in certain implementations of the first aspect, based on the association condition, if there are multiple key fields in a data set, when the data set generates key-value pairs, tags can be added to the multiple key fields to distinguish them.

[0013] In a second aspect, a data processing device is provided, which is used to execute the method in any possible implementation of the first aspect. Specifically, the device includes a module for executing the method in any possible implementation of the first aspect.

[0014] In a third aspect, another data processing device is provided, comprising a processor and a memory. The processor is used to read instructions stored in the memory to execute the method in any possible implementation of the first aspect.

[0015] Optionally, there are one or more processors and one or more memories.

[0016] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0017] In the specific implementation process, the memory can be a non-transitory memory, such as a read-only memory (ROM), which can be integrated with the processor on the same chip or can be set on different chips respectively. The embodiments of the present application do not limit the type of memory and the setting method of the memory and the processor.

[0018] The data processing device in the third aspect mentioned above can be a chip. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading the software code stored in the memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0019] In a fourth aspect, a computer-readable medium is provided, which stores a computer program (also referred to as code or instruction) which, when executed on a computer, enables the computer to execute a method in any possible implementation of any of the above aspects.

[0020] In a fifth aspect, a computer program product is provided, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute a method in any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic flow chart of a data processing method proposed in an embodiment of the present application;

[0022] Figure 2 is a schematic block diagram of a data processing device proposed in an embodiment of the present application;

[0023] Figure 3 It is a schematic block diagram of a data processing device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0025] Most of the related calculations on data sets including massive data are solved by distributed parallel programming, such as MapReduce. MapReduce is a parallel programming model used for parallel computing of large-scale data sets.

[0026] In the prior art, MapReduce is used to perform association calculations on data sets including massive data. First, the large data set is generally logically divided into independent small data blocks. These small data blocks are processed in parallel by multiple Map processes on multiple machines and generate corresponding intermediate results in the form of key-value pairs. Then, these intermediate results are processed in parallel by multiple Reduce processes on multiple machines, among which intermediate results with the same key value will be processed by the same Reduce process. Finally, the Reduce process will summarize and calculate the intermediate results according to the association conditions to obtain the final result of the association calculation.

[0027] The above divides the large data set into small data blocks. The starting position, data length and other information of the small data block in the large data set need to be determined by the user. If the data volume of the small data block is too large, the parallel efficiency will be low. If the data volume of the small data block is too small, data transmission will increase.

[0028] Therefore, dividing a large data set into small data blocks for association calculation is more random and less universal, and it takes a long time to complete the association calculation and occupies more processor resources.

[0029] In view of this, the embodiments of the present application provide a method and device for data processing, which are conducive to improving the efficiency of associative calculations on data tables containing massive data, saving processor resources, and improving the versatility of algorithms.

[0030] The data processing method provided in the embodiment of the present application can be implemented by MapReduce and can be applicable to any system that applies MapReduce, such as Hadoop, Hive, Spark, MR, Apache Pig, CDH (Cloudera's distribution including Apache Hadoop), Dpark based on Python and Mesos, etc. Among them, MapReduce is a software framework, based on which application programs can be easily written.

[0031] The data processing method provided in the embodiment of the present application can be developed using any development language, for example, python, java, C++, C#, C language, Scala, etc.

[0032] The data processing method provided in the embodiment of the present application does not limit the file system in which the processed data files are stored. The file system may be a file distributed storage (hadoop distributed file system, HDFS), a network file system (network file system, NFS) or a distributed file system (MooseFS, MFS), etc. The embodiment of the present application does not limit the file format of the data, for example, TextFile, SequenceFile, ParquetFile, ORCFile, etc. The method provided in the embodiment of the present application may also support file compression methods such as lzo, lz4, snappy, gzip, and bzip.

[0033] The massive data in the embodiment of the present application includes petabyte (PB), trillionbyte (TB), and gigabyte (GB) data. The embodiment of the present application has no restrictions on the number and size of the data sets for the association calculation.

[0034] The data processing method provided in the embodiment of the present application may also include looping and combining the method to implement a large number of complex, multi-data association calculations.

[0035] Figure 1 The present invention provides a schematic flow chart of a data processing method 100 provided in an embodiment of the present invention. The method 100 can be implemented by a data processing device, which can be any device that applies a MapReduce system. The method 100 can include the following steps:

[0036] S101, based on an association condition, determining a first key-value pair group from a first data set, wherein a first key-value pair in the first key-value pair group includes content of a first key field in the first data set and first data corresponding to the first key field in the first data set, the first data having a first tag, and the association condition includes that the content of the first key field is equal to content of the second key field in the second data set.

[0037] It should be understood that the data in the first data set can be converted into a first key-value pair, and the first key-value pair group includes at least one of the first key-value pairs.

[0038] Exemplarily, the first data set and the second data set are associated with each other based on the association condition, and the data of the first data set is preprocessed first, that is, a first key-value pair group is generated through a Map process, and the first key-value pair in the first key-value pair group can be (key1, "L~" + value1), wherein the content corresponding to key1 is the content of the first key field, the data corresponding to value1 is the first data corresponding to the first key field in the first data set, and "L~" is the first mark of the first data, and the first mark is used to indicate that the first data belongs to the first data set. It should be understood that "L~" is only an example.

[0039] The first mark can be a symbol, a letter, a number, or any combination of symbols, letters, and numbers. It should be understood that the embodiment of the present application does not limit the form of the first mark.

[0040] Optionally, if there is a filtering condition in the first data set, the data in the first data set may be filtered according to the filtering condition before determining the first key-value pair.

[0041] S102, based on the association condition, determine a second key-value pair group from the second data set, the second key-value pair in the second key-value pair group includes the content of the second key field and second data corresponding to the second key field in the second data set, and the second data has a second tag.

[0042] It should be understood that the data in the second data set can be converted into second key-value pairs, and the second key-value pair group includes at least one of the second key-value pairs.

[0043] Exemplarily, an association calculation is performed on the first data set and the second data set based on the association condition, and the data of the second data set is first preprocessed, that is, a second key-value pair group is generated through a Map process, and the second key-value pair in the second key-value pair group can be (key2, "R~" + value2), wherein the content corresponding to key2 is the content of the second key field, the data corresponding to value2 is the second data corresponding to the second key field in the second data set, and "R~" is a second mark of the second data, and the second mark is used to indicate that the second data belongs to the second data set.

[0044] The second mark can be a symbol, a letter, a number, or any combination of symbols, letters, and numbers. It should be understood that the embodiment of the present application does not limit the form of the second mark.

[0045] It should be understood that the second mark only needs to be different from the first mark.

[0046] Optionally, if there is a filtering condition in the second data set, the data in the second data set may be filtered according to the filtering condition before determining the second key-value pair.

[0047] S103, merging the key-value pairs in the first key-value pair group and the second key-value pair group that meet the association condition to obtain a third key-value pair group, wherein the third key-value pair in the third key-value pair group includes at least one of the following: the content of the first key field, the first data and the second data, the content of the first key field and the first data, or the content of the second key field and the second data.

[0048] Specifically, if the content of the first key field is equal to the content of the second key field, the third key-value pair includes the content of the first key field, the first data, and the second data; if the content of the first key field is not equal to the content of the second key field, the third key-value pair includes the content of the first key field and the first data, or the content of the second key field and the second data.

[0049] The embodiment of the present application does not limit the order of the first data and the second data in the third key-value pair.

[0050] Exemplarily, the first key-value pair and the second key-value pair can be merged through the Reduce process to obtain a third key-value pair group, and the third key-value pair in the third key-value pair group can be (key3, value3), wherein key3 can be key1 or key2, and value3 can be "L~"+value1, "R~"+value2, "L~"+value1+"R~"+value2.

[0051] If key1 is equal to key2, the third key-value pair can be (key1 or key2, "L~" + value1 + "R~" value2); if key1 is not equal to key2, the third key-value pair can be (key1, "L~" + value1 + null) or (key2, null + "R~" + value2).

[0052] It should be understood that the third key-value pair can be (key1 or key2, "R~" value2 + "L~" + value1); if key1 is not equal to key2, the third key-value pair can be (key1, null + "L~" + value1) or (key2, "R~" + value2 + null).

[0053] S104: Obtain an association result from the third key-value pair group based on the association mode, the first tag, and the second tag.

[0054] Optionally, the association mode may be left association, right association, inner association, full association or any combination thereof.

[0055] Based on the association mode, the first tag and the second tag, the key-value pairs that do not conform to the association mode are filtered out from the third key-value pair group.

[0056] Exemplarily, if the association mode is left association, the association results are (key1, "L~" + value1 + null) and (key1 or key2, "L~" + value1 + "R~" + value2).

[0057] If the association method is right association, the association results are (key2, null+"R~"+value2) and (key1 or key2, "L~"+value1+"R~"+value2).

[0058] If the association method is internal association, the association result is (key1 or key2, "L~" + value1 + "R~" + value2).

[0059] If the association mode is full association, the association results are (key1 or key2, "L~" + value1 + "R~" + value2), (key1, "L~" + value1 + null) and (key2, null + "R~" + value2).

[0060] The method provided in the embodiment of the present application adds tags to the data in the key-value pairs to facilitate the distinction between data in different data sets, which is beneficial to improving the efficiency of associative calculations on data tables containing massive data, saving processor resources, and improving the versatility of the algorithm.

[0061] As an optional embodiment, in the third key-value pair, a first separator is included between the first data and the second data, and a second separator is included between the first data and different data in the second data.

[0062] Optionally, the first separator and the second separator can be ","," / ,"|,",",",","~", etc. The embodiment of the present application is not limited to this. It should be understood that the first separator and the second separator can be different.

[0063] Exemplarily, the third key-value pair may be (key2, null+"|"+"R~"+value2). If value2 contains 3 data, namely value21, value22, and value23, the third key-value pair may be (key2, null+"|"+"R~"+value21+"~"+value22+"~"+value23). Among them, "|" is the first separator, and "~" is the second separator.

[0064] As an optional embodiment, the association condition also includes: the content of the third key field in the first data set is equal to the content of the fourth key field in the second data set; the first key-value pair also includes the content of the third key field, and the content of the third key field is connected to the content of the first key field through a third separator; the second key-value pair also includes the content of the fourth key field, and the content of the fourth key field is connected to the content of the second key field through the third separator.

[0065] Specifically, the association condition includes: the content of the first key field in the first data set is equal to the content of the second key field in the second data set, and the content of the third key field in the first data set is equal to the content of the fourth key field in the second data set.

[0066] Based on the association condition, the first data set is generated into a first key-value pair group through the Map process, and the first key-value pair in the first key-value pair group can be (key1+"~"+key3, "L~"+value1), wherein key1 is the content of the first key field, key3 is the content of the third key field, and "~" is the third separator mentioned above.

[0067] Based on the association condition, the second data set is generated into a second key-value pair group through the Map process, and the second key-value pair in the second key-value pair group can be (key2+"~"+key4, "R~"+value2), wherein key2 is the content of the second key field, key4 is the content of the fourth key field, and "~" is the third separator mentioned above.

[0068] The third separator can be ","," / ,"|,",",","~", etc. The embodiment of the present application is not limited to this. It should be understood that the third separator can be the same as or different from the above-mentioned second separator; the third separator is different from the above-mentioned first separator.

[0069] The first key-value pair and the second key-value pair are merged through the Reduce process to obtain a third key-value pair group, in which the third key-value pair can be (key1+"~"+key3 or key2+"~"+key4, "L~"+value1+"|"+"R~"+value2), (key1+"~"+key3, "L~"+value1+"|"+null), (key2+"~"+key4, null+"|"+"R~"+value2).

[0070] Based on the association mode, the first tag and the second tag, the key-value pairs that do not meet the association condition are filtered out from the third key-value pair group.

[0071] Exemplarily, if the association mode is left association, the association results are (key1+"~"+key3, "L~"+value1+"|"+null) and (key1+"~"+key3 or key2+"~"+key4, "L~"+value1+"|"+"R~"+value2).

[0072] If the association method is right association, the association results are (key2+"~"+key4, null+"|"+"R~"+value2) and (key1+"~"+key3 or key2+"~"+key4, "L~"+value1+"|"+"R~"+value2).

[0073] If the association mode is internal association, the association result is (key1+"~"+key3 or key2+"~"+key4, "L~"+value1+"|"+"R~"+value2).

[0074] If the association mode is full association, the association results are (key1+"~"+key3 or key2+"~"+key4, "L~"+value1+"|"+"R~"+value2), (key1+"~"+key3, "L~"+value1+"|"+null) and (key2+"~"+key4, null+"|"+"R~"+value2).

[0075] As an optional embodiment, the association condition also includes: the content of the fifth key field in the second data set is equal to the content of the sixth key field in the third data set; method 100 also includes: determining a fourth key-value pair group from the association result, the key-value pairs in the fourth key-value pair group include the content of the fifth key field and the third data corresponding to the fifth key field in the association result, and the third data has a third tag; determining a fifth key-value pair group from the third data set, the key-value pairs in the fifth key-value pair group include the content of the sixth key field and the fourth data corresponding to the sixth key field in the fifth data set, and the fourth data has a fourth tag; merging the key-value pairs in the fourth key-value pair group and the fifth key-value pair group that meet the association condition to obtain a sixth key-value pair group, the sixth key-value pair group includes the content of the fifth key field, the content of the sixth key field, the third data and the fourth data, the content of the fifth key field and the third data, or the content of the sixth key field and the fourth data; based on the association method, the third tag and the fourth tag, obtaining an association result from the sixth key-value pair group.

[0076] Specifically, the embodiment of the present application includes two association conditions: the first association condition is that the content of the first key field in the first data set is equal to the content of the second key field in the second data set, and the second association condition is that the content of the fifth key field in the second data set is equal to the content of the sixth key field in the third data set.

[0077] Based on the association condition, the first data set and the second data set may be associated based on the first association condition to obtain an association result, and then the association result and the third data set may be associated based on the second association condition.

[0078] Exemplarily, the association result obtained based on the first association condition is (key2, null+“|”+value2), and the association result is used to generate a fourth key-value pair group through the Map process, and the fourth key-value pair in the fourth key-value pair group can be (key4, “T~”+value4), where key4 is the content of the fifth key field, value4 is the third data corresponding to the fifth key field in the association result, and “T~” is the third tag of the third data. The third data set is generated into a fifth key-value pair group through the Map process, and the fifth key-value pair in the fifth key-value pair group can be (key5, “F~”+value5), where key5 is the content of the sixth key field, value5 is the fourth data corresponding to the sixth key field in the fifth data set, and “F~” is the fourth tag of the fourth data. The fourth key-value pair and the fifth key-value pair are merged through the Reduce process to obtain a sixth key-value pair group, in which the sixth key-value pair can be (key4 or key5, "T~" + value4 + "|" + "F~" + value5), (key4, "T~" + value4 + "|" + null), (key5, null + "|" + "F~" + value5). Based on the association mode, the third tag and the fourth tag, the association result is obtained from the sixth key-value pair group. If the association mode is left association, the association results are (key4, "T~" + value4 + null) and (key4 or key5, "T~" + value4 + "|" + "F~" + value5). If the association mode is right association, the association results are (key5, null + "|" + "F~" + value5) and (key4 or key5, "T~" + value4 + "|" + "F~" + value5). If the association mode is internal association, the association result is (key4 or key5, "T~" + value4 + "|" + "F~" + value5). If the association mode is full association, the association result is (key4 or key5, "T~" + value4 + "|" + "F~" + value5), (key4, "T~" + value4 + "|" + null), (key5, null + "|" + "F~" + value5).

[0079] Optionally, if there is a filtering condition in the third data set, the data in the third data set may be filtered according to the filtering condition before determining the fifth key-value pair.

[0080] Optionally, based on the association condition, if there are multiple key fields in a data set, when the data set generates key-value pairs, tags can be added to the multiple key fields to distinguish them.

[0081] Exemplarily, the first data set includes four fields, f1, f2, jl and jk, and the second data set includes four fields, f3, f4, jr and jh. The first data set and the second data set are associated based on an association condition jl=jr or jk=jh.

[0082] The first data set is generated into a first key-value pair group through the Map process, in which the first key-value pair may include ("S1~"+jl, "L~"+f1+"~"+f2+"~"+jk), ("S2~"+jk, "L~"+f1+"~"+f2+"~"+jl), and ("S3~"+jl+"~"+jk, "L~"+f1+"~"+f2).

[0083] The second data set is generated into a second key-value pair group through the Map process, and the second key-value pairs in the second key-value pair group may include ("S1~"+jr, "R~"+f3+"~"+f4+"~"+jh), ("S2~"+jh, "R~"+f3+"~"+f4+"~"+jr), ("S3~"+jr+"~"+jh, "R~"+f3+"~"+f4).

[0084] Among them, "S1~", "S2~" and "S3~" are used to mark different key fields to facilitate subsequent merging and obtaining associated results.

[0085] In addition, the association calculation can also be decomposed into three association conditions: jl=jr, jk=jh, jl=jr and jk=jh for association calculation.

[0086] Combined with the above Figure 1 , describes the method of the embodiment of the present application in detail, and will be combined with Figure 2 and Figure 3 , describe in detail the device of the embodiment of the present application.

[0087] Figure 2 A data processing device 200 provided in an embodiment of the present application is shown. The device 200 includes: a determination module 210 and an acquisition module 220.

[0088] The determination module 210 is used to: determine a first key-value pair group from a first data set based on an association condition, wherein a first key-value pair of the first key-value group includes content of a first key field in the first data set and first data corresponding to the first key field in the first data set, the first data having a first tag, and the association condition includes that the content of the first key field is equal to content of a second key field in a second data set; determine a second key-value pair group from a second data set based on the association condition, wherein a second key-value pair in the second key-value pair group includes content of the second key field and second data corresponding to the second key field in the second data set, the second data having a second tag;

[0089] The acquisition module 220 is used to: merge the key-value pairs in the first key-value pair group and the second key-value pair group that meet the association condition to obtain a third key-value pair group, wherein the third key-value pair in the third key-value pair group includes at least one of the following: the content of the first key field, the first data and the second data, the content of the first key field and the first data, or the content of the second key field and the second data; based on the association method, the first tag and the second tag, obtain the association result from the third key-value pair group.

[0090] Optionally, in the third key-value pair, a first separator is included between the first data and the second data, and a second separator is included between the first data and different data in the second data.

[0091] Optionally, the association condition also includes: the content of the third key field in the first data set is equal to the content of the fourth key field in the second data set; the first key-value pair also includes the content of the third key field, and the content of the third key field is connected to the content of the first key field through a third delimiter; the second key-value pair also includes the content of the fourth key field, and the content of the fourth key field is connected to the content of the second key field through a third delimiter.

[0092] Optionally, the association condition also includes: the content of the fifth key field in the second data set is equal to the content of the sixth key field in the third data set; the determination module 210 is specifically used to: determine a fourth key-value pair group from the association result, the key-value pairs in the fourth key-value pair group include the content of the fifth key field and the third data corresponding to the fifth key field in the association result, and the third data has a third tag; determine a fifth key-value pair group from the third data set, the key-value pairs in the fifth key-value pair group include the content of the sixth key field and the fourth data corresponding to the sixth key field in the fifth data set, and the fourth data has a fourth tag; merge the key-value pairs in the fourth key-value pair group and the fifth key-value pair group that meet the association condition to obtain a sixth key-value pair group, the sixth key-value pair group includes the content of the fifth key field, the content of the sixth key field, the third data and the fourth data, the content of the fifth key field and the third data, or the content of the sixth key field and the fourth data; based on the association method, the third tag and the fourth tag, obtain the association result from the sixth key-value pair group.

[0093] Optionally, the data included in the first data set and / or the second data set is data filtered according to a filtering condition.

[0094] Optionally, the association mode is left association, right association, inner association, or full association.

[0095] It should be understood that the device here is embodied in the form of a functional module. The term "module" here may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a proprietary processor or a group processor, etc.) and a memory for executing one or more software or firmware programs, a merged logic circuit and / or other suitable components that support the described functions. In an optional example, those skilled in the art can understand that the device can be specifically a data processing device in the above-mentioned embodiment, or the function of the data processing device in the above-mentioned embodiment can be integrated in the device, and the device can be used to execute the various processes and / or steps corresponding to the data processing device in the above-mentioned method embodiment. To avoid repetition, it will not be repeated here.

[0096] The above-mentioned device has the function of implementing the corresponding steps executed by the data processing device in the above-mentioned method 100; the above-mentioned function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned function. For example, the above-mentioned acquisition module can be a communication interface, such as a transceiver interface.

[0097] Figure 3A data processing device 300 provided in an embodiment of the present application is shown. The device 300 includes: a processor 310, a communication interface 320 and a memory 330. The processor, the communication interface and the memory communicate with each other through an internal connection path, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to control the communication interface to send signals and / or receive signals.

[0098] The above-mentioned device is used to execute each process and step in the above-mentioned data processing method. The processor 310 is used to: determine a first key-value pair group from a first data set based on an association condition, wherein the first key-value pair of the first key-value group includes the content of a first key field in the first data set and first data corresponding to the first key field in the first data set, the first data having a first tag, and the association condition including that the content of the first key field is equal to the content of a second key field in the second data set; determine a second key-value pair group from a second data set based on the association condition, wherein the second key-value pair in the second key-value pair group includes the content of the second key field and second data corresponding to the second key field in the second data set, the second data having a second tag; merge the key-value pairs in the first key-value pair group and the second key-value pair group that meet the association condition to obtain a third key-value pair group, wherein the third key-value pair in the third key-value pair group includes at least one of the following: the content of the first key field, the first data and the second data, the content of the first key field and the first data, or the content of the second key field and the second data; obtain an association result from the third key-value pair group based on the association method, the first tag and the second tag.

[0099] It should be understood that the device 300 can be used to execute the various steps and / or processes corresponding to the data processing device in the above method embodiment. Optionally, the memory 330 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the type of device. The processor 310 may be used to execute instructions stored in the memory, and when the processor 310 executes instructions stored in the memory, the processor 310 is used to execute the various steps and / or processes of the above method embodiment corresponding to the data processing device. The communication interface 320 may include a transmitter and a receiver, the transmitter may be used to implement the various steps and / or processes corresponding to the above communication interface for performing the sending action, and the receiver may be used to implement the various steps and / or processes corresponding to the above communication interface for performing the receiving action.

[0100] It should be understood that in the embodiments of the present application, the processor of the above-mentioned device may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0101] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software units in a processor for execution. The software unit can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0102] The present application provides a computer-readable storage medium, which is used to store a computer program. The computer program is used to implement the data processing method shown in various possible implementations of the above embodiments.

[0103] The present application provides a chip system, which is used to support the above-mentioned data processing method shown in various possible implementations of the embodiments.

[0104] The present application provides a computer program product, which includes a computer program (also referred to as code or instruction). When the computer program runs on a computer, the computer can execute various possible data processing methods in the above embodiments.

[0105] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0107] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0110] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0111] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for data processing, characterized in that: include: Based on the association condition, determining a first key-value pair group from the first data set, wherein a first key-value pair in the first key-value pair group includes content of a first key field in the first data set and first data corresponding to the first key field in the first data set, the first data having a first tag, and the association condition including that the content of the first key field is equal to the content of a second key field in the second data set; Based on the association condition, determine a second key-value pair group from the second data set, where a second key-value pair in the second key-value pair group includes content of the second key field and second data corresponding to the second key field in the second data set, where the second data has a second tag; Merging the key-value pairs that meet the association condition in the first key-value pair group and the second key-value pair group to obtain a third key-value pair group, wherein the third key-value pair in the third key-value pair group includes at least one of the following: the content of the first key field, the first data, and the second data, the content of the first key field and the first data, or the content of the second key field and the second data; Obtaining an association result from the third key-value pair based on the association mode, the first tag, and the second tag; In the third key-value pair, a first separator is included between the first data and the second data, and a second separator is included between different data in the first data and the second data.

2. The method according to claim 1, characterized in that: The association condition also includes: the content of the third key field in the first data set is equal to the content of the fourth key field in the second data set; The first key-value pair also includes the content of the third key field, and the content of the third key field is connected with the content of the first key field through a third separator; The second key-value pair also includes the content of the fourth key field, and the content of the fourth key field is connected with the content of the second key field through the third separator.

3. The method according to claim 1, characterized in that The association condition also includes: the content of the fifth key field in the second data set is equal to the content of the sixth key field in the third data set; The method further comprises: Determine a fourth key-value pair group from the association result, wherein the key-value pairs in the fourth key-value pair group include the content of the fifth key field and third data corresponding to the fifth key field in the association result, and the third data has a third tag; Determine a fifth key-value pair group from the third data set, wherein the key-value pairs in the fifth key-value pair group include the content of the sixth key field and fourth data corresponding to the sixth key field in the fifth data set, and the fourth data has a fourth tag; Merging the key-value pairs that meet the association condition in the fourth key-value pair group and the fifth key-value pair group to obtain a sixth key-value pair group, wherein the key-value pairs in the sixth key-value pair group include at least one of the following: the content of the fifth key field, the third data, and the fourth data, the content of the fifth key field and the third data, or the content of the sixth key field and the fourth data; Based on the association mode, the third tag and the fourth tag, an association result is obtained from the sixth key-value pair group.

4. The method according to claim 1, characterized in that: The data included in the first data set and / or the second data set is data filtered according to the filtering condition.

5. The method according to claim 1, characterized in that The association mode is left association, right association, inner association, or full association.

6. A data processing device, characterized in that: include: a determining module, configured to determine a first key-value pair group from a first data set based on an association condition, wherein a first key-value pair in the first key-value pair group includes content of a first key field in the first data set and first data corresponding to the first key field in the first data set, the first data having a first tag, and the association condition including that the content of the first key field is equal to the content of a second key field in a second data set; Based on the association condition, determine a second key-value pair group from the second data set, where a second key-value pair in the second key-value pair group includes content of the second key field and second data corresponding to the second key field in the second data set, where the second data has a second tag; an acquisition module, configured to merge the key-value pairs satisfying the association condition in the first key-value pair group and the second key-value pair group to obtain a third key-value pair group, wherein the third key-value pair in the third key-value pair group includes at least one of the following: the content of the first key field, the content of the second key field, the first data and the second data, the content of the first key field and the first data, or the content of the second key field and the second data; Obtaining an association result from the third key-value pair group based on the association mode, the first tag, and the second tag; In the third key-value pair, a first separator is included between the first data and the second data, and a second separator is included between different data in the first data and the second data.

7. A data processing device, characterized in that: include: A processor, wherein the processor is coupled to a memory, wherein the memory is used to store a computer program, and when the processor calls the computer program, the device executes the method according to any one of claims 1 to 5.

8. A chip system, characterized in that: It comprises: a processor, used to call and run a computer program from a memory, so that a device equipped with the chip system executes the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images