Data collision processing method and system and electronic equipment

Abstract

The invention provides a data collision processing method and system and electronic equipment, and relates to the technical field of data collision process.The method comprises the steps that firstly, configuration information of a data source during data collision is determined based on attribute information of the data source; key data fields in the data source are obtained, and field relation information of the key data fields is determined based on the configuration information; determining a corresponding collision main body table and a to-be-collided data table in the data source according to the field relation information, and determining a collision rule between the collision main body table and the to-be-collided data table based on the configuration information; then determining to-be-collided data corresponding to the collision main body table and to-be-collided fields corresponding to the collision data based on a collision rule; and finally, controlling the data table to be collided and the collision main body table to perform data collision by utilizing a collision rule to obtain a collision result of the data source. According to the method, data mining can be carried out on the data sources of various sources to obtain the corresponding collision rules, the collision rules are automatically utilized to complete the data collision process, and the processing logic of data collision is greatly simplified.

Description

Technical Field

[0001] The present invention relates to the technical field of data collision processing, and in particular to a data collision processing method, system and electronic equipment. Background Art

[0002] With the rapid development of big data, data construction and storage methods have gradually diversified, making data integration and processing increasingly difficult. In the process of data collision processing, operation and maintenance personnel are often required to first collect the data into a designated database, and then query the data one by one through database query, and then process the query results. This process will waste a lot of storage space locally, and specific management of the received data is required. When processing data with high timeliness, the data synchronization mechanism must also be considered. The increase in these processing links will inevitably lead to an increase in the probability of errors and a waste of resources. At the same time, when the data structure or mechanism of the two parties of the data collision changes, the operation and maintenance personnel of both parties are also required to participate in the adjustment at the same time, which increases the complexity of later operation and maintenance. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a data collision processing method, system and electronic device, which can perform data mining on data sources from multiple sources to obtain corresponding collision rules, and automatically use the collision rules to complete the data collision process, thereby greatly simplifying the processing logic of data collision, reducing the complexity of operation and maintenance, and reducing the workload of operation and maintenance personnel.

[0004] In a first aspect, an embodiment of the present invention provides a data collision processing method, the method comprising:

[0005] Determine the configuration information of the data source when performing data collision based on the attribute information of the data source;

[0006] Obtain key data fields in the data source, and determine field relationship information of the key data fields based on the configuration information;

[0007] Determine the corresponding collision subject table and to-be-collided data table in the data source according to the field relationship information, and determine the collision rule between the collision subject table and to-be-collided data table based on the configuration information;

[0008] Determine the to-be-collided data corresponding to the collision subject table based on the collision rule, and determine the to-be-collided fields corresponding to the to-be-collided data based on the collision rule;

[0009] By using the collision rules, the data table to be collided is controlled to collide with the collision subject table according to the fields to be collided, and then the collision result of the data source is obtained.

[0010] Optionally, the configuration information for the data source when performing data collision is determined based on the attribute information of the data source, including:

[0011] Determine the configured local data source and external data source according to the attribute information of the data source, and obtain the data table information and structure information corresponding to the local data source and the external data source;

[0012] Determine the configured interface information and file information according to the attribute information of the data source, and obtain the interface configuration information corresponding to the interface information and the file configuration information corresponding to the file information;

[0013] Based on the data table information and its corresponding structure information, the data source configuration information corresponding to the local data source and the external data source is determined, and the configuration information of the data source when data collision occurs is determined using the interface configuration information and the file configuration information.

[0014] Optionally, key data fields in the data source are obtained, and field relationship information of the key data fields is determined based on the configuration information, including:

[0015] Get the configuration file corresponding to the data source and read the key data fields contained in the configuration file;

[0016] Get the configuration parameters corresponding to the key data fields in the configuration file, and determine the field relationship information corresponding to the configuration parameters based on the configuration information.

[0017] Optionally, key data fields in the data source are obtained, and field relationship information of the key data fields is determined based on the configuration information, including:

[0018] Get the configuration file corresponding to the data source, and read the associated policy field contained in the configuration file;

[0019] Obtain the association policy corresponding to the association policy field in the configuration file, and obtain the corresponding key data field based on the association policy;

[0020] Determine the field relationship information corresponding to the key data fields based on the configuration information.

[0021] Optionally, determining the corresponding collision subject table and to-be-collided data table in the data source according to the field relationship information, and determining the collision rule between the collision subject table and the to-be-collided data table based on the configuration information includes:

[0022] Determine the corresponding collision subject table in the data source according to the corresponding collision subject parameters and clue table parameters in the field relationship information;

[0023] Determine the corresponding data table to be collided in the data source according to the corresponding data source information and the collision field parameters in the field relationship information;

[0024] The collision rule is determined according to the corresponding collision rule parameters and matching parameters in the field relationship information.

[0025] Optionally, determining the to-be-collided data corresponding to the collision subject table based on the collision rule, and determining the to-be-collided fields corresponding to the to-be-collided data based on the collision rule, includes:

[0026] Determine the collision configuration parameters corresponding to the collision subject table based on the collision rules, and obtain the data to be collided from the collision subject table according to the collision configuration parameters;

[0027] The fields contained in the data to be collided are obtained, and the fields to be collided corresponding to the fields are determined based on the collision rules.

[0028] Optionally, obtaining fields contained in the data to be collided, and determining the fields to be collided corresponding to the fields based on the collision rule, including:

[0029] Traverse the fields contained in the data to be collided and obtain the field data corresponding to the fields;

[0030] The non-empty data in the field data is used to verify the collision rules, the non-empty data that meets the verification is obtained, and the field to be collided is determined using the field corresponding to the non-empty data.

[0031] Optionally, using the collision rule, the collision result of the data source is obtained by controlling the data table to be collided to collide with the collision subject table according to the field to be collided, including:

[0032] Get the data volume of the data table to be collided;

[0033] If the data volume is greater than a preset quantity threshold, paging the to-be-collided data table based on the data volume to obtain multiple pages corresponding to the to-be-collided data table;

[0034] After cyclically reading multiple pages according to the collision rules, the fields to be collided are obtained, and the multiple pages are controlled to collide with the collision subject table according to the fields to be collided in turn to obtain the corresponding collision data;

[0035] After the collision data is named and processed according to the data source, the collision result is obtained.

[0036] In a second aspect, an embodiment of the present invention provides a data collision processing system, the system comprising:

[0037] A configuration information determination module, used to determine the configuration information of the data source when performing data collision based on the attribute information of the data source;

[0038] A field relationship information determination module, used to obtain key data fields in a data source and determine field relationship information of the key data fields based on configuration information;

[0039] A collision rule determination module is used to determine the corresponding collision subject table and to-be-collided data table in the data source according to the field relationship information, and to determine the collision rule between the collision subject table and to-be-collided data table based on the configuration information;

[0040] A data collision setting module, used to determine the to-be-collided data corresponding to the collision subject table based on the collision rule, and to determine the to-be-collided fields corresponding to the to-be-collided data based on the collision rule;

[0041] The data collision execution module is used to use the collision rules to control the data table to be collided to perform data collision with the collision subject table according to the fields to be collided to obtain the collision result of the data source.

[0042] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the data collision processing method provided in the first aspect.

[0043] In a fourth aspect, an embodiment of the present invention further provides a storage medium storing computer executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions prompt the processor to implement the steps of the data collision processing method provided in the first aspect.

[0044] The embodiment of the present invention provides a data collision processing method, system and electronic device. In the process of data collision, the method first determines the configuration information of the data source when performing data collision based on the attribute information of the data source; then obtains the key data field in the data source, and determines the field relationship information of the key data field based on the configuration information; then determines the corresponding collision subject table and the data table to be collided in the data source according to the field relationship information, and determines the collision rule between the collision subject table and the data table to be collided based on the configuration information; then determines the data to be collided corresponding to the collision subject table based on the collision rule, and determines the field to be collided corresponding to the data to be collided based on the collision rule; finally, the collision result of the data source is obtained after the data table to be collided with the collision subject table according to the field to be collided is controlled by the collision rule. The method can perform data mining on data sources of multiple sources to obtain corresponding collision rules, and automatically use the collision rules to complete the data collision process, thereby greatly simplifying the processing logic of data collision, reducing the complexity of operation and maintenance, and reducing the workload of operation and maintenance personnel.

[0045] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of a data collision processing method provided by an embodiment of the present invention;

[0049] Figure 2 A flowchart of step S101 in a data collision processing method provided by an embodiment of the present invention;

[0050] Figure 3 A flowchart of step S102 in a data collision processing method provided by an embodiment of the present invention;

[0051] Figure 4 A flowchart of step S102 in another data collision processing method provided by an embodiment of the present invention;

[0052] Figure 5 A flowchart of step S103 in a data collision processing method provided by an embodiment of the present invention;

[0053] Figure 6 A flowchart of step S104 in a data collision processing method provided by an embodiment of the present invention;

[0054] Figure 7 A flowchart of step S602 in a data collision processing method provided by an embodiment of the present invention;

[0055] Figure 8 A flowchart of step S105 in a data collision processing method provided by an embodiment of the present invention;

[0056] Fig. 9 A flowchart of another data collision processing method provided by an embodiment of the present invention;

[0057] Fig.10 A schematic diagram of the structure of a data collision processing system provided by an embodiment of the present invention;

[0058] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0059] icon:

[0060] 1010-configuration information determination module; 1020-field relationship information determination module; 1030-collision rule determination module; 1040-data collision setting module; 1050-data collision execution module;

[0061] 101 - processor; 102 - memory; 103 - bus; 104 - communication interface. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] With the rapid development of big data, data construction methods and storage methods have gradually diversified, making data integration and processing increasingly difficult. In the process of data collision processing, it is often necessary for operation and maintenance personnel to first collect the data into a designated database, and then query one by one through database query, and then process the query results. This process will waste a lot of storage space locally, and it is also necessary to perform specific management on the received data. When it is necessary to process data with high timeliness, it is also necessary to consider the data synchronization mechanism. The increase in these processing links will inevitably lead to an increase in the probability of errors, and also cause a waste of resources. At the same time, when the data structure or mechanism of both parties of the data collision changes, it is also necessary for the operation and maintenance personnel of both parties to participate in the adjustment at the same time, which increases the complexity of later operation and maintenance. Based on this, the present invention implements a data collision processing method, system and electronic device, which can perform data mining on data sources from multiple sources to obtain corresponding collision rules, and automatically use the collision rules to complete the data collision process, thereby greatly simplifying the processing logic of data collision, reducing the complexity of operation and maintenance, and reducing the workload of operation and maintenance personnel.

[0064] To facilitate understanding of this embodiment, a data collision processing method disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following steps:

[0065] Step S101, determining configuration information of a data source when performing data collision based on attribute information of the data source;

[0066] Step S102, acquiring key data fields in the data source, and determining field relationship information of the key data fields based on the configuration information;

[0067] Step S103, determining the corresponding collision subject table and to-be-collided data table in the data source according to the field relationship information, and determining the collision rule between the collision subject table and the to-be-collided data table based on the configuration information;

[0068] Step S104, determining the to-be-collided data corresponding to the collision subject table based on the collision rule, and determining the to-be-collided fields corresponding to the to-be-collided data based on the collision rule;

[0069] Step S105 , using the collision rule, controlling the to-be-collided data table to perform data collision with the collision subject table according to the to-be-collided fields to obtain a collision result of the data source.

[0070] Specifically, after obtaining the data source, the method uses its corresponding attribute information to configure the data collision process, thereby completing the process of configuring the data source information; then the data source is parsed to obtain the key data fields contained therein, and the field relationship information corresponding to the key data fields is determined based on the aforementioned configuration information, thereby realizing the process of configuring the data source's data relationship.

[0071] By using the field relationship information to determine the corresponding collision subject table and the data table to be collided in the data source, the collision subject table and the data table to be collided are configured based on the configuration information, and finally the corresponding collision rules between the two are obtained. After the collision rule is obtained, the rule can be used to determine the data to be collided corresponding to the collision subject table, and at the same time determine the field to be collided corresponding to the data to be collided. Finally, the collision rule is used to control the data table to be collided and the collision subject table to collide according to the field to be collided, so as to obtain the collision result corresponding to the data source.

[0072] Optionally, step S101 of determining the configuration information of the data source when performing data collision based on the attribute information of the data source is as follows: Figure 2 As shown, including:

[0073] Step S201, determining the configured local data source and external data source according to the attribute information of the data source, and obtaining the data table information and structure information corresponding to the local data source and the external data source;

[0074] Step S202, determining the configured interface information and file information according to the attribute information of the data source, and obtaining the interface configuration information corresponding to the interface information and the file configuration information corresponding to the file information;

[0075] Step S203, based on the data table information and its corresponding structure information, determine the data source configuration information corresponding to the local data source and the external data source, and use the interface configuration information and the file configuration information to determine the configuration information of the data source when data collision occurs.

[0076] The process of obtaining configuration information mainly targets local data sources and external data sources. After the local data sources and external data sources that have been configured are determined through the attribute information of the data source, the corresponding data table information and structure information of the two can be obtained. Then, the interface information and file information are determined according to the attribute information of the data source, thereby realizing the configuration process of the interface and file. Through the data table information involved in the above process and its corresponding structure information, the configuration information of the data source is obtained, and on the basis of the interface configuration information and file configuration information, the configuration information involved in the data collision process of the data source is determined.

[0077] Optionally, step S102 of obtaining key data fields in the data source and determining field relationship information of the key data fields based on the configuration information is as follows: Figure 3 As shown, including:

[0078] Step S301, obtaining a configuration file corresponding to a data source, and reading key data fields contained in the configuration file;

[0079] Step S302: Obtain configuration parameters corresponding to key data fields in the configuration file, and determine field relationship information corresponding to the configuration parameters based on the configuration information.

[0080] There are two types of field key information, one involving configuration parameters in the configuration file and the other not involving configuration parameters. When configuration parameters are involved, first obtain the configuration file corresponding to the data source, which contains specific key data fields. After reading the key data fields and obtaining the configuration parameters corresponding to the fields, the field relationship information corresponding to the configuration parameters can be obtained based on the configuration information.

[0081] When no configuration parameters are involved, step S102 is as follows: Figure 4 As shown, including:

[0082] Step S401, obtaining a configuration file corresponding to the data source, and reading an associated policy field contained in the configuration file;

[0083] Step S402, obtaining the association policy corresponding to the association policy field in the configuration file, and obtaining the corresponding key data field based on the association policy;

[0084] Step S403: determining field relationship information corresponding to the key data field based on the configuration information.

[0085] At this time, the configuration file does not involve key data fields, but contains key policy fields. By reading the associated policies corresponding to the key policy fields, the corresponding key data fields are obtained, and finally the field relationship information corresponding to the key data fields is determined based on the configuration information.

[0086] Optionally, the step S103 of determining the corresponding collision subject table and the to-be-collided data table in the data source according to the field relationship information, and determining the collision rule between the collision subject table and the to-be-collided data table based on the configuration information, such as Figure 5 As shown, including:

[0087] Step S501, determining the corresponding collision subject table in the data source according to the corresponding collision subject parameters and clue table parameters in the field relationship information;

[0088] Step S502, determining the corresponding to-be-collided data table in the data source according to the corresponding data source information and the collision field parameters in the field relationship information;

[0089] Step S503: determining a collision rule according to the corresponding collision rule parameters and matching parameters in the field relationship information.

[0090] In the process of obtaining collision rules, the collision subject parameters and clue table parameters contained in the field relationship information are first obtained, and the collision subject table in the data source is determined using the above parameters. Then the data source information and collision field parameters contained in the field relationship information are obtained, and the corresponding data table to be collided in the data source is determined using the above information parameters. After the collision subject table and the data table to be collided are obtained, the collision rules of the two are determined using the corresponding collision rule parameters and matching parameters in the field relationship information. The collision rules may involve data collision processing strategies such as intersection, union, and difference, while the matching process involves complete matching, fuzzy matching, etc.

[0091] Optionally, step S104 of determining the to-be-collided data corresponding to the collision subject table based on the collision rule, and determining the to-be-collided fields corresponding to the to-be-collided data based on the collision rule, such as Figure 6 As shown, including:

[0092] Step S601, determining the collision configuration parameters corresponding to the collision subject table based on the collision rule, and obtaining the to-be-collided data from the collision subject table according to the collision configuration parameters;

[0093] Step S602: Acquire fields included in the to-be-collided data, and determine the to-be-collided field corresponding to the field based on the collision rule.

[0094] After the collision rules are obtained, the fields to be collided corresponding to the data to be collided can be determined, thereby guiding the collision execution position of the collision subject table. Specifically, the collision rules are first used to determine the collision configuration parameters corresponding to the collision subject table, and then the parameters are used to obtain the data to be collided from the collision subject table. For example, the collision configuration parameters are to obtain the first 10,000 data in the collision subject table. After intercepting and processing the data, the first 10,000 data can be determined as the data to be collided. Subsequently, the field information contained in the data to be collided is obtained, and then the fields are processed using the collision rules to obtain the corresponding fields to be collided. Specifically, the fields contained in the data to be collided are obtained, and the step S602 of determining the fields to be collided corresponding to the fields based on the collision rules is as follows. Figure 7 As shown, including:

[0095] Step S701, traverse the fields contained in the data to be collided, and obtain the field data corresponding to the fields;

[0096] Step S702: Use the non-empty data in the field data to verify with the collision rule, obtain the non-empty data that meets the verification, and use the field corresponding to the non-empty data to determine the field to be collided.

[0097] In the process of obtaining the fields to be collided, the field data corresponding to each field is obtained by traversing all the fields contained in the data to be collided; then the non-empty data in these field data are obtained, and these data are verified using the collision rules. After obtaining the non-empty data that meets the verification, the corresponding field is determined as the field to be collided.

[0098] Optionally, using the collision rule, the collision table to be collided is controlled to perform data collision with the collision subject table according to the collision field to obtain the collision result of the data source in step S105, such as Figure 8 As shown, including:

[0099] Step S801, obtaining the data volume of the to-be-collided data table;

[0100] Step S802, if the data volume is greater than a preset quantity threshold, paging the to-be-collided data table based on the data volume to obtain a plurality of pages corresponding to the to-be-collided data table;

[0101] Step S803, after cyclically reading multiple pages according to the collision rule, a field to be collided is obtained, and after controlling multiple pages to collide data with the collision subject table in turn according to the field to be collided, corresponding collision data is obtained;

[0102] Step S804: After the collision data is named according to the data source, a collision result is obtained.

[0103] After the collision rules are obtained, the data table to be collided can be collided with the collision subject table according to the fields to be collided. In this process, when the amount of data in the data table to be collided is large, it can be paged, and the corresponding collision data can be obtained after the paging is looped. The obtained collision data can be named according to the name of the data source to obtain the final collision result.

[0104] like Fig. 9 A flowchart of another data collision processing method is shown in FIG. Fig. 9 The corresponding content in the first row mainly implements the process of configuring data sources, including local business library data sources, other heterogeneous data sources, interfaces provided by other manufacturers as agreed, system information of other manufacturers (crawler configuration information), and files in agreed formats stored locally / FTP / HDFS.

[0105] The content in the second row mainly realizes the relationship between the data in the configuration data source. This step is optional. If configured, the configuration information is used. If not configured, it is automatically associated according to the value of the data. It is not configured by default.

[0106] The content in the third row is the user operation process. Users can import clue information as the first data table (collision subject table), or use a table specified by the above data source as the first data table. After specifying multiple data source information and collision fields, collision rules (intersection, union, difference) (exact match, fuzzy match) to be collided, the collision task can be started.

[0107] The content in the fourth row is the process of pulling data into memory for processing based on user operation information and configuration information. According to the information input by the user, the data source type is determined, and the first 10,000 data (configurable) are obtained based on the data source type. Then verify whether there is relationship information between the configured data source data fields. If not, execute the steps in the fifth row; if so, directly execute the steps in the sixth row.

[0108] The content in the fifth row is used to verify the relationship between the configured data sources. If not configured, the rule verification is performed based on the first 10 non-empty data in the data. If the verification passes, it will automatically be used as the field to be collided. According to the rules of the clues, the first 10 non-empty values ​​of each data are traversed. If there is one that meets the rules, it is considered that the field can be used as the field to be collided for comparison.

[0109] The content in the sixth row is the process of collision according to the rules. The comparison is performed according to the matching rules based on the comparable fields. After the comparison is completed, if the paging is not completed, the corresponding processes of the fourth and sixth rows will continue to be executed.

[0110] The content in the seventh row is the result output process, which outputs the collision result, adjusts the name of the collision result to the data set name, and finally outputs it to the relevant mongo library.

[0111] It can be seen from the data collision processing method described in the above embodiment that the method can perform data mining on data sources from multiple sources to obtain corresponding collision rules, and automatically use the collision rules to complete the data collision process, thereby greatly simplifying the processing logic of data collision, reducing the complexity of operation and maintenance, and reducing the workload of operation and maintenance personnel.

[0112] Corresponding to the above data collision processing method embodiment, the present invention also provides a data collision processing system, such as Fig.10 As shown, the system includes:

[0113] A configuration information determination module 1010, used to determine configuration information when a data source performs data collision based on attribute information of the data source;

[0114] A field relationship information determination module 1020 is used to obtain key data fields in a data source and determine field relationship information of the key data fields based on configuration information;

[0115] The collision rule determination module 1030 is used to determine the corresponding collision subject table and the to-be-collided data table in the data source according to the field relationship information, and determine the collision rule between the collision subject table and the to-be-collided data table based on the configuration information;

[0116] The data collision setting module 1040 is used to determine the to-be-collided data corresponding to the collision subject table based on the collision rule, and determine the to-be-collided fields corresponding to the to-be-collided data based on the collision rule;

[0117] The data collision execution module 1050 is used to use the collision rule to control the to-be-collided data table to perform data collision with the collision subject table according to the to-be-collided fields to obtain the collision result of the data source.

[0118] From the data collision processing system mentioned in the above embodiment, it can be seen that the system is able to perform data mining on data sources from multiple sources to obtain corresponding collision rules, and automatically use the collision rules to complete the data collision process, thereby greatly simplifying the processing logic of data collision, reducing the complexity of operation and maintenance, and reducing the workload of operation and maintenance personnel.

[0119] The data collision processing system provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned data collision processing method embodiment. For the sake of brief description, for matters not mentioned in the method embodiment, reference may be made to the corresponding contents in the aforementioned data collision processing method embodiment.

[0120] This embodiment also provides an electronic device. The structural diagram of the electronic device is as follows: Fig.11 As shown, the device includes a processor 101 and a memory 102; wherein the memory 102 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the steps of the above-mentioned data collision processing method.

[0121] Fig.11 The electronic device shown further includes a bus 103 and a communication interface 104 , and the processor 101 , the communication interface 104 and the memory 102 are connected via the bus 103 .

[0122] The memory 102 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The bus 103 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0123] The communication interface 104 is used to connect to at least one user terminal and other network units through a network interface, and send the encapsulated IPv4 message or IPv4 message to the user terminal through the network interface.

[0124] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 102, and the processor 101 reads the information in the memory 102 and completes the steps of the method of the above embodiment in combination with its hardware.

[0125] An embodiment of the present invention further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data collision processing method in the above embodiment are executed.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0129] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0130] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data collision processing method, characterized in that: The method comprises: Determining configuration information of the data source when performing data collision based on attribute information of the data source; Acquire key data fields in the data source, and determine field relationship information of the key data fields based on the configuration information; Determine the corresponding collision subject table and to-be-collided data table in the data source according to the field relationship information, and determine the collision rule between the collision subject table and the to-be-collided data table based on the configuration information; Determine the to-be-collided data corresponding to the collision subject table based on the collision rule, and determine the to-be-collided fields corresponding to the to-be-collided data based on the collision rule; The collision rule is used to control the to-be-collided data table to perform data collision with the collision subject table according to the to-be-collided fields to obtain the collision result of the data source.

2. The data collision processing method according to claim 1, characterized in that: Determining configuration information of the data source when performing data collision based on attribute information of the data source includes: Determine the configured local data source and external data source according to the attribute information of the data source, and obtain data table information and structure information corresponding to the local data source and the external data source; Determine the configured interface information and file information according to the attribute information of the data source, and obtain the interface configuration information corresponding to the interface information and the file configuration information corresponding to the file information; Based on the data table information and the corresponding structure information, the data source configuration information corresponding to the local data source and the external data source is determined, and the configuration information of the data source when data collision occurs is determined using the interface configuration information and the file configuration information.

3. The data collision processing method according to claim 1, characterized in that: Acquiring key data fields in the data source, and determining field relationship information of the key data fields based on the configuration information, including: Obtaining a configuration file corresponding to the data source, and reading the key data field contained in the configuration file; The configuration parameters corresponding to the key data fields in the configuration file are obtained, and the field relationship information corresponding to the configuration parameters is determined based on the configuration information.

4. The data collision processing method according to claim 1, characterized in that: Acquiring key data fields in the data source, and determining field relationship information of the key data fields based on the configuration information, including: Obtain a configuration file corresponding to the data source, and read an associated policy field contained in the configuration file; Obtaining the association policy corresponding to the association policy field in the configuration file, and obtaining the corresponding key data field based on the association policy; The field relationship information corresponding to the key data field is determined based on the configuration information.

5. The data collision processing method according to claim 1, characterized in that: Determining the corresponding collision subject table and to-be-collided data table in the data source according to the field relationship information, and determining the collision rule between the collision subject table and the to-be-collided data table based on the configuration information, including: Determine the corresponding collision subject table in the data source according to the corresponding collision subject parameters and clue table parameters in the field relationship information; Determine the to-be-collided data table corresponding to the data source in the data source according to the corresponding data source information and the collision field parameter in the field relationship information; The collision rule is determined according to the corresponding collision rule parameters and matching parameters in the field relationship information.

6. The data collision processing method according to claim 1, characterized in that: Determining the to-be-collided data corresponding to the collision subject table based on the collision rule, and determining the to-be-collided fields corresponding to the to-be-collided data based on the collision rule, including: Determine the collision configuration parameters corresponding to the collision subject table based on the collision rule, and obtain the to-be-collided data from the collision subject table according to the collision configuration parameters; The fields included in the to-be-collided data are acquired, and the to-be-collided field corresponding to the field is determined based on the collision rule.

7. The data collision processing method according to claim 6, characterized in that: Acquiring the fields contained in the to-be-collided data, and determining the to-be-collided field corresponding to the field based on the collision rule, includes: Traversing the fields contained in the data to be collided, and acquiring field data corresponding to the fields; The non-empty data in the field data is used to perform verification with the collision rule to obtain the non-empty data that meets the verification, and the field to be collided is determined using the field corresponding to the non-empty data.

8. The data collision processing method according to claim 1, characterized in that: Using the collision rule, controlling the to-be-collided data table to perform data collision with the collision subject table according to the to-be-collided field to obtain the collision result of the data source, including: Obtaining the data volume of the to-be-collided data table; If the data volume is greater than a preset quantity threshold, paging the to-be-collided data table based on the data volume to obtain a plurality of pages corresponding to the to-be-collided data table; After cyclically reading the multiple pages according to the collision rule, the to-be-collided field is obtained, and after controlling the multiple pages to perform data collision with the collision subject table according to the to-be-collided field in sequence, corresponding collision data is obtained; After the collision data is named according to the data source, the collision result is obtained.

9. A data collision processing system, characterized in that: The system comprises: A configuration information determination module, used to determine the configuration information of the data source when data collision occurs based on the attribute information of the data source; A field relationship information determination module, used to obtain key data fields in the data source, and determine field relationship information of the key data fields based on the configuration information; A collision rule determination module, used to determine the corresponding collision subject table and to-be-collided data table in the data source according to the field relationship information, and determine the collision rule between the collision subject table and the to-be-collided data table based on the configuration information; A data collision setting module, used to determine the to-be-collided data corresponding to the collision subject table based on the collision rule, and to determine the to-be-collided fields corresponding to the to-be-collided data based on the collision rule; The data collision execution module is used to use the collision rule to control the to-be-collided data table to perform data collision with the collision subject table according to the to-be-collided fields to obtain the collision result of the data source.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the data collision processing method according to any one of claims 1 to 8.

Images