Data fusion method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a data fusion method and device, equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining the current business data of a current business system; filtering the current service data based on a preset filtering rule to obtain intermediate data, and adding stowage information for each intermediate data according to the data type of the intermediate data to obtain effective data corresponding to each intermediate data; and performing data processing on the effective data according to business requirements to obtain fused data corresponding to the current business data. According to the technical scheme, the current service data is fused to the target service system, and data fusion of the current service system and the target service system is further realized.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a data fusion method, device, equipment and storage medium. Background Art

[0002] In the context of business mergers and acquisitions, it is particularly important to integrate the current business data of the current business system and the target business data of the target business system. For example, in the logistics business, when integrating the logistics data of the first logistics system and the second logistics system, the resource reuse of the first logistics system and the second logistics system based on site sublease, line filling, air-rail cooperation and site physical integration is particularly important for achieving normalized resource reuse and mutual support in scenarios such as sudden natural disasters and special protection during the Spring Festival.

[0003] In the prior art, current business data of the current business system is simply sent to the target business system to achieve data fusion of the current business data of the current business system and the target business data of the target business system in the target business system.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following technical problems in the prior art:

[0005] The data formats of the current business data of the current business system and the target business of the target business system are quite different, and it is difficult to achieve resource reuse with existing technologies. Summary of the invention

[0006] The present invention provides a data fusion method, device, equipment and storage medium to achieve data fusion of a current business system into a target business system, and to achieve resource reuse of the current business system and the target business system.

[0007] In a first aspect, an embodiment of the present invention provides a data fusion method, including:

[0008] Get the current business data of the current business system;

[0009] After filtering the current business data based on the preset filtering rules to obtain intermediate data, adding loading information to each intermediate data according to the data type of the intermediate data to obtain effective data corresponding to each intermediate data;

[0010] Data processing is performed on each of the effective data according to business requirements to obtain fused data corresponding to the current business data.

[0011] In a second aspect, an embodiment of the present invention further provides a data fusion device, including:

[0012] The acquisition module is used to obtain the current business data of the current business system;

[0013] A filtering module, configured to add configuration information to each intermediate data according to the data type of the intermediate data after filtering the current business data based on a preset filtering rule to obtain the intermediate data, so as to obtain the effective data corresponding to each intermediate data;

[0014] The processing module is used to process the effective data according to business requirements to obtain fused data corresponding to the current business data.

[0015] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the data fusion method as described in any one of the first aspects.

[0019] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the data fusion method as described in any one of the first aspects.

[0020] The embodiments of the above invention have the following advantages or beneficial effects:

[0021] The embodiment of the present invention provides a data fusion method, which is characterized by comprising: obtaining the current business data of the current business system; filtering the current business data based on preset filtering rules to obtain intermediate data, adding loading information to each intermediate data according to the data type of the intermediate data, and obtaining effective data corresponding to each intermediate data; performing data processing on each effective data according to business requirements to obtain fused data corresponding to the current business data. In the above technical scheme, after the target business system obtains the current business data of the current business system, it can filter the current business data according to preset filtering rules to obtain the intermediate data required by the target business system, match the loading information for the intermediate data according to the data type of the intermediate data, obtain the effective data corresponding to the intermediate data, and realize the supplement of the loading information of the intermediate data; it can also perform data processing on the effective data according to business requirements to obtain the fused data corresponding to the current business data required by the target business system, realize the fusion of the current business data into the target business system, and further realize the data fusion of the current business system and the target business system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A flow chart of a data fusion method provided by an embodiment of the present invention;

[0023] Figure 2 A flowchart of another data fusion method provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of interaction between a current business system and a target business system in a data fusion method provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the structure of a data fusion device provided by an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0029] Figure 1 The flowchart of a data fusion method provided by an embodiment of the present invention is applicable to the situation where data fusion of the current business system and the target business system is required. The method can be executed by a data fusion device, which can be implemented by software and / or hardware. Figure 1 Said method specifically comprises the following steps:

[0030] Step 110: Obtain current business data of the current business system.

[0031] Among them, the current business system can be the first logistics system, the current business data can be the first logistics data, and in the scenario where the first logistics data of the first logistics system and the second logistics data of the second logistics system need to be merged, the second logistics system can be the target business system.

[0032] The current business system and the target business system can communicate based on the current gateway corresponding to the current business system and the target gateway corresponding to the target business system. In order to realize the reuse of resources between the current business system and the target business system, it is necessary to perform data fusion on the current business system and the target business system, that is, it is necessary to perform data fusion on the first logistics data of the first logistics system and the second logistics data of the second logistics system.

[0033] In practical applications, it can be understood that the current business data of the current business system needs to be integrated into the target business system, and data filtering is performed in the target business data of the target business system according to the filtering conditions sent by the current business system, and the filtered target data is sent to the current business system.

[0034] Specifically, the target business system can call the current business system based on the target gateway and the current gateway to obtain the current business data of the current business system. Specifically, after the target business system sends the data request to the current business system through the target gateway and the current gateway, the current business system can respond to the data request and send the current business data to the target business system through the current gateway and the target gateway. At this point, the target business system can perform data fusion processing on the acquired current business data.

[0035] In the embodiment of the present invention, the target business system obtains current business data of the current business system.

[0036] Step 120: After filtering the current business data based on the preset filtering rules to obtain intermediate data, add loading information to each intermediate data according to the data type of the intermediate data to obtain effective data corresponding to each intermediate data.

[0037] The preset filtering rules may at least include a preset originating node, a preset destination node and a preset load.

[0038] Specifically, in a logistics scenario where the current business system is the first logistics system and the target business system is the second logistics system, after the target business system obtains the current business data of the current business system, it can first determine the nodes that need to be supplemented with data according to actual needs, and use these nodes as preset originating nodes and / or preset destination nodes to initially screen the current business data to obtain transition data. At this time, the transition data is configured with initial loading information, which can include the originating destination node, the belonging network, the expiration time, the flight schedule information, etc. The transition data with the expiration time within the validity period can be determined as intermediate data to achieve filtering and screening of the current business data.

[0039] Intermediate data includes original data and supplementary data. Original data can be understood as data in the current business system, and supplementary data can be understood as fused data formed by merging the data of the target business system into the business system. When adding configuration information to intermediate data, you first need to determine whether the intermediate data is original data or supplementary data. When the intermediate data is determined to be original data, you can match the loading information for the intermediate data according to the product type of the original data to obtain the effective data corresponding to the intermediate data; when the intermediate data is determined to be supplementary data, you can parse the original loading information of the supplementary data to obtain the whitelist information and blacklist information corresponding to the supplementary data, and determine the whitelist information and blacklist information corresponding to the supplementary data as the loading information of the intermediate data to obtain the effective data corresponding to the intermediate data.

[0040] In an embodiment of the present invention, after the target business system obtains the current business data of the current business system, it filters the current business data through preset filtering rules to obtain the intermediate data required by the target business system, and then matches the loading information for the intermediate data according to the data type of the intermediate data to obtain the effective data corresponding to the intermediate data, thereby supplementing the loading information of the intermediate data.

[0041] Step 130: Process the effective data according to business requirements to obtain fused data corresponding to the current business data.

[0042] Among them, business needs can be understood as the data that the target business system needs to supplement and integrate.

[0043] After the effective data is determined, it is necessary to convert the effective data into integrated data that can flow in the target business system.

[0044] Specifically, if the effective data includes effective data corresponding to the supplementary data, data conversion and data supplementation can be performed on the effective data based on the standard line data format to obtain fused data corresponding to the current business data. If the effective data includes effective data corresponding to the original data and the supplementary data, data processing can be performed on the effective data according to the data change type and the standard line data format to obtain fused data corresponding to the current business data.

[0045] In actual applications, after receiving the data screening conditions sent by the current business system, the target business system can filter the target business data in the target business system according to the data screening conditions, obtain the target data, and send the target data to the current business system. Of course, the current business system can also process the target data according to business needs, integrate the target data into the current business system, and further integrate the data of the current business system and the target business system.

[0046] In an embodiment of the present invention, the target business system implements data processing on the effective data according to business requirements, obtains fused data corresponding to the current business data required by the target business system, integrates the current business data into the target business system, and further realizes data fusion of the current business system and the target business system.

[0047] The data fusion method provided by the embodiment of the present invention includes: obtaining the current business data of the current business system; filtering the current business data based on preset filtering rules to obtain intermediate data, adding loading information to each intermediate data according to the data type of the intermediate data, and obtaining effective data corresponding to each intermediate data; processing each effective data according to business requirements to obtain fused data corresponding to the current business data. In the above technical scheme, after the target business system obtains the current business data of the current business system, it can filter the current business data according to preset filtering rules to obtain the intermediate data required by the target business system, match the loading information for the intermediate data according to the data type of the intermediate data, obtain the effective data corresponding to the intermediate data, and supplement the loading information of the intermediate data. It can also process the effective data according to business requirements to obtain the fused data corresponding to the current business data required by the target business system, and integrate the current business data into the target business system, and further realize the data fusion of the current business system and the target business system.

[0048] Figure 2A flowchart of another data fusion method provided for an embodiment of the present invention. The embodiment of the present invention is applicable to situations where data fusion of the current business system and the target business system is required. Based on the above embodiments, after obtaining the effective data corresponding to each intermediate data, the embodiment of the present invention adds "supplementing the fourth-level address information of the loading information corresponding to each intermediate data to the effective data corresponding to each intermediate data." After parsing the business file, it adds "uploading the business file to the server after determining that the parsing result of the business file is a parsing failure." It also adds "filtering the data in the target business data according to the filtering conditions sent by the current business system to obtain the target data; sending the target data to the current business system." The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. See. Figure 2 , the data fusion method provided by the embodiment of the present invention includes:

[0049] Step 210: Acquire current business data of the current business system.

[0050] In one implementation, step 210 may specifically include:

[0051] After receiving the file upload information sent by the current business system, the business file corresponding to the file upload information is searched and downloaded in the server; the business file is parsed to obtain the current business data corresponding to the file upload information.

[0052] Figure 3 A schematic diagram of the interaction between the current business system and the target business system in a data fusion method provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the current business system and the target business system can communicate and transmit data through the current gateway and the target gateway.

[0053] Both the current gateway and the target gateway have data encryption functions. When the request sent by the current business system to the target business system contains sensitive information, the request needs to be encrypted based on the current gateway and sent to the target gateway. The target gateway can decrypt the encrypted request and send the decrypted request to the target business system. The target business system determines the data corresponding to the request and sends the data to the target gateway. The target gateway encrypts the data and sends the encrypted data to the current gateway. The current gateway decrypts the encrypted data and sends the encrypted data to the current business system, realizing encrypted transmission of requests and data between the current business system and the target business system. Similarly, encrypted transmission of requests and data between the target business system and the current business system can also be realized.

[0054] Due to the performance issues of the current gateway and the target gateway, when the amount of data transmitted between the gateways exceeds 3M, data transmission often times out. Therefore, data transmission between gateways can be performed in the form of files. Specifically, the current business system can write the current business data into business files, for example, the current business data can be written into CSV files (Comma-Separated Values, a simple and practical file format for storing and representing various types of data including text, numerical values, etc.), compress the business files and upload them to the server, for example, it can be uploaded to OSS (ObjectStorage Service, a massive, secure, low-cost, and highly reliable cloud storage service). When the current business system needs to merge 1 million pieces of current business data into the target business system, the data volume is 1G. It takes no more than 0.2s for the current business system to write 1G of current business data into a CSV file, compress it, and upload it to OSS, thereby improving data transmission efficiency.

[0055] After the current business system compresses the business file and uploads it to the server, it can send the file upload information containing the file name and file address to the target business system. After the target business system receives the file upload information sent by the current business system, it can search and download the corresponding business file in the server based on the file upload information. Furthermore, the target business system can parse the business file based on the asynchronous task to obtain the current business data. Specifically, the data in the business file can be read row by row, and data deduplication and data verification can be performed to obtain the current business data corresponding to the file upload information.

[0056] In addition, when the target business system obtains the current business data corresponding to the file upload information, if the target business system has not yet completed parsing the business file corresponding to the file upload information, that is, if the status of the asynchronous task corresponding to the business file corresponding to the file upload information is processing, the current business data corresponding to the file upload information can be obtained again after a preset time until the current business data corresponding to the file upload information is obtained.

[0057] Furthermore, after parsing the business file, the method further includes:

[0058] After determining that the parsing result of the business file is a parsing failure, the business file is uploaded to the server.

[0059] After parsing the business file, the target business system can verify the parsing result. If it is determined that any field in the parsing result is empty and / or the data format is wrong, the parsing result of the business file is determined to be a parsing failure. At this time, the business file can be uploaded to OSS to realize the backup of the business file. Of course, the parsing result of the business file can also be sent to the current business system.

[0060] In actual applications, after the target business system receives the business file, it also needs to perform a pre-verification on the business file. Specifically, it can verify whether the file exists, the file size, and the required value verification. When performing the required value verification, it can determine the version number, file type, file name, upload time, number of data items and other required information. When performing the pre-verification, the target business system can also generate the pre-verification task ID and pre-verification task version number corresponding to the business file. Specifically, the timestamp can be used as the pre-verification task version number, and the pre-verification task ID and pre-verification task version number corresponding to the business file are returned to the current business system.

[0061] When the current business system is the first logistics system and the target business system is the second logistics system, the second logistics system can obtain the first logistics data of the first logistics system.

[0062] In an embodiment of the present invention, after receiving the file upload information sent by the current business system, the target business system searches for and downloads the business file corresponding to the file upload information in the server to obtain the business file. After determining that the business file passes the pre-verification, the business file is parsed based on the asynchronous task processing method. After determining that the parsing result is successful, the current business data corresponding to the business file is obtained to obtain the current business data of the current business system.

[0063] Step 220: Filter the current business data based on preset filtering rules to obtain intermediate data.

[0064] In one implementation, the preset filtering rule at least includes a preset originating node, a preset destination node, and a preset load. Accordingly, step 220 may specifically include:

[0065] The current business data is initially filtered based on the preset originating node and the preset destination node to obtain transition data; and according to the preset loading, the transition data with a loading attribute of being in effect is determined as the intermediate data.

[0066] Specifically, after the target business system obtains the current business data of the current business system, it needs to filter the current business data based on preset filtering rules to obtain intermediate data that can be converted into corresponding data of the target business system.

[0067] When the current business system is the first logistics system and the target business system is the second logistics system, the preset filtering rules may at least include the preset originating node, the preset destination node, and the preset loading. Therefore, the nodes that need to be supplemented with data can be determined according to actual needs, and these nodes are used as the preset originating nodes and / or preset destination nodes to perform initial filtering on the current business data to obtain transition data. The transition data is configured with initial loading information, which may include the originating destination node, the belonging network, the expiration time, the flight schedule information, etc. Furthermore, the transition data with the expiration time within the validity period can be determined as intermediate data to achieve filtering and screening of the current business data.

[0068] In actual applications, the intermediate data can be written into the successful data set, and the other data in the current business data except the intermediate data can be written into the failed data set. The failed data set can also be uploaded to OSS.

[0069] In the embodiment of the present invention, after the target business system obtains the current business data of the current business system, it can filter the current business data based on preset filtering rules to obtain intermediate data, and determine the intermediate data required by the target business system through data filtering.

[0070] Step 230: Add loading information to each intermediate data according to the data type of the intermediate data to obtain the effective data corresponding to each intermediate data.

[0071] In one implementation, step 230 may specifically include:

[0072] When it is determined that the data type of the intermediate data is original data, the loading information is matched for the original data according to the product type of the intermediate data to obtain the effective data corresponding to the original data; when it is determined that the data type of the intermediate data is supplementary data, the supplementary data is parsed to obtain the whitelist information and blacklist information corresponding to the supplementary data, and the whitelist information and blacklist information corresponding to the supplementary data are used as the loading information of the supplementary data to obtain the effective data corresponding to the supplementary data.

[0073] Among them, the loading information is used to describe the cargo transportation capacity from the origin node to the destination node, including information such as product type, outlets, birth and expiration time, etc.

[0074] Specifically, the intermediate data written into the successful data set includes original data and supplementary data. For each intermediate data of the successful data set, when it is determined that the data type of the intermediate data is original data, the loading information can be matched for the original data according to the product type of the intermediate data to obtain the effective data corresponding to the original data. Specifically, the search can be performed in all the loading information of the target business system based on the originating node and destination node contained in the original data. If the originating node and destination node of any loading information of the target business system are consistent with the originating node and destination node contained in the original data, then the configuration information is determined as the loading information of the original data. If there is no originating node and destination node in the loading information of the target business system that are consistent with the originating node and destination node contained in the original data, then the configuration information in which the destination node in the loading information of the target business system is consistent with the destination node contained in the original data is determined as the loading information of the original data to obtain the effective data corresponding to the original data. When it is determined that the data type of the intermediate data is supplementary data, the original configuration information contained in the supplementary data can be parsed to obtain the blacklist information and whitelist information corresponding to the supplementary data. Specifically, the original configuration information in the form of "xxx#xxx" contained in the supplementary data can be split according to "#", and the information before "#" and the information after "#" can be respectively determined as the whitelist information and blacklist information corresponding to the supplementary data, and the whitelist information and blacklist information corresponding to the supplementary data can be determined as the loading information of the supplementary data to obtain the effective data corresponding to the supplementary data.

[0075] In an embodiment of the present invention, the loading information corresponding to the intermediate data can be determined according to the data type of the intermediate data, and the loading information corresponding to the intermediate data can be added to the intermediate data to obtain the effective data corresponding to the intermediate data, thereby supplementing the loading information of the intermediate data.

[0076] Step 240: Supplement the fourth-level address information of the loading information corresponding to each of the intermediate data to the effective data corresponding to each of the intermediate data.

[0077] The fourth-level address information of the loading information can be understood as the fourth-level address of the network point included in the configuration information, that is, the province, city, district and county to which the network point belongs.

[0078] Specifically, after adding the loading information corresponding to the intermediate data to the intermediate data and obtaining the effective data corresponding to the intermediate data, the fourth-level address information of the loading information corresponding to the intermediate data can also be determined, and the fourth-level address information of the loading information corresponding to the intermediate data can be supplemented to the effective data corresponding to the intermediate data, so as to further supplement the effective data corresponding to the intermediate data.

[0079] In actual applications, after obtaining the effective data, the loading information of the effective data can be checked. If the configuration information of the effective data is empty, the effective data is transferred to the failed data set. The uid, buinessId, task id, task version, mileage, departure time, origin and destination network code corresponding to the intermediate data can also be added to the effective data corresponding to the intermediate data.

[0080] In the embodiment of the present invention, by supplementing the fourth-level address information of the loading information corresponding to the intermediate data to the effective data corresponding to the intermediate data, further supplementation of the effective data corresponding to the intermediate data is achieved.

[0081] Step 250: Process the effective data according to business requirements to obtain fused data corresponding to the current business data.

[0082] In one implementation, step 250 may specifically include:

[0083] When it is determined that the effective data includes the effective data corresponding to the supplementary data, data conversion and data supplement are performed on the effective data based on the standard line data format to obtain the fused data corresponding to the current business data; when it is determined that the effective data includes the original data and the effective data corresponding to the supplementary data, data processing is performed on the effective data according to the data change type and the standard line data format to obtain the fused data corresponding to the current business data.

[0084] Specifically, when the effective data includes effective data corresponding to the supplementary data, the effective data can be converted and supplemented based on the standard line data format to obtain the fused data corresponding to the current business data. For example, the nodes contained in the effective data can be encoded based on the node encoding method of the target business system. When the effective data includes effective data corresponding to the original data and the supplementary data, the effective data can be processed according to the data change type and the standard line data format to obtain the fused data corresponding to the current business data. For example, when the data change type is data addition, the effective data can be converted and supplemented according to the standard line data format to obtain the fused data corresponding to the effective data; when the data change type is data update, the corresponding existing line data can be updated according to the effective data to obtain the fused data corresponding to the effective data; when the data change type is data deletion, the data corresponding to the effective data can be deleted in the target business data of the target business system, and the data expiration time of the target business data corresponding to the effective data can be set to 23:59:59 of the previous day; when the data change type is unchanged, the effective data can be converted and supplemented according to the standard line data format to obtain the fused data corresponding to the effective data.

[0085] It should be noted that when supplementing the effective data according to the standard line data format, the supplemented information includes: the four-level address of the outlet, the outlet type, the fence time, the network type, the plan type, the area type, the line type, the transportation method, the vehicle type, etc. After converting and supplementing the effective data according to the standard line data format and obtaining the fused data corresponding to the effective data, the fused data can be verified at the first level, specifically, the format, value logic, etc. of the fused data can be verified.

[0086] In an embodiment of the present invention, the target business system implements data processing on the effective data according to business requirements, obtains fused data corresponding to the current business data required by the target business system, integrates the current business data into the target business system, and further realizes data fusion of the current business system and the target business system.

[0087] Step 260: Filter the target business data according to the filtering conditions sent by the current business system to obtain target data; and send the target data to the current business system.

[0088] The data fusion of the target business system and the current business system requires not only the fusion of the data of the current business system into the target business system, but also the fusion of the data of the target business system into the current business system.

[0089] Among them, the screening condition is the data screening condition of the target business data of the target business system, which may include but is not limited to at least one of the originating node type, the destination node type, the node type blacklist and whitelist, and the node blacklist.

[0090] Specifically, after the target business system receives the data screening conditions sent by the current business system, it can screen and filter the target business data of the target business system according to the data screening conditions including but not limited to at least one of the originating node type, destination node type, node type blacklist and whitelist, and node blacklist to obtain the target data, and further send the target data to the current business system.

[0091] Of course, after the current business system receives the target data sent by the target business system, it can process the target data according to the standard data format corresponding to the current business system to process the target data into a format compatible with the current business system, thereby integrating the target data into the current business system and further realizing data integration of the current business system and the target business system.

[0092] In an embodiment of the present invention, target data required by the current business system is determined from the target business data of the target business system according to the screening conditions sent by the current business system. After the target data is sent to the current business system, the current business system can also process the target data according to business needs to process the target data into a format compatible with the current business system, thereby integrating the target business data into the current business system and further realizing data integration of the current business system and the target business system.

[0093] It should be noted that when the current business system is the first logistics system and the target business system is the second logistics system, in order to ensure business independence, when the first logistics data of the first logistics system and the second logistics data of the second logistics system are integrated, the terminal of the collection and dispatch is not integrated, and only the transportation and transshipment related data are integrated. In the port entry scenario, on the one hand, the transshipment node of the second logistics system can be shut down, and the business of the transshipment node of the second logistics system can be integrated into the transshipment node of the first logistics system. On the other hand, the intermediate node of the second logistics system can be integrated with the intermediate node of the first logistics system. On the other hand, the urban distribution business of the second logistics system can be integrated with the urban distribution business of the first logistics system. In the port departure scenario, on the one hand, the large-piece collection business of the second logistics system can be integrated into the collection business of the first logistics system. On the other hand, the small-piece collection business of the first logistics system can be integrated into the collection business of the second logistics system. On the other hand, the express routes of the second logistics system can be integrated with the express routes of the first logistics system.

[0094] The data fusion method provided by the embodiment of the present invention includes: acquiring current business data of the current business system; filtering the current business data based on preset filtering rules to obtain intermediate data; adding loading information to each intermediate data according to the data type of the intermediate data to obtain effective data corresponding to each intermediate data; supplementing the fourth-level address information of the loading information corresponding to each intermediate data to the effective data corresponding to each intermediate data; performing data processing on each effective data according to business needs to obtain fused data corresponding to the current business data; performing data filtering in the second logistics data according to the filtering conditions sent by the current business system to obtain target data; and sending the target data to the current business system. According to the above technical solution, after receiving the file upload information sent by the current business system, the target business system can first search and download the business file corresponding to the file upload information in the server to obtain the business file. After determining that the business file passes the pre-check, the business file is parsed based on the asynchronous task processing method. After determining that the parsing result is successful, the current business data corresponding to the business file is obtained to obtain the current business data of the current business system. Secondly, the current business data can be filtered based on the preset filtering rules to obtain the intermediate data. The intermediate data required by the target business system can be determined through data filtering. The loading information corresponding to the intermediate data can also be determined according to the data type of the intermediate data, and the loading information corresponding to the intermediate data can be added to the intermediate data to obtain the effective data corresponding to the intermediate data to supplement the loading information of the intermediate data. The effective data corresponding to the intermediate data can also be further supplemented by supplementing the fourth-level address information of the loading information corresponding to the intermediate data to the effective data corresponding to the intermediate data. Then, data processing of the effective data can be realized according to business needs to obtain the fused data corresponding to the current business data required by the target business system, so as to integrate the current business data into the target business system and further realize the data integration of the current business system and the target business system.

[0095] In addition, the target business system can also determine the target data required by the current business system in the target business data of the target business system according to the screening conditions sent by the current business system. After sending the target data to the current business system, the current business system can also process the target data according to business needs to process the target data into a format compatible with the current business system, thereby integrating the target business data into the current business system, further realizing data integration of the current business system and the target business system, and realizing resource reuse of the current business system and the target business system.

[0096] Of course, the data fusion of the current business data of the current business system and the target business data of the target business system has reference value for connecting the data of other current business systems and the corresponding target business systems. In the process of data fusion, it solves problems such as data consistency, data change status processing, data processing failure, etc., and realizes full-process lifecycle management of data synchronization, which has strong practical value.

[0097] Figure 4 The structure diagram of a data fusion device provided by an embodiment of the present invention is shown in FIG. The device and the data fusion methods of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the data fusion device, reference can be made to the embodiment of the above data fusion method.

[0098] The specific structure of the data fusion device is as follows: Figure 4 As shown, including:

[0099] The acquisition module 410 is used to acquire the current business data of the current business system;

[0100] A filtering module 420, configured to add configuration information to each intermediate data according to the data type of the intermediate data after filtering the current business data based on a preset filtering rule to obtain intermediate data, so as to obtain effective data corresponding to each intermediate data;

[0101] The processing module 430 is used to process the effective data according to business requirements to obtain fused data corresponding to the current business data.

[0102] Based on the above embodiment, the acquisition module 410 is specifically used for:

[0103] After receiving the file upload information sent by the current business system, the business file corresponding to the file upload information is searched and downloaded in the server; the business file is parsed to obtain the current business data corresponding to the file upload information.

[0104] Based on the above embodiment, the preset filtering rules at least include a preset originating node, a preset destination node and a preset load. Accordingly, the filtering module 420 is specifically used to:

[0105] The current business data is initially filtered based on the preset originating node and the preset destination node to obtain transition data; according to the preset loading, the transition data with the loading attribute being in effect is determined as the intermediate data; when it is determined that the data type of the intermediate data is original data, the loading information is matched for the original data according to the product type of the intermediate data to obtain the effective data corresponding to the original data; when it is determined that the data type of the intermediate data is supplementary data, after parsing the supplementary data to obtain the whitelist information and blacklist information corresponding to the supplementary data, the whitelist information and blacklist information corresponding to the supplementary data are used as the loading information of the supplementary data to obtain the effective data corresponding to the supplementary data.

[0106] Based on the above embodiment, the filtering module 420 is further used for:

[0107] After obtaining the effective data corresponding to each of the intermediate data, the fourth-level address information of the loading information corresponding to each of the intermediate data is supplemented to the effective data corresponding to each of the intermediate data.

[0108] Based on the above embodiment, the processing module 430 is specifically used for:

[0109] When it is determined that the effective data includes the effective data corresponding to the supplementary data, data conversion and data supplement are performed on the effective data based on the standard line data format to obtain the fused data corresponding to the current business data; when it is determined that the effective data includes the original data and the effective data corresponding to the supplementary data, data processing is performed on the effective data according to the data change type and the standard line data format to obtain the fused data corresponding to the current business data.

[0110] Based on the above embodiment, the acquisition module 410 is further used for:

[0111] After determining that the parsing result of the business file is a parsing failure, the business file is uploaded to the server.

[0112] Based on the above embodiment, the device further includes:

[0113] The screening module is used to screen the target business data according to the screening conditions sent by the current business system to obtain the target data; and send the target data to the current business system.

[0114] The data fusion device provided in the embodiment of the present invention can execute the data fusion method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data fusion method.

[0115] It is worth noting that in the embodiment of the above-mentioned data fusion device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0116] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 5 A block diagram of an exemplary computer device 5 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The computer device 5 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] like Figure 5 As shown, the computer device 5 is in the form of a general-purpose computing electronic device. The components of the computer device 5 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0118] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0119] The computer device 5 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 5, including volatile and non-volatile media, removable and non-removable media.

[0120] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 5 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0121] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0122] The computer device 5 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the computer device 5, and / or any device that enables the computer device 5 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the computer device 5 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. Figure 5 As shown, the network adapter 20 communicates with other modules of the computer device 5 via the bus 18. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with computer device 5, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0123] The processing unit 16 executes various functional applications and page displays by running the programs stored in the system memory 28, for example, implementing the data fusion method provided in the embodiment of the present invention, which includes:

[0124] Get the current business data of the current business system;

[0125] After filtering the current business data based on the preset filtering rules to obtain intermediate data, adding loading information to each intermediate data according to the data type of the intermediate data to obtain effective data corresponding to each intermediate data;

[0126] Data processing is performed on each of the effective data according to business requirements to obtain fused data corresponding to the current business data.

[0127] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data fusion method provided by any embodiment of the present invention.

[0128] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data fusion method provided by the embodiment of the present invention is implemented. The method includes:

[0129] Get the current business data of the current business system;

[0130] After filtering the current business data based on the preset filtering rules to obtain intermediate data, adding loading information to each intermediate data according to the data type of the intermediate data to obtain effective data corresponding to each intermediate data;

[0131] Data processing is performed on each of the effective data according to business requirements to obtain fused data corresponding to the current business data.

[0132] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0133] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0134] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0135] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0137] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data fusion method, characterized in that: include: Get the current business data of the current business system; After filtering the current business data based on the preset filtering rules to obtain intermediate data, adding loading information to each intermediate data according to the data type of the intermediate data to obtain effective data corresponding to each intermediate data; Data processing is performed on each of the effective data according to business requirements to obtain fused data corresponding to the current business data.

2. The data fusion method according to claim 1, characterized in that: Get the current business data of the current business system, including: After receiving the file upload information sent by the current business system, searching and downloading the business file corresponding to the file upload information in the server; The business file is parsed to obtain the current business data corresponding to the file upload information.

3. The data fusion method according to claim 1, characterized in that: The preset filtering rules at least include a preset originating node, a preset destination node, and a preset load. Accordingly, filtering the current service data based on the preset filtering rules to obtain intermediate data includes: Performing a primary filtering on the current service data based on the preset originating node and the preset destination node to obtain transition data; According to the preset loading, the transition data with the loading attribute of being in effect is determined as the intermediate data.

4. The data fusion method according to claim 1, characterized in that: Adding loading information to each intermediate data according to the data type of the intermediate data to obtain the effective data corresponding to each intermediate data includes: When it is determined that the data type of the intermediate data is original data, matching the loading information for the original data according to the product type of the intermediate data to obtain the effective data corresponding to the original data; When it is determined that the data type of the intermediate data is supplementary data, after parsing the supplementary data to obtain the whitelist information and blacklist information corresponding to the supplementary data, the whitelist information and blacklist information corresponding to the supplementary data are used as the loading information of the supplementary data to obtain the effective data corresponding to the supplementary data.

5. The data fusion method according to claim 1, characterized in that: After obtaining the effective data corresponding to each of the intermediate data, the method further includes: The fourth-level address information of the loading information corresponding to each of the intermediate data is supplemented to the effective data corresponding to each of the intermediate data.

6. The data fusion method according to claim 4, characterized in that: Processing each of the effective data according to business requirements to obtain fused data corresponding to the current business data includes: When it is determined that the effective data includes the effective data corresponding to the supplementary data, data conversion and data supplementation are performed on the effective data based on a standard line data format to obtain the fused data corresponding to the current service data; When it is determined that the effective data includes the original data and the effective data corresponding to the supplementary data, the effective data is processed according to the data change type and the standard line data format to obtain the fused data corresponding to the current business data.

7. The data fusion method according to claim 2, characterized in that: After parsing the business file, the method further includes: After determining that the parsing result of the business file is a parsing failure, the business file is uploaded to the server.

8. The data fusion method according to claim 1, characterized in that: Also includes: Filtering the target business data according to the filtering conditions sent by the current business system to obtain the target data; The target data is sent to the current business system.

9. A data fusion device, characterized in that: include: The acquisition module is used to obtain the current business data of the current business system; A filtering module, configured to add configuration information to each intermediate data according to the data type of the intermediate data after filtering the current business data based on a preset filtering rule to obtain the intermediate data, so as to obtain the effective data corresponding to each intermediate data; The processing module is used to process the effective data according to business requirements to obtain fused data corresponding to the current business data.

10. A computer device, characterized in that: The computer device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data fusion method as described in any one of claims 1 to 8.

11. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the data fusion method according to any one of claims 1 to 8.

Images