Multi-source data processing method and device

Abstract

The invention provides a multi-source data processing method and device, and the method comprises the steps: firstly, carrying out the unified mapping of the data names and data formats of multi-source data based on a preset data entity dictionary, enabling the same data items to have the same names and the same data expression formats, then extracting the operation data belonging to the same data object, and carrying out the unified mapping of the data names and data formats of the multi-source data. The multi-source data fusion method comprises the steps that a data object is obtained, all possible record data is formed based on multi-valued data existing in the same operation data, then the weight of each record data is calculated in combination with a set business data graph structure, and finally the record data with the maximum weight serves as real data of the data object, so that effective fusion of the multi-source data can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a multi-source data processing method and device. Background Art

[0002] In some business scenarios, the same business data may exist in multiple data sources. Since the information systems of each data source are independently built, the same business data is named and stored differently in different data sources, and different data sources record the same data item differently. This will result in inaccurate business data actually obtained, affecting subsequent applications.

[0003] flight Summary of the invention

[0004] In view of the above technical problems, the technical solution adopted by the present invention is:

[0005] According to a first aspect of the present invention, a multi-source data processing method is provided, the method comprising the following steps:

[0006] S100, obtaining a multi-source data set that currently needs to be processed, wherein the multi-source data set is formed by operation data of multiple data objects provided by multiple data sources within a set time period, and the operation data includes attribute data and business data.

[0007] S200, mapping the multi-source data set that currently needs to be processed using the data entity dictionary corresponding to the multi-source data set that currently needs to be processed, so that the same data items in the mapped multi-source data set have the same name and the same data value expression method, thereby obtaining the mapped multi-source data set.

[0008] S300, classifying the multi-source data set after the mapping process according to the data identifier to obtain a classified data set DC, DC = {DC i} i＝1......E , E is the number of data identifiers in the multi-source data set after mapping, DC i The running data set corresponding to the i-th data identifier, DC i = {DC ij} j＝1......n(i) , D.C. ij is the j-th running data corresponding to the ith data identifier, the value of j is 1 to n(i), and n(i) is the number of running data corresponding to the ith data identifier.

[0009] S400, DC-based i , get the business data record set R corresponding to the i-th data identifier i ={ED is} s＝1......q(i) , E.D.is For R i The sth record in R, q(i) is i The number of records in q(i) is based on DC i The number of different data values ​​of non-empty data items in all corresponding business data is determined, ED is =((EN is h , D.N. is h )) h＝1......z , EN is h For ED i The name of the hth data item in DN is h For ED i The data value of the hth data item in the business data, where h ranges from 1 to z, and z is the number of data items in the business data.

[0010] S500, traverse R i , for ED is , obtain ED based on the set business data graph structure is The corresponding weight W is The set business data graph structure includes a node network and free nodes that represent static properties of data objects. The nodes in the node network represent data items of business data, and the edges of the set business data graph structure represent time interval constraints between two nodes.

[0011] S600, get max{W is} s＝1......q(i) The corresponding record is used as the real record corresponding to the i-th data identifier.

[0012] According to a second aspect of the present invention, there is provided a multi-source data processing device, the device comprising:

[0013] The data acquisition module is used to acquire the multi-source data set that needs to be processed currently. The multi-source data set is formed by the operation data of multiple data objects provided by multiple data sources within a set time period. The operation data includes attribute data and business data.

[0014] The mapping processing module is used to map the multi-source data set that needs to be processed currently using the data entity dictionary corresponding to the multi-source data set that needs to be processed currently, so that the same data items in the mapped multi-source data set have the same name and the same data value expression method, thereby obtaining the multi-source data set after mapping processing.

[0015] The classification data determination module is used to classify the multi-source data set after the mapping process according to the data identifier to obtain a classification data set DC, DC = {DCi} i＝1......E , E is the number of data identifiers in the multi-source data set after mapping, DC i The running data set corresponding to the i-th data identifier, DC i = {DC ij} j＝1......n(i) , D.C. ij The jth operating data corresponding to the ith data identifier, where j ranges from 1 to n(i), and n(i) is the number of operating data corresponding to the ith data identifier. The data identifier is determined by the ID of the data object and the planned occurrence time of the last business corresponding to the business data.

[0016] Business data record determination module, used to determine the business data based on DC i , get the business data record set R corresponding to the i-th data identifier i ={ED is} s＝1......q(i) , E.D. is For R i The sth record in R, q(i) is i The number of records in q(i) is based on DC i The number of different data values ​​of non-empty data items in all corresponding business data is determined, ED is =((EN is h , D.N. is h )) h＝1......z , EN is h For ED i The name of the hth data item in DN is h For ED i The data value of the hth data item in the business data, where h ranges from 1 to z, and z is the number of data items in the business data.

[0017] Weight acquisition module, used to traverse R i , for ED is , obtain ED based on the set business data graph structure is The corresponding weight W is The set business data graph structure includes a node network and free nodes that represent static properties of data objects. The nodes in the node network represent data items of business data, and the edges of the set business data graph structure represent time interval constraints between two nodes.

[0018] Real record acquisition module, used to obtain max{W is} s＝1......q(i) The corresponding record is used as the real record corresponding to the i-th data identifier.

[0019] The present invention has at least the following beneficial effects:

[0020] The multi-source data processing method provided by the embodiment of the present invention first uniformly maps the multi-source data in terms of data name and data format based on a preset data entity dictionary, so that the same data items have the same name and the same data expression format, then extracts the operating data belonging to the same data object, and forms all possible record data based on the multi-valued data existing in the same operating data, and then calculates the weight of each record data in combination with the set business data graph structure, and finally uses the record data with the largest weight as the real data of the data object, thereby enabling effective fusion of multi-source data.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flowchart of a multi-source data processing method provided by an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of a business data flow provided by an embodiment of the present invention;

[0025] Figures 3 to 5 This is a schematic diagram of the comparison effect. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0028] It should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0029] The inventors of the present invention have discovered that in some business scenarios, such as airport flight operation support business scenarios, since the airport flight operation support business is complicated and involves multiple participants, multiple parties need to collaborate to complete multiple support operations. The large amount of relevant business data generated during the operation is an important basis for intelligent collaborative decision-making. These data mainly come from multi-type information systems of multiple participants such as various airport departments, aviation fuel companies, air traffic control departments, airlines, ground service companies, etc. Due to historical reasons such as the lack of unified technical standards of different systems, incompatible data formats, and lack of basis for data benefit distribution, there is currently no unified data management and exchange platform, and the data obtained by each participant in the flight operation support process is incomplete.

[0030] The embodiment of the present invention provides a multi-source data processing method, which aims to fuse the multi-source data to obtain higher quality fused data by performing underlying data level processing on the multi-source data. Figure 1 As shown, the multi-source data processing method provided by the embodiment of the present invention may include the following steps:

[0031] S100, obtaining a multi-source data set that currently needs to be processed, wherein the multi-source data set is formed by operation data of multiple data objects provided by multiple data sources within a set time period, and the operation data includes attribute data and business data.

[0032] In the embodiment of the present invention, the multi-source data set can be obtained from the intelligent terminal of the corresponding data source.

[0033] In the embodiment of the present invention, the length of the set time period may be determined based on actual conditions. For example, in an illustrative embodiment, the length of the set time period may be 3 months.

[0034] In an embodiment of the present invention, attribute data is data that reflects the static attributes of a data object, and business data includes the actual occurrence time of corresponding multiple businesses, that is, the actual execution time of the data object executing the corresponding business. In an illustrative embodiment, the operation data may be flight data. Flight data may be data related to a flight, and may include data such as flight number, aircraft model, flight type, route, domestic and international, direction, aircraft position, runway, planned takeoff, and planned landing. In the case where the operation data is flight data, the attribute data may include flight number, aircraft model, flight type, aircraft position, runway, direction, domestic and international, etc., and the business data may include the specific time when each business is executed during the actual operation of the flight, such as the occurrence time of opening the cabin door, closing the cabin door, starting refueling, and ending refueling. The various businesses in the actual operation of the flight may be as follows: Figure 2 shown.

[0035] When the operation data is flight data, the data source may include the airport A-CDM system, ground service department, aviation fuel company system, etc. The data object is the flight.

[0036] In the embodiment of the present invention, when the operation data is flight data, the multi-source data set that needs to be processed currently consists of flight data within a set time period. S200, mapping the multi-source data set that needs to be processed currently using the data entity dictionary corresponding to the multi-source data set that needs to be processed currently, so that the same data items in the mapped multi-source data set have the same name and the same data value expression method, and the mapped multi-source data set is obtained.

[0037] In an embodiment of the present invention, the data entity dictionary corresponding to the multi-source data set that needs to be processed can be determined based on the corresponding business scenario. In the case where the operating data is flight data, the corresponding data entity dictionary can be constructed based on the collected relevant data. In an exemplary embodiment, the constructed data font dictionary can be shown in Table 1 below. Among them, the data entity dictionary defines the standard name of each flight data item, the name set used for the same data item from different data sources, and the data value expression format. Among them, the standard name is determined according to the official documents of the Civil Aviation Administration, and the name set is determined according to the set of names of corresponding data in different databases.

[0038] Table 1

[0039]

[0040] In the embodiment of the present invention, the type of the data value may include character type, numeric type, timestamp type, etc.

[0041] S300, classifying the multi-source data set after the mapping process according to the data identifier to obtain a classified data set DC, DC = {DC i} i＝1......E, E is the number of data identifiers in the multi-source data set after mapping, DC i The running data set corresponding to the i-th data identifier, DC i = {DC ij} j＝1......n(i) , D.C. ij is the j-th operating data corresponding to the ith data identifier, where j ranges from 1 to n(i), and n(i) is the number of operating data corresponding to the ith data identifier. That is, for the ith data identifier, if there are operating data provided by n(i) data sources, then the number of operating data corresponding to the ith data identifier is n(i), that is, there are n(i) data records.

[0042] In the embodiment of the present invention, the data identifier is determined by the ID of the data object and the planned occurrence time of the last business corresponding to the business data, and the multi-source data set after the mapping process can be classified using the data identifier as an index to obtain the operation data belonging to each data identifier. For example, in the case where the operation data is flight data, the multi-source data set after the mapping process can be classified using "flight number" + "planned departure time" as the data identifier to obtain the flight data belonging to each flight.

[0043] S400, DC-based i , get the business data record set R corresponding to the i-th data identifier i ={ED is} s＝1......q(i) , E.D. is For R i The sth record in R, q(i) is i The number of records in , q(i) is based on DC i The number of different data values ​​of non-empty data items in all corresponding business data is determined, ED is =((EN is h , D.N. is h )) h＝1......z , EN is h For ED i The name of the hth data item in DN is h For ED i The data value of the hth data item in , where h ranges from 1 to z, and z is the number of data items in the business data. Figure 2 In the business process shown, the number of business data items may be 32.

[0044] Take the sth record of a flight on a certain day as an example, ED is=((start refueling, 1693560748), (end refueling, 1693561423), ..., (actual takeoff, 1693565220)).

[0045] In the embodiment of the present invention, q(i)=m i1 ×……×m ig ×……×m ip , m ig is the number of different data values ​​(excluding null values) corresponding to the gth non-null business data item in the business data corresponding to the i-th data identifier, g ranges from 1 to p, and p is the number of non-null data items in the business data of the i-th data identifier. In this way, all possible data records can be retained. For example, in the case where the operating data is flight data, assuming that a flight has a total of 7 business data items with values, two of which have 2 values, and the remaining 5 business data have only 1 value, then after multi-value processing, a total of 4 possible records of the flight can be obtained.

[0046] S500, traverse R i , for ED is , obtain ED based on the set business data graph structure is The corresponding weight W is The business data graph structure is set to include a node network and free nodes representing static attributes of data objects. The nodes in the node network represent data items of business data, and the edges of the business data graph structure are set to represent the time interval constraints between two nodes. In the embodiment of the present invention, when the operation data is flight data, the business data graph structure is set to be based on Figure 2 The data graph structure constructed by the business process shown.

[0047] In the embodiment of the present invention, the time interval constraint corresponding to any edge in the node network is obtained by the following steps:

[0048] S10, based on the starting point and the end point of any edge, query the preset constraint rules. If the corresponding constraint rules can be obtained, the time interval constraint corresponding to the obtained constraint rules will be used as the time interval constraint of any edge. If it cannot be obtained, execute S20.

[0049] In an embodiment of the present invention, the preset constraint rules can be obtained by obtaining the relevant regulations corresponding to the business scenarios corresponding to the multi-source data sets, for example, they can be obtained based on the relevant regulations for flight operation support. In an exemplary embodiment, by analyzing and sorting out the relevant regulations for flight operation support, 26 rules with clear numerical constraints on various business activities of flight operation support are extracted, and examples are shown in Table 2. These rules clarify the specific constraints between the businesses with time-series relationships in the flight operation support business process diagram, for example, the duration between "start of the upper gear" and "end of the upper gear" should satisfy the rule numbered 26 in Table 1.

[0050] Table 2:

[0051]

[0052]

[0053] In the embodiment of the present invention, the minimum value of the time interval constraint in the preset constraint rules is 0, and the maximum value is the time specified in the rules. For example, the maximum value of the time interval constraint between the completion of the passenger elevator docking and the opening of the passenger cabin door is 4 minutes.

[0054] S20, based on the multi-source data set that needs to be processed, obtain the time difference set △T corresponding to any side = {△t x} x＝1……Y , △t x The time difference between the data value of the starting point and the data value of the ending point corresponding to any side is obtained based on the x-th data record in the Y data records corresponding to the multi-source data set currently to be processed, and the value of x ranges from 1 to Y; execute S30.

[0055] In the embodiment of the present invention, for the business data that needs to be calculated by classification, such as the refueling time related to the type of aircraft, when calculating the time difference set, it is necessary to calculate separately according to the type of aircraft. At this time, the corresponding data record is the data record of the corresponding type of aircraft. For example, taking the edges corresponding to the start of refueling and the end of refueling as an example, firstly, according to the aircraft type corresponding to the currently processed flight, the data record to be used is determined, and then the time difference between the start of refueling and the end of refueling in each data record in the determined data record is obtained, and the time difference between the start of refueling and the end of refueling corresponding to all data records in the multi-source data set that needs to be processed is obtained. If the business data does not need to be calculated by classification, the corresponding data record is all data records in the multi-source data set that needs to be processed. For example, taking the edges corresponding to the start of upper gear and the end of upper gear as an example, according to the currently processed multi-source data set, the time difference between the start of upper gear and the end of upper gear in each data record is obtained, and the time difference between the start of upper gear and the end of upper gear corresponding to all data records in the current data set to be processed is obtained.

[0056] S30, sort △T in ascending order, and obtain the m1th time difference in the sorted time difference set as the minimum value of the time interval constraint on either side, and obtain the m2th time difference in the sorted time difference set as the maximum value of the time interval constraint on either side, m1=[a1*Y], m2=[a2*Y], a1 is the first coefficient, a2 is the second coefficient, 0<a1<0.5, 0.5<a2<1, [] indicates rounding.

[0057] In the embodiment of the present invention, a1=0.1, a2=0.9.

[0058] S40, obtaining the time interval constraint of any side based on the maximum value and the minimum value of the time interval constraint of any side.

[0059] Further, in the embodiment of the present invention, W is You can obtain it by following the steps below:

[0060] S401, if ED is There is no business data with empty data value in ED is , assign a value to each node in the node network, obtain the assigned node network, and execute S402; if ED is There are business data with empty data values ​​in the ED is , obtaining nodes whose data values ​​are not empty in the node network, forming a non-empty node set, and executing S404.

[0061] S402, traverse all edges of the node network after the assignment, and for the dth edge traversed, obtain the time deviation dif corresponding to the dth edge is d , and based on dif is d Get the weight W of the dth edge is d , execute S403; wherein, dif is d =(a is d-end -a is d-start ), a is d-end is the data value of the end point of the dth edge, a is d-start is the data value of the starting point of the dth edge, the data value range of d is 1 to m, and m is the number of edges in the node network after assignment.

[0062] S403, Get ED is The corresponding weight W is=W is 1 ×W is 2 ×……×W is d ×……×W is z .

[0063] S404, traverse the non-empty node set, for the u-th non-empty node NN traversed u , get NN from the node network u The first non-empty successor node of , if the corresponding first non-empty successor node is obtained, the obtained first non-empty successor node is used as NN u The non-empty successor node NN u back , forming the uth effective edge e is u And add it to the effective edge set EI, e is u ={a is u-start , a is u-end ,fe is u}, a is u-start NN u The data value of a is u-end NN u back The data value of fe is u NN u and NN u back The time interval constraint is set, the data value of u is 1 to H, and H is the number of non-empty nodes in the non-empty node set; execute S405.

[0064] In the embodiment of the present invention, is u =∑ f(u) c＝1 f is c , f is c is the node in the network located at NN u and NN u back The time interval constraint corresponds to the cth edge among the f(u) edges between them, and the data value range of c is 1 to f(u).

[0065] Taking a record of a flight on a certain day as an example, there are only five data items, namely, start of refueling, end of refueling, closing of cabin door, closing of cargo door, and actual takeoff. Therefore, there are four valid edges in total. The starting point of the valid edge corresponding to the valid path from closing of cabin door to actual takeoff is the data value of closing of cabin door, and the ending point is the data value of actual takeoff. The time interval constraint between the starting point and the ending point is the edge passed by the valid edge, namely, closing of cabin door to removal of boarding bridge / passenger ladder, removal of boarding bridge / passenger ladder to removal of wheel block, and removal of wheel block to actual takeoff. The rule functions corresponding to these three edges are added together.

[0066] S405, traverse EI, for the traversed e is u , get e is u The corresponding time deviation dif is u , and based on dif is u Get e is u The weight W is u , execute S406.

[0067] S406, Get ED is The corresponding weight W is =W is 1 ×W is 2 ×……×W is u ×……×W is H .

[0068] The present invention analyzes from the perspective of actual business logic, uses specific business rules combined with business logic, and based on the original distribution of business data, believes that the business data should satisfy the normal distribution in terms of distribution, and uses the upper and lower limit values ​​calculated in the time interval constraint of the effective edge as the 90% and 10% quantiles of the normal distribution. Specifically, W is d The following conditions must be met:

[0069] If dif is d ≤0,W is d =0, if dif is d >0,W is d =e -g1 / (σ is d ×(2π) 1 / 2), where g1 is the intermediate quantity, g1=(dif is d -ψ is d ) 2 / 2(σ is d ) 2 , σ is d is the time deviation corresponding to the dth edge, specifically the standard deviation of the time distribution, ψ is d is the mean of the time distribution corresponding to the d-th edge, determined based on the time interval constraint corresponding to the d-th edge.

[0070] Specifically, σ is d Satisfy the following conditions: is d =(max is d -min is d )×k1,max is d is the maximum value of the time interval constraint corresponding to the dth edge, min is d is the minimum value in the time interval constraint corresponding to the d-th edge, k1 is a first set value, and in an exemplary embodiment, k1=4.66.

[0071] ψ is d The following conditions must be met:

[0072] ψ is d =min is d +k2×σ is d , k2 is a second set value. In an exemplary embodiment, k2=2.33.

[0073] Furthermore, W is u The following conditions must be met:

[0074] If dif is u ≤0,W is u =0, if dif is u >0,W is u =e -g2 / (σ is u ×(2π)1 / 2 ), where g2 is the intermediate quantity, g2 = (dif is u -ψ is u ) 2 / 2(σ is u ) 2 , σ is u for e is u The corresponding e is u The corresponding time deviation, ψ is u for e is u The corresponding time distribution mean, based on e is u The corresponding time interval constraints are determined.

[0075] Specifically, σ is u =(max is u -min is u )×k1,max is u For is u The maximum value, min is u For is u The minimum value in .

[0076] ψ is u The following conditions must be met:

[0077] ψ is u =min is u +k2×σ is u .

[0078] S500, get max{W is} s＝1......q(i) The corresponding record is taken as the real record corresponding to the i-th data identifier. max{} means taking the maximum value, that is, taking the record with the largest weight among all possible records corresponding to the i-th data identifier as the real record of the i-th data identifier.

[0079] (Example)

[0080] The operation data used in the embodiment of the present invention is the actual operation data of a certain airport in September 2023, with a total of 15,321 flights and a total of 175,890 cumulative records of all flights, including 7 items of data: start of refueling, end of refueling, closing of passenger cabin door, closing of cargo cabin door, removal of boarding bridge / passenger ladder, removal of wheel blocks, and actual takeoff. Therefore, when actually constructing the data structure, Figure 2 The nodes in are simplified, and only 7 nodes are constructed, with a total of 7 edges.

[0081] In the actual fusion process, Python 3.9 was used as the code language to complete the codes for data preprocessing and data fusion. After fusion, one record was retained for each flight.

[0082] In multi-sensor data fusion, when the true value is unknown, the mean method is often used to fuse the data as the data value, and the mean is used as the baseline for comparative analysis to evaluate the data quality from two perspectives: one is to directly analyze the fused data and conduct qualitative analysis on the data through visualizing the data, and the other is to conduct quantitative analysis based on the amount of data information.

[0083] Since there is no real value for comparison, it is impossible to directly evaluate the quality of the data fusion result. Therefore, the present invention conducts a comparative experiment to predict the sliding time on the data set fused by the method in this paper and the averaging method, and further evaluates the method in this paper through the prediction effect.

[0084] In the comparative experiment, mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE) and mean absolute percentage error (MAPE) were selected as evaluation indicators. MSE and RMSE can capture the overall fluctuation of prediction errors, are particularly sensitive to large errors, and are suitable for measuring the model's ability to predict extreme values; MAE focuses more on the average level of all prediction errors, which can avoid the excessive impact of extreme errors on the evaluation results; MAPE introduces the concept of relative error, which can measure the relative accuracy of the model under different magnitudes of data. The combined use of these indicators can comprehensively evaluate the advantages and disadvantages of the model from different angles and ensure a comprehensive understanding of the prediction performance.

[0085] Figure 3 is the data before fusion, Figure 4 For the fused data obtained by the mean method, Figure 5 It is the fusion data obtained by the method provided by the present invention. Figure 3 It can be seen that the data distribution before fusion is very wide, and the left and right end values ​​are obviously beyond the normal range of the data. Figure 4 Although the distribution range of the data shown is slightly reduced, some of them are still less than 0 and do not meet the business logic. Figure 5The data distribution shown removes all abnormal data, which conforms to the business logic, and the distribution of each process time greater than 0 is approximately normally distributed, which is consistent with the assumptions of the present invention when calculating the data distribution parameters, indicating that the method of this article is logically correct and more reasonable than the method of directly taking the mean.

[0086] The information in the A-CDM system is a collection of information from various participants. It has the largest amount of information and is widely used as the data basis for civil aviation research. Therefore, the A-CDM system data is used as a comparison object. The fused data has increased the total amount of information by 12.6% compared to the original data.

[0087] Table 3 compares the number of 6 business process time data after fusion by the mean method and the method of this paper (excluding the data less than 0), and it is found that the two are basically close. In general, the method of this paper has 40 more than the mean method. The method of the present invention does not significantly increase the amount of data. The reason for this situation is that the original data itself is seriously missing.

[0088] Table 3:

[0089] Business Process Mean method (number) The method of the present invention A 15003 15062 B 14771 14747 C 3292 3299 D 870 835 E 3544 3574 F 3612 3597

[0090] In summary, the multi-source data processing method provided by the embodiment of the present invention completely retains the business data and business rules by converting the business data from a linear structure to a graph structure representation, where vertices represent businesses and edges represent the relationship between businesses, and more intuitively represents the relationship between data and rules. It provides a flexible data structure that can adapt to different actual operating conditions. In addition, the rules in the business data of the constructed graph structure are used as parameters for weight calculation to achieve fast and simple fusion of multi-source business data from different systems, which can improve the accuracy of the data.

[0091] Based on the same inventive concept, an embodiment of the present invention provides a multi-source data processing device, the device comprising:

[0092] The data acquisition module is used to acquire the multi-source data set that needs to be processed currently. The multi-source data set is formed by the operation data of multiple data objects provided by multiple data sources within a set time period. The operation data includes attribute data and business data.

[0093] The mapping processing module is used to map the multi-source data set that needs to be processed currently using the data entity dictionary corresponding to the multi-source data set that needs to be processed currently, so that the same data items in the mapped multi-source data set have the same name and the same data value expression method, thereby obtaining the multi-source data set after mapping processing.

[0094] The classification data determination module is used to classify the multi-source data set after the mapping process according to the data identifier to obtain a classification data set DC, DC = {DC i} i＝1......E , E is the number of data identifiers in the multi-source data set after mapping, DC i The running data set corresponding to the i-th data identifier, DC i = {DC ij} j＝1......n(i) , D.C. ij The jth operating data corresponding to the ith data identifier, where j ranges from 1 to n(i), and n(i) is the number of operating data corresponding to the ith data identifier. The data identifier is determined by the ID of the data object and the planned occurrence time of the last business corresponding to the business data.

[0095] Business data record determination module, used to determine the business data based on DC i , get the business data record set R corresponding to the i-th data identifier i ={ED is} s＝1......q(i) , E.D. is For R i The sth record in R, q(i) is i The number of records in , q(i) is based on DC i The number of different data values ​​of non-empty data items in all corresponding business data is determined, ED is =((EN is h , D.N. is h )) h＝1......z , EN is h For ED i The name of the hth data item in DN is h For ED i The data value of the hth data item in the business data, where h ranges from 1 to z, and z is the number of data items in the business data.

[0096] Weight acquisition module, used to traverse R i , for ED is , obtain ED based on the set business data graph structure is The corresponding weight W is The set business data graph structure includes a node network and free nodes that represent static properties of data objects. The nodes in the node network represent data items of business data, and the edges of the set business data graph structure represent time interval constraints between two nodes.

[0097] Real record acquisition module, used to obtain max{Wis} s＝1......q(i) The corresponding record is used as the real record corresponding to the i-th data identifier.

[0098] The device can be used to perform Figure 1 Therefore, for the functions that can be realized by each functional module of the device, please refer to Figure 1 The description of the illustrated embodiment is not repeated in detail.

[0099] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0100] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-source data processing method, characterized in that: The method comprises the following steps: S100, obtaining a multi-source data set that needs to be processed currently, wherein the multi-source data set is formed by operation data of multiple data objects provided by multiple data sources within a set time period, wherein the operation data includes attribute data and business data, and the business data includes actual occurrence time of corresponding multiple businesses; S200, mapping the multi-source data set that needs to be processed currently using the data entity dictionary corresponding to the multi-source data set that needs to be processed currently, so that the same data items in the mapped multi-source data set have the same name and the same data value expression method, thereby obtaining the mapped multi-source data set; S300, classifying the multi-source data set after the mapping process according to the data identifier to obtain a classified data set DC, DC = {DC i } i＝1......E , E is the number of data identifiers in the multi-source data set after mapping, DC i The running data set corresponding to the i-th data identifier, DC i = {DC ij } j＝1......n(i) , D.C. ij is the jth running data corresponding to the i-th data identifier, where j is a value from 1 to n(i), and n(i) is the number of running data corresponding to the i-th data identifier, and the data identifier is determined by the ID of the data object and the planned occurrence time of the last business corresponding to the business data; S400, DC-based i , get the business data record set R corresponding to the i-th data identifier i ={ED is } s＝1......q(i) , E.D. is For R i The sth record in R, q(i) is i The number of records in q(i) is based on DC i The number of different data values ​​of non-empty data items in all corresponding business data is determined, ED is =((EN is h , D.N. is h )) h＝1......z , EN is h For ED i The name of the hth data item in DN is h For ED i The data value of the hth data item in , where h ranges from 1 to z, and z is the number of data items in the business data; S500, traverse R i , for ED is , obtain ED based on the set business data graph structure is The corresponding weight W is , the set business data graph structure includes a node network and free nodes representing static attributes of data objects, the nodes in the node network represent data items of business data, and the edges of the set business data graph structure represent time interval constraints between two nodes; S600, get max{W is } s＝1......q(i) The corresponding record is used as the real record corresponding to the i-th data identifier.

2. The method according to claim 1, characterized in that W is Obtain it by following the steps below: S401, if ED is There is no business data with empty data value in ED is , assign a value to each node in the node network, obtain the assigned node network, and execute S402; if ED is There are business data with empty data values ​​in the ED is , obtaining nodes whose data values ​​are not empty in the node network to form a non-empty node set, and executing S404; S402, traverse all edges of the node network after the assignment, and for the dth edge traversed, obtain the time deviation dif corresponding to the dth edge is d , and based on dif is d Get the weight W of the dth edge is d , execute S403; wherein, dif is d =(a is d-end -a is d -start ), a is d-end is the data value of the end point of the dth edge, a is d-start is the data value of the starting point of the dth edge, the data value range of d is 1 to m, and m is the number of edges in the node network; S403, Get ED is The corresponding weight W is =W is 1 ×W is 2 ×……×W is d ×……×W is z ; S404, traverse the non-empty node set, for the u-th non-empty node NN traversed u , get NN from the node network u The first non-empty successor node of , if the corresponding first non-empty successor node is obtained, the obtained first non-empty successor node is used as NN u The non-empty successor node NN u back , forming the uth effective edge e is u And add it to the effective edge set EI, e is u ={a is u -start , a is u-end ,fe is u }, a is u-start NN u The data value of a is u-end NN u back The data value of fe is u NN u and NN u back The time interval constraint between them, the data value of u is 1 to H, and H is the number of non-empty nodes in the non-empty node set; execute S405; S405, traverse EI, for the traversed e is u , get e is u The corresponding time deviation dif is u , and based on dif is u Get e is u The weight W is u , execute S406; S406, Get ED is The corresponding weight W is =W is 1 ×W is 2 ×……×W is u ×……×W is H .

3. The method according to claim 2, characterized in that W is d The following conditions must be met: If dif is d ≤0,W is d =0, if dif is d >0,W is d =e -g1 / (σ is d ×(2π) 1 / 2 ), where g1 is the intermediate quantity, g1=(dif is d -ψ is d ) 2 / 2(σ is d ) 2 , σ is d is the time deviation corresponding to the dth edge, ψ is d is the mean of the time distribution corresponding to the d-th edge, determined based on the time interval constraint corresponding to the d-th edge.

4. The method according to claim 3, characterized in that σ is d The following conditions must be met: σ is d =(max is d -min is d )×k1,max is d is the maximum value of the time interval constraint corresponding to the dth edge, min is d is the minimum value of the time interval constraint corresponding to the dth edge, and k1 is the first set value; ψ is d The following conditions must be met: ψ is d =min is d +k2×σ is d , k2 is the second set value.

5. The method according to claim 1, characterized in that W is u The following conditions must be met: If dif is u ≤0,W is u =0, if dif is u >0,W is u =e -g2 / (σ is u ×(2π) 1 / 2 ), where g2 is the intermediate quantity, g2 = (dif is u -ψ is u ) 2 / 2(σ is u ) 2 , σ is u for is u The corresponding time deviation, ψ is u for e is u The corresponding time distribution mean, based on e is u The corresponding time interval constraints are determined.

6. The method according to claim 5, characterized in that σ is u =(max is u -min is u )×k1,max is u For is u The upper limit value in, min is u For is u The lower limit value in, k1 is the first set value; ψ is u The following conditions must be met: ψ is u =min is u +k2×σ is u , k2 is the second set value.

7. The method according to claim 5, characterized in that f is u =∑ f(u) c＝1 f is c , f is c is the node in the network located at NN u and NN u back The time interval constraint corresponds to the cth edge among the f(u) edges between them, and the data value range of c is 1 to f(u).

8. The method according to claim 2, characterized in that: The time interval constraint corresponding to any edge in the node network is obtained by the following steps: S10, based on the starting point and the end point of any edge, query the preset constraint rules, if the corresponding constraint rules can be obtained, use the time interval constraint corresponding to the obtained constraint rules as the time interval constraint of any edge, if not obtained, execute S20; S20, based on the multi-source data set that needs to be processed, obtain the time difference set △T corresponding to any side = {△t x } x＝1……Y , △t x The time difference between the data value of the starting point and the data value of the ending point corresponding to any side obtained based on the x-th data record in the Y records corresponding to the multi-source data set currently to be processed; executing S30; S30, sort △T in order from small to large, and obtain the m1th time difference in the sorted time difference set as the minimum value of the time interval constraint of any side, and obtain the m2th time difference in the sorted time difference set as the maximum value of the time interval constraint of any side, m1=[a1*Y], m2=[a2*Y], a1 is the first coefficient, a2 is the second coefficient, 0<a1<0.5, 0.5<a2<1, [] indicates rounding; S40, obtaining the time interval constraint of any side based on the maximum value and the minimum value of the time interval constraint of any side.

9. The method according to claim 1, characterized in that: The operation data is flight data.

10. A multi-source data processing device, characterized in that: The device comprises: A data acquisition module is used to acquire a multi-source data set that needs to be processed currently. The multi-source data set is formed by operation data of multiple data objects provided by multiple data sources within a set time period. The operation data includes attribute data and business data. The business data includes actual occurrence time of corresponding multiple businesses. A mapping processing module is used to map the multi-source data set that needs to be processed currently using the data entity dictionary corresponding to the multi-source data set that needs to be processed currently, so that the same data items in the mapped multi-source data set have the same name and the same data value expression method, thereby obtaining the multi-source data set after mapping processing; The classification data determination module is used to classify the multi-source data set after the mapping process according to the data identifier to obtain a classification data set DC, DC = {DC i } i＝1......E , E is the number of data identifiers in the multi-source data set after mapping, DC i The running data set corresponding to the i-th data identifier, DC i = {DC ij } j＝1......n(i) , D.C. ij is the jth running data corresponding to the i-th data identifier, where j is a value from 1 to n(i), and n(i) is the number of running data corresponding to the i-th data identifier, and the data identifier is determined by the ID of the data object and the planned occurrence time of the last business corresponding to the business data; Business data record determination module, used to determine the business data based on DC i , get the business data record set R corresponding to the i-th data identifier i ={ED is } s＝1......q(i) , E.D. is For R i The sth record in R, q(i) is i The number of records in q(i) is based on DC i The number of different data values ​​of non-empty data items in all corresponding business data is determined, ED is =((EN is h , D.N. is h )) h＝1......z , EN is h For ED i The name of the hth data item in DN is h For ED i The data value of the hth data item in , where h ranges from 1 to z, and z is the number of data items in the business data; Weight acquisition module, used to traverse R i , for ED is , obtain ED based on the set business data graph structure is The corresponding weight W is , the set business data graph structure includes a node network and free nodes representing static attributes of data objects, the nodes in the node network represent data items of business data, and the edges of the set business data graph structure represent time interval constraints between two nodes; Real record acquisition module, used to obtain max{W is } s＝1......q(i) The corresponding record is used as the real record corresponding to the i-th data identifier.

Images