An event stream data field completion method, electronic equipment and storage medium

Abstract

This invention provides a method, electronic device, and storage medium for completing event stream data fields, relating to the field of event stream data field completion. The method includes: obtaining a list of standard fields corresponding to the original message sent by each data source to obtain a set of standard field lists D; traversing D... r If D r，p If it is a preset primary key field, then retrieve D. r，p To obtain D r The corresponding primary key field list ZD r ; Traverse ZD r If ZD r，a If the ID number field is a preset field, then ZD is obtained. r，a Corresponding ID number HD r，a According to HD r，a And the preset database mapping table, determine D r The corresponding user data corresponds to the database QD r The preset database mapping table includes several databases and the corresponding ID number for each database; QD r Add the target primary key field and its corresponding field value to ZD. r The present invention can improve the completeness of information display on the business platform.

Description

Technical Field

[0001] This invention relates to the field of event stream data field completion, and in particular to an event stream data field completion method, electronic device, and storage medium. Background Technology

[0002] In the civil aviation sector, civil aviation management platforms typically receive data streams from multiple different data sources to monitor flight dynamics or user travel event information in real time. Each data stream from each data source is then processed to monitor flight or user travel information. When an event occurs, corresponding event stream data is generated and sent to the business platform in real time. However, event stream data includes several fields and their corresponding values. Because the event stream data is pushed from different data sources, the data from each source may be incomplete. For example, some event stream data may only contain flight numbers and ticket numbers but not the user's identification number. When the business side uses this event stream data, it cannot determine the specific user information, resulting in incomplete information display on the business platform. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0004] According to a first aspect of this application, a method for completing field information in an event stream is provided, the method comprising the following steps:

[0005] Q100, obtain the standard field list corresponding to the original message sent by each data source, so as to obtain the standard field list set D = (D1, D2, ..., D...). r D s ), r = 1, 2, ..., s; where, D r Here is a list of standard fields corresponding to the original message sent from the r-th data source, where s is the number of data sources; D r C r Corresponding standard field list; D r =(D r，1 D r，2 D r，p ,…,D r，g(r) ), p=1, 2,..., g(r); D r，p Let g(r) be the p-th standard field in the standard field list corresponding to the original message sent by the r-th data source, and g(r) be the number of standard fields in the standard field list corresponding to the original message sent by the r-th data source.

[0006] Q200, traversing D r If D r，p If it is a preset primary key field, then retrieve D. r，p To obtain Dr a corresponding first primary key field list ZD r = (ZD r，1 , ZD r，2 , …, ZD r，a , …, ZD r，h(r) ), a = 1, 2, …, h(r); wherein, ZD r，a is a D r corresponding a-th primary key field, and h(r) is a number of D r corresponding primary key fields; the primary key fields are used to determine corresponding trip information.

[0007] Q300, traversing ZD r , if ZD r，a is a preset certificate number field, obtaining a ZD r，a corresponding certificate number HD r，a .

[0008] Q400, determining a database QD r，a corresponding to D r corresponding user data according to HD r and a preset database mapping table; wherein, the preset database mapping table includes a plurality of databases and a certificate number corresponding to each database.

[0009] Q500, adding a target primary key field in QD r and a corresponding field value to ZD r ; wherein, the target primary key field in QD r is a primary key field not in ZD r in the preset primary key field.

[0010] According to another aspect of the present application, a non-transitory computer readable storage medium is also provided, the storage medium storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by a processor to implement the above-mentioned event flow data field completion method.

[0011] According to another aspect of the present application, an electronic device is also provided, including a processor and the above-mentioned non-transitory computer readable storage medium.

[0012] The present application has at least the following beneficial effects:

[0013] The event stream data field completion method provided by the application obtains a standard field list corresponding to an original message sent by each data source; obtains a primary key field in each standard field list and an identification number field in the primary key field to obtain a corresponding identification number; determines a database corresponding to user data corresponding to the identification number according to the identification number and a preset database mapping table; determines a target primary key field in the database according to other primary key fields in the standard field list, and then adds the target primary key field and a corresponding field value to the standard field list to complete the completion of the standard field list; so that the business end can obtain complete user data when using the event stream data, thereby improving the completeness of information display of the business platform. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0015] Figure 1 The flow chart of the event stream data field completion method provided by the embodiments of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0017] It should be noted that based on the present disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, an apparatus and / or a method can be implemented using any number of the aspects set forth herein. In addition, this apparatus and / or method can be implemented using other structures and / or functionalities in addition to or other than one or more of the aspects set forth herein.

[0018] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application. Figure 1 The flow chart of the event stream data field completion method is described.

[0019] The event stream data field completion method can include the following steps:

[0020] Q100, obtain the standard field list corresponding to the original message sent by each data source, so as to obtain the standard field list set D = (D1, D2, ..., D...). r D s ), r = 1, 2, ..., s; where, D r Here is a list of standard fields corresponding to the original message sent from the r-th data source, where s is the number of data sources; D r C r Corresponding standard field list; D r =(D r，1 D r，2 D r，p ,…,D r，g(r) ), p=1, 2,..., g(r); D r，p Let g(r) be the p-th standard field in the standard field list corresponding to the original message sent by the r-th data source, and g(r) be the number of standard fields in the standard field list corresponding to the original message sent by the r-th data source.

[0021] In this embodiment, the list of standard fields corresponding to the original message sent by each data source can be obtained through the following steps:

[0022] Q10, obtain each target field corresponding to each preset fusion project to obtain a target field list set A = (A1, A2, ..., A...). i A n ), i=1, 2,...,n; where, A i A represents the list of target fields corresponding to the i-th fusion project, where n is the preset number of fusion projects; i =(A i，1 A i，2 A i，j A i，f(i) ), j=1,2,…,f(i);A i，j Let f(i) be the j-th target field corresponding to the i-th fusion project, and f(i) be the number of target fields corresponding to the i-th fusion project; the target fields are used to provide corresponding field values ​​for the fusion project.

[0023] In this embodiment, the fusion project can be understood as the project by which the business end merges and displays data sent from different data sources; for example, a fusion project for displaying user travel information; each fusion project corresponds to a target field list, which can be in the form of a full Java bean, and includes all the standard fields required by the corresponding fusion project; for example, the target field list corresponding to the fusion project for displaying user travel information includes standard fields such as flight number field, ID number field, and flight segment number field.

[0024] It is understood that, since the number of target fields corresponding to each fusion project is different in this embodiment, f(i) does not refer to a specific function or function result value, but to a possible value that varies with the specific value of i. For example, when i=1, f(i)=3; when i=2, f(i)=4; when i=3, f(i)=3.

[0025] Q20. Perform deduplication on A to obtain a list of specified fields B = (B1, B2, ..., B...). k B m ), k = 1, 2, ..., m; where B k The specified field is the k-th field obtained after deduplication of A, and m is the number of specified fields obtained after deduplication of A.

[0026] In this embodiment, it is understood that the types of data processed by each fusion project may be different, and the required standard fields may not be exactly the same. For example, the fusion project for displaying user travel information mainly processes fields related to user travel, while the fusion project for displaying flight information processes fields related to flights. The two may be dealing with the same fields.

[0027] Q30 retrieves the original message sent by each data source; each original message includes several original fields, and each original field has a corresponding field value.

[0028] In this embodiment, each data source will push relevant information in real time in the form of raw messages. The raw messages will have multiple data formats, such as JSON or XML. It can be understood that, regardless of the format, the raw messages will contain several raw fields and the field values ​​corresponding to each raw field.

[0029] Q40, Based on B, obtain the target original field corresponding to each original message to obtain the target original field list set C = (C1, C2, ..., C... r C s ), r = 1, 2, ..., s; where, C r C represents the list of target raw fields corresponding to the raw message sent from the r-th data source, where s is the number of data sources; r =(C r，1 C r，2 C r，p ,…,C r，g(r) ), p=1, 2,..., g(r); C r，pThe pth target original field in the target original field list corresponding to the original message sent by the rth data source, g(r) is the number of the target original field list corresponding to the original message sent by the rth data source; C r，p Corresponding to a specified field in B.

[0030] In this embodiment, a certain target field in step Q10 is corresponding to a certain original field in the original message; for example, the flight number corresponding original field in the original message is Flightno, and the flight number corresponding standard field is flightnumber; the meanings represented by Flightno and flightnumber are the same; therefore, the target original field can be determined in the original field of each original message according to B; the target original field can be understood as the original field having a corresponding standard field in B.

[0031] It should be noted that when obtaining the target original field, if any two original fields in an original message are not the same, then the target corresponding target original field can be uniquely determined and obtained according to the target field; and if there are multiple original fields representing different meanings in an original message, then the specific original field needs to be specified according to the meaning represented by the original field.

[0032] Q50, according to the preset normalization rule, converting each target original field in C into a corresponding standard field to obtain a standard field list set D=(D1, D2, …, D r , …, D s ) corresponding to C; wherein D r is the standard field list corresponding to C r ; D r =(D r，1 , D r，2 , …, D r，p , …, D r，g(r) ); D r，p is the standard field corresponding to C r，p .

[0033] In this embodiment, it can be understood that different data sources may have different field names for the same meaning field, and in order to facilitate data acquisition in the fusion project, the original fields corresponding to different data sources need to be normalized; specifically, step Q50 can include the following steps:

[0034] Q51, traversing C, if the first letter of C r，p is a capital letter, then changing the first letter of C r，p to a lowercase letter; otherwise, not changing C r，pthe first letter of the target original field corresponding to the original message is changed to obtain the intermediate original field list corresponding to C, QC=(QC1, QC2, …, QC r , …, QC s ); wherein, QC r is the intermediate original field list corresponding to C r ; QC r =(QC r，1 , QC r，2 , …, QC r，p , …, QC r，g(r) ); QC r，p is the intermediate original field corresponding to C r，p .

[0035] In the embodiment, the first letter of the target original field corresponding to the original message is a capital letter, in order to make the original field meet the standardized structure of the java bean processing, the first letter of the target original field with the capital letter is converted to the corresponding lower case letter.

[0036] Q52, according to the preset standard field mapping table, each intermediate original field in QC is converted to the corresponding standard field to obtain D; wherein, the preset standard field mapping table includes a plurality of intermediate original fields and the standard field corresponding to each intermediate original field.

[0037] In the embodiment, the standard field mapping table contains each intermediate original field corresponding to each original message and the standard field corresponding to each original field; for example, flightno and flight both represent the flight number, so the standard field corresponding to flightno and flight is both flightnumber; thus, each intermediate original field in QC can be converted to the corresponding standard field to obtain D.

[0038] Further, after step Q50, the method can further include the following steps:

[0039] Q53, determining the standard field with time attribute in D r as the time standard field.

[0040] The standard field with time attribute can be the takeoff time field, the ticketing time field, etc.

[0041] Q54, obtaining each time standard field in D r and the event time information corresponding to each time standard field.

[0042] In the embodiment, the time information corresponding to each time standard field can be obtained through the original message, and the time information corresponding to the standard field includes the occurrence time of the corresponding event.

[0043] Q55, determine D according to the time information corresponding to each time standard field r corresponding pending time relationship GX1, and the standard time relationship GX2 of the event corresponding to each time standard field; wherein, GX1 is used to represent the D r corresponding to each time standard field in D, GX2 is used to represent the D r corresponding to each time standard field in D, the real occurrence time sequence of the event corresponding to each time standard field.

[0044] Q56, determine D according to the time information corresponding to each time standard field in the original message record r corresponding pending time relationship GX1; it can be understood that GX1 is determined according to the D r corresponding to each time standard field in D, and GX2 is determined according to the real occurrence time sequence of the event corresponding to each time standard field in D r corresponding to each time standard field in D; for example, the time corresponding to the takeoff time field is yesterday, and the time corresponding to the ticket time field is today, which can be concluded that the takeoff time is earlier than the ticket time; and the corresponding standard time relationship should be that the ticket time is earlier than the takeoff time.

[0045] Q56, GX1 and GX2 are different, then determine D r abnormal.

[0046] In this embodiment, GX1 and GX2 are different, indicating that the field value corresponding to the target field in D r is abnormal; therefore, through the above method, it can be determined that the field value corresponding to each standard field in D r is correct.

[0047] Q60, according to D, update the field value corresponding to each target field in the target field list corresponding to each fusion project.

[0048] In this embodiment, each fusion project corresponds to a target field list, and only the field value corresponding to the standard field identical to the target field in D needs to be added to the target field, so as to complete the data update of the target field corresponding to the fusion project.

[0049] The data normalization method of the multiple data sources in the embodiment performs deduplication processing on each target field corresponding to each preset fusion project, obtains a specified field list commonly corresponding to all fusion projects, and determines a target original field in an original message sent by each data source according to a specified field in the specified field list; the target original field is normalized to obtain a standard field list corresponding to each data source; and the field value corresponding to each target field in the target field list corresponding to each fusion project is updated according to the standard field list corresponding to each data source. In the method, an original field in an original message in different data formats can be processed using one data processing rule, and only the target field required by the fusion project needs to be processed, without processing all original fields in the original message, so that the data parsing is simpler, and the data processing efficiency is higher.

[0050] In addition, the method in the embodiment can make the form of the standard field corresponding to each original target field be a unified data format, and different fusion projects only need to process the standard field when using the data in the original message, without considering the format of the message sent by the data source and the form of the original field, so as to further simplify the complexity of data parsing and improve the data processing efficiency.

[0051] Further, after step Q60, the method can include the following steps:

[0052] Q70, obtaining each new target field corresponding to the new fusion project to obtain a new target field list QA=(QA1, QA2, …, QA u , …, QA v ), u=1, 2, …, v; wherein QA u is the u-th new target field corresponding to the new fusion project, and v is the number of new target fields corresponding to the new fusion project.

[0053] Q71, traversing QA, if each new target field in QA exists in B, the new fusion project uses D to update the corresponding field value.

[0054] Q72, if QA u does not exist in B, QA u is added to B, and step Q40 is entered.

[0055] In the embodiment, if a fusion project is added, whether the original field of the data source needs to be newly acquired can be determined according to the new target field corresponding to the new fusion project; even if the new acquisition is required, the new target field is only added to B, without the need to reset the normalization rule of the original field, so that when the fusion project is added, the data parsing workload is greatly simplified.

[0056] Furthermore, after step Q60, the method further includes the following steps:

[0057] Q80, retrieve each original field corresponding to the original message sent by the newly added data source to obtain the list of newly added original fields QB = (QB1, QB2, ..., QB...). x QB y ), x = 1, 2, ..., y; where QB x The x-th original field corresponding to the original message sent to the newly added data source, where y is the number of original fields corresponding to the original message sent to the newly added data source.

[0058] Q81, iterate through QB, if QB x If the specified field exists in B, then QB will be... x Identify the target original fields corresponding to the newly added data source to obtain the target original field list WB corresponding to the newly added data source.

[0059] Q82, add WB to C, then proceed to step Q50.

[0060] In this embodiment, if a new data source is added, it is only necessary to determine whether there is an original field in the original message sent by the new data source that corresponds to the specified field in B; if there is, the original field in the original message that corresponds to the specified field in B is determined as the target original field and added to C; there is no need to reset the normalization rules. Therefore, when adding a new data source, the workload of data parsing can be greatly simplified and the efficiency of data parsing can be improved.

[0061] Q200, traversing D r If D r，p If it is the preset primary key field, then retrieve D. r，p To obtain D r The corresponding primary key field list ZD r =(ZD r，1 ZD r，2 , ..., ZD r，a , ..., ZD r，h(r) ), a = 1, 2, ..., h(r); where ZD r，a D r The corresponding a-th primary key field, h(r), is D. r The number of corresponding primary key fields; primary key fields are used to determine the corresponding trip information.

[0062] In this embodiment, D r There will be primary key fields and non-primary key fields, which can be used to divide D r Retrieve all primary key fields to obtain ZDr .

[0063] Q300, traverse ZD r If ZD r，a If the ID number field is a preset field, then ZD is obtained. r，a Corresponding ID number HD r，a .

[0064] Understandably, ZD r The primary key field may contain an ID number field; the ID number can be the user's national ID number; the corresponding ID number can be obtained from the original message based on the ID number field.

[0065] Q400, according to HD r，a And the preset database mapping table, determine D r The corresponding user data corresponds to the database QD r The preset database mapping table includes several databases and the corresponding ID number for each database.

[0066] In this embodiment, the historical travel information of each user sent by each data source is stored in the corresponding database. It should be noted that there can be multiple databases, and the travel information of the same user is stored in the same database. The database corresponding to the user can be determined based on the ID number.

[0067] Furthermore, the ID number can be converted into a corresponding hash value before being mapped to the database to improve the privacy of the mapping and prevent the ID number from being leaked.

[0068] Q500, QD r Add the target primary key field and its corresponding field value to ZD. r In China; among them, QD r The target primary key field in the data is not in the preset primary key field ZD. r The primary key field in the database.

[0069] In this embodiment, QD r HD is stored in r，a The corresponding list of several standard fields and the field value corresponding to each standard field; capable of being used in QD r The primary key field determines that it is not in ZD r The primary key field in the QD; then QD r The primary key field is not in ZD r Add the primary key field to ZD r In order to achieve the effect of primary key field completion, thereby enabling ZD r Includes all preset primary key fields to ensure ZD r The completeness of the information.

[0070] The event stream data field completion method of the embodiment obtains a standard field list corresponding to each raw message sent by each data source; obtains a primary key field in each standard field list and a certificate number field in the primary key field to obtain a corresponding certificate number; determines a database corresponding to user data corresponding to the certificate number according to the certificate number and a preset database mapping table; determines a target primary key field in the database according to other primary key fields in the standard field list, and then adds the target primary key field and a corresponding field value to the standard field list to complete the completion of the standard field list; so that the business end can obtain complete user data when using the event stream data, thereby improving the completeness of information display of the business platform.

[0071] Further, after Q200, the method can include the following steps:

[0072] Q210, traversing ZD r If the preset certificate number field does not exist in ZD r , the certificate number field corresponding to ZD r and the certificate number HD r are determined according to ZD r，a and a preset certificate number mapping table; step Q400 is entered; wherein the preset certificate number mapping table includes a plurality of primary key field groups and a certificate number corresponding to each primary key field group, and the primary key field group includes a plurality of primary key fields.

[0073] In the embodiment, for the case that the preset certificate number field does not exist in ZD r , the certificate number field corresponding to ZD r and the certificate number HD r，a can be determined by the above method; for example, ZD r contains a flight number field and a seat number field, and the primary key field group formed by the flight number field and the seat number field can determine the corresponding certificate number in the preset certificate number mapping table; then step Q400 is entered.

[0074] Further, for the completed standard field list, some completed standard field lists may have logical errors of event occurrence, therefore, the fusion process needs to determine whether to adopt the completed standard field list before use; specifically, the adoption judgment of the completed standard field list can include the following steps:

[0075] Q600, obtaining an event QE corresponding to a fusion data list RT currently corresponding to a target fusion project; wherein RT includes a plurality of standard fields and a field value currently corresponding to each standard field.

[0076] In this embodiment, the target fusion project can be any one of a plurality of fusion projects. It can be understood that a fusion project corresponds to a target field, and each data source corresponds to a standard field. If the meaning of a target field is the same as that of another standard field, the target field and the standard field are the same, except that the names are different. The data in the target field list corresponding to a fusion project corresponds to an event. For example, the event currently corresponding to the fusion project is check-in. Q610, according to QE and a preset transitionable event mapping table, determining a transitionable event list LE=(LE1, LE2,..., LE c ,..., LE d ), c=1, 2,..., d; wherein LE c is the cth transitionable event corresponding to QE, and d is the number of transitionable events corresponding to QE; LE c occurs after QE.

[0077] In this embodiment, each event corresponds to a subsequent event that can occur, referred to as a transitionable event. For example, the transitionable events corresponding to the check-in event include the cancel check-in event, the take-off event, and the like. The ticketing event is not a transitionable event of the check-in event.

[0078] Q620, obtaining the completed standard field list corresponding to the original message sent by each data source to obtain a completed standard field list set E=(E1, E2,..., E r ,..., E s ), r=1, 2,..., s; wherein E r is the completed standard field list corresponding to the original message sent by the rth data source, and s is the number of data sources; E r =(E r，1 , E r，2 ,..., E r，q ,..., E r，y(r) ); E r，q is the qth standard field in the completed standard field list corresponding to the original message sent by the rth data source, and y(r) is the number of standard fields in the completed standard field list corresponding to the original message sent by the rth data source. Each original message includes a plurality of original fields, and each original field corresponds to a field value. The standard field is obtained according to the corresponding original field. E is used to update RT.

[0079] In this embodiment, the completed standard field list set E can be obtained by the method in the above embodiment, which is not described here.

[0080] Further, after step Q620 and before step Q630, the method can include the following steps:

[0081] Q621, obtaining the data source quality index currently corresponding to each data source to obtain a data source quality index list η = (η1, η2, …, η r , …, η s ); wherein η r is the data source quality index currently corresponding to the rth data source; the data source quality index currently corresponding to each data source is obtained according to a preset data source quality index determination rule.

[0082] Q622, traversing η, if η r < LF, it is determined that the event data stream corresponding to E r is not adoptable; otherwise, step T400 is entered; wherein LF is a preset data source quality index threshold.

[0083] In this embodiment, first, the data source quality index of each completed standard field list corresponding data source is obtained, if the data source quality index of a certain data source is less than the preset data source quality index threshold, it is determined that the data source is abnormal, then the data in the original message sent by the data source also exists abnormality, it is determined that the event stream data corresponding to the completed standard field list of the abnormal data source is not adoptable; the data source quality index is obtained by the preset evaluation system according to the preset evaluation rule.

[0084] Q630, according to the event field corresponding to the field value in each completed standard field list in E, the original message corresponding to each data source is determined to obtain the message event list RA = (RA1, RA2, …, RA r , …, RA s ) corresponding to E; wherein RA r is the message event corresponding to E r .

[0085] It can be understood that each completed standard field list corresponds to a field representing time, which can directly determine the event corresponding to each completed standard field list.

[0086] Q640, traversing RA, if RA r ∈ LE, it is determined that the event data stream corresponding to E r is adoptable.

[0087] In this embodiment, if RA r ∈ LE, it indicates that RA r is the event corresponding to E r that can be transitioned, that is, RA r occurs after QE is in line with the logic of event occurrence, therefore, it is determined that the event data stream corresponding to E r is adoptable; E rThe corresponding event stream data can be used to update RT.

[0088] The event stream data acceptance and judgment method in this embodiment determines the corresponding jumpable event list LE based on the event QE currently corresponding to the fusion project; then it determines whether the event corresponding to each completed standard field is in the LE. If any event corresponding to any completed standard field is in the LE, it means that the event corresponding to the completed standard field occurred after the QE, which is logically consistent, thereby determining the E. r The corresponding event data stream is reliable; the method in this embodiment can avoid the problem of data display disorder or error caused by the business platform processing the received stream data in the order they are received.

[0089] Furthermore, after step Q640, the method may include the following steps:

[0090] Q700, if Then E r Transfer to a preset intermediate data list; where E r The storage duration in the intermediate data list is the preset duration.

[0091] In this embodiment, if Represents RA r This is not the transitional event corresponding to QE, but at this point, E cannot be determined. r The corresponding event data stream is unreliable because the actual order of events may differ from the order of the received corresponding messages. For example, QE represents a ticketing event, while the check-in event occurs before the check-out event. However, the fusion project receives the original message corresponding to the check-out event first. Therefore, the check-out event is not a transitional event to the ticketing event, leading to... The preset duration can be 500ms, which is to set a waiting time window.

[0092] Q710, obtain the occurrence time of the event corresponding to each standard field list in the intermediate data list, to obtain the event occurrence time list t = (t1, t2, ..., t... e , ..., t z ), e = 1, 2, ..., z; where t e Let z be the occurrence time of the event corresponding to the e-th standard field list in the intermediate data list, and z be the number of standard field lists in the intermediate data list.

[0093] Q720. Sort the occurrence times in t according to their chronological order to obtain the sorted list of event occurrence times t' = (t'1, t'2, ..., t'...). e ,…,t'z ); wherein, t' α Before t' α+1 ; α=1, 2, …, z-1.

[0094] In the waiting time window, the events are sorted according to the actual occurrence time, without considering the time when the original packet is received, so as to adjust the real order of the event occurrence.

[0095] Q730, updating the RT according to t'.

[0096] Further, the step Q730 comprises the following steps:

[0097] Q731, obtaining a preset value M=1.

[0098] Q732, updating the RT using t' M .

[0099] Q733, if t' M+1 corresponds to the event in the QE corresponding transition event list, obtaining M=M+1, and entering the step Q740; otherwise, deleting the t' M+1 corresponding standard field list.

[0100] In the embodiment, it can be understood that the QE is dynamically changed, and the RT is updated once, and the QE will follow the corresponding update; through the above method, all events which are suspected to be not transition events can be reordered according to the real occurrence time of the events, and then the RT is updated in turn, so as to ensure the integrity of the data and avoid the data being discarded by mistake.

[0101] In addition, although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. In addition or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps, etc.

[0102] The embodiment of the present application also provides a non-transitory computer readable storage medium, which can be arranged in an electronic device to save at least one instruction or at least one program related to a method in the method embodiment, and the at least one instruction or the at least one program is loaded and executed by the processor to realize the method provided by the above-mentioned embodiment.

[0103] The program product can employ any combination of one or more computer-readable media. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] The computer-readable signal medium can include a computer-readable storage medium that is propagated as a carrier wave. The computer-readable signal medium can further be any computer-readable medium that is not a storage medium. The computer-readable signal medium can be a computer-readable storage medium that is a propagated signal.

[0105] The program code embodied on the computer-readable media can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0106] The program code can be executed by one or more programmable processors, which can be implemented using one or more microprocessors, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any other devices suitable for retrieval and execution of instructions. The program code can execute entirely on a user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0107] Embodiments of the present application also provide an electronic device including a processor and the aforementioned non-transitory computer-readable storage medium.

[0108] The electronic device according to the embodiment of the present application. The electronic device is merely an example, and should not bring any limitation to the function and use range of the embodiments of the present application.

[0109] The electronic device is in the form of a general purpose computing device. Components of the electronic device can include, but are not limited to, the at least one processor described above, the at least one memory described above, a bus that connects the different system components including the memory and the processor.

[0110] The memory stores a program code that can be executed by the processor, such that the processor performs the steps in the various embodiments described in this specification.

[0111] The memory can include a readable medium in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and can further include read only memory (ROM).

[0112] The memory can also include a program / utility having a set of program modules that include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, which can include implementation of a network environment, each or a combination of these examples.

[0113] The bus can be representative of one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or local bus using any of a variety of bus structures.

[0114] The electronic device can also communicate with one or more external devices such as a keyboard or a pointing device, a Bluetooth device, etc. through an input / output (I / O) interface. The electronic device can also communicate with one or more devices that enable a user to interact with the electronic device, and / or one or more devices (e.g., a router, a modem, etc.) that enable the electronic device to communicate with one or more other computing devices. Such communication can occur via an I / O interface. The electronic device can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or the public network, such as the Internet) through a network adapter. The network adapter communicates with the other modules of the electronic device via the bus. It should be appreciated that although not shown, other hardware and / or software modules could be used in conjunction with the electronic device. Such hardware and / or software modules can include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0115] Those skilled in the art can clearly understand the example embodiments described herein through the above description of the example embodiments, and the example embodiments described herein can be implemented by software or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0116] Embodiments of the present disclosure also provide a computer program product comprising program code for causing an electronic device to perform the steps of the methods according to the various example embodiments of the present disclosure described above in the specification when the program product is run on the electronic device.

[0117] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art should understand that the above examples are only for illustration, and are not intended to limit the scope of the present disclosure. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present disclosure.

Claims

1. An event stream data field completion method, characterized by, The method comprises the following steps: Q100, obtaining the standard field list corresponding to the original message sent by each data source to obtain the standard field list set D=(D1, D2, …, D r , …, D s ), r=1, 2, …, s; wherein D r is the standard field list corresponding to the original message sent by the rth data source, and s is the number of data sources; D r is the standard field list corresponding to C r ; D r =(D r，1 , D r，2 , …, D r，p , …, D r，g（r） ), p=1, 2, …, g(r); D r，p is the pth standard field in the standard field list corresponding to the original message sent by the rth data source, and g(r) is the number of standard fields in the standard field list corresponding to the original message sent by the rth data source; C r is the target original field list corresponding to the original message sent by the rth data source; Q200, traversing D r , if D r，p is a preset primary key field, obtaining D r，p to obtain a corresponding first primary key field list ZD r r = (ZD r，1 , ZD r，2 , …, ZD r，a , …, ZD r，h（r） ), a = 1, 2, …, h (r); wherein ZD r，a is the a-th primary key field corresponding to D r , and h (r) is the number of primary key fields corresponding to D r ; the primary key field is used to determine the corresponding journey information;​ Q300, traversing ZD r , if ZD r，a is a preset certificate number field, obtaining ZD r，a corresponding certificate number HD r，a ; Q400, according to HD r，a and a preset database mapping table, determine D r corresponding to the user data corresponding to the database QD r ; wherein the preset database mapping table includes a number of databases and a corresponding certificate number corresponding to each database; Q500, add the target primary key field in QD r to ZD r ; wherein the target primary key field in QD r is a primary key field in the preset primary key fields that is not in the primary key fields in ZD r . 2.The event stream data field completion method of claim 1, wherein, After Q200, the method comprises the following steps: Q210, Traverse ZD r If ZD r If there is no preset ID number field, then it will be based on ZD. r Based on the preset ID number mapping table, determine ZD r The corresponding ID number field and ID number HD r，a Proceed to step Q400; The preset ID number mapping table includes several primary key field groups and the ID number corresponding to each primary key field group. The primary key field group includes several primary key fields. 3.The event stream data field completion method of claim 1, wherein, After step Q500, the method comprises the following steps: Q600, acquiring an event QE corresponding to a fusion data list RT corresponding to the target fusion project at present; wherein RT comprises a plurality of standard fields and a field value corresponding to each standard field at present; Q610, Based on QE and the preset transitional event mapping table, determine the transitional event list LE = (LE1, LE2, ..., LE) corresponding to QE. c , ..., LE d ), c=1,2,…,d; where LE c Let d be the c-th transitional event corresponding to QE, and d be the number of transitional events corresponding to QE; LE c It occurred after QE; Q620, obtain the completed standard field list corresponding to the original message sent by each data source, so as to obtain the completed standard field list set E = (E1, E2, ..., E...). r , ..., E s ), r=1,2,…,s; where, E r The complete list of standard fields corresponding to the original message sent to the r-th data source, where s is the number of data sources; E r =(E r，1 E r，2 , ..., E r，q ,…,E r，y（r） ); E r，q Let y(r) be the q-th standard field in the completed standard field list corresponding to the original message sent by the r-th data source, and let y(r) be the number of standard fields in the completed standard field list corresponding to the original message sent by the r-th data source. Each original message includes several original fields, and each original field has a corresponding field value. The standard fields are obtained based on the corresponding original fields. E is used to update RT. Q630, according to the field value corresponding to the event field in each completed standard field list in E, determine the event corresponding to the original message sent by each data source to obtain the message event list RA=(RA1, RA2, …, RA r , …, RA s ) corresponding to E; wherein RA r is the message event corresponding to E r . Q640, traverse RA, if RA r ∈ LE, then determine E r The corresponding event data stream is admissible.

4. The event stream data field completion method of claim 3, wherein, After step Q610 and before step Q620, the method comprises the following steps: Q611, obtaining a data source quality index currently corresponding to each data source to obtain a data source quality index list η = (η1, η2, …, η r ) s ); wherein η r is a data source quality index currently corresponding to the rth data source; the data source quality index currently corresponding to each data source is obtained according to a preset data source quality index determination rule; Q612, traverse η, if η r < LF, then E r The corresponding event data stream is not credible; otherwise, go to step Q620; wherein, LF is a preset data source instruction index threshold.

5. The event stream data field completion method of claim 3, wherein, After step Q640, the method comprises the following steps: Q700, if RA r ∉LE, then E r is stored in a preset intermediate data list; wherein E r the storage duration of E in the intermediate data list is a preset duration; Q710, obtain the occurrence time of the event corresponding to each standard field list in the intermediate data list, to obtain the event occurrence time list t = (t1, t2, ..., t...). e , ..., t z ), e = 1, 2, ..., z; where t e Let z be the occurrence time of the event corresponding to the e-th standard field list in the intermediate data list, and z be the number of standard field lists in the intermediate data list. Q720. Sort the occurrence times in t according to their chronological order to obtain the sorted list of event occurrence times t' = (t'1, t'2, ..., t''). e ,…,t' z ); where t' α Earlier than t' α+1 ; α = 1, 2, ..., z-1; Q730, updating RT according to t'.

6. The event stream data field completion method of claim 5, wherein, Step Q730 comprises the following steps: Q731, acquiring a preset value M=1; Q732, using t' M Update RT; Q733, if t' M+1 If the corresponding event is in the list of QE's corresponding transitable events, then M = M + 1 is obtained and step Q740 is entered; otherwise, t' M+1 the corresponding list of standard fields.

7. The event stream data field completion method of claim 1, wherein, The certificate number comprises an identity card number of the user. 8.A non-transitory computer readable storage medium having stored therein at least one instruction or at least one piece of program, characterized in that, The at least one instruction or the at least one program is loaded and executed by the processor to realize the event flow data field completion method as claimed in any one of claims 1-7.

9. An electronic device, comprising: It comprises a processor and the non-transitory computer readable storage medium of claim 8.

Images