A method for accepting and judging the reliability of event stream data, an electronic device and a storage medium

Abstract

This invention provides a method, electronic device, and storage medium for accepting and judging event stream data, relating to the field of accepting and judging event stream data. The method includes: obtaining the event QE corresponding to the target field list RT currently corresponding to the target fusion project; determining the jumpable event list LE corresponding to QE according to QE and a preset jumpable event mapping table; obtaining the completed standard field list corresponding to the original message sent by each data source to obtain the completed standard field list set E; determining the event corresponding to the original message sent by each data source according to the field value corresponding to the event field in each completed standard field list in E to obtain the event list RA corresponding to E; traversing RA, if RA r If ∈LE, then determine E. r The corresponding event data stream is reliable; this invention can prevent business platforms from processing the received stream data in the order they are received, which could lead to data display disorder or errors.

Description

Technical Field

[0001] This invention relates to the field of event stream data acceptance and judgment, and in particular to an event stream data acceptance and judgment method, electronic device and storage medium. Background Technology

[0002] In the civil aviation sector, civil aviation management platforms typically receive data streams from multiple different data sources to monitor flight dynamics or user travel event information in real time. Each data stream from each data source is then processed to monitor flight or user travel information. When an event occurs, corresponding event stream data is generated and sent to the business platform in real time. However, during the transmission of event stream data, network fluctuations may occur, causing the time stream data corresponding to earlier events to arrive at the business platform later, and vice versa. If the business platform processes the received stream data in the order they appear, it will lead to disordered or incorrect data display. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0004] According to a first aspect of this application, a method for accepting and judging the credibility of event stream data is provided, the method comprising the following steps:

[0005] T100 retrieves the event QE corresponding to the target field list RT currently corresponding to the target fusion project; where RT includes several target fields and the field value currently corresponding to each target field.

[0006] T200, based on QE and the preset transitional event mapping table, determines the transitional event list LE = (LE1, LE2, ..., LE...). c , ..., LE d ), c = 1, 2, ..., d; where LE c Let d be the c-th transitional event corresponding to QE, and d be the number of transitional events corresponding to QE; LE c It happened after QE.

[0007] T300 retrieves the completed standard field list corresponding to the original message sent by each data source, to obtain the completed standard field list set E = (E1, E2, ..., E...). r , ..., E s ), r = 1, 2, ..., s; where, E r The complete list of standard fields corresponding to the original message sent to the r-th data source, where s is the number of data sources; E r =(E r，1 E r，2 , ..., Er，q ,…,E r，y(r) ); E r，q Let q be the qth standard field in the complete standard field list corresponding to the original message sent by the r-th data source, and y(r) be the number of standard fields in the complete standard field list corresponding to the original message sent by the r-th data source. Each original message includes several original fields, and each original field has a corresponding field value. The standard fields are obtained based on the corresponding original fields. E is used to update RT.

[0008] T400 determines the event corresponding to each original message sent by each data source based on the field value corresponding to the event field in each completed standard field list in E, thus obtaining the event list RA = (RA1, RA2, ..., RA3) for E. r , ...,RA s ); where RA r For E r The corresponding event.

[0009] T500, iterate through RA, if RA r If ∈LE, then determine E. r The corresponding event data stream is acceptable.

[0010] According to another aspect of this application, a non-transitory computer-readable storage medium is also provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described event stream data acceptance and judgment method.

[0011] According to another aspect of this application, an electronic device is also provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0012] The present invention has at least the following beneficial effects:

[0013] The event stream data acceptance and judgment method of the present invention determines the corresponding jumpable event list LE based on the event QE corresponding to the current fusion project; then it determines whether the event corresponding to each completed standard field is in the LE. If the event corresponding to any completed standard field is in the LE, it means that the event corresponding to the completed standard field occurred after QE, which is logically consistent, thereby determining E. r The corresponding event data stream is reliable; the method in this invention can avoid the problem of data display disorder or error caused by the business platform processing the received stream data in the order it is received. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of the event stream data acceptance and judgment method provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be noted that, based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Furthermore, this device and / or practice the method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0018] The following will refer to Figure 1 The flowchart of the event stream data acceptance and judgment method introduces an event stream data acceptance and judgment method.

[0019] The method for accepting and judging the reliability of event stream data may include the following steps:

[0020] T100 retrieves the event QE corresponding to the target field list RT currently corresponding to the target fusion project; where RT includes several target fields and the field value currently corresponding to each target field.

[0021] In this embodiment, the target fusion project can be any one of several fusion projects; it can be understood that the fusion project corresponds to the target field, and each data source corresponds to the standard field. If the meaning of one target field is the same as the meaning of another standard field, then the target field and the standard field are the same, only the names are different; the data in the target field list corresponding to the fusion project corresponds to an event; for example, the event currently corresponding to the fusion project is check-in.

[0022] T200, based on QE and the preset transitional event mapping table, determines the transitional event list LE = (LE1, LE2, ..., LE...). c , ..., LE d ), c = 1, 2, ..., d; where LE c Let d be the c-th transitional event corresponding to QE, and d be the number of transitional events corresponding to QE; LE c It happened after QE.

[0023] In this embodiment, each event corresponds to a subsequent event that can occur, which is called a transitional event; for example, the transitional events corresponding to the check-in event include the check-out event, the take-off event, etc.; the ticketing event is not a transitional event of the check-in event.

[0024] T300 retrieves the completed standard field list corresponding to the original message sent by each data source, to obtain the completed standard field list set E = (E1, E2, ..., E...). r , ..., E s ), r = 1, 2, ..., s; where, E r The complete list of standard fields corresponding to the original message sent to the r-th data source, where s is the number of data sources; E r =(E r，1 E r，2 , ..., E r，q ,…,E r，y(r) ); E r，q Let q be the qth standard field in the complete standard field list corresponding to the original message sent by the r-th data source, and y(r) be the number of standard fields in the complete standard field list corresponding to the original message sent by the r-th data source. Each original message includes several original fields, and each original field has a corresponding field value. The standard fields are obtained based on the corresponding original fields. E is used to update RT.

[0025] Furthermore, after step T300 and before step T400, the method may include the following steps:

[0026] T310, obtain the current data source quality index for each data source to obtain a list of data source quality indices η = (η1, η2, ..., η...). r , ..., η s ); where η r This represents the current data source quality index corresponding to the r-th data source. The current data source quality index for each data source is obtained according to the preset data source quality index determination rules.

[0027] T320, iterate through η, if η r If <LF, then E is determined. rThe corresponding event data stream is unreliable; otherwise, proceed to step T400; where LF is a preset data source quality index threshold.

[0028] In this embodiment, the data source quality index of the data source corresponding to each completed standard field list is first obtained. If the data source quality index of a certain data source is less than the preset data source quality index threshold, it is determined that the data source is abnormal. Then, the data in the original message sent by the data source is also abnormal. Therefore, it is determined that the event stream data corresponding to the completed standard field list of the abnormal data source is unreliable. The data source quality index is obtained by the preset data source quality evaluation system according to the preset evaluation rules.

[0029] T400 determines the event corresponding to each original message sent by each data source based on the field value corresponding to the event field in each completed standard field list in E, thus obtaining the event list RA = (RA1, RA2, ..., RA3) for E. r , ...,RA s ); where RA r For E r The corresponding event.

[0030] Understandably, each completed list of standard fields corresponds to a field representing time, which can directly determine the event corresponding to each completed list of standard fields.

[0031] T500, iterate through RA, if RA r If ∈LE, then determine E. r The corresponding event data stream is acceptable.

[0032] In this embodiment, if RA r ∈LE, representing RA r For E r The corresponding jumpable event, i.e., RA r It is logical for E to occur after QE; therefore, E is determined. r The corresponding event data stream is reliable; E can be used directly. r The corresponding event stream data can be used to update RT.

[0033] The event stream data acceptance and judgment method in this embodiment determines the corresponding jumpable event list LE based on the event QE currently corresponding to the fusion project; then it determines whether the event corresponding to each completed standard field is in the LE. If the event corresponding to any completed standard field is in the LE, it means that the event corresponding to the completed standard field occurred after the QE, which is logically consistent, thereby determining the E. rThe corresponding event data stream is reliable; the method in this embodiment can avoid the problem of data display disorder or error caused by the business platform processing the received stream data in the order they are received.

[0034] Furthermore, after step T500, the method may include the following steps:

[0035] T600, if Then E r Transfer to a preset intermediate data list; where E r The storage duration in the intermediate data list is the preset duration.

[0036] In this embodiment, if Represents RA r This is not the transitional event corresponding to QE, but at this point, E cannot be determined. r The corresponding event data stream is unreliable because the actual order of events may differ from the order of the received corresponding messages. For example, QE represents a ticketing event, while the check-in event occurs before the check-out event. However, the fusion project receives the original message corresponding to the check-out event first. Therefore, the check-out event is not a transitional event to the ticketing event, leading to... The preset duration can be 500ms, which is to set a waiting time window.

[0037] T610, retrieve the occurrence time of the event corresponding to each standard field list in the intermediate data list, to obtain the event occurrence time list t = (t1, t2, ..., t... e , ..., t z ), e = 1, 2, ..., z; where t e Let z be the occurrence time of the event corresponding to the e-th standard field list in the intermediate data list, and z be the number of standard field lists in the intermediate data list.

[0038] T620, sort the occurrence times in t according to their chronological order to obtain the sorted list of event occurrence times t' = (t'1, t'2, ..., t'...). e ,…,t' z ); where t' α Earlier than t' α+1 ; α = 1, 2, ..., z-1.

[0039] Within the waiting time window, events are sorted according to their actual occurrence time, without considering the time when the original message was received, thus adjusting the true order of events.

[0040] T630, retrieve the preset value M=1.

[0041] T640, using t' M Update RT.

[0042] T650, if t' M+1 If the corresponding event is in the list of transitionable events corresponding to QE, then obtain M = M + 1 and proceed to step T640; otherwise, delete t' M+1 The corresponding list of standard fields.

[0043] In this embodiment, it can be understood that QE is dynamically changing. Every time RT is updated, QE will be updated accordingly. Through the above method, all events that are suspected to be non-transitionable events can be reordered according to the actual occurrence time of the events, and then RT can be updated in sequence, thereby ensuring data integrity and avoiding data being discarded incorrectly.

[0044] In an exemplary embodiment, the complete list of standard fields corresponding to the original message sent by each data source in the above embodiments can be obtained through the following steps:

[0045] T010, obtain each target field corresponding to each preset fusion project to obtain a target field list set A = (A1, A2, ..., A...). i A n ), i=1, 2,...,n; where, A i A represents the list of target fields corresponding to the i-th fusion project, where n is the preset number of fusion projects; i =(A i，1 A i，2 A i，j A i，f(i) ), j=1,2,…,f(i);A i，j Let f(i) be the j-th target field corresponding to the i-th fusion project, and f(i) be the number of target fields corresponding to the i-th fusion project; the target fields are used to provide corresponding field values ​​for the fusion project.

[0046] In this embodiment, the fusion project can be understood as the project by which the business end merges and displays data sent from different data sources; for example, a fusion project for displaying user travel information; each fusion project corresponds to a target field list, which can be in the form of a full Java bean, and includes all the standard fields required by the corresponding fusion project; for example, the target field list corresponding to the fusion project for displaying user travel information includes standard fields such as flight number field, ID number field, and flight segment number field.

[0047] It is understood that, since the number of target fields corresponding to each fusion project is different in this embodiment, f(i) does not refer to a specific function or function result value, but to a possible value that varies with the specific value of i. For example, when i=1, f(i)=3; when i=2, f(i)=4; when i=3, f(i)=3.

[0048] T020, perform duplicate processing on A to obtain a list of specified fields B = (B1, B2, ..., B...). k B m ), k = 1, 2, ..., m; where B k The specified field is the k-th field obtained after deduplication of A, and m is the number of specified fields obtained after deduplication of A.

[0049] In this embodiment, it is understood that the types of data processed by each fusion project may be different, and the required standard fields may not be exactly the same. For example, the fusion project for displaying user travel information mainly processes fields related to user travel, while the fusion project for displaying flight information processes fields related to flights. The two may be dealing with the same fields.

[0050] T030 retrieves the original message sent by each data source; each original message includes several original fields, and each original field has a corresponding field value.

[0051] In this embodiment, each data source will push relevant information in real time in the form of raw messages. The raw messages will have multiple data formats, such as JSON or XML. It can be understood that, regardless of the format, the raw messages will contain several raw fields and the field values ​​corresponding to each raw field.

[0052] T040, according to B, obtain the target original field corresponding to each original message to obtain the target original field list set C = (C1, C2, ..., C... r C s ), r = 1, 2, ..., s; where, C r C represents the list of target raw fields corresponding to the raw message sent from the r-th data source, where s is the number of data sources; r =(C r，1 C r，2 C r，p ,…,C r，g(r) ), p=1, 2,..., g(r); C r，pLet g(r) be the p-th target original field in the target original field list corresponding to the original message sent by the r-th data source, and g(r) be the number of target original fields corresponding to the original message sent by the r-th data source; C r，p This corresponds to a specified field in B.

[0053] In this embodiment, a target field in step T010 corresponds to a certain original field in the original message; for example, the original field corresponding to the flight number in the original message is Flightno, and the standard field corresponding to the flight number is flightnumber; Flightno and flightnumber have the same meaning; therefore, the target original field can be determined from the original fields of each original message according to B; the target original field can be understood as the original field that has a corresponding standard field in B.

[0054] It should be noted that when obtaining the target original field, if any two original fields in an original message are different, then the target original field can be uniquely determined and obtained based on the target field; however, if there are multiple original fields in an original message that have the same original field but different meanings, then it is necessary to specify which original field it is based on the meaning of the original field.

[0055] T050, according to the preset normalization rules, convert each target original field in C into the corresponding standard field to obtain the standard field list set D = (D1, D2, ..., D...). r D s ); where D r C r Corresponding standard field list; D r =(D r，1 D r，2 D r，p ,…,D r，g(r) );D r，p C r，p The corresponding standard fields.

[0056] In this embodiment, it is understood that different data sources may name fields with the same meaning differently. In order to facilitate the acquisition of data in the project, it is necessary to normalize the original fields corresponding to different data sources. Specifically, step T050 may include the following steps:

[0057] T501, iterate through C, if C r，p If the first letter of the letter is capitalized, then C will be... r，p Change the first letter of C to a lowercase; otherwise, do not use C. r，pThe first letter is changed to obtain the intermediate original field list QC = (QC1, QC2, ..., QC) corresponding to C. r QC s ); where QC r C r The corresponding list of intermediate raw fields; QC r =(QC r，1 QC r，2 QC r，p QC r，g(r) QC r，p C r，p The corresponding intermediate original field.

[0058] In this embodiment, the first letter of the target original field corresponding to the original message may be a capital letter. In order to make the original field conform to the standardized structure of Java bean processing, the first letter of the target original field that starts with a capital letter needs to be converted to the corresponding lowercase letter.

[0059] T502, according to the preset standard field mapping table, converts each intermediate original field in QC into the corresponding standard field to obtain D; wherein, the preset standard field mapping table includes several intermediate original fields and the standard field corresponding to each intermediate original field.

[0060] In this embodiment, the standard field mapping table contains each intermediate original field corresponding to each original message and the standard field corresponding to each original field; for example, if both flightno and flight represent flight number, then the standard field corresponding to both flightno and flight is flightnumber; thus, each intermediate original field in QC can be converted into the corresponding standard field to obtain D.

[0061] Furthermore, for different fusion projects, there are several preset main standard fields, defined as primary key fields; the field values ​​corresponding to all primary key fields can uniquely identify the user's itinerary information; for example, primary key fields include flight number field, ID number field, flight segment number field, etc.; however, the original message sent by each data source may not necessarily contain all the preset primary key fields; for example, the original message sent by a certain data source only contains flight number field and flight segment number field; in order for the business side to be able to fully display the user's itinerary information, it is necessary to complete the primary key field of each standard field list in D. Specifically, after step T050, the method may also include the following steps:

[0062] T510, Traversing D r If D r，p If it is the preset primary key field, then retrieve D. r，p To obtain Dr The corresponding primary key field list ZD r =(ZD r，1 ZD r，2 , ..., ZD r，a , ..., ZD r，h(r) ), a = 1, 2, ..., h(r); where ZD r，a D r The corresponding a-th primary key field, h(r), is D. r The number of corresponding primary key fields; primary key fields are used to determine the corresponding trip information.

[0063] In this embodiment, D r There will be primary key fields and non-primary key fields, which can be used to divide D r Retrieve all primary key fields to obtain ZD r .

[0064] T520, traversing ZD r If ZD r，a If the ID number field is a preset field, then ZD is obtained. r，a Corresponding ID number HD r，a .

[0065] Understandably, ZD r The primary key field may contain an ID number field; the ID number can be the user's national ID number; the corresponding ID number can be obtained from the original message based on the ID number field.

[0066] T530, according to HD r，a And the preset database mapping table, determine D r The corresponding database QD for the corresponding user r The preset database mapping table includes several databases and the corresponding ID number for each database.

[0067] In this embodiment, the historical travel information of each user sent by each data source is stored in the corresponding database. It should be noted that there can be multiple databases, and the travel information of the same user is stored in the same database. The database corresponding to the user can be determined based on the ID number.

[0068] Furthermore, the ID number can be converted into a corresponding hash value before being mapped to the database to improve the privacy of the mapping and prevent the ID number from being leaked.

[0069] T540, QD r Add the target primary key field and its corresponding field value to ZD. r In China; among them, QD r The target primary key field in the ZD is not among the preset primary key fields. rThe primary key field in the database.

[0070] In this embodiment, QD r HD is stored in r，a The corresponding list of several standard fields and the field value corresponding to each standard field; capable of being used in QD r The primary key field determines that it is not in ZD r The primary key field in the QD; then QD r The primary key field is not in ZD r Add the primary key field to ZD r In order to achieve the effect of primary key field completion, thereby enabling ZD r Includes all preset primary key fields to ensure ZD r The completeness of the information.

[0071] Furthermore, after step T510, the method includes the following steps:

[0072] T511, traversing ZD r If ZD r If there is no preset ID number field, then it will be based on ZD. r Based on the preset ID number mapping table, determine ZD r The corresponding ID number field and ID number HD r，a Proceed to step S530; wherein, the preset document number mapping table includes several primary key field groups and the document number corresponding to each primary key field group, and the primary key field group includes several primary key fields.

[0073] In this embodiment, for D r If the preset ID number field is not available, the method in step S511 can be used to first determine ZD. r The corresponding ID number field and ID number HD r，a For example, ZD r It includes a flight number field and a seat number field. Based on the primary key field group composed of the flight number field and the seat number field, the corresponding ID number can be determined in the preset ID number mapping table; then proceed to step S530.

[0074] In this embodiment, a list of standard fields corresponding to the original message sent by each data source is obtained; the primary key field in each list of standard fields is obtained, and the ID number field in the primary key field is obtained to obtain the corresponding ID number; based on the ID number and a preset database mapping table, the database corresponding to the user data of the ID number is determined; based on other primary key fields in the list of standard fields, the target primary key field is determined in the database, and then the target primary key field and its corresponding field value are added to the list of standard fields to complete the list of standard fields; thereby enabling the business end to obtain complete user data when using the event stream data, thus improving the completeness of information display on the business platform.

[0075] Furthermore, after step T050, the method may further include the following steps:

[0076] T550, D r The standard field with the time attribute is determined as the time standard field.

[0077] Standard fields with time attributes can include departure time, ticket issuance time, etc.

[0078] S551, obtain D r Each time standard field and the event time information corresponding to each time standard field.

[0079] In this embodiment, the time information corresponding to each time standard field can be obtained through the original message. The time information corresponding to the standard field includes the occurrence time of the corresponding event.

[0080] T552, determine D based on the time information corresponding to each time standard field. r The corresponding undetermined time relationship GX1, and the standard time relationship GX2 for each time standard field; where GX1 is used to characterize D r The chronological order of each time standard field in GX2 is used to characterize D. r The actual order of the occurrence times of the events corresponding to each time standard field in the data.

[0081] D is determined based on the time information corresponding to each time standard field in the original message record. r The corresponding undetermined time relation GX1; it can be understood that GX1 is based on D r The order of time corresponding to each time standard field in the standard is determined by the time sequence, while GX2 is based on D. rThe actual order of events corresponding to each time standard field in the system is determined; for example, the departure time field corresponds to yesterday's time, while the ticket issuance time field corresponds to today's time, which can lead to a pending time relationship where the departure time is earlier than the ticket issuance time; while the corresponding standard time relationship should be that the ticket issuance time is earlier than the departure time.

[0082] If T553, GX1, and GX2 are different, then determine D. r abnormal.

[0083] In this implementation, GX1 and GX2 are different, representing D r The value of the target field in the data is abnormal; therefore, the above method can be used to determine D. r The field values ​​corresponding to each standard field in the code are correct.

[0084] T600, based on D, updates the field values ​​corresponding to each target field in the target field list for each fusion project.

[0085] In this embodiment, each fusion project corresponds to a target field list. Simply add the field value corresponding to the standard field in D that is the same as the target field to the target field to complete the data update of the target field corresponding to the fusion project.

[0086] The data normalization method for multiple data sources in this embodiment performs deduplication processing on each target field corresponding to each preset fusion project to obtain a list of specified fields common to all fusion projects. Based on the specified fields in the list of specified fields, the target original fields in the original message sent by each data source are determined. The target original fields are then normalized to obtain a list of standard fields corresponding to each data source. The field values ​​corresponding to each target field in the target field list corresponding to each fusion project are updated based on the standard field list corresponding to each data source. In this method, a single data processing rule can be used to process the original fields in original messages of different data formats, and only the target fields required by the fusion project are processed, without processing all original fields in the original message. This simplifies data parsing and increases data processing efficiency.

[0087] In addition, the method in this embodiment ensures that the standard fields corresponding to each original target field are in a uniform data format. When different fusion projects use data from the original message, they only need to process the standard fields without having to consider the format of the message sent by the data source or the form of the original fields, thereby further simplifying the complexity of data parsing and improving data processing efficiency.

[0088] Furthermore, after step T050, the method may include the following steps:

[0089] T070, retrieve each newly added target field corresponding to the newly added fusion project to obtain a list of newly added target fields QA = (QA1, QA2, ..., QA...). u QA v ), u=1, 2,...,v; among them, QA u Let v be the u-th newly added target field corresponding to the newly added fusion project, and v be the number of newly added target fields corresponding to the newly added fusion project.

[0090] T071, iterate through QA. If every newly added target field in QA exists in B, then the newly added fusion project updates the corresponding field value using D.

[0091] T072, if QA u If it does not exist in B, then QA will be... u Add it to B and proceed to step S400.

[0092] In this embodiment, if a new fusion project is added, it can be determined whether to add or retrieve the original fields of the data source based on the new target fields corresponding to the new fusion project. Even if it is necessary to add or retrieve the original fields, it is only necessary to add the new target fields to B. There is no need to reset the normalization rules of the original fields, which greatly simplifies the workload of data parsing when adding a new fusion project.

[0093] Furthermore, after step T050, the method further includes the following steps:

[0094] T080, retrieve each original field corresponding to the original message sent by the newly added data source to obtain the list of newly added original fields QB = (QB1, QB2, ..., QB...). x QB y ), x = 1, 2, ..., y; where QB x The x-th original field corresponding to the original message sent to the newly added data source, where y is the number of original fields corresponding to the original message sent to the newly added data source.

[0095] T081, iterate through QB, if QB x If a corresponding specified field exists in B, then QB will be... x Identify the target original fields corresponding to the newly added data source to obtain the target original field list WB corresponding to the newly added data source.

[0096] T082, add WB to C, then proceed to step T050.

[0097] In this embodiment, if a new data source is added, it is only necessary to determine whether there is an original field in the original message sent by the new data source that corresponds to the specified field in B; if there is, the original field in the original message that corresponds to the specified field in B is determined as the target original field and added to C; there is no need to reset the normalization rules. Therefore, when adding a new data source, the workload of data parsing can be greatly simplified and the efficiency of data parsing can be improved.

[0098] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0099] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0100] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0101] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0102] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0103] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0104] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0105] An electronic device according to this embodiment of the present application. The electronic device is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0106] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and buses connecting different system components (including memory and processor).

[0107] The memory stores program code that can be executed by the processor, causing the processor to perform the steps described in the various embodiments of this specification.

[0108] The storage may include readable media in the form of volatile storage, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0109] The storage may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more applications, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0110] A bus can represent one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus architectures.

[0111] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0112] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0113] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0114] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A method for accepting and judging the reliability of event stream data, characterized in that, The method includes the following steps: T100 retrieves the event QE corresponding to the target field list RT currently corresponding to the target fusion project; where RT includes several target fields and the field value currently corresponding to each target field; T200, based on QE and the preset transitional event mapping table, determines the transitional event list LE = (LE1, LE2, ..., LE...). c , ..., LE d ), c = 1, 2, ..., d; where LE c Let d be the c-th transitional event corresponding to QE, and d be the number of transitional events corresponding to QE; LE c It occurred after QE; T300 retrieves the completed standard field list corresponding to the original message sent by each data source, to obtain the completed standard field list set E = (E1, E2, ..., E...). r , ..., E s ), r = 1, 2, ..., s; where, E r The complete list of standard fields corresponding to the original message sent to the r-th data source, where s is the number of data sources; E r =(E r，1 E r，2 , ..., E r，q ,…,E r，y(r) ); E r，q Let y(r) be the q-th standard field in the completed standard field list corresponding to the original message sent by the r-th data source, and let y(r) be the number of standard fields in the completed standard field list corresponding to the original message sent by the r-th data source. Each original message includes several original fields, and each original field has a corresponding field value. The standard fields are obtained based on the corresponding original fields. E is used to update RT. T400 determines the event corresponding to each original message sent by each data source based on the field value corresponding to the event field in each completed standard field list in E, thus obtaining the event list RA = (RA1, RA2, ..., RA3) for E. r , ...,RA s ); where RA r For E r The corresponding event; T500, iterate through RA, if RA r If ∈LE, then determine E. r The corresponding event data stream is acceptable.

2. The event stream data acceptance and judgment method according to claim 1, characterized in that, After step T300 and before step T400, the method includes the following steps: T310, obtain the current data source quality index for each data source to obtain a list of data source quality indices η = (η1, η2, ..., η...). r , ..., η s ); where η r This is the data source quality index corresponding to the r-th data source; the data source quality index corresponding to each data source is obtained according to the preset data source quality index determination rules. T320, iterate through η, if η r If <LF, then E is determined. r The corresponding event data stream is unreliable; otherwise, proceed to step T400; where LF is a preset data source quality index threshold.

3. The event stream data acceptance and judgment method according to claim 1, characterized in that, After step T500, the method includes the following steps: T600, if Then E r Transfer to a preset intermediate data list; where E r The storage duration in the intermediate data list is the preset duration; T610, retrieve the occurrence time of the event corresponding to each standard field list in the intermediate data list, to obtain the event occurrence time list t = (t1, t2, ..., t... e , ..., t z ), e = 1, 2, ..., z; where t e Let z be the occurrence time of the event corresponding to the e-th standard field list in the intermediate data list, and z be the number of standard field lists in the intermediate data list. T620, sort the occurrence times in t according to their chronological order to obtain the sorted list of event occurrence times t' = (t'1, t'2, ..., t'...). e ,…,t' z ); where t' α Earlier than t' α+1 ; α = 1, 2, ..., z-1; T630, obtain the preset value M=1; T640, using t' M Update RT; T650, if t' M+1 If the corresponding event is in the list of transitionable events corresponding to QE, then obtain M = M + 1 and proceed to step T640; otherwise, delete t' M+1 The corresponding list of standard fields.

4. The event stream data acceptance and judgment method according to claim 1, characterized in that, Prior to step T100, the method includes the following steps: T010, obtain each target field corresponding to each preset fusion project to obtain a target field list set A = (A1, A2, ..., A...). i A n ), i=1, 2,...,n; where, A i A represents the list of target fields corresponding to the i-th fusion project, where n is the preset number of fusion projects; i =(A i，1 A i，2 A i，j A i，f(i) ), j=1,2,…,f(i);A i，j Let f(i) be the j-th target field corresponding to the i-th fusion project, and f(i) be the number of target fields corresponding to the i-th fusion project; the target fields are used to provide corresponding field values ​​for the fusion project. T020, perform duplicate processing on A to obtain a list of specified fields B = (B1, B2, ..., B...). k B m ), k = 1, 2, ..., m; where B k The specified field is the k-th field obtained after deduplication of A, and m is the number of specified fields obtained after deduplication of A; T030, obtain the original message sent by each data source; each original message includes several original fields, and each original field has a corresponding field value; T040, according to B, obtain the target original field corresponding to each original message to obtain the target original field list set C = (C1, C2, ..., C... r C s ), r = 1, 2, ..., s; where, C r C represents the list of target raw fields corresponding to the raw message sent from the r-th data source, where s is the number of data sources; r =(C r，1 C r，2 C r，p ,…,C r，g(r) ), p=1, 2,..., g(r); C r，p Let g(r) be the p-th target original field in the target original field list corresponding to the original message sent by the r-th data source, and g(r) be the number of target original fields corresponding to the original message sent by the r-th data source; C r，p Corresponding to a specified field in B; T050, according to the preset normalization rules, convert each target original field in C into the corresponding standard field to obtain the standard field list set D = (D1, D2, ..., D...). r D s ); where D r C r Corresponding standard field list; D r =(D r，1 D r，2 D r，p ,…,D r，g(r) );D r，p C r，p The corresponding standard fields.

5. The event stream data acceptance and judgment method according to claim 4, characterized in that, The completed standard field list set E is obtained from the standard field list set D.

6. The event stream data acceptance and judgment method according to claim 4, characterized in that, After step T050, the method includes the following steps: T070, retrieve each newly added target field corresponding to the newly added fusion project to obtain a list of newly added target fields QA = (QA1, QA2, ..., QA...). u QA v ), u=1, 2,...,v; among them, QA u This represents the u-th newly added target field corresponding to the newly added fusion project, and v represents the number of newly added target fields corresponding to the newly added fusion project. T071, Traverse QA. If every newly added target field in QA exists in B, then the newly added fusion project uses D to update the corresponding field value. T072, if QA u If it does not exist in B, then QA will be... u Add it to B and proceed to step T040.

7. The event stream data acceptance and judgment method according to claim 4, characterized in that, After step T050, the method further includes the following steps: T080, retrieve each original field corresponding to the original message sent by the newly added data source to obtain the list of newly added original fields QB = (QB1, QB2, ..., QB...). x QB y ), x = 1, 2, ..., y; where QB x The x-th original field corresponding to the original message sent to the newly added data source, where y is the number of original fields corresponding to the original message sent to the newly added data source; T081, iterate through QB, if QB x If a corresponding specified field exists in B, then QB will be... x Identify the target original fields corresponding to the newly added data source to obtain the list of target original fields WB corresponding to the newly added data source; T082, add WB to C, then proceed to step T050.

8. A non-transitory computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the event stream data acceptance and judgment method as described in any one of claims 1-7.

9. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 8.

Images