Rail transit service data quality checking method and device and electronic equipment

Abstract

The application provides a rail transit service data quality checking method and device and electronic equipment, the method comprises the following steps: obtaining target service data through a data bus, the data bus is constructed based on a message middleware; checking the quality of the target service data based on a distributed processing engine Flink and data quality checking configuration, and obtaining a quality checking result; the data quality checking configuration is used to indicate the data quality checking logic adopted by the distributed processing engine Flink. By constructing the data bus based on the message middleware, the target service data can be obtained in real time by using the data bus, and then the data quality checking logic adopted by the distributed processing engine Flink can be indicated by using the data quality checking configuration, and the rail transit service data is parsed by using the distributed processing engine Flink, so that the service data can be matched in a pipeline mode in real time, the data quality is checked in real time, and the efficiency of the data quality checking can be improved.

Description

Methods, devices and electronic equipment for verifying the quality of rail transit business data Technical Field

[0001] This invention relates to the field of rail transit technology, and in particular to a method, apparatus, and electronic device for verifying the quality of rail transit business data. Background Technology

[0002] Urban rail transit operations involve the real-time or near-real-time collection, processing, and storage of various types of urban rail business data. The processing and storage of large volumes of real-time data for urban rail operations necessitates the verification of data quality and the filtering of abnormal data to support research and development or on-site testing needs.

[0003] In related technologies, the main solution is to copy the full amount of data that has already been stored offline, process it offline, or process it manually using an Excel tool. The offline mode results in high latency, leading to low efficiency in the data quality verification process. Summary of the Invention

[0004] To address the problems existing in the prior art, embodiments of the present invention provide a method, apparatus, and electronic device for verifying the quality of rail transit business data.

[0005] In a first aspect, the present invention provides a method for verifying the quality of rail transit business data, comprising:

[0006] The target business data is acquired through a data bus, which is built based on message middleware and is used to collect rail transit business data in real time.

[0007] Based on the distributed processing engine Flink and data quality verification configuration, the target business data is subjected to quality verification, and the quality verification result is obtained.

[0008] The data quality verification configuration is used to indicate the data quality verification logic adopted by the distributed processing engine Flink.

[0009] Optionally, according to the rail transit business data quality verification method provided by the present invention, the distributed processing engine Flink includes a Filter operator and a Complex Event Processing (CEP) module, the data quality verification configuration includes a Filter operator verification configuration and a CEP module verification configuration, the Filter operator verification configuration is used to indicate the data quality verification logic adopted by the Filter operator, the CEP module verification configuration is used to indicate the data quality verification logic adopted by the CEP module, and the quality verification result includes a first type of quality verification result and a second type of quality verification result;

[0010] The process, based on the distributed processing engine Flink and data quality verification configuration, performs quality verification on the target business data and obtains the quality verification results, including:

[0011] Based on the Filter operator and the Filter operator verification configuration, the target business data is subjected to quality verification, and the first type of quality verification result is obtained;

[0012] Based on the CEP module and the CEP module verification configuration, the target business data is subjected to quality verification to obtain the second type of quality verification result.

[0013] Optionally, according to the rail transit business data quality verification method provided by the present invention, when the Filter operator verification configuration includes a uniqueness verification configuration, the first type of quality verification result includes a uniqueness verification result, and the uniqueness verification configuration is used to indicate the uniqueness verification logic of one or more data fields in the rail transit business data;

[0014] The step of performing quality verification on the target service data based on the Filter operator and the Filter operator verification configuration, and obtaining the first type of quality verification result, includes:

[0015] Based on the Filter operator and the uniqueness verification configuration, the target business data is uniquely verified, and the uniqueness verification result is obtained.

[0016] Optionally, according to the rail transit business data quality verification method provided by the present invention, when the Filter operator verification configuration includes a validity verification configuration, the first type of quality verification result includes a validity verification result, and the validity verification configuration is used to indicate the validity verification logic of one or more data fields in the rail transit business data;

[0017] The step of performing quality verification on the target service data based on the Filter operator and the Filter operator verification configuration, and obtaining the first type of quality verification result, includes:

[0018] Based on the Filter operator and the validity verification configuration, the target business data is validated, and the validity verification result is obtained.

[0019] Optionally, according to the rail transit business data quality verification method provided by the present invention, when the Filter operator verification configuration includes a reliability verification configuration, the first type of quality verification result includes a reliability verification result, and the reliability verification configuration is used to indicate the reliability verification logic of one or more data fields in the rail transit business data;

[0020] The step of performing quality verification on the target service data based on the Filter operator and the Filter operator verification configuration, and obtaining the first type of quality verification result, includes:

[0021] Based on the Filter operator and the reliability verification configuration, the target business data is subjected to reliability verification, and the reliability verification result is obtained.

[0022] Optionally, according to the rail transit business data quality verification method provided by the present invention, when the CEP module verification configuration includes a first event matching configuration, the second type of quality verification result includes the first event matching verification result. The first event matching configuration is used to indicate the verification logic between the business data of the first rail transit event and the business data of the second rail transit event, wherein the first rail transit event and the second rail transit event are events that are adjacent in time.

[0023] The process of performing quality verification on the target business data based on the CEP module and the CEP module verification configuration, and obtaining the second type of quality verification result, includes:

[0024] Based on the CEP module and the first event matching configuration, the target business data is subjected to event matching and quality verification to obtain the first event matching verification result.

[0025] Optionally, according to the rail transit business data quality verification method provided by the present invention, when the CEP module verification configuration includes a second event matching configuration, the second type of quality verification result includes the second event matching verification result;

[0026] The second event matching configuration is used to indicate the verification logic between the service data of the third rail transit event and the service data of the fourth rail transit event. N fifth rail transit events occur between the event time corresponding to the third rail transit event and the event time corresponding to the fourth rail transit event. The fifth rail transit events are different from the third rail transit event and the fourth rail transit event, and N is an integer greater than or equal to 0.

[0027] The process of performing quality verification on the target business data based on the CEP module and the CEP module verification configuration, and obtaining the second type of quality verification result, includes:

[0028] Based on the CEP module and the second event matching configuration, event matching and quality verification are performed on the target business data to obtain the second event matching verification result.

[0029] Secondly, the present invention also provides a rail transit business data quality verification device, comprising:

[0030] The first acquisition module is used to acquire target business data through a data bus, which is built based on message middleware and is used to collect rail transit business data in real time.

[0031] The second acquisition module is used to perform quality verification on the target business data based on the distributed processing engine Flink and the data quality verification configuration, and to obtain the quality verification result.

[0032] The data quality verification configuration is used to indicate the data quality verification logic adopted by the distributed processing engine Flink.

[0033] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the rail transit business data quality verification method as described above.

[0034] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the rail transit business data quality verification method as described above.

[0035] The rail transit business data quality verification method, apparatus, and electronic device provided by this invention, by constructing a data bus based on message middleware, can acquire target business data in real time using the data bus. Then, it can use the data quality verification configuration to instruct the distributed processing engine Flink to adopt the data quality verification logic, and use the distributed processing engine Flink to parse the rail transit business data, perform pipeline matching of the business data in real time, and output the matched data to obtain the quality verification result. This avoids the use of offline mode for data quality verification, realizes real-time data quality verification, and can improve the efficiency of the data quality verification process. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 is a flowchart illustrating the data quality verification method for rail transit business provided by related technologies;

[0038] Figure 2 is one of the flowcharts of the rail transit business data quality verification method provided by the present invention;

[0039] Figure 3 is a second flowchart of the rail transit business data quality verification method provided by the present invention;

[0040] Figure 4 is a flowchart of the rail transit business data quality verification method provided by the present invention (Part 3).

[0041] Figure 5 is a flowchart of the rail transit business data quality verification method provided by the present invention (fourth one).

[0042] Figure 6 is a flowchart of the rail transit business data quality verification method provided by the present invention (the fifth one).

[0043] Figure 7 is a flowchart of the rail transit business data quality verification method provided by the present invention (the sixth one).

[0044] Figure 8 is a flowchart illustrating the event matching process based on strict nearest neighbor relationships provided by the present invention;

[0045] Figure 9 is a flowchart of the rail transit business data quality verification method provided by the present invention (the seventh one).

[0046] Figure 10 is a flowchart illustrating the event matching process based on the relaxed nearest neighbor relationship provided by the present invention.

[0047] Figure 11 is the eighth flowchart of the rail transit business data quality verification method provided by the present invention;

[0048] Figure 12 is a schematic diagram of the structure of the rail transit business data quality verification device provided by the present invention;

[0049] Figure 13 is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To facilitate a clearer understanding of the various embodiments of the present invention, some relevant background knowledge will be introduced as follows.

[0051] Figure 1 is a flowchart illustrating the data quality verification method for rail transit business provided by related technologies. As shown in Figure 1, in these technologies, for urban rail transit industry business log data, the traditional approach involves incrementally writing the data to disk, then periodically starting offline tasks to process the entire data based on offline batch data processing technology. Finally, an offline framework is used to load an offline algorithm to perform a full quality verification on the already written data, identifying any abnormal data. The offline execution mode results in high latency, leading to low efficiency in the data quality verification process.

[0052] To overcome the above-mentioned shortcomings, the present invention provides a method, apparatus and electronic device for verifying the quality of rail transit business data, which can improve the efficiency of the data quality verification process.

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] Figure 2 is one of the flowcharts of the rail transit business data quality verification method provided by the present invention. As shown in Figure 2, the executing entity of the rail transit business data quality verification method can be an electronic device, such as a server. The method includes:

[0055] Step 201: Obtain target business data through a data bus. The data bus is built based on message middleware and is used to collect rail transit business data in real time.

[0056] Specifically, one can choose the Kafka message middleware, an internet technology, to build a real-time data bus and access urban rail transit industry business data in real time.

[0057] It is understandable that using the Internet technology Kafka to build a real-time data bus can meet the needs of collecting and storing real-time business data, providing strong support for subsequent real-time framework programs. The data bus built on Kafka as a message middleware has the following advantages: (1) Kafka supports a high throughput of millions of messages per second; (2) It provides message persistence through an O(1) disk data structure, which can maintain stable performance for a long time even for message storage at the TB level; (3) It supports partitioning messages through Kafka servers and consumer clusters.

[0058] Step 202: Based on the distributed processing engine Flink and the data quality verification configuration, perform quality verification on the target business data and obtain the quality verification result;

[0059] The data quality verification configuration is used to indicate the data quality verification logic adopted by the distributed processing engine Flink.

[0060] Specifically, data quality verification rules can be defined at the beginning of development (that is, by defining custom data quality verification configurations to indicate the data quality verification logic adopted by the distributed processing engine Flink). After the application goes online, the distributed processing engine Flink consumes real-time data sources and loads verification rules, which can realize real-time data quality verification without the need for manual offline execution of verification programs or offline scheduled execution programs.

[0061] It is understandable that, based on Flink real-time computing technology, it is possible to access urban rail transit industry data sources and process data streams in real time. Compared with traditional offline batch data processing technology, Flink real-time technology has the following advantages: (1) It has high throughput, low latency, and high performance real-time processing capabilities; (2) It supports large-scale cluster mode and multiple running models; (3) It supports stream processing and window processing with event time semantics. The event time semantics makes the results of stream computing more accurate, especially when events arrive out of order or are delayed; (4) It supports the savepoints mechanism, which can save the running state of the application and achieve stateless loss and minimum downtime when upgrading the application or processing historical data; (5) It implements fault-tolerant and consistency (exactly-once) semantics based on lightweight distributed snapshots (CheckPoint); (6) Flink supports complex event processing (CEP), which means that data that meets the detection mode in an infinite event stream can be output through a custom detection model.

[0062] Optionally, the quality verification results may include anomaly details, anomaly statistics, and data quality reports.

[0063] The rail transit business data quality verification method provided by this invention constructs a data bus based on message middleware. It can acquire target business data in real time using the data bus, and then use the data quality verification configuration to instruct the distributed processing engine Flink to adopt the data quality verification logic. By using the distributed processing engine Flink to parse the rail transit business data, the method performs pipeline matching of the business data in real time, outputs the matched data, and obtains the quality verification result. This avoids the use of offline mode for data quality verification, realizes real-time data quality verification, and improves the efficiency of the data quality verification process.

[0064] Optionally, according to the rail transit business data quality verification method provided by the present invention, the distributed processing engine Flink includes a Filter operator and a Complex Event Processing (CEP) module, the data quality verification configuration includes a Filter operator verification configuration and a CEP module verification configuration, the Filter operator verification configuration is used to indicate the data quality verification logic adopted by the Filter operator, the CEP module verification configuration is used to indicate the data quality verification logic adopted by the CEP module, and the quality verification result includes a first type of quality verification result and a second type of quality verification result;

[0065] The process, based on the distributed processing engine Flink and data quality verification configuration, performs quality verification on the target business data and obtains the quality verification results, including:

[0066] Based on the Filter operator and the Filter operator verification configuration, the target business data is subjected to quality verification, and the first type of quality verification result is obtained;

[0067] Based on the CEP module and the CEP module verification configuration, the target business data is subjected to quality verification to obtain the second type of quality verification result.

[0068] Specifically, Figure 3 is a second flowchart of the rail transit business data quality verification method provided by the present invention. As shown in Figure 3, the method includes steps 301 to 303.

[0069] Step 301: Obtain target business data through the data bus, then execute steps 302 and 303.

[0070] Step 302: Based on the Filter operator and Filter operator verification configuration, perform quality verification on the target business data and obtain the first type of quality verification result.

[0071] Step 303: Based on the CEP module and CEP module verification configuration, perform quality verification on the target business data and obtain the second type of quality verification result.

[0072] Understandably, Flink's Filter operator combined with CEP (Complex Event Processing) technology performs real-time data quality verification in the data stream, enabling pipelined data quality verification—that is, anomalies are output as soon as abnormal data is generated. The Filter operator performs quality checks on aspects such as uniqueness, validity, or reliability; while CEP technology performs quality checks from a logical perspective, allowing the detection of complex patterns in the data stream without requiring manual implementation of pattern matching logic.

[0073] By leveraging Flink's real-time computing capabilities and utilizing its built-in Filter operator and Complex Event Processing (CEP) capabilities, and based on the requirements of urban rail transit industry business data quality verification rules, custom pattern detection rules are defined (i.e., by customizing data quality verification configurations to instruct the data quality verification logic adopted by the distributed processing engine Flink). This enables real-time quality verification of the data stream after real-time access to the urban rail transit industry data stream, real-time statistical analysis and output of detailed data that meet the quality verification conditions, and acquisition of quality verification results.

[0074] Optionally, the Filter operator verification configuration may include any one or more of the following configurations: uniqueness verification configuration, validity verification configuration, or reliability verification configuration, etc.

[0075] Optionally, Flink's CEP can support online modification, addition, or deletion of detection rules (i.e., CEP module verification configuration). Updating matching rules online makes Flink's CEP more flexible, allowing it to adopt corresponding complex patterns for event processing based on constantly changing business needs.

[0076] Therefore, the Flink real-time computing engine can consume data bus topic data (i.e., target business data) in real time. By referencing the Filter operator and Complex Event Processing (CEP), it defines the verification logic (or verification algorithm) in the Filter and CEP modes, thereby realizing real-time verification of simple streaming data and complex event stream data and improving the efficiency of data quality verification.

[0077] Optionally, according to the rail transit business data quality verification method provided by the present invention, when the Filter operator verification configuration includes a uniqueness verification configuration, the first type of quality verification result includes a uniqueness verification result, and the uniqueness verification configuration is used to indicate the uniqueness verification logic of one or more data fields in the rail transit business data;

[0078] The step of performing quality verification on the target service data based on the Filter operator and the Filter operator verification configuration, and obtaining the first type of quality verification result, includes:

[0079] Based on the Filter operator and the uniqueness verification configuration, the target business data is uniquely verified, and the uniqueness verification result is obtained.

[0080] Specifically, Figure 4 is a flowchart of the rail transit business data quality verification method provided by the present invention. As shown in Figure 4, the method includes steps 401 to 403.

[0081] Step 401: Obtain the target business data through the data bus, and then execute steps 402 and 403.

[0082] Step 402: Based on the Filter operator and uniqueness verification configuration, perform uniqueness verification on the target business data and obtain the uniqueness verification result.

[0083] For uniqueness verification, let's take the quality inspection of key fields in urban rail transit vehicle logs as an example. For instance, for the head direction data field (head_direction), the data quality requirements include that, when the train is traveling from platform A to platform B (stations A and B are adjacent), if the train is located between stations A and B, the value of the head direction data field should remain unique. Based on this, a corresponding uniqueness verification configuration can be generated to perform uniqueness verification on the head direction data field and obtain the uniqueness verification result.

[0084] Step 403: Based on the CEP module and CEP module verification configuration, perform quality verification on the target business data and obtain the second type of quality verification result.

[0085] Therefore, by referencing the Filter operator and defining the uniqueness verification logic under the Filter mode, it is possible to perform uniqueness verification on one or more data fields in rail transit business data in real time, thereby improving the efficiency of data quality verification.

[0086] Optionally, according to the rail transit business data quality verification method provided by the present invention, when the Filter operator verification configuration includes a validity verification configuration, the first type of quality verification result includes a validity verification result, and the validity verification configuration is used to indicate the validity verification logic of one or more data fields in the rail transit business data;

[0087] The step of performing quality verification on the target service data based on the Filter operator and the Filter operator verification configuration, and obtaining the first type of quality verification result, includes:

[0088] Based on the Filter operator and the validity verification configuration, the target business data is validated, and the validity verification result is obtained.

[0089] Specifically, Figure 5 is a flowchart of the rail transit business data quality verification method provided by the present invention, as shown in Figure 5. The method includes steps 501 to 503.

[0090] Step 501: Obtain the target business data through the data bus, and then execute steps 502 and 503.

[0091] Step 502: Based on the Filter operator and validity verification configuration, perform validity verification on the target business data and obtain the validity verification result.

[0092] It is understandable that validity check configurations can include configurations for validating field length, validating field content, validating the range of field values, and validating whether a field is null.

[0093] For validity verification, let's take the quality inspection of key fields in urban rail transit vehicle logs as an example. For instance, for the time data field (log_time), data quality requirements include output details for data with a length of 12 and data with a length other than 12. Based on this, a corresponding validity verification configuration can be generated to perform validity verification on the time data field and obtain the validity verification results.

[0094] For example, for the parking accuracy data field (park_accuracy_value), data quality requirements include outputting a parking accuracy of 99999 in abnormal situations, and providing detailed information. Based on this, a corresponding validity verification configuration can be generated to perform validity verification on the parking accuracy data field and obtain the validity verification results.

[0095] Step 503: Based on the CEP module and CEP module verification configuration, perform quality verification on the target business data and obtain the second type of quality verification result.

[0096] Therefore, by referencing the Filter operator and defining the validity verification logic in the Filter mode, it is possible to perform validity verification on one or more data fields in rail transit business data in real time, thereby improving the efficiency of data quality verification.

[0097] Optionally, according to the rail transit business data quality verification method provided by the present invention, when the Filter operator verification configuration includes a reliability verification configuration, the first type of quality verification result includes a reliability verification result, and the reliability verification configuration is used to indicate the reliability verification logic of one or more data fields in the rail transit business data;

[0098] The step of performing quality verification on the target service data based on the Filter operator and the Filter operator verification configuration, and obtaining the first type of quality verification result, includes:

[0099] Based on the Filter operator and the reliability verification configuration, the target business data is subjected to reliability verification, and the reliability verification result is obtained.

[0100] Specifically, Figure 6 is the fifth flowchart of the rail transit business data quality verification method provided by the present invention. As shown in Figure 6, the method includes steps 601 to 603.

[0101] Step 601: Obtain target business data through the data bus, then execute steps 602 and 603.

[0102] Step 602: Based on the Filter operator and reliability verification configuration, perform reliability verification on the target business data and obtain the reliability verification result.

[0103] For reliability verification, let's take the quality inspection of key fields in urban rail transit vehicle logs as an example. For instance, for the front train number data field (front_train_id), data quality requirements include an invalid value of -1, and the need to output detailed data when the front train number is -1, to assist in verifying the reliability of data interaction between the front and rear trains. Based on this, a corresponding reliability verification configuration can be generated to perform reliability verification on the front train number data field and obtain the reliability verification results.

[0104] Step 603: Based on the CEP module and CEP module verification configuration, perform quality verification on the target business data and obtain the second type of quality verification result.

[0105] Therefore, by referencing the Filter operator and defining the reliability verification logic in the Filter mode, it is possible to perform reliability verification on one or more data fields in rail transit business data in real time, thereby improving the efficiency of data quality verification.

[0106] Optionally, according to the rail transit business data quality verification method provided by the present invention, when the CEP module verification configuration includes a first event matching configuration, the second type of quality verification result includes the first event matching verification result. The first event matching configuration is used to indicate the verification logic between the business data of the first rail transit event and the business data of the second rail transit event, wherein the first rail transit event and the second rail transit event are events that are adjacent in time.

[0107] The process of performing quality verification on the target business data based on the CEP module and the CEP module verification configuration, and obtaining the second type of quality verification result, includes:

[0108] Based on the CEP module and the first event matching configuration, the target business data is subjected to event matching and quality verification to obtain the first event matching verification result.

[0109] Specifically, Figure 7 is a flowchart of the rail transit business data quality verification method provided by the present invention, as shown in Figure 7. The method includes steps 701 to 703.

[0110] Step 701: Obtain target business data through the data bus, then execute steps 702 and 703.

[0111] Step 702: Based on the Filter operator and Filter operator verification configuration, perform quality verification on the target business data and obtain the first type of quality verification result.

[0112] Step 703: Based on the CEP module and the first event matching configuration, perform event matching and quality verification on the target business data, and obtain the first event matching verification result.

[0113] It is understandable that Flink's CEP technology provides three types of nearest neighbor relationships: strict nearest neighbor, loose nearest neighbor, and nondeterministic loose nearest neighbor. Figure 8 is a flowchart illustrating the event matching process for strict nearest neighbor relationships provided by this invention. As shown in Figure 8, for strict nearest neighbors, the matched events appear strictly in sequence, one after another, without any other events in between. This can be determined using the `.next` method ("next" method) of the complex event handling matching pattern.

[0114] For the service data quality verification of matching events that meet the strict nearest neighbor relationship, a first event matching configuration can be set. The first event matching configuration can indicate the verification logic between the service data of the first rail transit event and the service data of the second rail transit event. The first rail transit event and the second rail transit event are events that are adjacent in time.

[0115] For example, taking the quality inspection of key fields in urban rail transit vehicle logs as an example, for field logic checks and data quality verification, such as the continuous monitoring of the interval_id data field in the vehicle logs, the data quality requirements include that adjacent train data have continuous interval numbers; discontinuous interval numbers are considered abnormal data, and detailed discontinuity data must be output. Based on this, a corresponding first event matching configuration can be generated to perform event matching and quality verification on the interval number data field and obtain the first event matching verification result.

[0116] By referencing the Filter operator and defining the reliability verification logic in the Filter mode, the reliability verification of one or more data fields in rail transit business data can be performed in real time, improving the efficiency of data quality verification.

[0117] Therefore, by referencing the Complex Event Processing (CEP) module and defining the verification logic for strictly nearest neighbor matching events under the CEP mode, the quality verification of rail transit business data that meets the strictly nearest neighbor matching event requirement can be performed in real time, thereby improving the efficiency of data quality verification.

[0118] Optionally, according to the rail transit business data quality verification method provided by the present invention, when the CEP module verification configuration includes a second event matching configuration, the second type of quality verification result includes the second event matching verification result;

[0119] The second event matching configuration is used to indicate the verification logic between the service data of the third rail transit event and the service data of the fourth rail transit event. N fifth rail transit events occur between the event time corresponding to the third rail transit event and the event time corresponding to the fourth rail transit event. The fifth rail transit events are different from the third rail transit event and the fourth rail transit event, and N is an integer greater than or equal to 0.

[0120] The process of performing quality verification on the target service data based on the CEP module and the CEP module verification configuration, and obtaining the second type of quality verification result, includes:

[0121] Based on the CEP module and the second event matching configuration, event matching and quality verification are performed on the target business data to obtain the second event matching verification result.

[0122] Specifically, Figure 9 is a flowchart of the rail transit business data quality verification method provided by the present invention, as shown in Figure 9. The method includes steps 901 to 903.

[0123] Step 901: Obtain the target business data through the data bus, and then execute steps 902 and 903.

[0124] Step 902: Based on the Filter operator and Filter operator verification configuration, perform quality verification on the target business data and obtain the first type of quality verification result.

[0125] Step 903: Based on the CEP module and the second event matching configuration, perform event matching and quality verification on the target business data, and obtain the second event matching verification result.

[0126] Understandably, Flink's CEP technology provides three types of nearest neighbor relationships: strict nearest neighbor, loose nearest neighbor, and nondeterministic loose nearest neighbor. Figure 10 is a flowchart illustrating the event matching process for loose nearest neighbor relationships provided by this invention. As shown in Figure 10, for loose nearest neighbor, the focus is on the order in which events occur, while relaxing the "distance" requirement for matching events. In other words, other non-matching events can occur between two matching events. In the code, this corresponds to the `.followedBy` method, which clearly indicates that "following" is sufficient; they do not need to be directly adjacent.

[0127] For the service data quality verification of matching events that satisfy the loose nearest neighbor relationship, a second event matching configuration can be set. The second event matching configuration can indicate the verification logic between the service data of the third rail transit event and the service data of the fourth rail transit event. N fifth rail transit events occur between the event time corresponding to the third rail transit event and the event time corresponding to the fourth rail transit event. The fifth rail transit events are different from the third rail transit event and the fourth rail transit event, and N is an integer greater than or equal to 0.

[0128] For example, taking the quality inspection of key fields in urban rail transit vehicle logs as an example, for real-time verification of the sequential data of fields, such as the formation identifier (formation_state) field in the vehicle log, the data quality requirements include real-time output of data details at two time points when the train enters the formation state from a single-car state, without focusing on data from other formation processes in between, and statistically outputting the time consumed. Based on this, a corresponding second event matching configuration can be generated to perform event matching and quality verification on the formation identifier data field and obtain the second event matching verification results.

[0129] Therefore, by referencing the Complex Event Processing (CEP) module and defining the verification logic for loosely neighbored matching events under the CEP mode, the quality verification of rail transit business data that meets the loosely neighbored matching event requirement can be performed in real time, thereby improving the efficiency of data quality verification.

[0130] Figure 11 is a flowchart of the rail transit business data quality verification method provided by the present invention, the eighth of which is shown in Figure 11. The method includes steps 1101 to 1103.

[0131] Step 1101: Monitor and collect real-time data from urban rail transit industry operations to the data bus.

[0132] Alternatively, a data bus can be built based on the Kafka middleware, an internet technology.

[0133] Step 1102: Use the Flink real-time computing framework to connect to the data bus topic data in real time.

[0134] Understandably, by using the Flink real-time computing framework to connect to the data bus topic data in real time, the target business data to be verified can be obtained in real time.

[0135] Step 1103: Based on two types of data quality verification modes, detect data streams that conform to the modes.

[0136] Optionally, based on business needs, two types of data quality verification modes are defined: (1) a normal data quality verification mode is defined based on Flink's Filter operator technology; (2) a complex event processing matching mode (Pattern) is defined based on Flink's CEP technology, and the Pattern is applied to the data stream to obtain a pattern stream (PatternStream).

[0137] Optionally, for data streams that match the detected pattern, a data selection command (such as the select command) can be invoked to return the matching result (i.e., the quality verification result). The matching result can be output to the alert stream for downstream services to implement monitoring and alert functions.

[0138] Understandably, based on Flink's Filter operator and Complex Event Processing (CEP), data quality verification patterns are defined in multiple dimensions to ensure that data streams conforming to quality rules are output from complex data streams, thus completing the data verification of specific patterns in complex data streams.

[0139] The rail transit business data quality verification method provided by this invention constructs a data bus based on message middleware. It can acquire target business data in real time using the data bus, and then use the data quality verification configuration to instruct the distributed processing engine Flink to adopt the data quality verification logic. By using the distributed processing engine Flink to parse the rail transit business data, the method performs pipeline matching of the business data in real time, outputs the matched data, and obtains the quality verification result. This avoids the use of offline mode for data quality verification, realizes real-time data quality verification, and improves the efficiency of the data quality verification process.

[0140] The following describes the rail transit business data quality verification device provided by the present invention. The rail transit business data quality verification device described below and the rail transit business data quality verification method described above can be referred to in correspondence.

[0141] Figure 12 is a schematic diagram of the structure of the rail transit business data quality verification device provided by the present invention. As shown in Figure 12, the device includes: a first acquisition module 1201 and a second acquisition module 1202, wherein:

[0142] The first acquisition module 1201 is used to acquire target business data through a data bus. The data bus is built based on message middleware and is used to collect rail transit business data in real time.

[0143] The second acquisition module 1202 is used to perform quality verification on the target business data based on the distributed processing engine Flink and data quality verification configuration, and obtain the quality verification result.

[0144] The data quality verification configuration is used to indicate the data quality verification logic adopted by the distributed processing engine Flink.

[0145] The rail transit business data quality verification device provided by this invention constructs a data bus based on message middleware. It can acquire target business data in real time using the data bus, and then use the data quality verification configuration to instruct the distributed processing engine Flink to adopt the data quality verification logic. By using the distributed processing engine Flink to parse the rail transit business data, it performs pipeline matching of the business data in real time, outputs the matched data, and obtains the quality verification result. This avoids the use of offline mode for data quality verification, realizes real-time data quality verification, and can improve the efficiency of the data quality verification process.

[0146] Figure 13 is a schematic diagram of the electronic device provided by the present invention. As shown in Figure 13, the electronic device may include: a processor 1310, a communication interface 1320, a memory 1330, and a communication bus 1340. The processor 1310, the communication interface 1320, and the memory 1330 communicate with each other via the communication bus 1340. The processor 1310 can call logical instructions in the memory 1330 to execute a rail transit business data quality verification method, which includes:

[0147] The target business data is acquired through a data bus, which is built based on message middleware and is used to collect rail transit business data in real time.

[0148] Based on the distributed processing engine Flink and data quality verification configuration, the target business data is subjected to quality verification, and the quality verification result is obtained.

[0149] The data quality verification configuration is used to indicate the data quality verification logic adopted by the distributed processing engine Flink.

[0150] Furthermore, the logical instructions in the aforementioned memory 1330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the rail transit business data quality verification method provided by the above methods, the method comprising:

[0152] The target business data is acquired through a data bus, which is built based on message middleware and is used to collect rail transit business data in real time.

[0153] Based on the distributed processing engine Flink and data quality verification configuration, the target business data is subjected to quality verification, and the quality verification result is obtained.

[0154] The data quality verification configuration is used to indicate the data quality verification logic adopted by the distributed processing engine Flink.

[0155] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for verifying the quality of rail transit business data, characterized in that, include: The target business data is acquired through a data bus, which is built based on message middleware and is used to collect rail transit business data in real time. Based on the distributed processing engine Flink and data quality verification configuration, the target business data is subjected to quality verification, and the quality verification results are obtained. The data quality verification configuration is used to indicate the data quality verification logic adopted by the distributed processing engine Flink. The distributed processing engine Flink includes a Filter operator and a Complex Event Processing (CEP) module. The data quality verification configuration includes a Filter operator verification configuration and a CEP module verification configuration. The Filter operator verification configuration is used to indicate the data quality verification logic adopted by the Filter operator, and the CEP module verification configuration is used to indicate the data quality verification logic adopted by the CEP module. The quality verification results include a first type of quality verification result and a second type of quality verification result. The process of performing quality verification on the target business data based on the distributed processing engine Flink and data quality verification configuration, and obtaining quality verification results, includes: performing quality verification on the target business data based on the Filter operator and the Filter operator verification configuration, and obtaining the first type of quality verification result; performing quality verification on the target business data based on the CEP module and the CEP module verification configuration, and obtaining the second type of quality verification result; wherein, when the CEP module verification configuration includes a second event matching configuration, the second type of quality verification result includes a second event matching verification result; the second event matching configuration is used to indicate the verification logic between the business data of the third rail transit event and the business data of the fourth rail transit event, wherein N fifth rail transit events occur between the event time corresponding to the third rail transit event and the event time corresponding to the fourth rail transit event, and the fifth rail transit events are different from the third rail transit event and the fourth rail transit event, and N is an integer greater than or equal to 0.

2. The method for verifying the quality of rail transit business data according to claim 1, characterized in that, When the Filter operator verification configuration includes a uniqueness verification configuration, the first type of quality verification result includes a uniqueness verification result, and the uniqueness verification configuration is used to indicate the uniqueness verification logic of one or more data fields in the rail transit business data; The step of performing quality verification on the target service data based on the Filter operator and the Filter operator verification configuration, and obtaining the first type of quality verification result, includes: performing unique verification on the target service data based on the Filter operator and the uniqueness verification configuration, and obtaining the uniqueness verification result.

3. The method for verifying the quality of rail transit business data according to claim 1, characterized in that, When the Filter operator verification configuration includes a validity verification configuration, the first type of quality verification result includes a validity verification result, and the validity verification configuration is used to indicate the validity verification logic of one or more data fields in the rail transit business data; The step of performing quality verification on the target service data based on the Filter operator and the Filter operator verification configuration, and obtaining the first type of quality verification result, includes: performing validity verification on the target service data based on the Filter operator and the validity verification configuration, and obtaining the validity verification result.

4. The method for verifying the quality of rail transit business data according to claim 1, characterized in that, When the Filter operator verification configuration includes a reliability verification configuration, the first type of quality verification result includes a reliability verification result, and the reliability verification configuration is used to indicate the reliability verification logic of one or more data fields in the rail transit business data; The step of performing quality verification on the target service data based on the Filter operator and the Filter operator verification configuration, and obtaining the first type of quality verification result, includes: performing reliability verification on the target service data based on the Filter operator and the reliability verification configuration, and obtaining the reliability verification result.

5. The method for verifying the quality of rail transit business data according to claim 1, characterized in that, When the CEP module verification configuration includes a first event matching configuration, the second type of quality verification result includes the first event matching verification result. The first event matching configuration is used to indicate the verification logic between the business data of the first rail transit event and the business data of the second rail transit event. The first rail transit event and the second rail transit event are events that are adjacent in time. The step of performing quality verification on the target service data based on the CEP module and the CEP module verification configuration to obtain the second type of quality verification result includes: performing event matching and quality verification on the target service data based on the CEP module and the first event matching configuration to obtain the first event matching verification result.

6. The method for verifying the quality of rail transit business data according to claim 1, characterized in that, The step of performing quality verification on the target service data based on the CEP module and the CEP module verification configuration, and obtaining the second type of quality verification result, includes: performing event matching and quality verification on the target service data based on the CEP module and the second event matching configuration, and obtaining the second event matching verification result.

7. A rail transit business data quality verification device, used to implement the rail transit business data quality verification method as described in any one of claims 1-6, characterized in that, include: The first acquisition module is used to acquire target business data through a data bus, which is built based on message middleware and is used to collect rail transit business data in real time. The second acquisition module is used to perform quality verification on the target business data based on the distributed processing engine Flink and the data quality verification configuration, and to obtain the quality verification result; the data quality verification configuration is used to indicate the data quality verification logic adopted by the distributed processing engine Flink.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the rail transit business data quality verification method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the rail transit business data quality verification method as described in any one of claims 1 to 6.

Images