Railway data auditing method and device and electronic equipment
By automatically determining verification rules and executable methods in railway data auditing, the complexity and high cost problems caused by manual dependence in the prior art are solved, and efficient and accurate data audit is achieved.
Patent Information
- Application Number
- CN202510254997.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-05
- Publication Date
- 2025-07-25
AI Technical Summary
The existing railway data audit process relies heavily on manual intervention, resulting in high complexity, high risk of errors and high labor costs.
By determining the target fields based on business needs, and using the execution rule model to automatically determine the verification rules and executable methods, automatic audit of railway data is realized.
It reduces the complexity of the audit process and the risk of errors, reduces labor costs, and improves audit efficiency and accuracy.
Smart Images
Figure CN120372228A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of transportation technologies, and in particular, to a method, apparatus, and electronic device for auditing railway data. Background Art
[0002] With the continuous development of rail transit technologies and the continuous improvement of the informatization level, the railway industry has accumulated a vast amount of railway data resources. How to effectively manage the railway data resources and rationally develop and utilize them has become an urgent problem to be solved in the railway industry. In the process of railway data management, data auditing is not only the basis for the effectiveness and accuracy of data analysis and data mining conclusions, but also the premise for data-driven decision-making.
[0003] In the existing auditing process of railway data, it highly depends on manual intervention. Both rule configuration and auditing tasks require a large amount of manual participation. This dependence on manual labor not only increases complexity and error risks, but also has a high labor cost. Summary of the Invention
[0004] The present invention provides a method, apparatus, and electronic device for auditing railway data to solve the defect that in the existing technology, the auditing process of railway data highly depends on manual intervention, both rule configuration and auditing tasks require a large amount of manual participation, this dependence on manual labor not only increases complexity and error risks, but also has a high labor cost. The technical solution of the present invention does not need to rely on manual intervention, thereby reducing the complexity and error risks of the auditing process and reducing the labor cost.
[0005] The present invention provides a method for auditing railway data, including the following steps.
[0006] Determine a plurality of target fields in the target railway data set based on the business requirements corresponding to the target railway data set; Determine the verification rules and executable manners respectively corresponding to each of the target fields; the executable manner represents the manner in which the railway data corresponding to the target field can be executed; Input the railway data, verification rules, and executable manners respectively corresponding to each of the target fields into an execution rule model to obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules, and executable manners respectively corresponding to a plurality of sample fields in a sample railway data set; Determine a target auditing result of the railway data corresponding to all the target fields based on the target execution rule.
[0007] According to the method for auditing railway data provided by the present invention, the determining the verification rules respectively corresponding to each of the target fields includes: For each of the target fields, based on the preset matching relationship between the target field and the verification rule, determine the verification rule corresponding to the target field; or, input the target field into a rule recommendation model to obtain the verification rule corresponding to the target field output by the rule recommendation model; Wherein, the rule recommendation model is trained based on each of the sample fields and their respective preset verification rules.
[0008] According to a railway data auditing method provided by the present invention, determining the executable manners respectively corresponding to each of the target fields includes: For the railway data respectively corresponding to each of the target fields, in the case where there is no association relationship between the sub-railway data in the railway data, determine the executable manners corresponding to the target field as serial and parallel; In the case where there is an association relationship between the sub-railway data in the railway data, determine the executable manner corresponding to the target field as serial.
[0009] According to a railway data auditing method provided by the present invention, the determining the target auditing result of the railway data corresponding to all the target fields based on the target execution rule includes: For the railway data respectively corresponding to each of the target fields, based on the target execution rule, determine the execution manner and execution order of the railway data; Audit all the railway data based on the verification rules, execution manners and execution orders corresponding to all the railway data, and determine the auditing results of all the railway data; Based on the auditing results of the railway data corresponding to all the target fields, determine the target auditing result.
[0010] According to a railway data auditing method provided by the present invention, the auditing all the railway data based on the verification rules, execution manners and execution orders corresponding to all the railway data, and determining the auditing results of all the railway data includes: Audit all the railway data based on the verification rules, execution manners and execution orders corresponding to all the railway data, and determine the auditing accuracy rate of all the railway data; For each of the railway data, in the case where the auditing accuracy rate of the railway data is greater than the accuracy rate threshold corresponding to the railway data, determine the auditing result of the railway data as qualified; In the case where the auditing accuracy rate of the railway data is less than or equal to the accuracy rate threshold corresponding to the railway data, determine the auditing result of the railway data as unqualified.
[0011] A railway data auditing method provided by the present invention, determining the target auditing result based on the auditing results of the railway data corresponding to all the target fields, includes: Determine the railway data corresponding to the unqualified auditing results among the auditing results as error-reporting railway data, and determine the error causes of each error-reporting railway data; Determine the auditing results of the railway data corresponding to all the target fields and the error causes of all the error-reporting railway data as the target auditing result.
[0012] A railway data auditing method provided by the present invention, the method further includes: Input the railway industry specification documents and rule template prompts corresponding to the target railway data set into a preset natural language model, obtain multiple initial verification rules output by the preset natural language model, and construct a verification rule library based on all the initial verification rules; Determine the verification rules respectively corresponding to each of the target fields from the verification rule library.
[0013] A railway data auditing method provided by the present invention, the verification rule library at least includes: digital verification, letter verification, value range verification, comparison verification with a preset data set, numerical range verification, uppercase letter verification, lowercase letter verification, Chinese character verification, date format verification, fixed content verification, uniqueness verification, non-empty verification, logical verification, length verification, and data timestamp verification.
[0014] The present invention also provides a railway data auditing device, including the following modules: The first determination module is used to determine multiple target fields in the target railway data set based on the service requirements corresponding to the target railway data set; The second determination module is used to determine the verification rules and executable manners respectively corresponding to each of the target fields; the executable manner represents the manner in which the railway data corresponding to the target field can be executed; The input module is used to input the railway data, verification rules, and executable manners respectively corresponding to each of the target fields into an execution rule model, and obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules, and executable manners corresponding to multiple sample fields in a sample railway data set; The auditing module is used to determine the target auditing result of the railway data corresponding to all the target fields based on the target execution rule.
[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the railway data auditing method as described in any one of the above is implemented.
[0016] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the railway data auditing method as described in any one of the above is implemented.
[0017] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the railway data auditing method as described in any one of the above is implemented.
[0018] For the railway data auditing method, device, and electronic device provided by the present invention, a plurality of target fields in a target railway data set are determined based on the service requirements corresponding to the target railway data set; verification rules and executable manners respectively corresponding to each target field are determined; the railway data, verification rules, and executable manners respectively corresponding to each target field are input into an execution rule model to obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules, and executable manners respectively corresponding to a plurality of sample fields in a sample railway data set; and a target auditing result of the railway data corresponding to all target fields is determined based on the target execution rule. The technical solution of the present invention inputs the railway data, verification rules, and executable manners respectively corresponding to each target field in the target railway data set into the execution rule model to obtain the target execution rule output by the execution rule model, and then determines the target auditing result of the railway data corresponding to all target fields based on the target execution rule, without relying on manual intervention, capable of automatically determining the target auditing result, thereby reducing the complexity and error risk of the auditing process and reducing the labor cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0020] Figure 1 is a schematic flowchart of the railway data auditing method provided by the present invention.
[0021] Figure 2 is a schematic structural diagram of the railway data auditing device provided by the present invention.
[0022] Figure 3 is a schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners
[0023] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without making creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.
[0024] Figure 1 is a schematic flowchart of the railway data auditing method provided by the present invention. As Figure 1 shown, the method includes the following steps 110, 120, 130 and 140.
[0025] Step 110: Determine a plurality of target fields in the target railway data set based on the service requirements corresponding to the target railway data set.
[0026] Specifically, the target railway data set includes a plurality of fields. For example, the fields in the target railway data set may include fields such as train number, train ID, departure station, terminal station, departure time, and arrival time. The target railway data set can be represented by the following formula: Wherein, represents the first field in the target railway data set represents the second field in the target railway data set represents the th field in the target railway data set represents the th field in the target railway data set represents the th field in the target railway data set represents the number of fields in the target railway data set.
[0027] Furthermore, the fields in the target railway data set that meet the service requirements can be determined as target fields based on the service requirements corresponding to the target railway data set. For example, if the target railway data set is a train operation data set, the train ID, train position, and time can be determined as target fields to ensure the accuracy of train operation data. Another example is that if the target railway data set is a freight transportation data set, the cargo ID, load, loading time, and destination can be determined as target fields to ensure the consistency of cargo information.
[0028] Step 120: Determine the verification rules and executable manners respectively corresponding to each of the target fields; the executable manner characterizes the manner in which the railway data corresponding to the target field can be executed.
[0029] Specifically, the validation rules and executable manners respectively corresponding to each target field can be determined. The validation rules are used to validate the railway data corresponding to the target field, and the executable manner is used to represent the manner in which the railway data corresponding to the target field can be executed. The executable manner is serial and parallel, or, serial.
[0030] In addition, there is railway data corresponding to the target field, and the railway data corresponding to the target field may include multiple sub-railway data. For example, the railway data corresponding to the train number may be a set of {G06, Z08, G1895}, and the multiple sub-railway data included therein are G06, Z08, and G1895 respectively. Another example is that the railway data corresponding to the departure station may be a set of {Beijing, Guangzhou, and Shanghai}, and the multiple sub-railway data included therein are Beijing, Guangzhou, and Shanghai respectively.
[0031] In one embodiment, the method further includes: Inputting the railway industry specification file and rule template prompt corresponding to the target railway data set into a preset natural language model to obtain multiple initial validation rules output by the preset natural language model, and constructing a validation rule library based on all the initial validation rules; Determining the validation rules respectively corresponding to each of the target fields from the validation rule library.
[0032] Specifically, the railway industry specification file and rule template prompt corresponding to the target railway data set can be input into a preset natural language model, and further, multiple initial validation rules output by the preset natural language model can be obtained. Both the rule template prompt and the preset natural language model can be set as needed, and the embodiments of the present invention do not make specific limitations here. Further, a validation rule library can be constructed based on all the initial validation rules. It is easy to understand that the validation rule library can be updated in real time to adapt to the latest railway industry specification file. Validation rule library It can be represented by the following formula: Wherein, represents the first validation rule in the validation rule library represents the second validation rule in the validation rule library represents the validation rule library represents the second validation rule in the validation rule library represents the validation rule library represents the th validation rule in the validation rule library represents the validation rule library represents the validation rule for the field in the validation rule library.
[0033] Furthermore, when determining the validation rule corresponding to the target field, the validation rules respectively corresponding to each of the target fields can be selected and determined from the validation rule library.
[0034] In the above embodiments, a verification rule library adapted to railway industry specification documents is determined by using a natural language model, and then, from the verification rule library, the verification rules corresponding to each target field are determined, so that the verification rules of the target field are more accurate and timely.
[0035] In one embodiment, the verification rule library at least includes: digital verification, letter verification, value range verification, comparison verification with a preset data set, numerical range verification, uppercase letter verification, lowercase letter verification, Chinese character verification, date format verification, fixed content verification, uniqueness verification, non-empty verification, logical verification, length verification, and data timestamp verification.
[0036] Specifically, the verification rule library can at least include the following 14 verification rules, and the following exemplarily describes each verification rule: 1) Digital verification indicates that the railway data is a number and the verification is qualified when the number meets the preset precision requirements.
[0037] 2) Letter verification indicates that the verification is qualified when the railway data is a letter.
[0038] 3) Value range verification indicates that the verification is qualified when the railway data conforms to the range corresponding to the railway specification document.
[0039] 4) Comparison verification with a preset data set indicates that the verification is qualified when the railway data is the same as the data in the preset data set. The preset data set can be set according to the target railway data set to be audited, and the embodiments of the present invention do not make specific limitations here.
[0040] 5) Numerical range verification indicates that the verification is qualified when the railway data is within the preset numerical range.
[0041] 6) Uppercase letter verification indicates that the verification is qualified when the railway data contains uppercase letters.
[0042] 7) Lowercase letter verification indicates that the verification is qualified when the railway data contains lowercase letters.
[0043] 8) Chinese character verification indicates that the verification is qualified when the railway data is a Chinese character.
[0044] 9) Date format verification indicates that the verification is qualified when the railway data conforms to the preset date format.
[0045] 10) Fixed content verification indicates that the verification is qualified when the railway data contains the preset fixed content.
[0046] 11) Uniqueness verification indicates that the verification is qualified when each sub-railway data in the railway data corresponding to a certain field is different from each other.
[0047] 12) Non-empty verification indicates that the verification is passed when each sub-railway data in the railway data corresponding to a certain field is not empty.
[0048] 13) Logical verification can verify the logical relationship between two or more fields. For example, for the data corresponding to the same train, its departure time must be earlier than its arrival time. In the case where the departure time is earlier than the arrival time, the verification is passed. Another example is that for the data corresponding to the same train, there is a matching relationship between the time and the train position. In the case where the time and the train position match, the verification is passed.
[0049] 14) Length verification indicates that the verification is passed when the length of the railway data meets the preset length.
[0050] 15) Data timestamp verification indicates that the verification is passed when the data timestamp corresponding to the railway data conforms to the timing logic.
[0051] It should be noted that the verification rules corresponding to the target field can be one or multiple. For example, the verification rules corresponding to the departure time can be date format verification, or date format verification and numeric verification. The unified verification rules can also correspond to multiple target fields. For example, numeric verification can be used for verifying the train number and also for verifying the departure time.
[0052] In the prior art, a single rule is usually used to audit railway data. In the above embodiments of the present invention, it is set that the verification rule library at least includes: numeric verification, alphabetic verification, value range verification, comparison verification with a preset data set, numeric range verification, uppercase letter verification, lowercase letter verification, Chinese character verification, date format verification, fixed content verification, uniqueness verification, non-empty verification, length verification, and data timestamp verification. Based on the above verification rules, the target railway data set can be accurately audited, ensuring the audit quality. The application of multiple verification rules can also meet the audit requirements of various data and is more flexible.
[0053] In one embodiment, determining the verification rules respectively corresponding to each of the target fields includes: For each of the target fields, based on the preset matching relationship between the target field and the verification rules, determine the verification rules corresponding to the target field; or, input the target field into a rule recommendation model to obtain the verification rules corresponding to the target field output by the rule recommendation model; Wherein, the rule recommendation model is trained based on each of the sample fields and their respective preset verification rules.
[0054] Specifically, the preset matching relationship between the target fields and the verification rules can be preset as needed. Then, for each target field, based on the preset matching relationship between the target field and the verification rule, the verification rule corresponding to the target field can be determined. For example, the train number can correspond to two verification rules: digital verification and letter verification. The time can correspond to two verification rules: digital verification and date format verification.
[0055] Alternatively, a rule recommendation model can also be trained in advance based on each sample field in the sample railway dataset and the preset verification rules corresponding to all the sample fields. The rule recommendation model can be trained based on one of a random forest model, an Extreme Gradient Boosting (XGBoost) model, and a neural network model. Then, the target field can be input into the rule recommendation model to obtain the verification rule corresponding to the target field output by the rule recommendation model. Exemplarily, this process can also be represented by the following formula: Among them, represents the verification rule corresponding to the target field output by the rule recommendation model corresponding to the target field.
[0056] In the above embodiments, determining the verification rule corresponding to the target field based on the preset matching relationship between the target field and the verification rule can ensure the flexibility of the target field and the verification rule. Users can control the set of railway data by modifying the preset matching relationship. Inputting the target field into the rule recommendation model to obtain the verification rule corresponding to the target field output by the rule recommendation model can improve the verification accuracy of the railway data of the target field by using artificial intelligence.
[0057] In one embodiment, determining the executable manner corresponding to each of the target fields includes: For the railway data corresponding to each of the target fields, when there is no association relationship between the sub-railway data in the railway data, determining that the executable manner corresponding to the target field is serial and parallel; When there is an association relationship between the sub-railway data in the railway data, determining that the executable manner corresponding to the target field is serial.
[0058] Specifically, for the railway data corresponding to each target field, when there is no association relationship between the sub-railway data in the railway data, it can be determined that the executable manner corresponding to the target field is serial and parallel. When there is an association relationship between the sub-railway data in the railway data, it is determined that the executable manner corresponding to the target field is serial.
[0059] Exemplarily, the railway data corresponding to the train numbers is the set {G06, Z08, G1895}. Since there is no association relationship among the train numbers, the three sub-railway data of G06, Z08, and G1895 can be verified simultaneously in parallel to improve the verification efficiency. It is easy to understand that the three sub-railway data of G06, Z08, and G1895 can also be verified serially.
[0060] For another example, the railway data corresponding to the departure times is the set {8:46, 14:56, 17:10}. Since there is an association relationship among the departure times, and this association relationship is that the previous sub-railway data must be earlier than the subsequent sub-railway data on the time scale, the three sub-railway data of 8:46, 14:56, and 17:10 need to be verified serially and cannot be verified in parallel.
[0061] It should be noted that in the case of auditing railway data through a parallel execution method, it can be implemented through multi-core processing or distributed computing.
[0062] In the prior art, only a serial auditing method is used to audit railway data, resulting in a slow auditing process and making it difficult to meet the requirements of real-time data quality monitoring. In the above embodiments of the present invention, in the case where there is no association relationship among the sub-railway data in the railway data, the determined executable methods corresponding to the target fields are serial and parallel, enabling subsequent parallel auditing of railway data and accelerating the auditing rate compared with the existing data.
[0063] Step 130: Input the railway data, verification rules, and executable methods respectively corresponding to each of the target fields into an execution rule model to obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules, and executable methods respectively corresponding to multiple sample fields in a sample railway data set.
[0064] Specifically, an initial deep learning model can be trained in advance based on the sample railway data, verification rules, and executable methods respectively corresponding to multiple sample fields in a sample railway data set to obtain an execution rule model. The initial deep learning model can be, for example, a Long Short-Term Memory (LSTM) network, or it can also be a Transformer model. The embodiments of the present invention do not make specific limitations here. Further, the execution rule model can also introduce Reinforcement Learning (RL) to optimize the process of determining the target execution rule. During the process of training the execution rule model, the reward function can be set to one or more of the qualification rate, error rate, and execution speedup ratio.
[0065] Exemplarily, the execution speedup ratio can be determined by the following formula: wherein, represents the total time for auditing railway data with a serial execution mode, the total time for auditing railway data with a parallel execution mode, represents the verification rule verification field the time taken for the corresponding railway data.
[0066] Furthermore, after obtaining the execution rule model, the railway data, verification rules, and executable modes corresponding to each target field can be input into the execution rule model, which is used to determine the actual execution mode and execution order corresponding to the target field, so that the target execution rule output by the execution rule model can be obtained.
[0067] Step 140: Determine the target audit results of the railway data corresponding to all the target fields based on the target execution rule.
[0068] In one embodiment, the determining the target audit results of the railway data corresponding to all the target fields based on the target execution rule includes: For the railway data corresponding to each target field, determine the execution mode and execution order of the railway data based on the target execution rule; Audit all the railway data based on the verification rules, execution modes, and execution orders corresponding to all the railway data to determine the audit results of all the railway data; Determine the target audit results based on the audit results of the railway data corresponding to all the target fields.
[0069] Specifically, for the railway data corresponding to each target field, the execution mode and execution order of the railway data can be determined based on the target execution rule. It is easy to understand that if the executable mode of the railway data is serial and parallel, the execution mode of the railway data can be determined to be serial or parallel based on the target execution rule. If the executable mode of the railway data is serial, its execution mode must be serial. The execution order of the railway data represents the order of verification one after another among the railway data corresponding to multiple fields. For example, the execution order corresponding to the departure time should be before the execution order corresponding to the arrival time.
[0070] After determining the verification rules, execution methods, and execution order corresponding to all railway data, it is possible to verify whether all railway data conforms to the verification rules according to the execution methods and execution order, and determine the audit results of all railway data. For example, the execution order corresponding to the train number is the first to execute, and the execution order corresponding to the departure time is the second to execute. The execution method corresponding to the train number is parallel, and the verification rules are letter verification and number verification. Then, it is possible to simultaneously determine whether multiple train number data (sub-railway data) contain both letters and numbers through distributed computing to obtain the audit results corresponding to the train number. After the data audit corresponding to the train number is completed, since the execution method corresponding to the departure time is serial, it is necessary to verify multiple departure time data one by one to obtain the audit results corresponding to the departure time. It should be noted that if logical verification is used to verify the departure time and arrival time, it is based on serial verification, and the departure time and arrival time are alternately verified, that is, first verify the departure time and arrival time of Train A, and then verify the departure time and arrival time of Train B until all railway data corresponding to the departure time and arrival time are verified.
[0071] Exemplarily, the audit result of railway data can also be represented by any of the following formulas: Among them, represents the audit result after railway data are all verified through the verification rule .
[0072] Among them, represents the audit result after railway data passes all verification rules .
[0073] Among them, represents the audit result after all railway data are all verified through their respective corresponding verification rules .
[0074] Furthermore, it is possible to determine the target audit result based on the audit results of the railway data corresponding to all target fields.
[0075] In the above embodiments, for the railway data corresponding to each target field, the execution method and execution order of the railway data are determined based on the target execution rule, and then all the railway data are audited based on the verification rules, execution methods, and execution orders corresponding to all the railway data. Since the target execution rule is the optimal execution rule determined by the execution rule model, the execution method and execution order determined on this basis are also the most efficient execution method and execution order, thereby improving the efficiency of railway data auditing.
[0076] In one embodiment, auditing all the railway data based on the verification rules, execution methods, and execution orders corresponding to all the railway data to determine the auditing results of all the railway data includes: Auditing all the railway data based on the verification rules, execution methods, and execution orders corresponding to all the railway data to determine the auditing accuracy rate of all the railway data; For each piece of the railway data, when the auditing accuracy rate of the railway data is greater than the accuracy rate threshold corresponding to the railway data, determining that the auditing result of the railway data is qualified; When the auditing accuracy rate of the railway data is less than or equal to the accuracy rate threshold corresponding to the railway data, determining that the auditing result of the railway data is unqualified.
[0077] Specifically, all the railway data can be audited based on the verification rules, execution methods, and execution orders corresponding to all the railway data to determine the auditing accuracy rate of all the railway data. For example, the railway data corresponding to the train number includes a total of 10 sub-railway data, and 9 sub-railway data pass the digital verification and letter verification among the 10 sub-railway data, then the auditing accuracy rate of the railway data corresponding to the train number is 90%.
[0078] Exemplarily, it can be determined whether each sub-railway data in a piece of railway data conforms to the verification rule, and this process can be represented by the following formula: The verification value of each sub-railway data can be determined through the above formula, and the auditing accuracy rate of the railway data can be determined according to the verification values of all the sub-railway data.
[0079] For each piece of railway data, when the verification accuracy rate of the railway data is greater than the accuracy rate threshold corresponding to the railway data, it is determined that the verification result of the railway data is qualified. When the verification accuracy rate of the railway data is less than or equal to the accuracy rate threshold corresponding to the railway data, it is determined that the verification result of the railway data is unqualified. It is easy to understand that since different railway data have different degrees of impact on the business, different railway data can correspond to different accuracy rate thresholds. The accuracy rate threshold can be set according to business requirements, and the embodiments of the present invention do not make specific limitations here. For example, the accuracy rate threshold for the railway data corresponding to the departure time can be 100%, and the accuracy rate threshold for the railway data corresponding to the train number can be 90%.
[0080] In the above embodiment, the verification accuracy rate of all railway data is determined, and further, whether the verification result of the railway data is qualified is determined according to the comparison with the accuracy rate threshold, so that the verification result can more intuitively reflect the data quality of the railway data.
[0081] In one embodiment, determining the target verification result based on the verification results of the railway data corresponding to all the target fields includes: Determining the railway data corresponding to the unqualified verification results among the verification results as error-reporting railway data, and determining the error reasons for each of the error-reporting railway data; Determining the verification results of the railway data corresponding to all the target fields and the error reasons for all the error-reporting railway data as the target verification result.
[0082] Specifically, the railway data corresponding to the unqualified verification results among the verification results can be determined as error-reporting railway data, and the error reasons for each of the error-reporting railway data can be determined. The error reason indicates which sub-railway data in the error-reporting railway data fails to pass the verification. For example, if there is an eighth sub-railway data of 18:62 in the railway data corresponding to the time, then this railway data will be determined as error-reporting railway data, and the error reason for this error-reporting railway data is that the eighth sub-railway data fails to pass the numerical verification.
[0083] Further, the verification results of the railway data corresponding to all the target fields and the error reasons for all the error-reporting railway data can be determined as the target verification result. The target verification result can be in the form of a report or in the form of a table. The embodiments of the present invention do not make specific limitations here.
[0084] In the above embodiments, the railway data corresponding to the unqualified audit results among the audit results is determined as the reported error railway data, and the reasons for the reported errors of the reported error railway data are determined. Further, the audit results of the railway data corresponding to all target fields and the reasons for the reported errors of all reported error railway data are determined as the target audit results, which can help business personnel quickly locate data problems and correct the relevant data.
[0085] The railway data auditing method provided by the present invention determines multiple target fields in a target railway data set based on the business requirements corresponding to the target railway data set; determines the verification rules and executable methods respectively corresponding to each target field; inputs the railway data, verification rules, and executable methods respectively corresponding to each target field into an execution rule model to obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules, and executable methods respectively corresponding to multiple sample fields in a sample railway data set; and determines the target audit results of the railway data corresponding to all target fields based on the target execution rule. The technical solution of the present invention inputs the railway data, verification rules, and executable methods respectively corresponding to each target field in the target railway data set into the execution rule model to obtain the target execution rule output by the execution rule model, and then determines the target audit results of the railway data corresponding to all target fields based on the target execution rule, without relying on manual intervention, can automatically determine the target audit results, thereby reducing the complexity and error risk of the auditing process and reducing the labor cost.
[0086] Optionally, the technical solution of the present invention can also audit multiple target railway data sets in real time, update the verification rule library according to each target audit result, and can also correspondingly adjust the parameters of the rule execution model to ensure that the method of the present invention is adapted to the auditing of the latest railway data. The technical solution of the present invention can also construct the relationship between the target railway data set, business requirements, target fields, and verification rules into a knowledge graph, and input the knowledge graph into the execution rule model and the rule recommendation model to improve the output accuracy of the above models.
[0087] The railway data auditing device provided by the present invention will be described below. The railway data auditing device described below can be correspondingly referred to the railway data auditing method described above.
[0088] Figure 2 is a schematic structural diagram of the railway data auditing device provided by the present invention, as Figure 2 shown, the railway data auditing device 200 includes the following modules: The first determination module 210 is used to determine multiple target fields in the target railway data set based on the business requirements corresponding to the target railway data set; The second determination module 220 is configured to determine the verification rules and executable manners respectively corresponding to each of the target fields; the executable manner characterizes the manner in which the railway data corresponding to the target field can be executed. The input module 230 is configured to input the railway data, verification rules, and executable manners respectively corresponding to each of the target fields into an execution rule model, and obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules, and executable manners respectively corresponding to multiple sample fields in a sample railway dataset. The auditing module 240 is configured to determine a target auditing result of the railway data corresponding to all the target fields based on the target execution rule.
[0089] In one embodiment, the second determination module 220 is specifically configured to: For each of the target fields, based on a preset matching relationship between the target field and the verification rule, determine the verification rule corresponding to the target field; or input the target field into a rule recommendation model, and obtain the verification rule corresponding to the target field output by the rule recommendation model. Wherein, the rule recommendation model is trained based on each of the sample fields and their corresponding preset verification rules.
[0090] In one embodiment, the second determination module 220 is further specifically configured to: For the railway data respectively corresponding to each of the target fields, in the case where there is no association relationship between the sub-railway data in the railway data, determine that the executable manners corresponding to the target fields are serial and parallel; In the case where there is an association relationship between the sub-railway data in the railway data, determine that the executable manner corresponding to the target field is serial.
[0091] In one embodiment, the auditing module 240 is specifically configured to: For the railway data respectively corresponding to each of the target fields, determine the execution manner and execution order of the railway data based on the target execution rule; Audit all the railway data based on the verification rules, execution manners, and execution orders corresponding to all the railway data, and determine the auditing results of all the railway data; Determine the target auditing result based on the auditing results of the railway data corresponding to all the target fields.
[0092] In one embodiment, the auditing module 240 is further specifically configured to: Audit all the railway data based on the verification rules, execution manners, and execution orders corresponding to all the railway data, and determine the auditing accuracy rate of all the railway data. For each of the railway data, when the verification accuracy rate of the railway data is greater than the accuracy rate threshold corresponding to the railway data, determine that the verification result of the railway data is qualified; When the verification accuracy rate of the railway data is less than or equal to the accuracy rate threshold corresponding to the railway data, determine that the verification result of the railway data is unqualified.
[0093] In one embodiment, the verification module 240 is further specifically configured to: Determine the railway data corresponding to the unqualified verification results among the verification results as error-reporting railway data, and determine the error causes of the error-reporting railway data; Determine the verification results of the railway data corresponding to all the target fields and the error causes of all the error-reporting railway data as the target verification results.
[0094] In one embodiment, the railway data verification device further includes a construction module, and the construction module is specifically configured to: Input the railway industry specification file and rule template prompt corresponding to the target railway data set into a preset natural language model, obtain multiple initial verification rules output by the preset natural language model, and construct a verification rule library based on all the initial verification rules; Determine the verification rules respectively corresponding to the target fields from the verification rule library.
[0095] In one embodiment, the verification rule library at least includes: digital verification, letter verification, value range verification, comparison verification with a preset data set, numerical range verification, uppercase letter verification, lowercase letter verification, Chinese character verification, date format verification, fixed content verification, uniqueness verification, non-empty verification, logical verification, length verification, and data timestamp verification.
[0096] The railway data auditing device provided by the present invention determines multiple target fields in a target railway data set based on the business requirements corresponding to the target railway data set; determines the verification rules and executable manners respectively corresponding to each target field; inputs the railway data, verification rules and executable manners respectively corresponding to each target field into an execution rule model to obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules and executable manners respectively corresponding to multiple sample fields in a sample railway data set; and determines a target auditing result of the railway data corresponding to all target fields based on the target execution rule. The technical solution of the present invention inputs the railway data, verification rules and executable manners respectively corresponding to each target field in the target railway data set into the execution rule model to obtain the target execution rule output by the execution rule model, and then determines the target auditing result of the railway data corresponding to all target fields based on the target execution rule, without relying on manual intervention, can automatically determine the target auditing result, thereby reducing the complexity and error risk of the auditing process and reducing the labor cost.
[0097] Figure 3 An entity structure diagram of an electronic device is exemplified, as Figure 3 shown. The electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340. Among them, the processor 310, the communication interface 320, and the memory 330 complete communication with each other through the communication bus 340. The processor 310 may call logical instructions in the memory 330 to execute a railway data auditing method, and the method includes: Determining multiple target fields in the target railway data set based on the business requirements corresponding to the target railway data set; Determining the verification rules and executable manners respectively corresponding to each of the target fields; the executable manner represents the manner in which the railway data corresponding to the target field can be executed; Inputting the railway data, verification rules and executable manners respectively corresponding to each of the target fields into an execution rule model to obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules and executable manners respectively corresponding to multiple sample fields in a sample railway data set; Determining a target auditing result of the railway data corresponding to all of the target fields based on the target execution rule.
[0098] In addition, when the logical instructions in the above-mentioned memory 330 can be implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.
[0099] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the railway data auditing method provided by the above-mentioned various methods. The method includes: Determine multiple target fields in the target railway data set based on the business requirements corresponding to the target railway data set; Determine the verification rules and executable manners respectively corresponding to each of the target fields; the executable manner represents the manner in which the railway data corresponding to the target field can be executed; Input the railway data, verification rules, and executable manners respectively corresponding to each of the target fields into an execution rule model to obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules, and executable manners respectively corresponding to multiple sample fields in a sample railway data set; Determine the target auditing result of the railway data corresponding to all the target fields based on the target execution rule.
[0100] On yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the railway data auditing method provided by the above-mentioned various methods. The method includes: Determine multiple target fields in the target railway data set based on the business requirements corresponding to the target railway data set; Determine the verification rules and executable manners respectively corresponding to each of the target fields; the executable manner represents the manner in which the railway data corresponding to the target field can be executed; Input the railway data, verification rules, and executable methods corresponding to each of the target fields into an execution rule model to obtain the target execution rules output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules, and executable methods corresponding to multiple sample fields in a sample railway dataset; Determine the target audit results of the railway data corresponding to all the target fields based on the target execution rules.
[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.
[0102] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.
Claims
1. A railway data auditing method, characterized in that, Including: Determine multiple target fields in the target railway dataset based on the business requirements corresponding to the target railway dataset; Determine the verification rules and executable manners respectively corresponding to each of the target fields; The executable manner represents the manner in which the railway data corresponding to the target field can be executed; Input the railway data, verification rules, and executable manners respectively corresponding to each of the target fields into an execution rule model to obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules, and executable manners respectively corresponding to multiple sample fields in a sample railway dataset; Determine a target audit result of the railway data corresponding to all the target fields based on the target execution rule.
2. The railway data auditing method according to claim 1, wherein The determination of the verification rules respectively corresponding to each of the target fields includes: For each of the target fields, determine the verification rule corresponding to the target field based on the preset matching relationship between the target field and the verification rule; or input the target field into a rule recommendation model to obtain the verification rule corresponding to the target field output by the rule recommendation model; Wherein, the rule recommendation model is trained based on each of the sample fields and their corresponding preset verification rules.
3. The railway data auditing method according to claim 1, wherein Determine the executable manners respectively corresponding to each of the target fields, including: For the railway data respectively corresponding to each of the target fields, when there is no association relationship between the sub-railway data in the railway data, determine the executable manners corresponding to the target field as serial and parallel; When there is an association relationship between the sub-railway data in the railway data, determine the executable manner corresponding to the target field as serial.
4. The railway data auditing method according to claim 3, characterized in that, The determination of the target audit result of the railway data corresponding to all the target fields based on the target execution rule includes: For the railway data respectively corresponding to each of the target fields, determine the execution manner and execution order of the railway data based on the target execution rule; Audit all the railway data based on the verification rules, execution manners, and execution orders corresponding to all the railway data to determine the audit results of all the railway data; Determine the target audit result based on the audit results of the railway data corresponding to all the target fields.
5. The railway data auditing method according to claim 4, wherein The audit of all the railway data based on the verification rules, execution manners, and execution orders corresponding to all the railway data to determine the audit results of all the railway data includes: Audit all the railway data based on the verification rules, execution manners, and execution orders corresponding to all the railway data to determine the audit accuracy rate of all the railway data; For each of the railway data, when the audit accuracy rate of the railway data is greater than the accuracy rate threshold corresponding to the railway data, determine the audit result of the railway data as qualified; When the audit accuracy rate of the railway data is less than or equal to the accuracy rate threshold corresponding to the railway data, determine the audit result of the railway data as unqualified.
6. The railway data auditing method according to claim 5, wherein The determination of the target audit result based on the audit results of the railway data corresponding to all the target fields includes: Determine the railway data corresponding to the unqualified audit results among the respective audit results as error-reported railway data, and determine the reasons for the errors of the respective error-reported railway data; Determine the audit results of the railway data corresponding to all the target fields and the reasons for the errors of all the error-reported railway data as the target audit results.
7. The railway data auditing method according to any one of claims 1 to 6, characterized in that The method further includes: Input the railway industry specification file and rule template prompt corresponding to the target railway data set into a preset natural language model to obtain multiple initial verification rules output by the preset natural language model, and construct a verification rule library based on all the initial verification rules; Determine the verification rules respectively corresponding to the respective target fields from the verification rule library.
8. The railway data auditing method according to claim 7, wherein The verification rule library at least includes: digital verification, letter verification, value range verification, comparison verification with a preset data set, numerical range verification, uppercase letter verification, lowercase letter verification, Chinese character verification, date format verification, fixed content verification, uniqueness verification, non-empty verification, logical verification, length verification, and data timestamp verification.
9. A railway data auditing device, characterized in that, It includes: A first determination module, configured to determine multiple target fields in the target railway data set based on the service requirements corresponding to the target railway data set; A second determination module, configured to determine the verification rules and executable manners respectively corresponding to the respective target fields; The executable manner represents the executable manner of the railway data corresponding to the target field; An input module, configured to input the railway data, verification rules, and executable manners respectively corresponding to the respective target fields into an execution rule model to obtain a target execution rule output by the execution rule model; the execution rule model is trained based on the sample railway data, verification rules, and executable manners corresponding to multiple sample fields in a sample railway data set; An audit module, configured to determine the target audit results of the railway data corresponding to all the target fields based on the target execution rule.
10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the railway data audit method according to any one of claims 1 to 8.