Data extraction method based on data characteristics of signal system, medium and electronic equipment

Abstract

The invention discloses a data extraction method based on data characteristics of a signal system, a medium and electronic equipment. The data extraction method comprises the following steps: collecting multi-source data; classifying the multi-source data; identifying key data from the multi-source data; performing semantic association analysis on the multi-source data; generating a structured template and an initial strategy; generalization processing is carried out on the multi-source data based on the initial strategy and the structured template; verifying the generalized multi-source data to obtain a verification data report; and carrying out self-learning according to the verification data report. According to the data extraction method based on the data characteristics of the signal system, key information is quickly and accurately screened out from mass data based on frequency statistics and a dynamic distribution model; a semantic association analysis technology is introduced, internal logic of data is deeply understood, and a standardized key data extraction structure is automatically generated; and high reliability of output data is ensured while high automation of the whole process is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of rail transit data processing, and in particular to a data extraction method based on signal system data characteristics, a medium and an electronic device. BACKGROUND

[0002] In the field of key infrastructures such as rail transit and industrial control, the signal system as the core control center, its running state is directly related to the safety and efficiency of the whole system. In order to realize the comprehensive situation awareness of complex signal system, it is urgent to efficiently and accurately collect and process the massive, multi-source and heterogeneous operation logs and state data generated by the system. Such data is an important basis for supporting fault diagnosis, performance optimization and intelligent decision-making.

[0003] At present, the data collection methods commonly used in the industry mainly include two types: one is to directly read data through the simple API interface opened by the system; the other is to collect original log files through standard or manufacturer proprietary protocols such as Syslog. However, the above methods still rely heavily on manual intervention in the subsequent data processing link: usually, data engineers or domain experts need to manually identify key information related to safety instructions, device status, and train permission from the complex original messages based on experience, and manually write rules or scripts to clean, classify and "generalize" the data (i.e. convert to standardized and structured upper-layer usable format). This process is not only inefficient and time-consuming, but also difficult to meet the situation awareness needs in high real-time scenarios.

[0004] In addition, manual rules have strong subjectivity and vulnerability. On the one hand, rule making is easily affected by individual experience, making it difficult to guarantee the accuracy and consistency of the processing results; on the other hand, as the signal system version iterates or the data format evolves, the original rules often quickly become invalid and need to be frequently manually maintained and adjusted, which is prone to introduce errors or omissions, bringing potential risks to subsequent data analysis and decision-making.

[0005] More importantly, existing technologies generally lack self-learning and self-adaptive capabilities. In the face of new data sources, new log structures or dynamically changing data features, the traditional processing flow based on static preset rules cannot be automatically adjusted and must rely on manual reanalysis and configuration, resulting in high system operation and maintenance costs, poor scalability and weak robustness.

[0006] In summary, there is an urgent need in the field for a technical solution that can automatically and intelligently accurately identify and extract key information from signal system raw data, in order to effectively overcome the inherent defects of existing methods in terms of efficiency, accuracy, consistency and self-adaptive ability. SUMMARY

[0007] The application aims to provide a data extraction method, medium and electronic device based on signal system data characteristics, which has the advantage of high system reliability.

[0008] To achieve the above-mentioned purpose, the application provides a data extraction method based on signal system data characteristics, which comprises: S10, collecting multi-source data; S20, classifying the multi-source data; S30, identifying key data from the multi-source data; S40, performing semantic correlation analysis on the multi-source data; S50, generating a structured template and an initial strategy; S60, performing generalization processing on the multi-source data based on the initial strategy and the structured template; S70, verifying the multi-source data after generalization processing to obtain a verification data report; S80, self-learning according to the verification data report.

[0009] In some embodiments, the step S10 of collecting multi-source data comprises: S101, collecting the multi-source data from multiple interfaces; S102, sending the collected multi-source data to a data cache queue; and S103, standardizing and packaging the multi-source data.

[0010] In some embodiments, the step S20 of classifying the multi-source data comprises: S201, performing rule matching classification and machine learning clustering classification on the multi-source data; and S202, confirming and mapping the classification results.

[0011] In some embodiments, the step S30 of identifying key data from the multi-source data comprises: S301, extracting data characteristics from the multi-source data within a sliding time window; S302, establishing a probability distribution model based on the data characteristics, judging the keyness of each data according to the probability distribution model; and S303, updating the probability distribution model.

[0012] In some embodiments, in the step S303, the conditions for updating the probability distribution model include that N sliding time windows have passed or the probability distribution of the data characteristics has been detected to drift relative to the probability in the original probability distribution model.

[0013] In some embodiments, the probability distribution model is based on Poisson distribution.

[0014] In some embodiments, the updating process of the probability distribution model comprises: preparing a new training data set; updating the mean value of the probability distribution model based on the new training data set; updating the probability distribution model with the updated mean value; and retaining the old version of the probability distribution model.

[0015] In some embodiments, the step S40, the semantic correlation analysis on the multi-source data, comprises: S401, multi-level data preprocessing on the multi-source data; S402, entity and relationship extraction on the multi-source data based on natural language processing technology, and implicit relationship mining based on association rule mining; S403, constructing a knowledge graph; and S404, updating the knowledge graph.

[0016] In some embodiments, the step S80, self-learning according to the verification data report, comprises: S801, judging the error level of the verification result in the verification data report; S802, if it is a mild failure, returning to the step S60; if it is a severe failure, returning to the step S30; and if it is a persistent failure, the administrator intervenes.

[0017] In some embodiments, after the step S80, there is further a step S90 of self-iteration based on the learning result of the verification data report.

[0018] The application further provides a computer storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data extraction method based on signal system data features.

[0019] The application further provides an electronic device, which comprises a processor and a memory, and the memory stores a computer program, and the computer program is executed by the processor to implement the data extraction method based on signal system data features.

[0020] In summary, compared with the prior art, the data extraction method based on signal system data features, medium and electronic device provided by the application have the following beneficial effects: The data extraction method based on signal system data features provided by the application can quickly and accurately screen out key information from massive data based on frequency statistics and dynamic distribution models, introduce semantic correlation analysis technology to deeply understand the internal logic of data, automatically generate a standardized key data extraction structure, construct a refined process integrating strategy generalization, closed-loop verification and self-learning, and ensure the robustness of the processing process through a multi-layer return mechanism, thereby realizing high automation of the whole process and ensuring the high reliability of the output data. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1This is a flowchart of the data extraction method based on signal system data characteristics in this application.

[0022] Figure 2 This is a schematic diagram of the data flow of the data extraction method in the embodiments of this application. Detailed Implementation

[0023] The following will be combined with the appendix in the embodiments of the present invention. Figure 1 ~Attached Figure 2 The technical solutions, structural features, objectives and effects achieved in the embodiments of the present invention will be described in detail.

[0024] It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions. They are only used to facilitate and clarify the purpose of illustrating the embodiments of the present invention, and are not intended to limit the implementation conditions of the present invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationship, or adjustments to the size should still fall within the scope of the technical content disclosed in the present invention, provided that they do not affect the effects and objectives that the present invention can produce.

[0025] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only the expressly listed elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0026] like Figure 1 As shown, this invention provides a data extraction method based on signal system data features. The data extraction method includes: S10. Collect multi-source data. This includes: S101. Collecting multi-source data from multiple interfaces. In this embodiment, multiple data acquisition adapters are deployed within the system to collect data from multiple data interfaces. These data acquisition adapters include, but are not limited to: a RESTful API client for obtaining data from a modern signaling subsystem that supports Web services; a Syslog protocol listener for receiving system logs sent by network devices and servers; a direct database connection interface, such as JDBC / ODBC, for directly reading data from the business database; and a file collector for parsing specific log files, such as .log and .txt files.

[0027] After the data is collected from each data collection adapter, S102 is entered, and the collected multi-source data is sent to a data cache queue. In this step, the collected raw data is sent to a distributed message queue (such as Kafka or RabbitMQ). Thus, the data collection process and the data processing process are decoupled to cope with peak pressure of data inflow. When the instantaneous data collection amount is too large, the data can be stored in the cache queue to wait for processing, avoiding data loss during transmission.

[0028] S103, standardize and package multi-source data. Since multi-source data comes from different sources, the formats of data from different sources are different. In this step, the data items collected from various sources are packaged into an internal standard data object. The data object includes the following metadata: data_source_ip, indicating the data source IP, used to trace the data source; timestamp, used to record the data collection timestamp, thereby establishing a unified time axis; original_message, used to record the original log content, preserving the original appearance of the data, ensuring that information is not lost, and facilitating subsequent in-depth analysis and troubleshooting; source_system_type, indicating the signal subsystem type, such as ATS / CI / ATP, automatically inferred according to IP or message header, preparing for subsequent differentiated processing. In this step, by standardizing and packaging multi-source data with different sources and formats, the data processing process in the subsequent module is simplified, without the need to adapt different data formats in the subsequent steps, reducing the complexity of the system; and the data collection process and the data processing process are decoupled, the data collection process only focuses on data collection and labeling, and the data processing process focuses on data processing, without affecting each other; in addition, the original data is also preserved, which is convenient for subsequent tracing.

[0029] S20, classify multi-source data. In this step, the large amount of data collected in step S10 is preliminarily sorted. Specifically, S201, rule matching classification and machine learning clustering classification are performed on multi-source data. In this step, for logs with regular format and obvious features (such as messages with fixed header identification), the system first uses a predefined rule engine (such as Drools) for matching, i.e. rule matching classification. For example, all messages containing "ATS_CMD" are directly classified into "ATS dispatch instruction class". For logs without rules or irregular format, the system uses an unsupervised learning clustering algorithm (such as K-Means or DBSCAN) for classification. The unsupervised learning clustering algorithm takes the features of the log text after TF-IDF vectorization as input, and automatically aggregates similar data into clusters. Each cluster is assigned a temporary classification label (such as Cluster_1), thereby realizing machine learning clustering classification.

[0030] After the multi-source data is classified, the classification result is confirmed and mapped in step S202. The clustered classification result is audited by an administrator once, for example, a temporary label such as Cluster_1 is mapped to a business label (such as "Interlocking Communication Abnormal Class"), and the mapping relationship between the temporary label and the business label is recorded for automatic classification of the same feature data in the future.

[0031] S30, identifying key data from multi-source data. In this step, the multi-source data collected in the foregoing steps is subjected to data content frequency statistics and key marking based on a distribution model, thereby establishing a dynamic key evaluation model that can evolve and adapt to changes in data characteristics beyond a static threshold. It includes step S301, extracting data features from multi-source data in a sliding time window. Data is extracted based on a sliding time window. In this embodiment, the window size is 24 hours and the sliding interval is 1 hour. Real-time feature extraction is performed on the content under each data classification. The features can be specific error codes (such as "Error_504"), operation instruction verbs (such as "Set_Route"), or status keywords (such as "Signal_Red"). The frequency (Frequency, F) of each feature appearing in the current time window is continuously recorded.

[0032] After the data features are collected in the time window, step S302 is entered to establish a probability distribution model based on the data features to determine the key of each data. Without a probability distribution model based on existing data, an initial probability distribution model is used as a reference. In this embodiment, the probability distribution model is based on Poisson distribution, which is suitable for describing the probability of the number of random events occurring per unit time.

[0033] After the probability distribution model is determined, data of an initial time period (e.g., the first 24 hours) is collected, and the average occurrence frequency (λ) of each feature is calculated as the parameter of the Poisson distribution. After the parameters of the probability distribution model are determined, criticality determination is performed. The specific process of criticality determination is as follows: for the frequency F of a data feature in the current window, the probability P(X≥F) under the initial probability distribution model is calculated. If the probability value is lower than a pre-set significance level a (for example, a = 0.05), it is considered that the occurrence of the feature belongs to a small probability event, but the frequency is high, and therefore it is marked as “most critical”. Similarly, different confidence intervals can be set (for example, 0.05 < P < 0.20 is marked as “less critical”, and P≥0.20 is marked as “general”). If a data marked as “most critical” always successfully passes in subsequent generalization and verification, it proves that the marking of the model is accurate, and the effectiveness of the current model is strengthened. On the contrary, if a “most critical” data frequently fails in subsequent steps or even always needs manual intervention, such feedback is regarded as negative feedback. In the next model iteration, these features will be focused on to analyze whether the frequency pattern has changed essentially, thereby accelerating the model correction process.

[0034] S303, updating the probability distribution model. In the embodiment, after the probability distribution model is generated based on the collected data in the foregoing step, the probability distribution model is iteratively updated periodically or under specific trigger conditions after new multi-source data is collected, to ensure that it always conforms to the real running state of the current system. In step S303, the conditions for updating the probability distribution model include time-triggered updating and event-triggered updating. The time-triggered updating is to update after N sliding time windows, for example, in the embodiment, 24 time windows (i.e., one day) are selected for probability distribution model updating. The event-triggered updating is to detect that the probability distribution of a data feature deviates from the probability in the original probability distribution model. For example, if the frequency of a feature significantly deviates from the prediction range of the original model for a plurality of windows, the updating is immediately triggered.

[0035] Specifically, the updating process of the probability distribution model includes: preparing a new training data set, and selecting a data set that can reflect the recent behavior pattern, for example, the feature frequency data of the past M time windows (e.g., the past 7 days) is taken as the new training data set.

[0036] Based on the new training data set, the new average value “λ” of the frequency of each feature is recalculated, which represents the “normal” level of the occurrence of the feature at the current stage, and the average value of the probability distribution model is updated.

[0037] The probability distribution model is updated with the new average value "λ" and replaces the existing Poisson distribution model. Thereafter, critical flags are based on this new, more recent, distribution model.

[0038] After the probability distribution model is updated, the old version of the probability distribution model is retained so that it can be rolled back to the previous stable version in case of problems with the update.

[0039] In order to enable the machine to understand the business meaning behind the data fields and the inherent logic, rather than just superficial character processing, semantic correlation analysis is performed on the multi-source data in step S40; including: S401, multi-level data preprocessing is performed on the multi-source data, in this step, for semi-structured log text, firstly, the key-value pairs and fixed format fields (such as [Time: 10:00][Device: S07][Status: ON]) in the text are extracted using regular expressions or custom parsers, and are converted into structured data records. Natural language processing (NLP) is performed on the free text description (such as the alarm information "Train T123 is emergency braking outside signal S07 due to failure of turnout P5 locking") in the log.

[0040] S402, based on natural language processing technology, entity and relationship extraction is performed on multi-source data, and implicit relationships are mined based on association rules. The process of natural language processing of free text specifically includes word segmentation, stop word filtering, and part-of-speech tagging. Word segmentation refers to dividing a sentence into meaningful word units; stop word filtering includes removing words with no meaning such as "of", "in", "and" and the like; and part-of-speech tagging includes identifying nouns, verbs, etc. to help understand the role of words in the context. Then, entity and relationship extraction based on natural language processing (NLP) is performed, including named entity recognition, using a pre-trained or domain-tuned NER (Named Entity Recognition) model (BiLSTM-CRF or BERT-based model), to identify entities in the signal system field from the text. These entities are classified into predefined categories, such as device entities: signal machines (S07), turnouts (P5), trains (T123); state entities: red light, lock failure, emergency braking; operation entities: set route, cancel command; and location entities: XX station, XX section. After entity extraction, relationship extraction is performed, using dependency syntax analysis or a pre-trained language model-based relationship classifier to analyze the relationships between entities and form relationship triples (subject, predicate, object). For example, from the sentence "Train T123 is parked outside signal machine S07", triples such as (T123, parked at, S07 outside) and (S07, located in, XX section) can be extracted. These triples intuitively express the semantic network implied in the data.

[0041] Based on association rule mining, implicit relationships are mined for structured data fields by analyzing the statistical correlations between them. First, an association rule mining algorithm such as Apriori or FP-Growth is used to scan a large number of historical data records. The algorithm outputs association rules in the form of {Field_A = Value_A, Field_B = Value_B} => {Field_C = Value_C}. For example, a rule may be found: {device_type="signal", signal_status="red"} => {train_speed="0"} (support=0.8, confidence=0.99), where support represents the frequency of the rule pattern in the data set, and confidence represents the reliability of the rule. The method filters out strong rules with support and confidence higher than the set threshold. These rules reveal stable and reliable business logic constraints between data fields.

[0042] S403, constructing a knowledge graph, in this step, the semantic analysis result in the previous step is sublimated, and the results obtained in S401 and S402 are fused to construct a field knowledge graph, including taking the entity recognized by NER (Named Entity Recognition) as a node of the graph, taking the triplets obtained by relation extraction and the high-confidence rules mined by association rule mining (converted into graph relations) as edges of the graph. A knowledge graph instance is generated: (train T123) --[controlled by]--> (ATS system A) --[issues an instruction to]--> (signal S07) --[displays as]-->(red light state) --[causes]--> (train T123 speed = 0). The generated knowledge graph dynamically and structurally represents the complex relationship between key data points in the current signal system, and is the "brain" for subsequent generation of extraction templates and intelligent checking.

[0043] S404, updating the knowledge graph, updating the knowledge graph from the obtained feedback, for example, if a piece of data extracted based on the current semantic rule is judged by a human as incorrect, this case will be used as a negative sample, and new verified positive and negative samples will be used to incrementally train the NER model and the relation extraction model, so that the model can learn new entity appellations or relation expressions, and constantly adapt to changes in log description methods, and improve the accuracy of semantic understanding.

[0044] S50, generating a structured template and an initial strategy, in this step, the results of semantic analysis are solidified into executable rules. The specific implementation includes: template structure definition and field mapping and rule generation. Including: the extraction template adopts a structured definition in the form of JSON Schema or XML Schema, and defines a template object for each type of data, and the template object includes: required_fields: a list of fields that must be extracted (such as train_id, signal_status); extraction_rules: rules for extracting these fields from the original data (such as using a regular expression to match "Train ID: (\d+)" to obtain train_id); semantic_constraints: semantic constraints discovered in step S40 (such as when signal_status is "red", the speed field must be 0, otherwise it is marked as an exception).

[0045] S60, generalization of multi-source data based on initial strategy and structured template, including strategy library loading. Strategy library loading is to load a configurable generalization strategy library. Each strategy includes conditions and actions. Strategy example, condition: field name equals "timestamp". Action: call StandardizeTimeFormat() function to unify all time formats to "YYYY-MM-DD HH:MM:SS". Pipeline processing: data is sent to a generalization pipeline, which performs the following in turn: extraction: according to the structured template in step S50, the target field is extracted from the original log. Cleaning: apply the strategy library to perform format conversion, unit unification, invalid value filtering, etc. Enrichment: supplement some derived fields (such as generating day_of_week according to timestamp) as needed.

[0046] S70, verification of the generalization of multi-source data to obtain a verification data report. In this step, the quality control of the generalization of multi-source data is performed, including defining a standard data dictionary and an automatic verification process. Defining a standard data dictionary includes: maintaining a standard data dictionary, clearly defining the data type (String, Integer), length, value range (such as 0-100), and allowed enumeration values (such as "normal", "warning", "error") of each standard field. The automatic verification process includes: the verifier compares each data after generalization with the aforementioned maintained standard data dictionary field by field. Not only the syntax is checked (such as type, length), but also the business logic is checked (such as speed value cannot be negative). The verification result generates a detailed verification data report.

[0047] S80, self-learning according to the verification data report. In this step, based on the verification data report, self-learning and correction are performed, which is the quality control and intelligent evolution center of the entire data extraction method. In this step, it is not simply to judge whether each report in the verification data report is "passed" or "not passed", but a feedback control system with complex decision logic. According to the verification result, the optimal correction path is automatically selected, and only when it cannot be automatically processed does it request human intervention. Specifically, step S80 includes: S801, judging the error level of the verification result in the verification data report; in step S801, the verification data report is received from step S70. The verification data report generates a status code and detailed information for each data record.

[0048] The status codes include: CODE_SUCCESS, CODE_FORMAT_ERROR, CODE_LOGIC_ERROR, and CODE_CRITICAL_ERROR. Among them, CODE_SUCCESS indicates that all fields pass the verification; CODE_FORMAT_ERROR indicates a field-level syntax error (such as data type inconsistency, length overrun, value out of enumeration range); CODE_LOGIC_ERROR indicates a record-level business logic error (such as triggering a brake alarm when the speed is not zero, which conflicts with the semantic rules); and CODE_CRITICAL_ERROR indicates a serious error or unknown error (such as an unresolvable data format, missing critical fields).

[0049] S802, if the status code is CODE_SUCCESS, the data is marked as qualified and output to the downstream situation awareness application. The successful processing record (including the classification used, criticality level, and generalization strategy) is stored as a high-quality positive sample in the learning sample library. These samples will be used for reinforcement learning of the strategy library and model in subsequent steps, consolidating the correct processing path.

[0050] If the status code is CODE_FORMAT_ERROR or part of CODE_LOGIC_ERROR, it indicates that the problem may be in the data generalization link, which is a mild failure. The data is returned to the generalization pipeline entrance of step S60 again, and the generalization pipeline entrance of step S60 is not simply repeated the last failed strategy, but selects a backup strategy from the generalization strategy library for this error type. For example: if the date format "DD / MM / YY" fails to parse, the system will try "YYYY-MM-DD", "MM / DD / YYYY", etc. in turn. And in the retry process, monitor the number of retries, and set a retry counter for each data. If the number of retries for the same error exceeds the preset threshold (such as 3 times), it is determined that the strategy retry is invalid, and the error is upgraded. If the data passes the verification after a retry, record the mapping relationship of "problem-backup strategy", and enrich the decision-making knowledge of the strategy library.

[0051] If the data fails after multiple retries, or the status code is a severe CODE LOGIC ERROR / CODE CRITICAL ERROR, it is considered a severe failure. A severe failure indicates that the problem might originate from an even more upstream stage, i.e. a critical data recognition error or a template mismatch, and the data (or a batch of similar errors) is sent back to the input queue in step S30 for reprocessing. In step S30, the data will be re-evaluated based on the latest distribution model. Due to its frequent check failures, the model might be downgraded in its criticality level (e.g. from “most critical” to “general”). If a large amount of similar data is backtracked, a root cause analysis process is triggered to determine whether the classification rules in step S20 need to be adjusted or the structured templates in step S50 need to be updated, in order to proactively adapt to the drift of data characteristics.

[0052] If the data still fails after re-evaluation and re-generalization, it is considered a persistent failure, indicating that a completely new pattern or anomaly is encountered, and the administrator is involved. This includes automatically creating a high-priority alert ticket in the management platform, which contains not only the original data that failed, but also a complete “diagnosis report” that includes all processing steps the data has gone through, the generalization strategies tried, the details of the check error, and the backtracking history. After completing the above steps, the abnormal data is placed in the “quarantine zone” and waits for the administrator to handle it without affecting the normal data pipeline. After receiving the high-priority alert ticket, the administrator analyzes the diagnosis report and solves the problem (e.g. discovers a new device alarm code). After solving the problem, the administrator feeds back the solution to the problem, such as adding a new classification rule in step S20, or adding a new field extraction definition in the structured template in step S50, or adding a new generalization strategy in the strategy library in step S60. And this administrator handling process is recorded as a successful manual intervention case, and the corresponding knowledge base is updated, so that it can be automatically processed in the subsequent processing process without the need for manual intervention.

[0053] After step S80, there is also step S90, which is self-iteration based on the learning results of the check data report. In successful cases in step S80, the case is proven to be an effective generalization strategy, and its weight is increased. In “manual handling” cases, when the administrator solves an unknown problem (e.g. adds a new data format), the administrator is guided to solidify the solution (e.g. a new regular expression, a new classification rule) as a new strategy or classification rule and add it to the corresponding library. Based on various positive and negative samples obtained during use, new positive and negative samples are used to incrementally train the classification model obtained in step S20 and the semantic analysis model obtained in step S40, so that they can adapt to the evolution and changes of the signal system.

[0054] The data extraction method based on signal system data characteristics of the present application realizes the goal of quickly and accurately positioning core information from massive data through machine learning engine to traverse learning of full-quantity raw data. Firstly, the data is classified according to data sources, and then the classified results are statistically analyzed by using mathematical distribution model, and the data content with the highest occurrence frequency is automatically marked as key data to complete preliminary screening. For the data screened out preliminarily, the semantic learning engine is used to deeply analyze the internal semantic association between fields, and a standardized "key data structure" is automatically constructed. The structure clearly defines the standard fields to be extracted from various data sources, replacing the manual rule setting link, and realizing the intelligent and automatic generation of data extraction rules.

[0055] The present application provides a data extraction method integrating policy-based generalization, closed-loop verification and self-learning. The data extraction method automatically performs data generalization operation under the guidance of initial strategy according to the key data structure, and automatically compares the results with standard data. A multi-layer return mechanism is innovatively established: return to retry if verification fails; return to the first stage of re-screening if multiple failures occur, forming a closed-loop self-correction ability; if it still cannot be solved, trigger manual intervention alarm, thereby ensuring high degree of processing automation while guaranteeing the robustness of the entire system and the reliability of the output data.

[0056] As shown in Figure 2 The following illustrates the data extraction method based on signal system data characteristics of the present application with a specific embodiment. Figure 2 The main flow in S10-S90 shows the sequential execution process from data collection to final optimization, embodying the complete life cycle of data from collection to final optimization.

[0057] First, multi-source data is collected. In this embodiment, the data sources include ATS (Automatic Train Supervision), CI (Interlocking System), and ATP (Automatic Train Protection). Specifically, the ATS data source collects train operation plan data via a RESTful API. The raw data is: {"train_id": "T123", "planned_route": "StationA_to_StationC", "departure_time": "2024-06-20 08:00:00"}. The CI data source collects equipment status logs via Syslog. The raw data collected is... <134> 2024-06-20T08:00:01Z CI-ServerSignal_S07 Status=Red. The ATP system data source collects real-time monitoring data through the database interface. The raw data collected is: T123,45,2024-06-20 08:00:05 (representing train ID, speed, and timestamp, respectively). After collecting multi-source data, the collected data is standardized and encapsulated. Code example for standardized encapsulation: json { "data_source_ip": "192.168.1.10", "timestamp": "2024-06-20T08:00:05Z", "original_message": "Original collected data", "source_system_type": "ATP"} After completion, the collected multi-source data is classified, including rule-matching classification and machine learning clustering classification. For example, if a log contains "Signal_S07" and "Status=Red", the rule engine directly classifies it as "Signal Status". For example, after TF-IDF vectorization of the log "Point switch P05 operationdelay 1500ms", the clustering model clusters it with historical "Switch Operation Anomaly" logs, temporarily labeling it Cluster_5. After classification, manual mapping is performed; for example, the administrator maps Cluster_5 to "Switch Switching Anomaly".

[0058] Step S30, key data is identified from the multi-source data, including data content frequency statistics and key marking based on distribution model. In the exemplary embodiment, the statistical process includes: in the 08:00-09:00 time window, the number of times of occurrence of different states in the "signal status class" is counted: Signal_S07 Status=Red: 150 times Signal_S08 Status=Green: 30 times Other states: average 5-10 times Key marking: based on Poisson distribution, the frequency of occurrence of Signal_S07 Status=Red is much higher than the expected value (λ=50), P value <0.01, so it is marked as "most key" data.

[0059] Step S40, semantic correlation analysis is performed on the multi-source data, and in the exemplary embodiment, the identification result is as follows: entity recognition: from the log "Train T123 emergency stop at Signal_S07 due to pointP05 fault", the following is identified: Device entity: T123 (train), S07 (signal), P05 (turnout); State entity: emergency stop (emergency stop), fault (fault); Relation extraction: generate triplets: (T123, trigger, emergency_stop), (P05_fault, cause, emergency_stop); Association rule: the rule {signal_status=Red, point_status=fault} =>{train_speed=0} (confidence 98%) is mined.

[0060] Step S50, a structured template and an initial strategy are generated. In the exemplary embodiment, the generated JSON Schema template example is as follows: json { "class": "signal_status", "required_fields": ["signal_id", "status", "timestamp", "related_trains"], "extraction_rules": { "signal_id": "regex: Signal_(\\w+)", "status": "regex: Status=(\\w+)" }, "semantic_constraints": [ "if status == 'Red', then related_trains must not be empty" ]}.

[0061] After completion, enter step S60, and the multi-source data is generalized based on the initial strategy and the structured template. In the example embodiment, the generalization process is as follows: original data: Signal_S07 Status=Red 2024-06-20 08:00:01; extraction: signal_id: "S07", status: "Red" are extracted according to the template; generalization process includes: time format standardization: 2024-06-20T08:00:01Z; status value mapping: "Red" → "STOP"; data completion: according to the knowledge graph, related_trains: ["T123"] is automatically added.

[0062] After completion, enter step S70, and the multi-source data after the generalization is checked to obtain a checking data report. In the example embodiment, the standard data dictionary is defined as follows: json { "signal_id": {"type": "string", "pattern": "^S\\d{2}$"}, "status": {"type": "string", "enum": ["STOP", "CAUTION", "CLEAR"]}, "timestamp": {"type": "string", "format": "iso8601"}, "related_trains": {"type": "array", "minItems": 0}} The checking result is that the data after the generalization completely conforms to the standard format, and is marked as CODE_SUCCESS.

[0063] Subsequently, enter step S80, and perform self-learning according to the verification data report. The multi-layer feedback mechanism in step S80 is the core of the data extraction method of the present embodiment. The sub-process in step S80 demonstrates the multi-layer processing mode, including: First layer processing (strategy retry): for light errors, return to step S60 for retry; second layer processing (key re-evaluation): for severe errors, return to step S30 to re-evaluate the data key; third layer processing (manual intervention): for persistent failures, manually process and update the rule library, and the update content is directly fed back to steps S20, S50, and S60. In this step, a multi-layer feedback and closed-loop self-correction mechanism is realized. There are three cases, and the scene processing of light failure is as follows: the time format "20 / 06 / 2024 08:00" verification fails (CODE_FORMAT_ERROR). The system returns to step S60, tries "DD / MM / YYYY", "MM / DD / YYYY", and the third try "DD-MM-YYYY" succeeds.

[0064] The scene processing of severe failure is as follows: the log "Signal_S99 Mode=Test" still fails to verify after multiple generalizations (CODE_LOGIC_ERROR). Return to step S30, and the probability distribution model identifies "S99" as a test signal, and reduces its key from "most critical" to "general".

[0065] The scene processing of persistent failure is as follows: a brand new encrypted log format cannot be parsed. Create a work order, and the administrator intervenes to find that it is a log of a new security module. After adding a new decryption strategy, the problem is solved.

[0066] After completion, enter step S90, and perform self-iteration based on the learning results of the verification data report. The strategy library optimization results in step S90 are fed back to the classification model in step S20 and the semantic analysis model in step S40, which embodies the continuous self-learning ability. In addition, the successful strategies of step S80 also flow into step S90 for accumulation, continuously maintaining the learning ability for new data.

[0067] For cases that are successfully processed, positive learning is performed: the experience of successfully processing the "20 / 06 / 2024" format is recorded, and the mapping strategy weight of this format to the ISO format is increased. For negative samples obtained during use, inverse learning is performed: the manually processed encrypted log case is added to the training set to update the semantic analysis model in step S40, and the semantic analysis model is incrementally trained. The model is updated regularly, and the classification model is retrained with new data every week to enable it to recognize new log types.

[0068] The embodiment also provides a computer storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data extraction method based on signal system data features.

[0069] The embodiment also provides an electronic device, which comprises a processor and a memory, and the memory stores a computer program, and the computer program is executed by the processor to implement the data extraction method based on signal system data features.

[0070] To sum up, the application provides a data extraction method based on signal system data features, a medium and an electronic device. A key data automatic identification and quantitative evaluation mechanism of “dynamic distribution model + iterative learning” is innovatively designed. Unlike traditional methods that rely on fixed rules or static thresholds, the data extraction method of the application introduces a statistical analysis method based on a probability distribution model and has an iterative learning capability. By continuously counting the occurrence frequency of data features in a sliding time window, the parameters of the distribution model are dynamically updated, so that the system running state can be automatically adapted to changes, and accurate and adaptive identification and grade quantification of key data can be achieved, thereby fundamentally avoiding identification bias caused by outdated rules.

[0071] In addition, the application also creatively constructs a key data structure template automatic generation method of “multi-modal semantic analysis fusion”. The limitations of single analysis technology are broken through, the named entity recognition and relationship extraction technology in the natural language processing field are combined with the association rule mining technology in the data mining field, and the semantic association and statistical rules inside the data are deeply analyzed. Further, the analysis results are fused to construct a lightweight domain knowledge graph, a key data extraction template with semantic understanding ability and business logic constraints is automatically generated, and the mode of completely relying on manual experience to formulate rules is replaced.

[0072] Finally, the application initiatively implements a whole-process intelligent data refining and self-optimizing data extraction method of “strategy guidance, closed-loop verification and feedback learning”, which connects data generalization, verification and feedback into a complete intelligent closed loop. Not only is the data generalized according to the initial strategy and template, but also a multi-level and traceable closed-loop verification mechanism is designed. The verification result will trigger different feedback paths (such as retry and back to the screening stage), and finally be used for iterative optimization of the generalization strategy library and the upstream analysis model, so that the whole system has the ability to continuously learn from the processing results and self-improve, and realizes the leap from “automation” to “intelligence”.

[0073] Compared with the prior art, the data extraction method based on signal system data characteristics has a qualitative improvement in the efficiency of data extraction: through full-process automation, the traditional inefficient mode of relying on manual analysis and rule writing is completely changed. From data classification, key identification to template generation and generalization processing, all are automatically completed by the algorithm engine, greatly shortening the data processing cycle and meeting the needs of real-time or quasi-real-time situation awareness.

[0074] The data extraction method based on signal system data characteristics guarantees the accuracy and consistency of data. The key data identification based on dynamic models and semantic analysis avoids the randomness and errors of manual subjective judgment. The closed-loop verification mechanism acts as a "quality checkpoint" to ensure that only data meeting the standards can be output, effectively eliminating the inevitable negligence and inconsistency problems in manual processing, and significantly improving the reliability of the final data.

[0075] The data extraction method based on signal system data characteristics has strong adaptability and scalability, with iterative learning and feedback optimization capabilities, and can evolve itself following the evolution of the signal system. When new data sources appear or data formats change, it can gradually adapt through its learning mechanism without or with only a small amount of manual intervention to restore efficient processing, greatly reducing long-term operation and maintenance costs, and having good scalability and life cycle.

[0076] The data extraction method based on signal system data characteristics can realize the continuous growth of knowledge accumulation and intelligent level. In the processing process, successful strategies and manual disposal schemes are continuously converted into reusable knowledge and deposited in the strategy library and model. This makes the model not a fixed tool, but an intelligent agent that "gets smarter with use", with processing capacity and accuracy increasing over time, accumulating data resources into valuable digital assets.

[0077] Although the content of the present application has been described in detail through the above preferred embodiments, it should be recognized that the above description should not be considered as limiting the present application. After reading the above content, various modifications and alternatives of the present application will be apparent to those skilled in the art. Therefore, the scope of protection of the present application should be defined by the appended claims.

Claims

1. A data extraction method based on signal system data features, characterized in that, The data extraction method includes: S10, collecting multi-source data; S20, classifying the multi-source data; S30, identifying key data from the multi-source data; S40, performing semantic association analysis on the multi-source data; S50, generating a structured template and an initial strategy; S60, performing generalization processing on the multi-source data based on the initial strategy and the structured template; S70, verifying the generalized multi-source data to obtain a verification data report; and S80, performing self-learning based on the verification data report.

2. The data extraction method based on signal system data features as described in claim 1, characterized in that, Step S10, collecting multi-source data, includes: S101, collecting the multi-source data from multiple interfaces; S102, sending the collected multi-source data into a data cache queue; and S103, standardizing and encapsulating the multi-source data.

3. The data extraction method based on signal system data features as described in claim 1, characterized in that, Step S20, classifying the multi-source data, includes S201, performing rule matching classification and machine learning clustering classification on the multi-source data; and S202, confirming and mapping the classification results.

4. The data extraction method based on signal system data features as described in claim 1, characterized in that, Step S30, identifying key data from the multi-source data, includes S301, extracting data features from the multi-source data within a sliding time window; S302, establishing a probability distribution model based on the data features, and determining the criticality of each data point based on the probability distribution model; and S303, updating the probability distribution model.

5. The data extraction method based on signal system data features as described in claim 4, characterized in that, In step S303, the conditions for updating the probability distribution model include after N sliding time windows or after detecting that the probability distribution of the data feature has drifted relative to the probability in the original probability distribution model.

6. The data extraction method based on signal system data features as described in claim 5, characterized in that, The probability distribution model is based on the Poisson distribution.

7. The data extraction method based on signal system data features as described in claim 6, characterized in that, The update process of the probability distribution model includes: preparing a new training dataset; updating the mean of the probability distribution model based on the new training dataset; updating the probability distribution model with the updated mean; and retaining the old version of the probability distribution model.

8. The data extraction method based on signal system data features as described in claim 1, characterized in that, Step S40, performing semantic association analysis on the multi-source data, includes: S401, performing multi-level data preprocessing on the multi-source data; S402, extracting entities and relationships from the multi-source data based on natural language processing technology, and mining implicit relationships based on association rules; S403, constructing a knowledge graph; and S404, updating the knowledge graph.

9. The data extraction method based on signal system data features as described in claim 1, characterized in that, Step S80, self-learning based on the verification data report, includes: S801, determining the error level of the verification result in the verification data report; S802, if it is a minor failure, returning to step S60; if it is a severe failure, returning to step S30; if it is a persistent failure, the administrator intervenes.

10. The data extraction method based on signal system data features as described in claim 9, characterized in that, Following step S80, step S90 is further included, which involves self-iteration based on the learning results of the verification data report.

11. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data extraction method based on signal system data features as described in any one of claims 1-10.

12. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the data extraction method based on signal system data features as described in any one of claims 1-10.

Images