Method and system for constructing high-quality data set for large model in urban rail field

Abstract

The invention provides a method and system for constructing a high-quality data set for a large model in the urban rail field, and the method comprises the following steps: collecting and summarizing data meeting business requirements, and enabling the collected data to serve as an input data source; data is cleaned through data preprocessing, so that the quality of the data is improved; labels are added to the preprocessed data, so that key features of the data can be better understood and learned; and performing quality auditing and dynamic management on the annotated data through data maintenance. The invention provides a method and a system for constructing a high-quality data set for a large model in the urban rail field. The method and the system cover multiple aspects of data acquisition, preprocessing, labeling, maintenance, safety and the like. The systematization and standardization degree of the whole data processing process is improved, and the data quality and the model training effect can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of urban rail data processing, and more specifically, relates to a method and system for constructing a high-quality data set for a large model in the field of urban rail. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the excellent performance of big models, various industries are actively exploring how to use artificial intelligence to promote the intelligent transformation of their businesses. Focusing on the field of urban rail transit, big models also have great potential for development. They can not only improve operational management efficiency and optimize passenger experience, but also further enable the digital transformation of enterprises. However, to give full play to the advantages of big models in the field of urban rail transit, building high-quality data sets is a key step.

[0003] At present, one of the main challenges in building large models in the field of urban rail transit is how to effectively collect, clean, annotate and manage large-scale urban rail transit data, which usually has the characteristics of wide sources, diverse formats and different storage methods. Data information includes equipment and facility monitoring information, train operation status, passenger flow and other information.

[0004] At present, there is a lack of pre-processing technology specifically targeting the characteristics of urban rail data on the market, which makes it difficult to ensure the quality of the data set. General data annotation tools and platforms are difficult to meet the requirements of urban rail for specific scenarios and features. The existing data set maintenance methods are not perfect, and the lack of data security and privacy protection makes it impossible to form a reliable, high-quality urban rail large model data set. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for constructing a high-quality data set for a large model in the field of urban rail transit, aiming to solve the technical problem in the prior art that there is a lack of preprocessing technology specifically targeting the characteristics of urban rail transit data, which makes it difficult to ensure the quality of the data set.

[0006] To achieve the above object, the technical solution adopted by the present invention is: to provide a method for constructing a high-quality data set for a large model in the field of urban rail transit, comprising the following steps:

[0007] Collect and summarize data that meets business needs as input data sources;

[0008] Clean the data through data preprocessing to improve the data quality;

[0009] Add labels to the preprocessed data so that the key features of the data can be better understood and learned;

[0010] Conduct quality review and dynamic management of labeled data through data maintenance.

[0011] Preferably, the data is cleaned by data preprocessing to improve the quality of the data, including one or more of data desensitization processing, missing value processing, low-quality filtering processing, redundant removal processing, and outlier monitoring processing.

[0012] Preferably, the data desensitization processing includes:

[0013] Noise addition or hash desensitization strategy is used to protect sensitive data that needs to be processed to protect data sensitive information; wherein the noise includes one or more of Gaussian noise and Laplace noise.

[0014] Preferably, the outlier monitoring process includes using a K-nearest neighbor algorithm to perform data anomaly detection, and the using a K-nearest neighbor algorithm to perform data anomaly detection includes the following steps:

[0015] Choose Euclidean distance as the metric:

[0016] Where and represent two data sample points, represents the th eigenvalue, represents the number of data point features, represents the value of the th sample on the th feature, represents the value of the th sample on the th feature, and represents the distance between the two sample points and;

[0017] For each data point, find the nearest neighbors, where represents a neighbor of the data point, represents one of the neighbors of, represents any data point, and is used to compare the distance to the neighbor to determine the neighbor set, represents the neighbor set of the data point:

[0018] The local density is measured by using the average distance of the nearest neighbors. The local density, that is, the average distance of the calculated neighbors, reflects the data distribution in the area around the data point:

[0019] Calculate the anomaly score for each data point to measure how far the data point deviates from the normal pattern. The larger the value, the more likely the data point is to be an outlier:

[0020] A threshold is set based on the anomaly score, and data points with scores above the threshold are marked as anomalies.

[0021] Preferably, the cleaning of data by data preprocessing to improve the quality of data includes judging the quality of content, and the judging of the quality of content includes the following steps:

[0022] Find the optimal hyperplane to classify sample quality in the data distribution feature space.

[0023] The hyperplane formula is: , where is the normal vector, is the point of the hyperplane, and is a constant term.

[0024] The distance from the point to the plane is, the classification interval is, minimize the objective function, and at the same time satisfy the constraints, where is the feature vector of the data, is the label of the data, and the constraints are used to ensure that the data quality is correctly judged.

[0025] Preferably, the adding of labels to the preprocessed data so that key features of the data are better understood and learned comprises using a tool to add labels to the preprocessed data, wherein the tool comprises one or more of LabelBox, BRAT, WebAnno and Audacity.

[0026] Preferably, the preprocessed data is labeled so that the key features of the data can be better understood and learned, including one or more NLP text data annotation methods such as machine translation, named entity recognition, question-answer matching, association extraction, classification, text classification, and document summarization.

[0027] Preferably, the preprocessed data is labeled so that key features of the data can be better understood and learned, and the labeled objects are one or more of text, speech and image.

[0028] Preferably, it also includes data security services throughout the entire process of data set construction.

[0029] The present invention also provides a system for constructing a high-quality data set for a large model in the field of urban rail transit, which is used to execute the steps included in the method for constructing a high-quality data set for a large model in the field of urban rail transit as described in any one of the above items, including:

[0030] The data collection module is used to collect and summarize data that meets business requirements as an input data source;

[0031] The preprocessing module is used to clean the data through data preprocessing to improve the data quality;

[0032] The annotation module is used to add labels to the preprocessed data so that the key features of the data can be better understood and learned;

[0033] The maintenance module is used to perform quality review and dynamic management of annotation data through data maintenance.

[0034] The beneficial effect of the method and system for constructing a high-quality data set for a large model in the field of urban rail provided by the present invention is that: compared with the prior art, the method and system for constructing a high-quality data set for a large model in the field of urban rail provided by the present invention covers multiple aspects such as data collection, preprocessing, labeling, maintenance and security. It improves the systematization and standardization of the entire data processing process, which is conducive to improving data quality and the effect of model training. The two core service modules of data preprocessing and data labeling clarify the cleaning strategy and labeling process of urban rail data. Ensure that the data is accurately labeled, and enhance the learning ability and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 An architectural diagram of a method for constructing a high-quality data set for a large model in the urban rail field provided by an embodiment of the present invention;

[0037] Figure 2 A data annotation process chain diagram in a method for constructing a high-quality data set for a large model in the urban rail field provided by an embodiment of the present invention;

[0038] Figure 3 A structural block diagram of a system for constructing high-quality data sets for large models in the urban rail field provided by an embodiment of the present invention;

[0039] Figure 4 A structural block diagram of another system for constructing high-quality data sets for large models in the urban rail field provided by an embodiment of the present invention.

[0040] In the figure: 1. Data acquisition module; 2. Preprocessing module; 3. Labeling module; 4. Maintenance module; 5. Security service module. DETAILED DESCRIPTION

[0041] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] Please also read Figures 1 to 4Now, a method for constructing a high-quality data set for a large model in the field of urban rail provided by the present invention is described. The method for constructing a high-quality data set for a large model in the field of urban rail includes the following steps:

[0043] S1. Collect and summarize data that meets business requirements as input data sources;

[0044] In this step, the data is specifically data in the field of urban planning. Different collection methods are required for data from different sources in the field of urban rail transit. The data collection service collects and summarizes the data that meets the business needs as the input data source of the data preprocessing service.

[0045] The sources of data include big data platform data, data from various professional business systems, industry public data, and data transmitted by sensors. Different collection methods are used for urban rail data from different sources. Among them, the collection methods include one or more of API interface access, SDK method, data download, and data subscription service. Preferably, the collection method is API interface access, SDK method, data download, and data subscription service.

[0046] The API interface supports multiple API standards such as RESTful API and SOAP API. It can communicate using protocols such as HTTP and HTTPS, and obtain data in formats such as JSON and XML. It provides SDK to encapsulate API call logic, simplify the data acquisition process, and interact with the data acquisition platform more conveniently to achieve efficient data extraction. Under the premise of permission, users can directly download and export. It supports providing in TXT, CSV, JSON, Excel, XML and other file formats. The export function can be set to periodic (such as daily, weekly), manual trigger or regular generation, etc. It supports regular synchronization of data to the user's local storage or cloud storage service through special tools or services; the data subscription method supports users to set subscription rules, according to the preset update frequency or event trigger mechanism, through WebSocket, MQTT, Kafka and other message queue technologies to push data in real time, and supports configuration of data subscription topics and filtering conditions.

[0047] S2. Clean the data through data preprocessing to improve the data quality;

[0048] Data preprocessing strictly screens, amends and improves the original data provided by the data acquisition service. It is a key link in forming high-quality data available for the urban rail large model.

[0049] The implementation process of this step includes:

[0050] S2.1. Desensitize the data to protect sensitive information. The cleaning methods include one or more of data desensitization, missing value processing, low-quality filtering, redundancy removal, and outlier monitoring.

[0051] Data desensitization is the process of deforming sensitive data through desensitization rules without reducing data security, effectively reducing the exposure of sensitive data during collection, transmission, and use.

[0052] According to the differences in data sensitivity, usage scenario requirements and security levels in different urban rail scenarios, noise addition and / or hash desensitization strategies are used to protect sensitive data information for processing.

[0053] Specifically, Laplace noise is used to mask the real urban rail data for data desensitization.

[0054] Among them, is the original data value, is the data after adding noise, is the sensitivity of the dataset, that is, the maximum impact of individual data changes on the function output, and represents the differential privacy parameter.

[0055] Gaussian noise is used to retain the statistical characteristics of urban rail data for data desensitization;

[0056] Among them, is the original data, is the data after adding Gaussian noise, and is a Gaussian distribution with a mean of 0 and a variance of .

[0057] The privacy of urban rail data is protected by modifying and replacing specific fields, and hash desensitization is used:

[0058] Among them, is the original data, is the hash value after hashing, is the hash function also called hash function, such as SHA-256, and is the established rule for data mapping.

[0059] S2.2. Handling missing values

[0060] Identify records or fields with missing values ​​in the data set, including null values, special values, inconsistent formats, and outliers, and select appropriate processing strategies based on the characteristics of urban rail data, such as deletion, interpolation based on related variables, and other operations.

[0061] S2.2. Detect outliers in the data. Specifically, use the K nearest neighbor algorithm to detect data anomalies. More specifically, select the Euclidean distance as the metric:

[0062] Among them, and represent two data sample points, represents the th eigenvalue, represents the number of data point features, represents the value of the th sample on the th feature, represents the value of the th sample on the th feature, and represents the distance between the two sample points and;

[0063] For each data point, find the nearest neighbors, where represents a neighbor of the data point, represents one of the neighbors of, represents any data point, and is used to compare the distance to the neighbor to determine the neighbor set, represents the neighbor set of the data point:

[0064] The local density is measured by using the average distance of the nearest neighbors. The local density, that is, the average distance of the calculated neighbors, reflects the data distribution in the area around the data point:

[0065] Calculate the anomaly score for each data point to measure how far the data point deviates from the normal pattern. The larger the value, the more likely the data point is to be an outlier:

[0066] Set a threshold based on the anomaly score and mark data points with scores above the threshold as anomalies

[0067] The content quality is judged by the support vector machine method.

[0068] In view of the uneven quality of the collected data, it is necessary to judge the quality of the content, and the quality judgment is made through the support vector machine method.

[0069] Find the optimal hyperplane to classify sample quality in the data distribution feature space.

[0070] The hyperplane formula is: , where is the normal vector, is the point of the hyperplane, and is a constant term.

[0071] The distance from the point to the plane is, the classification interval is, minimize the objective function, and at the same time satisfy the constraints, where is the feature vector of the data, is the label of the data, and the constraints are used to ensure that the data quality is correctly judged.

[0072] Identify outliers in data sets based on business knowledge scenarios, use statistical methods (such as box plots, clustering, etc.) or set thresholds based on business knowledge to identify outliers in data sets, and set deletion, replacement or smoothing technology to correct outliers to ensure the stability of the data set.

[0073] The data preprocessing service completes data cleaning through multiple technical processes, further ensuring the high quality of the data, and at the same time enters the data labeling stage as a data source.

[0074] S3. Label the preprocessed data so that the key features of the data can be better understood and learned; the methods used include one or more NLP text data annotation methods such as machine translation, named entity recognition, question-answer matching, association extraction, classification, text classification, and document summarization.

[0075] The data collected are in various formats.

[0076] (1) Based on the file type, annotation objects are divided into text, voice and image annotation.

[0077] Text annotation uses NLP text data annotation methods such as machine translation, named entity recognition, question-answer matching, association extraction, taxonomy, text classification, and document summarization;

[0078] Speech annotation uses segmented speech automatic recognition, automatic speech recognition, intent classification, signal quality detection, sound event detection, semantic segmentation, speech transcription and other annotation methods;

[0079] Image annotation is based on the requirements of the annotation task, and appropriate annotation shapes (such as linear, rectangular, polygonal, etc.) are selected to add labels to the selected area.

[0080] (2) Based on the annotation form, the annotation can be divided into structured, semi-structured and unstructured annotation.

[0081] Structured annotation uses matching annotation objects with structured tags, and the corresponding rules are reasonable and clear;

[0082] For unstructured data that lacks a fixed format, we formulate binding annotation standards and guidelines to distinguish data annotation or match labels.

[0083] Semi-structured annotation targets data with certain structural characteristics. It associates structured tag values ​​with semi-structured data and annotates them using methods such as document structure recognition and content classification.

[0084] (3) Based on the annotation method, annotation methods can be divided into two categories: manual annotation and machine annotation.

[0085] For routine labeling tasks that require common sense understanding in the field of urban rail transit, labelers need to use labeling tools to label and select data, which is easy to use, simple to get started, and highly repeatable. However, complex labeling in the field of urban rail transit requires labelers to have rich professional capabilities. Manual labeling is difficult and the task is heavy. Automated labeling is performed through labeling tools. Different labeling tools are selected according to the labeling object and labeling requirements. The data labeling tools are shown in the table below, which improves labeling efficiency while reducing costs.

[0086]

[0087] The data annotation service reprocesses the data and converts it into information that can be recognized by the big model for the training of the big model.

[0088] Data labeling is usually implemented through pre-defined rules or machine learning methods, which is similar to the concept of data quality control and is based on predefined rules or machine learning methods.

[0089] Methods based on pre-set rules include hidden Markov models, conditional random fields, etc., and machine learning includes deep learning, naive Bayes and support vector machines, etc.

[0090] Different methods are selected and applied according to different business needs. There is no need to re-derive the formulas when applying these methods, which depends on the specific task requirements and available resources.

[0091] The tasks and distinctions of annotation are described in detail in the article.

[0092] S4. Conduct quality review and dynamic management of labeled data through data maintenance.

[0093] At this step, the quality audit implementation process includes:

[0094] The administrator decomposes the annotation tasks; the annotator submits the standard results to the auditor for review; the task administrator decomposes the tasks; the auditor can view the annotator's annotation content through the assigned review tasks, and supports operations such as marking the annotation content as wrong, rejecting and re-annotating. Ensure the rigor and accuracy of the model training data baseline, and be responsible for the final model training effect from the source.

[0095] The implementation process of dynamic management includes: creating a new data version for each data update or major change, recording the change log of each version (such as the amount of new data, data source, label modification, etc.), so as to track the change history of the data set. In order to manage and maintain the latest status of the data set, set a fixed update cycle according to business needs, determine the scope of incremental update or full update, and set monitoring indicators to track data quality and update progress.

[0096] S5. Data security services throughout the entire process of data set construction

[0097] Follow the architectural design of the internal production network, management network and external service network in the urban rail field, follow the principle of security isolation, and perform user access control permissions and log monitoring.

[0098] Allocate data access rights according to the principle of least privilege to ensure that only authorized personnel can access specific data, protect corporate rights and interests and corporate data privacy. Combine log monitoring indicators to obtain data operation logs to avoid data leakage.

[0099] The method and system provided by the present invention for constructing a high-quality data set for a large model in the field of urban rail transit cover multiple aspects such as data collection, preprocessing, labeling, maintenance and security compared with the prior art. The systematization and standardization of the entire data processing process are improved, which is conducive to improving data quality and the effect of model training. The two core service modules of data preprocessing and data labeling clarify the cleaning strategy and labeling process of urban rail transit data. Ensure that the data is accurately labeled, and enhance the learning and generalization capabilities of the model. By defining a clear data management mechanism and task decomposition process in the field of urban rail transit, the standardization and efficiency of data processing are ensured. Improve the overall efficiency of data processing and ensure the accuracy and timeliness of the data set.

[0100] The present invention also provides a system for constructing a high-quality data set for a large model in the field of urban rail transit, characterized in that the system is used to execute the steps included in the method for constructing a high-quality data set for a large model in the field of urban rail transit as described in any of the above items, including: a data acquisition module 1, a preprocessing module 2, a labeling module 3, and a maintenance module 4. The data acquisition module 1 is used to collect and summarize data that meets business requirements as an input data source; the preprocessing module 2 is used to clean the data through data preprocessing to improve the quality of the data; the labeling module 3 is used to add labels to the preprocessed data so that the key features of the data can be better understood and learned; the maintenance module 4 is used to perform quality review and dynamic management of the labeled data through data maintenance.

[0101] In some feasible embodiments, the system for constructing high-quality data sets for large models in the urban rail field also includes a security service module 5, which is used to run through the entire process of data set construction and perform safety monitoring on each link respectively.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a high-quality data set for a large model in the field of urban rail transit, characterized in that: The following steps are involved: Collect and summarize data that meets business needs as input data sources; Clean the data through data preprocessing to improve the data quality; Add labels to the preprocessed data so that the key features of the data can be better understood and learned; Conduct quality review and dynamic management of labeled data through data maintenance.

2. A method for constructing a high-quality data set for a large model in the field of urban rail transit as claimed in claim 1, characterized in that: The data preprocessing is used to clean the data and improve the data quality, including one or more of data desensitization, missing value processing, low-quality filtering, redundancy removal, and outlier monitoring.

3. A method for constructing a high-quality data set for a large model in the field of urban rail transit as claimed in claim 2, characterized in that: The data desensitization processing includes: Noise addition or hash desensitization strategy is used to protect sensitive data that needs to be processed to protect data sensitive information; wherein the noise includes one or more of Gaussian noise and Laplace noise.

4. The method for constructing a high-quality data set for a large model in the field of urban rail transit as claimed in claim 2, characterized in that: The outlier monitoring process includes using a K-nearest neighbor algorithm to perform data anomaly detection, and the use of a K-nearest neighbor algorithm to perform data anomaly detection includes the following steps: Choose Euclidean distance as the metric: Among them, and represent two data sample points, represents the th eigenvalue, represents the number of data point features, represents the value of the th sample on the th feature, represents the value of the th sample on the th feature, and represents the distance between the two sample points and; For each data point, find the nearest neighbors, where represents a neighbor of the data point, represents one of the neighbors of, represents any data point, and is used to compare the distance to the neighbor to determine the neighbor set, represents the neighbor set of the data point: The local density is measured by using the average distance of the nearest neighbors. The local density, that is, the average distance of the calculated neighbors, reflects the data distribution in the area around the data point: Calculate the anomaly score for each data point to measure how far the data point deviates from the normal pattern. The larger the value, the more likely the data point is to be an outlier: A threshold is set based on the anomaly score, and data points with scores above the threshold are marked as anomalies.

5. A method for constructing a high-quality data set for a large model in the field of urban rail transit as described in any one of claims 1 to 4, characterized in that: The cleaning of data by data preprocessing to improve the quality of data includes judging the content quality, and the judging of the content quality includes the following steps: Find the optimal hyperplane classification sample quality in the data distribution feature space; The hyperplane formula is: , where is the normal vector, is the point of the hyperplane, and is a constant term; The distance from the point to the plane is, the classification interval is, minimize the objective function, and at the same time satisfy the constraints, where is the feature vector of the data, is the label of the data, and the constraints are used to ensure that the data quality is correctly judged.

6. The method for constructing a high-quality data set for a large model in the field of urban rail transit as claimed in claim 1, characterized in that: The adding of labels to the preprocessed data so that key features of the data can be better understood and learned includes using a tool to add labels to the preprocessed data, and the tool includes one or more of LabelBox, BRAT, WebAnno, and Audacity.

7. The method for constructing a high-quality data set for a large model in the field of urban rail transit as claimed in claim 1, characterized in that: The step of adding labels to the preprocessed data so that the key features of the data can be better understood and learned includes adopting one or more NLP text data annotation methods such as machine translation, named entity recognition, question-answer matching, association extraction, taxonomy, text classification, and document summarization.

8. The method for constructing a high-quality data set for a large model in the field of urban rail transit as claimed in claim 1, characterized in that: The preprocessed data is labeled so that key features of the data can be better understood and learned, and the labeled objects are one or more of text, speech and image.

9. A method for constructing a high-quality data set for a large model in the field of urban rail transit according to any one of claims 1 to 8, characterized in that: It also includes data security services throughout the entire process of data set construction.

10. A system for building high-quality data sets for large models in the field of urban rail transit, characterized in that: The steps for executing the method for constructing a high-quality data set for a large model in the urban rail field as described in any one of claims 1 to 9 include: The data collection module (1) is used to collect and summarize data that meets business requirements as an input data source; A preprocessing module (2) is used to clean the data through data preprocessing to improve the quality of the data; The labeling module (3) is used to add labels to the preprocessed data so that the key features of the data can be better understood and learned; The maintenance module (4) is used to perform quality review and dynamic management of the labeled data through data maintenance.

Images