Subway operation and maintenance data classification method and system, electronic equipment and storage medium

Abstract

The invention provides a subway operation and maintenance data classification method and system, electronic equipment and a storage medium, and the method comprises the steps: collecting data packets of all subway lines in real time, and converting the data packets into feature vectors through a vector algorithm; cleaning the feature vectors through a cyclic utilization pool and vector retrieval; for the cleaned feature vectors, calculating a matching sequence of a pre-constructed feature set by adopting a cluster index algorithm and a distance weighted average value, matching the feature vectors with the feature set according to the matching sequence by utilizing a vector retrieval algorithm, and determining a data category to which each feature vector belongs; namely, the feature vectors which fail to classify can be utilized again to clean the original data by using a content similarity analysis technology based on vector retrieval; and through clustering indexing and distance sorting, data are classified into corresponding categories by using a vector retrieval algorithm, so that multivariate data classification of the subway line is realized, and the requirements of a subway line network operation and maintenance system on data processing are met.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and more specifically, particularly relates to a subway operation and maintenance data classification method and system, electronic equipment, and storage medium. Background Art

[0002] With the booming development of urban rail transit, the importance of subway network operation and maintenance systems has become increasingly prominent, and it has become an indispensable part of ensuring the safe and efficient operation of subways. However, in the actual application of intelligent operation and maintenance systems, especially when processing subway network operation and maintenance data, unprecedented difficulties have been encountered.

[0003] Given that the equipment and data categories involved in subway network operation and maintenance are numerous and complex, including hexadecimal byte data, JSON data, XML data, Modbus protocol data, video streams, pictures, Chinese text, logs and other data categories, an efficient data processing solution is needed. The solution should be able to cope with the processing, analysis and application of these diversified data to ensure the accuracy and effectiveness of the data. First, the diversity of subway network operation and maintenance data poses a great challenge to data processing. Although there are many data categories and different structures, the existing subway network usually only outputs data through a unified data gateway, resulting in the consistency of all data sources, making the identification of data sources extremely difficult. This not only increases the difficulty of data cleaning and classification, but also brings great inconvenience to subsequent data processing. Secondly, there may be differences in data format, encoding and expression between different subway lines. Even for the same type of data, the expression form on different lines may be very different. This inconsistency in data format further exacerbates the difficulty of data processing, making it difficult to effectively integrate and analyze data.

[0004] In addition, most of the existing related patents or papers focus on the knowledge base and question-answering areas, and do not involve data processing issues in the subway. These solutions cannot be directly applied to subway business, nor can they achieve effective data cleaning in complex multi-type data. At the same time, existing patents also lack effective methods for classifying multi-type data, and cannot meet the urgent needs of subway line network operation and maintenance systems for data processing. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a subway operation and maintenance data classification method and system, electronic equipment, and storage medium, which are used to realize multi-dimensional data classification of subway lines and meet the data processing requirements of the subway line network operation and maintenance system, which provides strong data support for subsequent data analysis, fault prediction, and maintenance decisions.

[0006] The first aspect of the present application discloses a method for classifying subway operation and maintenance data, comprising:

[0007] Collecting data packets of each subway line in real time, converting the data packets into feature vectors through a vector algorithm, and storing the feature vectors in a first message queue; the data packets include original subway operation and maintenance data;

[0008] Cleaning the feature vectors in the first message queue by recycling the pool and vector retrieval, and putting the cleaned feature vectors into the second message queue; wherein the recycling pool is constructed and updated by using the feature vectors that failed classification;

[0009] For the feature vectors in the second message queue, a clustering index algorithm and a distance weighted average are used to calculate the matching order of the pre-constructed feature set, and a vector retrieval algorithm is used to match the feature vectors with the feature set according to the matching order to determine the data category to which each feature vector in the second message queue belongs.

[0010] Optionally, the real-time collection of data packets of each subway line, converting the data packets into feature vectors by using a vector algorithm, and storing the feature vectors in a first message queue includes:

[0011] Collect data packets from each subway line in real time through data gateways;

[0012] Preliminarily marking the data packet by parsing at least one of a data header, a label, a protocol type, a source IP address, and a source port number in the data packet;

[0013] The data packet is converted into a vector representation in a high-dimensional space by using the vector algorithm to obtain a feature vector;

[0014] The feature vector is stored in the first message queue.

[0015] Optionally, the cleaning of the feature vectors in the first message queue by recycling the pool and vector retrieval, and placing the cleaned feature vectors into the second message queue, includes:

[0016] Determine the hit number of the feature vector currently processed in the first message queue in the recycling pool; wherein the hit number is the number of similarities between the feature vector currently processed and the feature vector in the recycling pool that is greater than a threshold;

[0017] If the hit number corresponding to the currently processed feature vector is greater than a threshold, it is determined that the currently processed feature vector needs to be cleaned;

[0018] If the hit number corresponding to the currently processed feature vector is less than a threshold, it is determined that the currently processed feature vector does not need to be cleaned;

[0019] The feature vectors that do not need to be cleaned are placed in the second message queue.

[0020] Optionally, the feature vectors in the second message queue are used to calculate the matching order of the pre-constructed feature set using a clustering index algorithm and a distance weighted average, and the feature vectors are matched with the feature set according to the matching order using a vector retrieval algorithm to determine the data category to which each feature vector in the second message queue belongs, including:

[0021] For any feature vector, the similarity between the currently processed feature vector and each feature set is determined through the clustering index algorithm and the distance weighted average;

[0022] According to the similarity between the feature vector currently being processed and each feature set, the matching priority between the feature vector currently being processed and each feature set is determined; wherein, the higher the similarity, the higher the matching priority;

[0023] According to the matching priority of each feature set, the feature vector currently being processed is matched using a vector retrieval algorithm, and the feature vector currently being processed is classified into the category of the successfully matched feature set; among them, if a feature set with a higher priority successfully matches the feature vector currently being processed, a feature set with a lower priority will no longer match the feature vector currently being processed.

[0024] Optionally, before collecting the data packets of each subway line in real time, converting the data packets into feature vectors by using a vector algorithm, and storing the feature vectors in the first message queue, the method further includes:

[0025] Obtain the historical subway operation and maintenance raw data that needs to be classified;

[0026] Annotating the historical subway operation and maintenance raw data;

[0027] Through vector algorithms, feature vectors are extracted from the annotated historical subway operation and maintenance raw data to obtain feature sets for each data category;

[0028] The feature sets of each data category are bound to different service nodes to obtain multiple independent data processing units.

[0029] Optionally, after binding the feature sets of each data category with different service nodes, the method further includes:

[0030] Dynamically adjust the feature set matched by the data processing unit.

[0031] Optionally, also include:

[0032] If any of the feature vectors in the second message queue fails to be classified when matched with each feature set, the feature vector that fails to be classified will be added to the learning sample of the recycling pool, and the recycling pool will be filled and optimized through the autonomous learning algorithm, and the data set will be cleaned or updated regularly.

[0033] The second aspect of the present application discloses a subway operation and maintenance data classification system, including:

[0034] A feature vector module is used to collect data packets of each subway line in real time, convert the data packets into feature vectors through a vector algorithm, and store the feature vectors in a first message queue; the data packets include original subway operation and maintenance data;

[0035] A cleaning module, used for cleaning the feature vectors in the first message queue by recycling the pool and vector retrieval, and putting the cleaned feature vectors into the second message queue; wherein the recycling pool is constructed and updated by using the feature vectors that failed classification;

[0036] The classification module is used to calculate the matching order of the pre-constructed feature set for the feature vectors in the second message queue using a clustering index algorithm and a distance weighted average, and to match the feature vectors with the feature set according to the matching order using a vector retrieval algorithm to determine the data category to which each feature vector in the second message queue belongs.

[0037] The third aspect of the present application discloses an electronic device, comprising:

[0038] one or more processors;

[0039] a storage device having one or more programs stored thereon;

[0040] When one or more programs are executed by one or more processors, the one or more processors implement the subway operation and maintenance data classification method as described in any one of the first aspects of the present application.

[0041] A fourth aspect of the present application discloses a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the subway operation and maintenance data classification method as described in any one of the first aspects of the present application is implemented.

[0042] It can be seen from the above technical scheme that a subway operation and maintenance data classification method provided by the present invention includes: collecting data packets of each subway line in real time, converting the data packets into feature vectors through a vector algorithm, and storing the feature vectors in a first message queue; the data packets include original subway operation and maintenance data; cleaning the feature vectors in the first message queue through a recycling pool and vector retrieval, and putting the cleaned feature vectors into a second message queue; wherein the recycling pool is constructed and updated using feature vectors that failed classification; for the feature vectors in the second message queue, a clustering index algorithm and a distance weighted average are used to perform a pre-constructed feature vector search. Collect and calculate the matching order, and use the vector retrieval algorithm to match the feature vector with the feature set according to the matching order, and determine the data category to which each feature vector in the second message queue belongs; that is, based on the content similarity analysis technology of vector retrieval, the feature vector that failed to be classified can be reused to clean the original data and remove redundant and erroneous information; and through clustering index and distance sorting, the data is classified into corresponding categories using the vector retrieval algorithm, that is, the multi-dimensional data of the subway line is classified, which meets the data processing needs of the subway line network operation and maintenance system, which provides strong data support for subsequent data analysis, fault prediction and maintenance decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 is a flow chart of a method for classifying subway operation and maintenance data provided by an embodiment of the present invention;

[0045] Figure 2 is a flow chart of another method for classifying subway operation and maintenance data provided by an embodiment of the present invention;

[0046] Figure 3 is a flow chart of another method for classifying subway operation and maintenance data provided by an embodiment of the present invention;

[0047] Figure 4 is a flow chart of another method for classifying subway operation and maintenance data provided by an embodiment of the present invention;

[0048] Figure 5 is a flow chart of another method for classifying subway operation and maintenance data provided by an embodiment of the present invention;

[0049] Figure 6is a flow chart of another method for classifying subway operation and maintenance data provided by an embodiment of the present invention;

[0050] Figure 7 is a flow chart of another method for classifying subway operation and maintenance data provided by an embodiment of the present invention;

[0051] Figure 8 is a schematic diagram of a subway operation and maintenance data classification system provided by an embodiment of the present invention;

[0052] Fig. 9 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In the present application, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "comprise one ..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. In addition, the terms "first", "second", "third", "fourth", etc. (if present) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here, for example.

[0055] The embodiment of the present application discloses a method for classifying subway operation and maintenance data, which is used for data processing in the subway that is not involved in the existing technology. These solutions cannot be directly applied to subway business, nor can they achieve effective data cleaning in complex multivariate data. At the same time, the existing patents also lack effective methods for classifying multivariate data, and cannot meet the urgent needs of the subway network operation and maintenance system for data processing. The subway operation and maintenance data classification method mainly collects the original data to be classified from the subway network, performs manual initial labeling and classification training, extracts the feature vectors of the data to obtain the feature set of the data category, and then matches and retrieves the feature sets of each category of data category in combination with the similarity analysis method for the real-time data of the subway, and cleans the unnecessary data in advance through the self-learning feature vector circulation pool of failed classification, which can achieve rapid cleaning, accurate identification and classification of subway operation and maintenance data.

[0056] See also Figure 1 , the subway operation and maintenance data classification method includes:

[0057] S101. Construct feature sets for each data category, and bind each feature set to a different service node to obtain multiple independent data processing units.

[0058] Specifically, historical subway operation and maintenance raw data that need to be classified are obtained; the historical subway operation and maintenance raw data are labeled; feature vectors are extracted from the labeled historical subway operation and maintenance raw data through a vector algorithm to obtain feature sets of each data category; the feature sets of each data category are bound to different service nodes to obtain multiple independent data processing units.

[0059] More specifically, historical subway operation and maintenance raw data that needs to be classified is collected. The raw data can be log data of vehicle dispatch, binary data of driving control, interlocking log files, etc., which will not be listed here one by one. It depends on the actual situation and is within the protection scope of this application.

[0060] The collected raw data of different data categories are manually annotated to clarify the initial characteristics of each raw data, such as start and end symbols, log naming method, protocol type (TCP, UDP, HTPP, etc.) and data receiving and sending ports.

[0061] Then, the feature vectors of the labeled raw data are extracted through vector algorithms to obtain feature sets of different data categories. The feature sets are distributed to the service nodes for data calculation, which can independently and parallelly process a large amount of real-time data in the next stage.

[0062] Collect the raw data to be classified, including vehicle dispatch logs, driving control binary information, and interlocking logs. For these diverse raw data, perform manual initial annotation to clarify their initial features, such as start and end marks, log naming rules, protocol categories (such as TCP, UDP, HTTP, etc.) and data sending and receiving ports. Subsequently, use vector algorithms to extract feature vectors of the raw data to form feature sets of different data categories.

[0063] Optionally, the specific process of step S101, constructing feature sets of various data categories, binding each feature set to a different service node, and obtaining multiple independent data processing units may include:

[0064] In the initialization phase, a feature set of data categories for classification is constructed. The starting point of this process is to comprehensively collect the information of each category to be classified and its corresponding raw data from major subway lines. To ensure the comprehensiveness and accuracy of the data, all relevant categories are covered in detail.

[0065] Next, an automated labeling process was implemented for the raw data under each category. This step is crucial because it involves identifying and labeling the key features common to each type of data. These features include but are not limited to source IP, source port, ground line number, log type, start and end characters of binary data, Chinese tags, timestamps, protocol types, etc. The accurate labeling of these features provides a solid foundation for subsequent data processing and classification.

[0066] Based on the labeled data, a vector training model was constructed and pre-trained in depth. The core of this model is to use vector algorithm embedding technology to convert the original data of each category into feature vectors. These feature vectors not only retain the core information of the original data, but also form a feature set that can represent each data category by combining feature vectors of the same category.

[0067] This feature set is the knowledge base for subsequent vector retrieval and classification tasks. It contains all relevant features of the category, ensuring the accuracy and efficiency of classification. Through the data pre-training process, high-quality feature sets of different data categories are successfully obtained on the basis of relatively accurate data. These feature sets not only provide strong support for current classification tasks, but also lay a solid foundation for future data processing and knowledge mining.

[0068] Finally, the feature sets of each category are assigned to different service nodes, thereby constructing multiple independent data processing units. Here, there is a one-to-many mapping relationship between the feature sets of each category and the data processing units, that is, the feature vector set of each category can be accessed and called by multiple data processing units, but each data processing unit is responsible for processing a feature vector set of a specific category. In order to ensure the load balancing and fault tolerance of the system, and at the same time improve the parallelism and overall efficiency of data processing, the number of data processing units must be no less than the number of category feature vector sets. Such a design not only ensures that the data of each category can be processed in a timely and effective manner, but also enhances the stability and reliability of the system through redundant configuration.

[0069] Specifically, Figure 2 As shown in the figure, the original data of dispatching management, vehicle management, alarm management and DCS (distributed control system) in the subway system are collected respectively, and these data are classified according to their categories. Subsequently, the Embedding technology is used to convert these data into feature sets under their respective categories. In this process, multiple service nodes are set up, each of which is responsible for processing a specific category of data and calling the feature set of the corresponding classification. In this way, multiple independent data processing units are formed.

[0070] In order to improve the efficiency of data processing, a feature vector set is allowed to be called by multiple service nodes at the same time. This design not only makes full use of computing resources, but also significantly improves the parallelism and overall efficiency of data processing. Through such an architecture, various types of data in the subway system can be processed and analyzed more efficiently, providing strong support for the safe operation and efficient management of the subway.

[0071] S102. Collect data packets of each subway line through a data gateway, convert the data packets into feature vectors through a vector algorithm, and store the feature vectors in a first message queue.

[0072] The data packet includes original subway operation and maintenance data.

[0073] Data acquisition is achieved through a data gateway, which is responsible for collecting data packets from various subway lines.

[0074] Once the data packets are collected, the vector algorithm embedding technology is used to convert the raw data in the data packets into feature vectors. These feature vectors not only concisely represent the key information of the raw data, but also facilitate subsequent data analysis and processing.

[0075] After the vectorization process is completed, these feature vectors will be stored in the first message queue. This queue plays the role of data buffering and transmission, ensuring the orderly flow and efficient processing of data.

[0076] Through such a process design, a complete chain from data reception to vector processing and then to feature vector storage is realized. This not only improves the speed and accuracy of data processing, but also provides a solid data foundation for real-time monitoring and efficient management of the subway system.

[0077] More specifically, at this stage, the real-time data of each subway line is received through the data gateway, and the real-time output information of all lines is obtained. The data gateway is centrally managed and preprocessed, and the feature vector of the real-time data is output after preprocessing for use in the next stage. These data come from multiple subway lines and different subway systems, such as vehicle dispatching, driving control, interlocking management, network exchange monitoring, LTE network operation and maintenance, regional controller, power distribution supervision, passenger information service, etc., and their forms are various, including but not limited to video files, images, hexadecimal data streams, Modbus communication protocol data, protobuf serialization information, log documents, HTTP data packets, TCP / UDP communication data, and XML and JSON structured data. After the data enters the gateway, each piece of data is initially automatically annotated, and the source of the data is identified through information such as data header, label, protocol type, source IP, source port, etc. For data whose source or classification can be determined, preliminary annotation is performed. The data annotation can more accurately identify and understand the key features in the data, and it is easier to identify useful information in the data, thereby speeding up the feature extraction. Then, through the embedding method of vector algorithm, feature extraction is performed for each piece of data, which is converted into a vector representation in a high-dimensional space, and a feature vector of the data is generated, and the generated data feature vector is sent to the message queue. Using feature vectors, data from different sources and types can be converted into a unified numerical representation through feature extraction, so that different types of data originally scattered in different systems can be processed and analyzed under a unified framework, which brings data integration and standardization integration for subsequent unified processing, and provides more comprehensive and accurate support for subsequent decision-making.

[0078] The specific process of step S102 may include:

[0079] The data packets of each subway line are collected in real time through the data gateway; the data packets are preliminarily labeled by parsing at least one of the data header, label, protocol type, source IP and source port in the data packets; the data packets are converted into vector representations in high-dimensional space using the vector algorithm to obtain feature vectors; and the feature vectors are stored in the first message queue.

[0080] Specifically, at this stage, the real-time data of each line is first received through the data gateway. This data covers the real-time output information of all lines and is a comprehensive reflection of the subway operation status. As the core node of data processing, the data gateway undertakes the task of centralized management and preprocessing. Through a series of complex algorithms and logics, the data gateway performs preliminary processing on these raw data and finally outputs the feature vector of real-time data. These feature information are properly stored for access and analysis in the subsequent stage.

[0081] The sources of these real-time data are extremely wide, not only involving multiple subway lines, but also deeply involved in various key links of subway operations. Whether it is vehicle dispatching, driving control, interlocking management, network exchange monitoring, or even LTE network operation and maintenance, regional controllers, power distribution supervision, and passenger information services, systems are constantly generating data. The forms of these data are also diverse, from video files, images to hexadecimal data streams, Modbus communication protocol data, to protobuf serialization information, log documents, as well as HTTP data packets, TCP / UDP communication data, and even structured data such as XML and JSON, everything is available, forming a huge and complex ocean of data.

[0082] When these data enter the data gateway, the system will automatically label them. This step is crucial because these labels can help the system quickly identify the source and type of data. By parsing information such as data headers, tags, protocol types, source IPs, and source ports, the system can accurately classify and label most data. Such labeling not only improves the efficiency of data processing, but also enables the system to more accurately identify and understand key features in the data, making it easier to extract useful information.

[0083] After completing the preliminary annotation, the system will use the advanced technology of vector algorithm embedding to extract features from these data. The core of this step is to convert each piece of data into a vector representation in a high-dimensional space, that is, to generate feature vectors of the data. These feature vectors not only retain the main information of the original data, but also enable data from different sources and types to be compared and analyzed in the same framework.

[0084] Finally, these generated data feature vectors will be sent to the message queue, waiting for subsequent processing and analysis. By using feature vectors, the system can integrate different types of data originally scattered in different systems into a unified framework for processing and analysis. This process of data integration and standardization not only improves the efficiency of data processing, but also provides more comprehensive and accurate support for subsequent decision-making. This data processing method undoubtedly provides a strong technical guarantee for the intelligent operation and management of the subway system.

[0085] For example, Figure 3 As shown in the figure, the data of Metro Line 1 and Metro Line 2 use an output gateway. The data gateway receives all categories of data from different subway lines, preliminarily marks the identifiable data, converts it into data feature vectors through the vector algorithm Embedding, and sends it to the message queue. Metro Line 3 and Metro Line 4 use independent output gateways respectively. These output gateways all output data to a unified data input gateway (data gateway) for data aggregation. The data gateway processes all the aggregated data, that is, preliminarily classifies and labels it, obtains feature vectors through vector algorithms, and puts the feature vectors into the message queue.

[0086] S103, cleaning the feature vector by recycling the pool and vector retrieval, and putting the cleaned data into the second message queue.

[0087] The recycling pool is constructed or updated by using the feature vectors of failed classification.

[0088] Feature extraction is performed using feature vectors that failed classification, a recycling pool is constructed, data is cleaned in advance, data is retrieved from the message queue, and screened and matched through vector retrieval and data in the pool. Successfully matched data is cleared in advance, and the cleaned data is deleted before being put into the message queue.

[0089] Before starting the processing, the recycling pool does not save the feature vectors that failed to be classified. In the sixth stage, the feature vectors that will be cleared after the matching classification fails will be stored, and the data that failed to be classified will be recycled. In the sixth stage, the data with low matching rate in the recycling pool will be deleted through autonomous learning and optimization, and the amount of data in the pool will be maintained at 10% of the total amount of the category feature set. Continuous optimization and screening can keep the data in the pool of high quality and relevance. Limiting the amount of data in the pool can avoid resource waste caused by data redundancy and invalid storage. Take the data to be cleaned from the message queue passed from the second stage, and perform vector retrieval in the vector data set of the recycling pool. The retrieval uses the cosine similarity distance measurement method to calculate the similarity angle between the query vector and the vector in the pool, and compares it with the preset value of the similarity distance. The number of features whose cosine value of the similarity angle of the query vector is greater than the preset value is counted as the number of hits, and then check whether the number of hits exceeds another preset threshold. If the number of features whose cosine value of the similarity angle retrieved is greater than the preset value is higher than the preset hit threshold, the match is considered successful. At this point, the data is considered problematic data or data that needs to be cleaned, and is removed from the message queue or marked as processed. If the match is unsuccessful (i.e., the similarity cosine distance or the number of hits does not meet the threshold requirement), the data is placed in a new message queue for subsequent processing or entering the next stage.

[0090] During the data cleaning process, the first thing that was noticed was a key improvement in the recycling pool: at the beginning of the process, there were no feature vectors in the recycling pool that failed to be classified. However, in step S106, the system will specifically store feature vectors that will be cleared after classification fails, with the aim of reincorporating these failed data into the recycling system. At this stage, through autonomous machine learning and optimization mechanisms, the system will further filter out data items with low matching rates in the recycling pool to ensure that the amount of data retained in the pool is always maintained at around 10% of the total category feature set. This continuous optimization and screening mechanism can effectively improve the quality and relevance of the data in the pool. At the same time, by reasonably limiting the amount of data, it can also effectively avoid the problem of resource waste caused by data redundancy and invalid storage.

[0091] The specific process of step S103 may include:

[0092] Determine the number of hits of the feature vector currently being processed in the first message queue in the recycling pool, the hit number being the number of similarities between the feature vector currently being processed and the feature vectors in the recycling pool being greater than a threshold; if the hit number corresponding to the feature vector currently being processed is greater than the threshold, determine that the feature vector currently being processed needs to be cleaned; if the hit number corresponding to the feature vector currently being processed is less than the threshold, determine that the feature vector currently being processed does not need to be cleaned; and put the feature vectors that do not need to be cleaned into the second message queue.

[0093] Specifically, the data cleaning process extracts the data to be processed (feature vector) from the message queue received in step S102, and then performs a vector search operation in the vector data set of the recycling pool. During the search process, the cosine similarity distance can be used as a metric to measure the similarity between the feature vector currently being processed and the vector in the recycling pool by calculating the similarity angle between them. Then, the cosine values ​​corresponding to these similarity angles are compared with a preset similarity distance threshold. At the same time, the number of features whose cosine values ​​of the similarity angles exceed the preset threshold, i.e., the number of hits, can also be counted, and it can be further checked whether this number of hits exceeds another preset threshold.

[0094] If the search results show that the cosine value of the similarity angle exceeds the preset number of features and is higher than the preset hit count threshold, the match is considered successful. In this case, the data is considered problematic data or data that needs to be cleaned, and will be removed from the original message queue or marked as processed. On the contrary, if the match is unsuccessful, that is, the similarity cosine distance or the hit count fails to meet the preset threshold requirements, the data will be placed in a new message queue for subsequent processing or entering the next processing stage.

[0095] In other words, the data cleaning stage occupies a vital position in the entire process of data processing. It can not only greatly improve the quality level of data, but also effectively shorten the time span of subsequent processing, thereby comprehensively improving the overall efficiency of data processing.

[0096] More specifically, Figure 4 As shown in FIG. 1 , a data cleaning mechanism based on autonomous learning is introduced, which can recycle the feature vectors that were originally about to be deleted, and is specifically used for pre-cleaning the data passed in from the second stage (step S102). The design of this recycling pool cleverly controls its scale, and only retains 10% of the total number of all feature sets to be classified in the first stage as the comparison benchmark. Figure 4As shown in the figure, the number of feature sets in each of the six categories, including dispatch management, vehicle management, regional control, DCS network, and LTE wireless network, has reached the million level, so the total number of feature sets in all categories is about six million; while the amount of data in the recycling pool is maintained at around 600,000. By continuously optimizing and strictly limiting the amount of data in the pool, the scale of data that needs to be traversed during the data matching process is significantly reduced, thereby greatly improving the matching rate. Using the streamlined data set for comparison not only ensures the accuracy and efficiency of cleaning, but also effectively reduces the resource usage of the system.

[0097] When data flows into the recycling pool, the similarity calculation will be started immediately to measure the cosine similarity between these data and the data stored in the pool, and the angle value, a pre-set standard, will be used as the basis for judging the similarity of the data. Specifically, if the cosine value of the similarity angle of a certain data exceeds the preset threshold, and the number of matching hits also exceeds the set hit threshold, then this data will be considered redundant or does not meet the processing requirements. Subsequently, the system will immediately mark it as processed and remove it from the entire processing flow to prevent it from entering the subsequent data classification link. This mechanism ensures that the data processing process can efficiently and accurately screen out valuable data, while eliminating unnecessary or duplicate data, laying a solid foundation for subsequent data analysis and utilization.

[0098] For those data that fail to find a match in the recycling pool, these data will not be cleared, and they will enter the next processing stage for further matching with the vector feature set to be classified. If the match is successful in the next stage, it will be classified; conversely, if it is still not successfully classified after a comprehensive screening of all feature sets in the next stage, the data will be regarded as failed classification data. These failed classification data will be used to supplement and update the content of the recycling pool in the sixth stage. At the same time, in the sixth stage, the data will be regularly optimized through autonomous learning. According to the actual operation situation and feedback on data quality, the data set in the pool will be regularly cleaned or updated to ensure that the most representative and effective feature set is always stored therein. This design enables the system to show high flexibility and accuracy when facing ever-changing data inputs.

[0099] In this embodiment, by introducing a recycling pool mechanism and combining it with autonomous learning for optimization design, it is possible to filter out unnecessary data at an early stage using fewer feature sets, thereby significantly improving the overall data quality, reducing processing time and data size. For data that fails to be classified, it is no longer simply deleted, but filled into the pool through autonomous learning for effective use. This measure further optimizes the data processing process and achieves efficient integration and recycling of resources.

[0100] S104. For the feature vectors in the second message queue, a clustering index algorithm and a distance weighted average are used to calculate a matching order for a pre-constructed feature set, and a vector retrieval algorithm is used to match the feature vectors with the feature set according to the matching order to determine the data category to which each feature vector in the second message queue belongs.

[0101] In other words, the purpose of this step is to classify the data. Specifically, first use the clustering index algorithm, which can extract common features based on the similarity within the feature set, and then use this common feature as the clustering index of the feature set; then, calculate the distance between the feature vector and the clustering index of each feature set, and further determine the classification possibility of the feature vector and the corresponding feature set based on the distance weighted average, and determine the priority matching order of the feature set for the currently processed feature vector. In addition, in order to improve the efficiency of data retrieval and matching, an advanced vector retrieval algorithm is used, which can quickly locate the data point closest to the target feature.

[0102] During the retrieval process, the algorithm can identify the specific type of data and trace its source to ensure the accuracy and traceability of the data. Once the data type and source are clear, the data is classified into the pre-set category that best matches its characteristics. This process not only improves the accuracy of data processing, but also ensures the orderliness and efficiency of data management.

[0103] Optionally, the specific process of step S104 may include:

[0104] First, for any feature vector, the similarity between the feature vector currently being processed and each feature set is determined through the clustering index algorithm and the distance weighted average; secondly, the matching priority between the feature vector currently being processed and each feature set is determined based on the similarity between the feature vector currently being processed and each feature set; wherein, the higher the similarity, the higher the matching priority; then, according to the matching priority of each feature set, the vector retrieval algorithm is used to match the feature vector currently being processed, and the feature vector currently being processed is classified into the category of the feature set that has been successfully matched; wherein, if the feature set with a higher priority successfully matches the feature vector currently being processed, the feature set with a lower priority will no longer match the feature vector currently being processed.

[0105] When dealing with the task of matching feature vectors with multiple feature sets of different categories, in order to significantly improve efficiency and accuracy, an advanced strategy is adopted: combining feature vector extraction with cluster index construction, supplemented by distance weighted sorting. The core of this strategy is to accurately calculate which category of feature set the current data has a higher similarity with, so as to give priority to vector retrieval with the feature set under this category.

[0106] Specifically, if Figure 5 As shown, first select the closest feature set, and then check the data processing unit bound to it. If there is an idle data processing unit, then use the vector retrieval technology to compare the current vector features to be classified with the feature set of the selected category vector. This comparison process includes calculating the similarity angle between the query vector and each vector in the classification vector feature set (usually measured by cosine similarity), and comparing these similarity angles with the preset similarity distance threshold one by one.

[0107] At the same time, it is also necessary to check whether the number of hits exceeds another preset threshold. Only when the similarity angle cosine value of the query vector is greater than the preset angle value cosine threshold and the number of hits is also higher than the hit threshold, the match is considered successful. Once the match is successful, the data is immediately classified into the current category and marked as classified in the message queue to ensure that other data processing units will not process this data repeatedly.

[0108] If the match is unsuccessful, the current classification is marked as processed but not matched, and other data processing units of the same feature set are prohibited from processing this data again. Then, the data processing units of other classification feature sets are waited to continue to execute the above processing flow. If this data is still not marked as classified after all classification data sets have been processed, it may be necessary to further analyze the reasons or take other measures to process this data.

[0109] If no match is successful, the data will be placed in the recycling pool for recycling.

[0110] Through this strategy, not only the accuracy and efficiency of data matching are improved, but also the repeated processing of data is avoided, ensuring the orderliness and efficiency of the data processing process.

[0111] It should be noted that the introduction of clustering index construction and distance-weighted sorting is to perform a higher-level aggregation on the feature vectors of large data volumes to obtain data with smaller data volumes but more obvious features as clustering indexes, perform distance-weighted calculations and sorting on the feature vectors to be matched, predict feature sets with high matching rates, give priority to retrieval and matching, and speed up data classification processing.

[0112] Specifically, we first conduct an in-depth analysis of the feature vector set that needs to be classified, extract the more representative feature vectors, and create a clustering index based on these feature vectors. After a high degree of aggregation, the amount of data in the clustering index is only one percent of the original feature set, thereby greatly reducing the amount of data required to be processed for subsequent calculations.

[0113] like Figure 6As shown in the figure, first, feature extraction is performed on the feature set that needs to be classified. The core purpose of feature extraction is to eliminate redundant information and implement data dimensionality reduction, so as to accurately retain common and key features and finally generate a more hierarchical, compact and efficient feature vector set. For example, considering Figure 6 The scheduling management vector feature set shown has an original data volume of up to millions, which undoubtedly increases the processing complexity and time cost.

[0114] After fine processing of feature extraction, these data were successfully reduced to a cluster index data volume of 10,000 levels. This significant data compression provides great convenience for subsequent data processing. Using these optimized cluster index feature vectors, the originally large and complex million-level vector feature set can be clustered into a more compact and structured 10,000-level cluster index in an efficient way.

[0115] The introduction of this clustering index not only effectively controls the data size of the clustering index, making data management and retrieval more convenient, but also greatly reduces the time of data calculation and improves the overall data processing efficiency. Therefore, feature extraction and clustering index construction show their indispensable importance and practicality when processing large-scale feature vector sets.

[0116] Next, the feature vectors to be matched are taken from the message queue, and the distances between these feature vectors and all category clustering indexes are calculated one by one. In order to achieve this process, the Euclidean distance is used as the calculation standard, thereby obtaining the distance set D between the feature vector to be matched and the m feature vectors in the clustering index, which is specifically expressed as D = (D1, D2, D3, D4, ... Dm). The number of feature vectors in different clustering indexes can be the same or different.

[0117] After calculating these distances, they are weighted according to the preset distance weight algorithm. This weight design strictly follows the principle of "the closer the distance, the higher the weight", that is, the closer the data item is to the cluster center, the greater the possibility of matching with the category, and therefore a higher weight should be given.

[0118] Specifically, the weight Wi of each distance is calculated using the distance weight algorithm according to the value of each distance Di. Through this weighted processing, the matching degree between the feature vector to be matched and each category can be more accurately evaluated, providing strong support for subsequent classification decisions.

[0119] In summary, by calculating the distance between the feature vector to be matched and the clustering index feature vector and performing weighted processing according to the distance weight algorithm, the category closest to the feature vector to be matched can be effectively screened out, thereby achieving efficient and accurate classification.

[0120] Specifically, the weighted weighting adopts a position-based normalized weighted conversion function, and the calculation formula for the weight Wi of the i-th distance Di in the distance set D is as follows:

[0121]

[0122] Where W is the preset total weight value. Then normalize the weight and calculate the weight W i ', the formula is as follows:

[0123]

[0124] Finally, the weighted average is calculated to obtain the similarity Davg. The calculation formula is as follows:

[0125]

[0126] By calculating the weighted average of the clustering index of each category, we can get D 调度管理avg , D 车辆管理avg , D 区域控制avg , D DCS网络avg Etc., and sorting these values ​​can clearly identify the clustering index with the highest match for a given data item, thereby giving priority to processing these high matching rate feature sets.

[0127] By calculating the weighted average of the clustering index of each category, a series of average distance values ​​related to the feature vector to be matched can be obtained, including D 调度管理avg , D 车辆管理avg , D 区域控制avg and D DCS网络avg Etc. These average distance values ​​reflect the overall closeness between the feature vector to be matched and the cluster center of each category.

[0128] Next, these average distance values ​​are sorted to clearly identify the cluster index that has the highest matching degree with the data item to be matched. The sorting is based on the size of the average distance value. The smaller the average distance, the higher the matching degree between the feature vector to be matched and the cluster center of the category.

[0129] Through the sorted results, those feature sets with high matching rates can be processed first. These feature sets not only have a high similarity with the feature vector to be matched, but are also likely to belong to the same category or have similar characteristics. Therefore, when processing the feature vector to be matched, these feature sets with high matching rates can be given priority to improve the accuracy and efficiency of classification or processing.

[0130] In summary, by calculating the weighted average and sorting, we can effectively identify the clustering index with the highest match to a given data item, and give priority to these high matching feature sets, thereby providing strong support for subsequent classification, processing or decision-making.

[0131] The strategy of extracting feature vectors and building cluster indexes has shown excellent performance improvement in matching tasks of multiple large-scale data sets. This strategy can accurately predict feature sets with high matching potential by using a small number of cluster indexes and weighted sorting techniques. On this basis, vector retrieval of these feature sets is prioritized, which significantly reduces the consumption of computing resources and time investment. In other words, the strategy of extracting feature vectors and building cluster indexes has brought significant performance improvement and resource saving when processing matching tasks of large-scale data sets with its high efficiency and accuracy.

[0132] S105. Dynamically adjust the feature set matched by the data processing unit.

[0133] This step focuses on the intelligent scheduling strategy of the data processing unit, where the feature set and the service node operate independently and in parallel. In order to achieve optimal utilization of resources, a real-time monitoring mechanism for load can be introduced. This mechanism can dynamically evaluate the current load status of the service node and flexibly allocate resources accordingly.

[0134] Specifically, when the load of a service node reaches the preset threshold, the system will automatically trigger the dynamic allocation process. In this process, the system can unbind the binding relationship between the feature set and the service node and reallocate the binding relationship, ensuring that computing resources can be evenly distributed among the nodes and avoiding excessive concentration or idleness of resources.

[0135] In addition, the intelligent scheduling strategy also supports flexible expansion of resources. When faced with large-scale data processing tasks, the system can automatically increase computing resources to meet task requirements; when the amount of tasks decreases, the system will reduce resources accordingly to maximize cost-effectiveness. This flexible resource management mechanism enables the data processing unit to always maintain efficient utilization, regardless of the size of the task.

[0136] In data processing scenarios, intelligent scheduling mechanisms play a vital role in solving the problem of uneven resource utilization caused by differences in the frequency of feature set usage. Specifically, some highly active feature sets may continue to consume a large amount of computing resources, while other feature sets may be in a low-load state for a long time. This unbalanced resource allocation seriously affects the overall processing efficiency.

[0137] To address this challenge, an intelligent scheduling strategy is adopted in which feature sets and service nodes are independent of each other. Under this strategy, each service node is designed to be bound to only a single feature set, while each feature set can be bound to multiple service nodes to form a data processing unit. This design allows the allocation of computing resources to be flexibly adjusted according to the actual needs of the feature set.

[0138] The intelligent scheduling algorithm plays a core role in this process. It dynamically allocates service nodes to form multiple efficient data processing units by analyzing the usage frequency of feature sets, the load status of service nodes, and the overall resource utilization. These processing units can maximize resource utilization while ensuring load balancing.

[0139] In addition, the binding relationship between service nodes and feature sets can be dynamically changed. When the usage frequency of a feature set changes, or the performance of a service node fluctuates, the intelligent scheduling algorithm can quickly adjust the binding relationship and reallocate computing resources to the feature set that needs them most. This flexible binding mechanism ensures that the data processing unit can continue to operate efficiently, regardless of how the usage of the feature set changes.

[0140] In summary, by adopting the strategy of independent feature sets and service nodes, dynamic allocation of service nodes by intelligent scheduling algorithms, and replaceable binding relationships, the problem of unbalanced resource utilization has been successfully solved, ensuring efficient utilization and load balancing of data processing units. These measures have jointly improved the overall performance of data processing and provided strong support for large-scale data processing tasks.

[0141] like Figure 7 As shown in Figure 1, the system can monitor the idle time and resource utilization of the data processing unit corresponding to each classification data set in real time. At regular intervals, the system will conduct a comprehensive evaluation of the load of all classification data sets and the average idle time of each processing unit.

[0142] During the evaluation process, if it is found that the processing unit of a certain data set is idle for too long or has an abnormally high load, while the processing unit of another data set is idle for a short time or has a relatively low load, the intelligent scheduling system will immediately trigger the dynamic allocation mechanism. This mechanism will release the processing units with lower loads and reallocate them to the data sets with continuous high loads and short idle times. In this way, the system will increase the service nodes of the high-load data sets, thereby effectively alleviating their processing pressure, improving the overall data processing efficiency, and achieving a dynamic balance of the processing units.

[0143] In addition, the system can dynamically adjust resources according to the overall load. When business demand increases, the system can automatically add servers to provide flexible increase in computing resources; when business demand decreases, the system will reduce servers accordingly to optimize costs. This flexible resource configuration method not only improves performance and resource utilization, but also meets diverse business needs and achieves dynamic expansion.

[0144] By intelligently scheduling data processing units, the system can continuously monitor and dynamically adjust the allocation of processing units, so that resources can be more fully utilized and resource waste can be reduced. This mechanism ensures that the load between each classification data set remains relatively balanced, effectively reducing the overall idle waiting time of the system and improving the smoothness and response speed of data processing.

[0145] At the same time, the system also has good scalability. According to the actual operation situation, the system can flexibly increase or decrease data nodes to adapt to the processing needs of data classification in different scenarios and businesses. This flexibility and scalability enables the system to always maintain efficient and stable operation, providing strong support for the business.

[0146] S106, updating the recycling pool.

[0147] If any of the feature vectors in the second message queue fails to be classified when matched with each feature set, the feature vector that fails to be classified will be added to the learning sample of the recycling pool, and the recycling pool will be filled and optimized through the autonomous learning algorithm, and the data set will be cleaned or updated regularly.

[0148] The recycling pool is an innovative resource management mechanism that uses a self-learning algorithm to deal with the problem of feature vectors that fail to be classified. When feature vectors fail to be successfully classified into the predetermined categories, they are not abandoned, but are reused and added to the recycling pool.

[0149] The feature vectors in this pool will then become the "teaching materials" for the self-learning algorithm. The algorithm will deeply analyze the characteristics of these feature vectors and optimize and fill the recycling pool based on these analysis results. This process is dynamic and adaptive, ensuring that the feature set in the recycling pool always remains highly representative and effective.

[0150] In order to ensure that the data in the recycling pool is always up to date, a regular data cleaning and updating mechanism is implemented. This step aims to remove feature vectors that are outdated or no longer representative, and introduce new and more representative data to enhance the overall quality of the recycling pool.

[0151] In summary, the recycling pool reuses the feature vectors that failed to be classified by using the autonomous learning algorithm, and continuously optimizes and fills them. At the same time, through regular data cleaning and updating, the high representativeness and effectiveness of the feature set are ensured, providing strong support for fields such as data analysis and machine learning.

[0152] More specifically, after processing all classified data sets, if the data cannot be successfully classified, then these data are regarded as unnecessary data for the system and need to be deleted. In order to improve the quality of data to be processed, in this stage, the vector features of the data to be deleted are stored in the recycling pool of the third stage through feature extraction, and the data in the pool will be continuously updated and expanded through autonomous learning to include more types of data features that need to be cleaned. In order to avoid the problem of storage resource shortage and reduced processing efficiency caused by the continuous expansion of the amount of data in the cleaning pool, the data in the pool is updated and optimized through the autonomous learning algorithm, the hit feature values ​​of the data in the pool are marked, and the position of the data in the pool is adjusted. The more active the features, the more advanced they are. The data in the pool is automatically removed through the predetermined threshold. For the new data entering the pool, its similarity with the data in the pool and the data to be removed is predicted. The position of the data entering the pool is determined by the configuration parameters of breadth and precision, as well as the activity of the current data in the pool, and the order of optimization is determined by the position in the pool. When the same type of data enters the system again, the system can identify and cleanse the unnecessary data in advance by recycling the information in the pool, and quickly identify and eliminate the data at an early stage, thereby improving the efficiency and accuracy of data processing.

[0153] This step focuses on the autonomous learning and optimization process of the vector feature recycling pool. In step S104 of the data processing flow, the data that failed to be successfully classified is often regarded as data that is not directly needed by the system and may face the fate of deletion. However, in this step, a more proactive strategy is adopted: through sophisticated feature extraction technology, the vector features of these data that are about to be abandoned are captured and stored in the recycling pool of the third stage.

[0154] The recycling pool is not just a static storage warehouse, but an intelligent system with autonomous learning capabilities. It continuously learns and updates the data in the pool to cover more types of data features that may need to be cleaned. This learning method enables the recycling pool to gradually accumulate a rich and diverse data feature library, providing strong support for subsequent data processing tasks.

[0155] However, as the amount of data in the recycling pool continues to increase, there is also the potential risk of tight storage resources and reduced processing efficiency. To address this challenge, an advanced self-learning algorithm is introduced to intelligently update and optimize the data in the pool. The algorithm can mark the hit feature values ​​of the data and dynamically adjust its position in the pool based on the activity of the data. Features with high activity will be placed in a more forward position so that they can be accessed and used more quickly in subsequent data processing.

[0156] At the same time, a strict threshold mechanism is set to automatically remove data from the recycling pool. When the activity of data decreases to a certain level, they will be automatically removed to free up storage space, ensuring that the recycling pool can always maintain the refinement and efficiency of its data.

[0157] For the newly entered data, a more sophisticated similarity prediction technology is used. By predicting the similarity between the newly entered data and the existing data in the pool and the removed data, and combining the configuration parameters of breadth and precision and the activity of the current data in the pool, the position of the newly entered data in the pool and the order of optimization can be determined. This sophisticated pool entry strategy ensures that new data can be quickly integrated into the data ecology of the recycling pool and provides strong support for subsequent data processing tasks.

[0158] When the same type of data enters the system again, the information in the recycling pool will play a huge role. The system can quickly identify and eliminate these unnecessary data, thereby improving the efficiency and accuracy of data processing at an early stage. This ability not only helps to reduce redundant operations and error rates in subsequent data processing, but also improves the performance and stability of the entire system.

[0159] Through the autonomous learning and optimization process of the vector feature recycling pool, non-critical data can be effectively managed to avoid ineffective resource occupation. At the same time, the system can also continuously learn and optimize the data features in the pool to make it more adaptable to actual data processing needs. This intelligent recycling mechanism not only improves the efficiency and accuracy of data processing, but also provides a strong guarantee for the scalability and stability of the system.

[0160] It should be noted that at least one of step S101, step S105 and step S104 can be omitted; the specific combination will not be described here one by one, and it will depend on the actual situation and is within the scope of protection of this application.

[0161] In this embodiment, data packets of each subway line are collected in real time, the data packets are converted into feature vectors through a vector algorithm, and the feature vectors are stored in a first message queue; the data packets include original subway operation and maintenance data; the feature vectors in the first message queue are cleaned through a recycling pool and vector retrieval, and the cleaned feature vectors are placed in a second message queue; wherein the recycling pool is constructed and updated using feature vectors that fail to be classified; for the feature vectors in the second message queue, a clustering index algorithm and a distance weighted average are used to calculate a matching order for a pre-constructed feature set, and a vector retrieval algorithm is used to match the feature vectors with the feature set according to the matching order, and the data category to which each feature vector in the second message queue belongs is determined; that is, the content similarity analysis technology based on vector retrieval can reuse the feature vectors that fail to be classified to clean the original data and remove redundant and erroneous information; and through clustering index and distance sorting, the vector retrieval algorithm is used to classify the data into corresponding categories, that is, the multi-dimensional data of the subway line is classified, and the data processing requirements of the subway line network operation and maintenance system are met, which provides strong data support for subsequent data analysis, fault prediction and maintenance decision-making.

[0162] It should be noted that some problems that may arise in some solutions, as well as how this application solves them and the effects obtained are explained.

[0163] (1) The data comes from multiple subway lines and systems, covering vehicle dispatching, interlocking management, network exchange monitoring, LTE network operation and maintenance, regional controllers, passenger information services, etc., in various forms, including images, hexadecimal data streams, Modbus protocol data, protobuf serialized information, log documents, HTTP data packets, TCP / UDP communication data, and structured data such as XML and JSON. These data and types are numerous and complex, making them difficult to uniformly identify and process.

[0164] However, this embodiment uses a vector algorithm to extract features from data, converts it into a vector form in a high-dimensional space, and generates vector features of the data. This method enables data from different sources and types to be converted into a unified numerical representation through feature extraction.

[0165] This move breaks the limitations of data formats and sources, and enables the processing and analysis of different types of data originally scattered in different systems under a unified framework. This not only promotes data integration and standardization, but also provides more comprehensive and accurate support for subsequent decision-making.

[0166] (2) Faced with a massive amount of data, a large amount of invalid data mixed in it leads to uneven data quality. These invalid data not only take up valuable storage space in vain, but also greatly increase the difficulty and time cost of data processing, seriously hindering the effective use and in-depth analysis of data.

[0167] In order to meet this challenge, this embodiment designs a recycling pool mechanism. This mechanism fills the feature vectors that failed to be classified into the recycling pool after feature extraction again, thereby realizing the reuse of feature vectors that are about to be discarded. At the same time, we also strictly control the data scale of the recycling pool to ensure that the total amount of feature sets retained in the pool does not exceed 10% of all the data that needs to be classified as a comparison benchmark. In this way, we can use fewer feature sets to effectively clean up invalid data in the early stages of data processing, thereby greatly reducing the amount of data that needs to be retrieved during the data matching process and significantly improving the matching efficiency.

[0168] For data that failed to be classified, we no longer simply delete them, but instead use them effectively through the recycling pool. This practice further optimizes the data processing process and achieves efficient integration and recycling of resources. When the same type of data enters the system again, they will be compared and cleaned with a small amount of benchmark data in the recycling pool, and invalid data can be quickly identified and eliminated at an early stage. This not only ensures the accuracy and efficiency of data cleaning, but also effectively reduces the resource consumption of the system and greatly shortens the time cycle of subsequent processing, thereby comprehensively improving the overall data processing efficiency.

[0169] (3) If the data in the recycling pool is not effectively managed, it will continue to accumulate and expand, resulting in the gradual accumulation of useless feature vectors, resulting in the problem of resource occupation and low utilization. Although some feature vectors occupy resources, their actual usage frequency is not high, which not only causes a waste of resources, but also gradually slows down processing efficiency.

[0170] To meet this challenge, this embodiment introduces an autonomous learning mechanism to regularly optimize the data in the recycling pool. The mechanism will mark and track the hit feature values ​​of the data in the pool, adjust the position of the data in the pool according to the actual situation, and ensure the reasonable layout of the data. At the same time, combined with the actual operation of the system and data quality feedback, the data set in the pool will be regularly cleaned or updated, and the newly added data will be predicted to determine its optimal position in the pool. In this way, the most representative and effective feature set is always stored in the recycling pool.

[0171] By optimizing the recycling pool of vector features that failed to be classified through an autonomous learning mechanism, the system can automatically identify and clean up useless or low-utilization feature vectors, avoiding ineffective occupation of resources. In addition, the system can continuously learn and optimize the data features in the pool to ensure that the pool stores data that has been recently active and is more critical to the current analysis or processing tasks. This mechanism makes the recycling pool more adaptable to actual data processing needs, thereby significantly improving overall data quality, reducing processing time and data size, while maintaining a high degree of flexibility and accuracy.

[0172] (4) When data needs to be matched with multiple feature sets of different categories, the traditional one-by-one retrieval and matching method is inefficient. In the worst case, the data item may need to traverse to the last feature set to complete the match, which will undoubtedly cause the system's processing time and resource consumption to reach the limit, seriously slowing down the overall efficiency of data processing. This inefficient matching mode not only increases the time cost, but also may weaken the system's responsiveness due to resource constraints.

[0173] In order to overcome this problem, this embodiment innovatively adopts feature vector extraction technology, combined with cluster index construction and distance weighted sorting strategy. First, the original data set is deeply analyzed, the most representative feature vectors are extracted by dimensionality reduction, and a cluster index is constructed accordingly. The amount of data in this index is only one percent of the original feature set, which greatly reduces the scale of data required for subsequent calculations. Subsequently, we use distance weighted calculation to sort the data and the feature set to be matched, predict the feature set with a higher matching rate and give priority to retrieval and matching, thereby significantly accelerating the classification and processing speed of the data.

[0174] This solution, which combines feature vector extraction with cluster index construction and distance weighted sorting, has shown excellent performance in matching multiple large-scale data sets. With a small number of cluster indexes and weighted sorting, we can accurately predict feature sets with high matching rates and prioritize vector retrieval, effectively reducing computing time and resource requirements and improving matching accuracy and efficiency.

[0175] (5) In scenarios involving multiple feature sets, the usage frequencies of each feature set often vary significantly, resulting in uneven resource allocation. Some highly active feature sets may continuously occupy a large amount of computing resources, while others remain idle for a long time. This imbalance in resource utilization seriously restricts the overall performance of the system.

[0176] To solve this problem, this embodiment adopts an architecture in which feature sets and computing nodes are independent and parallel, and introduces an intelligent scheduling mechanism. Through intelligent scheduling, each feature set can be flexibly bound to multiple computing nodes to form a data processing unit. At the same time, computing nodes can also unbind feature sets and reallocate them at any time to achieve dynamic adjustment of resources. We intelligently allocate resources based on the resource utilization of the processing unit, release idle processing units in time, and reallocate them to data sets with continuous high loads. In addition, we can also flexibly increase or decrease computing resources according to actual needs to optimize costs and meet the needs of different businesses.

[0177] Through the intelligent scheduling of processing units and the strategy of dynamically unbinding and reallocating idle computing nodes, we have achieved more efficient resource utilization and reduced resource waste. This solution ensures that the load between each classified data set remains relatively balanced, effectively shortens the overall idle waiting time of the system, and improves the smoothness and response speed of data processing. At the same time, the solution also has good scalability and can flexibly increase or decrease data nodes according to actual operating conditions to meet the data classification processing needs under different scenarios and businesses.

[0178] (6) Faced with a wide variety of data types, systems, and equipment types, the data processing process often only requires data provided by a specific system or device.

[0179] This embodiment shows a high degree of adaptability, and it can be customized closely to the actual needs of the subway network operation and maintenance system. Specifically, this embodiment can only classify and process specific classified data in need, and through flexible configuration of classification feature sets, it can realize the dynamic increase and decrease of feature data sets, thereby accurately screening out the required data resources.

[0180] As operation and maintenance requirements continue to change, whether it is the introduction of new systems or the addition of new data categories, this embodiment can respond quickly. By adjusting and reconfiguring the classification feature set, it can easily adapt to changes in data requirements and ensure that the classified data resources are highly consistent with the operation and maintenance decision requirements. This high degree of flexibility and adaptability enables this embodiment to perform excellent performance in the subway network operation and maintenance system.

[0181] Another embodiment of the present application provides a subway operation and maintenance data classification system.

[0182] See also Figure 8 , the subway operation and maintenance data classification system includes:

[0183] The feature vector module 101 is used to collect data packets of each subway line in real time, convert the data packets into feature vectors through vector algorithms, and store the feature vectors in the first message queue. The data packets include original subway operation and maintenance data.

[0184] The cleaning module 102 is used to clean the feature vectors in the first message queue by recycling the pool and searching the vectors, and put the cleaned feature vectors into the second message queue. The recycling pool is constructed and updated by using the feature vectors that failed classification.

[0185] The classification module 103 is used to calculate the matching order of the pre-constructed feature set for the feature vectors in the second message queue using a clustering index algorithm and a distance weighted average, and to match the feature vectors with the feature set according to the matching order using a vector retrieval algorithm to determine the data category to which each feature vector in the second message queue belongs.

[0186] For details on the specific working process and principle of each of the above modules, please refer to the subway operation and maintenance data classification method provided in the above embodiment, which will not be described here one by one. It depends on the actual situation and is within the protection scope of this application.

[0187] In this embodiment, the feature vector module 101 collects data packets of each subway line in real time, converts the data packets into feature vectors through a vector algorithm, and stores the feature vectors in a first message queue; the data packets include original subway operation and maintenance data; the cleaning module 102 cleans the feature vectors in the first message queue through a recycling pool and vector retrieval, and puts the cleaned feature vectors into a second message queue; wherein the recycling pool is constructed and updated using feature vectors that fail to be classified; the classification module 103 calculates the matching order of the feature vectors in the second message queue using a clustering index algorithm and a distance weighted average value for the pre-constructed feature set, and uses a vector retrieval algorithm to match the feature vectors with the feature set according to the matching order to determine the data category to which each feature vector in the second message queue belongs; that is, the content similarity analysis technology based on vector retrieval can reuse the feature vectors that fail to be classified to clean the original data and remove redundant and erroneous information; and through clustering index and distance sorting, the data is classified into corresponding categories using a vector retrieval algorithm; this provides strong data support for subsequent data analysis, fault prediction and maintenance decisions.

[0188] Another embodiment of the present application provides a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for classifying subway operation and maintenance data as described in any one of the above embodiments is implemented.

[0189] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0190] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0191] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0192] Another embodiment of the present invention provides an electronic device, such as Fig. 9 As shown, including:

[0193] One or more processors 201.

[0194] The storage device 202 stores one or more programs.

[0195] When one or more programs are executed by one or more processors 201, one or more processors 201 implement the subway operation and maintenance data classification method as described in any one of the above embodiments.

[0196] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart.

[0197] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0198] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0199] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0200] The features recorded in the various embodiments in this specification can be replaced or combined with each other, and the same and similar parts between the various embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0201] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0202] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for classifying subway operation and maintenance data, characterized in that: include: Collecting data packets of each subway line in real time, converting the data packets into feature vectors through a vector algorithm, and storing the feature vectors in a first message queue; the data packets include original subway operation and maintenance data; Cleaning the feature vectors in the first message queue by recycling the pool and vector retrieval, and putting the cleaned feature vectors into the second message queue; wherein the recycling pool is constructed and updated by using the feature vectors that failed classification; For the feature vectors in the second message queue, a clustering index algorithm and a distance weighted average are used to calculate the matching order of the pre-constructed feature set, and a vector retrieval algorithm is used to match the feature vectors with the feature set according to the matching order to determine the data category to which each feature vector in the second message queue belongs.

2. The subway operation and maintenance data classification method according to claim 1 is characterized in that: The real-time collection of data packets of each subway line, converting the data packets into feature vectors by using a vector algorithm, and storing the feature vectors in a first message queue includes: Collect data packets from each subway line in real time through data gateways; Preliminarily marking the data packet by parsing at least one of a data header, a label, a protocol type, a source IP address, and a source port number in the data packet; The data packet is converted into a vector representation in a high-dimensional space by using the vector algorithm to obtain a feature vector; The feature vector is stored in the first message queue.

3. The subway operation and maintenance data classification method according to claim 1 is characterized in that: The step of cleaning the feature vectors in the first message queue by recycling the pool and searching the vectors, and placing the cleaned feature vectors into the second message queue includes: Determine the hit number of the feature vector currently processed in the first message queue in the recycling pool; wherein the hit number is the number of similarities between the feature vector currently processed and the feature vector in the recycling pool that is greater than a threshold; If the hit number corresponding to the currently processed feature vector is greater than a threshold, it is determined that the currently processed feature vector needs to be cleaned; If the hit number corresponding to the currently processed feature vector is less than a threshold, it is determined that the currently processed feature vector does not need to be cleaned; The feature vectors that do not need to be cleaned are placed in the second message queue.

4. The subway operation and maintenance data classification method according to claim 1 is characterized in that: The feature vectors in the second message queue are calculated the matching order of the pre-constructed feature set by using a clustering index algorithm and a distance weighted average, and the feature vectors are matched with the feature set according to the matching order by using a vector retrieval algorithm to determine the data category to which each feature vector in the second message queue belongs, including: For any feature vector, the similarity between the currently processed feature vector and each feature set is determined through the clustering index algorithm and the distance weighted average; According to the similarity between the feature vector currently being processed and each feature set, the matching priority between the feature vector currently being processed and each feature set is determined; wherein, the higher the similarity, the higher the matching priority; According to the matching priority of each feature set, the feature vector currently being processed is matched using a vector retrieval algorithm, and the feature vector currently being processed is classified into the category of the successfully matched feature set; among them, if a feature set with a higher priority successfully matches the feature vector currently being processed, a feature set with a lower priority will no longer match the feature vector currently being processed.

5. The subway operation and maintenance data classification method according to claim 1 is characterized in that: Before collecting data packets of each subway line in real time, converting the data packets into feature vectors by using a vector algorithm, and storing the feature vectors in the first message queue, the method further includes: Obtain the historical subway operation and maintenance raw data that needs to be classified; Annotating the historical subway operation and maintenance raw data; Through vector algorithms, feature vectors are extracted from the annotated historical subway operation and maintenance raw data to obtain feature sets for each data category; The feature sets of each data category are bound to different service nodes to obtain multiple independent data processing units.

6. The method for classifying subway operation and maintenance data according to claim 5, characterized in that: After binding the feature sets of each data category with different service nodes, the method further includes: Dynamically adjust the feature set matched by the data processing unit.

7. The method for classifying subway operation and maintenance data according to claim 1, characterized in that: Also includes: If any of the feature vectors in the second message queue fails to be classified when matched with each feature set, the feature vector that fails to be classified will be added to the learning sample of the recycling pool, and the recycling pool will be filled and optimized through the autonomous learning algorithm, and the data set will be cleaned or updated regularly.

8. A subway operation and maintenance data classification system, characterized in that: include: A feature vector module is used to collect data packets of each subway line in real time, convert the data packets into feature vectors through a vector algorithm, and store the feature vectors in a first message queue; the data packets include original subway operation and maintenance data; A cleaning module, used for cleaning the feature vectors in the first message queue by recycling the pool and vector retrieval, and putting the cleaned feature vectors into the second message queue; wherein the recycling pool is constructed and updated by using the feature vectors that failed classification; The classification module is used to calculate the matching order of the pre-constructed feature set for the feature vectors in the second message queue using a clustering index algorithm and a distance weighted average, and to match the feature vectors with the feature set according to the matching order using a vector retrieval algorithm to determine the data category to which each feature vector in the second message queue belongs.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When one or more programs are executed by one or more processors, the one or more processors implement the subway operation and maintenance data classification method as described in any one of claims 1-7.

10. A storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the subway operation and maintenance data classification method as described in any one of claims 1-7 is implemented.

Images