A method, system, and medium for fast classification of traffic data

Abstract

The application discloses a fast classification and grading method and system of traffic data and a medium, mainly relates to the technical field of classification and grading, and is used to solve the problems that the existing scheme only supports structured data, imperfectly supports unstructured data in traffic data, and has high dependence on LLM. The method comprises the following steps: collecting text content data corresponding to traffic data; calculating data fingerprints in parallel according to text types; determining whether the data fingerprints exist in the similarity of historical fingerprints in a preset fingerprint cache area based on the text types of the text content data; when the similarity is greater than a preset similarity threshold, determining that the classification and grading result of the historical fingerprint cache is the classification and grading result corresponding to the traffic data; when there is no historical fingerprint greater than the preset similarity threshold, determining the corresponding classification and grading result in parallel according to the text content data and the corresponding text types; and caching the classification and grading result and the corresponding data fingerprint to the preset fingerprint cache area.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of classification and grading, and particularly relates to a rapid classification and grading method and system for traffic data and a medium. BACKGROUND

[0002] Traditional traffic data classification and grading schemes mainly include:

[0003] 1. According to the number of current database tables to be classified, a recognition processing module corresponding to the number of database tables is started in the form of a sub-thread, and the database tables are sent one by one to different recognition processing modules for processing. Each table corresponds to a recognition thread, and after the recognition results of all tables are obtained, a shunting analysis module is used for result splicing and storage. 2. The calling sequence and execution process of the process engine control scheme are used. The supported classification and grading methods include nearly 10 classification and grading methods such as logistic regression and decision tree, which are rich in variety. The overall parallel processing idea is adopted, and the final result is obtained by comparing the results output by each method. 3. Mainly rely on intelligent agents, and use the ability of a large language model (LLM) to classify and grade data.

[0004] However, the above scheme 1 only supports structured data. Scheme 2 does not support unstructured data in traffic data, and only supports processing of files in the form of cvs and xlsx table types. In addition, the real-time performance is limited, and the multi-method parallel computing overhead is large. Scheme 3 has high dependence on LLM. SUMMARY

[0005] The application provides a rapid classification and grading method, system and medium for traffic data to solve the problems of existing schemes that only support structured data, do not support unstructured data in traffic data, and have high dependence on LLM.

[0006] In a first aspect, the application provides a rapid classification and grading method for traffic data, which comprises the following steps:

[0007] Collecting plaintext data corresponding to the traffic data; based on the text type of the plaintext data, obtaining text content data corresponding to the plaintext data by parallel analysis;

[0008] According to the text content data and the corresponding text type, the data fingerprint is calculated in parallel; wherein the text content data is divided into structured file data and unstructured file data according to the text type;

[0009] Based on the text type corresponding to the text content data, it is determined whether the data fingerprint exists in the similarity of the historical fingerprint in the preset fingerprint cache area, and when the similarity is greater than the preset similarity threshold, the classification and grading result of the historical fingerprint cache is determined as the classification and grading result corresponding to the traffic data;

[0010] When there is no historical fingerprint greater than the preset similarity threshold, a corresponding classification grading result is determined in parallel according to the text content data and the corresponding text type;

[0011] The classification grading result and the corresponding data fingerprint are cached to a preset fingerprint cache area.

[0012] In an implementation manner of the present application, the plaintext data corresponding to the flow data is collected, specifically including:

[0013] The plaintext data corresponding to the encrypted flow data is obtained by deploying a decryption collection module on a server corresponding to the encrypted flow data, using the ebpf technology to Hook the core encryption and decryption functions of the user space SSL / TLS library;

[0014] Or, the plaintext data corresponding to the flow data on the network card of the application server is directly obtained by using the ebpf technology;

[0015] Or, a third-party message capture module deployed on the gateway or bypass of the application server is interfaced, and the plaintext data corresponding to the flow data transmitted in the network message is parsed out by using the communication protocol restoration technology.

[0016] In an implementation manner of the present application, the data fingerprint is calculated in parallel according to the text content data and the corresponding text type, specifically including:

[0017] When the text type corresponds to structured file data, a parallel calculation mode is used to extract all keys in each text content data to form a key set, and a TF-IDF algorithm is used to calculate a word frequency vector corresponding to each key; the data fingerprint is calculated by using the key set and the word frequency vector; wherein the weight of each key in the TF-IDF algorithm is the same;

[0018] When the text type corresponds to unstructured file data, a parallel calculation mode is used to calculate the text length of each unstructured file data;

[0019] When the text length is greater than a preset length threshold, the text abstract of the unstructured file data is extracted, and the data fingerprint is calculated by using the text abstract;

[0020] When the text length is less than or equal to the preset length threshold, the data fingerprint is directly calculated by using the unstructured file data.

[0021] In an implementation manner of the present application, the corresponding classification grading result is determined in parallel according to the text content data and the corresponding text type, specifically including:

[0022] When the text type corresponds to structured file data, a parallel computing mode is adopted, and a rule engine is used to match the key value and data label of each structured file data respectively to obtain the corresponding data label; when there is a key value and data label that does not match successfully, a named entity recognition technology is used to determine the key value to obtain the corresponding data label; all data labels are summarized and input into a trained classification program to obtain a classification grading result;

[0023] When the text type corresponds to unstructured file data, a parallel computing mode is adopted, and BERTopic topic modeling technology is used to perform topic segmentation according to the subject content described in the paragraph in the unstructured file data corresponding to the current thread; the contents in the topic segmentation are preprocessed, and a rule engine is used to scan the preprocessed file to obtain the corresponding data label; the topic segmentation that does not scan the data label is identified by NER to obtain the corresponding data label; after the data label identification of all topic segmentations is completed, the data labels are summarized and input into a trained classification program to obtain a classification grading result.

[0024] In an implementation manner of the present application, based on the text type corresponding to the text content data, whether the data fingerprint exists in the similarity of the historical fingerprint in the preset fingerprint cache area is determined, specifically including:

[0025] When the text type corresponds to structured file data, a cosine similarity algorithm is used to determine the similarity of the data fingerprint and the historical fingerprint;

[0026] When the text type corresponds to unstructured file data, a hybrid similarity algorithm is used to determine the similarity of the data fingerprint and the historical fingerprint by combining the semantic-based vector similarity and the structure-based set similarity.

[0027] In an implementation manner of the present application, a hybrid similarity algorithm is used to determine the similarity of the data fingerprint and the historical fingerprint by combining the semantic-based vector similarity and the structure-based set similarity, specifically including:

[0028] The first similarity of the data fingerprint and the historical fingerprint is calculated by using the SBERT semantic technology;

[0029] The second similarity of the data fingerprint and the historical fingerprint is calculated by using the TF-IDF algorithm;

[0030] The text length of the unstructured file data is obtained, and the first similarity weight and the second similarity weight are determined according to the preset relationship between the text length, the first similarity weight and the second similarity weight;

[0031] According to the formula:

[0032] ;

[0033] calculating a similarity of the data fingerprint and the historical fingerprint;

[0034] wherein, denotes a first similarity weight, denotes a first similarity, denotes a second similarity weight, denotes a second similarity.

[0035] In a second aspect, the present application provides a fast classification and grading system of traffic data, the system comprising:

[0036] a parsing module, configured to collect plaintext data corresponding to the traffic data; and based on a text type of the plaintext data, to obtain text content data corresponding to the plaintext data in parallel;

[0037] a calculating module, configured to calculate data fingerprints in parallel according to the text content data and the corresponding text type; wherein the text content data is divided into structured file data and unstructured file data according to the text type;

[0038] a determining module, configured to determine, based on the text type of the text content data, a similarity of whether the data fingerprint exists in historical fingerprints in a preset fingerprint cache area; when the similarity is greater than a preset similarity threshold, to determine that a classification and grading result of the historical fingerprint cache is a classification and grading result corresponding to the traffic data; when there is no historical fingerprint greater than the preset similarity threshold, to determine a corresponding classification and grading result in parallel according to the text content data and the corresponding text type;

[0039] a caching module, configured to cache the classification and grading result and the corresponding data fingerprint to the preset fingerprint cache area.

[0040] In an implementation manner of the present application, the calculating module comprises a calculating unit, configured to, when the text type corresponds to the structured file data, extract all keys in each text content data to form a key set in a parallel calculation manner, and calculate a word frequency vector corresponding to each key using a TF-IDF algorithm; and calculate the data fingerprint using the key set and the word frequency vector; wherein a weight corresponding to each key in the TF-IDF algorithm is the same.

[0041] When the text type corresponds to the unstructured file data, the calculating module is configured to calculate text lengths of each unstructured file data in a parallel calculation manner.

[0042] When the text length is greater than a preset length threshold, the calculating module is configured to extract a text abstract of the unstructured file data, and calculate the data fingerprint using the text abstract.

[0043] When the text length is less than or equal to the preset length threshold, the calculating module is configured to directly calculate the data fingerprint using the unstructured file data.

[0044] In an implementation form of the application, the determining module comprises a determining unit configured to determine, based on the text type corresponding to the text content data, whether the data fingerprint exists in the similarity of the historical fingerprints in the preset fingerprint cache area, specifically comprising:

[0045] When the text type corresponds to structured file data, the cosine similarity algorithm is used to determine the similarity of the data fingerprint and the historical fingerprint.

[0046] When the text type corresponds to unstructured file data, the hybrid similarity algorithm is used to determine the similarity of the data fingerprint and the historical fingerprint by combining the semantic-based vector similarity and the structure-based set similarity.

[0047] In a third aspect, the application provides a non-volatile computer storage medium having computer instructions stored thereon, the computer instructions being executed to implement the fast classification and grading method of traffic data according to any one of the above.

[0048] From the above technical solutions, the application has the following advantages:

[0049] Parallel parsing technology: by parallel parsing of plaintext data, the data processing efficiency is improved. This method can simultaneously process structured file data and unstructured file data, breaking through the limitation of traditional schemes that only support structured data, and realizing comprehensive coverage of various text contents in traffic data.

[0050] Data fingerprint cache mechanism: by storing historical fingerprints and classification and grading results in a preset fingerprint cache area, the overhead of repeated calculation is reduced. When the similarity of the new data fingerprint and the historical fingerprint exceeds the threshold, the cache result can be directly called, avoiding time-consuming calculation depending on large language models (LLM), and shortening the classification and grading time from tens of seconds to milliseconds.

[0051] Dynamic classification and grading optimization: for the fingerprint cache miss scenario, the topic segmentation idea is used to segment long text for parallel calculation, reducing the processing time of the entire text, thereby improving the processing efficiency of classification and grading. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 is a flow chart of a fast classification and grading method of traffic data provided by an embodiment of the present application.

[0054] Figure 2This is a schematic diagram of the internal structure of a rapid classification and grading system for traffic data provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.

[0057] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0058] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0059] The embodiment provides a method for rapid classification and grading of traffic data, such as Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps:

[0060] Step 110: Collect plaintext data corresponding to traffic data; based on the text type of plaintext data, parse in parallel to obtain the text content data corresponding to the plaintext data.

[0061] In some embodiments, the plaintext data corresponding to the collected traffic data specifically includes:

[0062] By deploying a decryption acquisition module on the server corresponding to the encrypted traffic data, the plaintext data corresponding to the encrypted traffic data is obtained by hooking the core encryption and decryption functions of the user-space SSL / TLS library using ebpf technology.

[0063] Alternatively, plaintext data corresponding to traffic data on the application server's network card can be directly obtained through EBPF technology;

[0064] Alternatively, it can interface with a third-party packet capture module deployed on the application server gateway or in the bypass, and use communication protocol restoration technology to parse out the plaintext data corresponding to the traffic data transmitted in the network packets.

[0065] In addition, plaintext data includes both structured and unstructured data.

[0066] Structured data includes database tables, key-value formats such as JSON, XML, and YAML, as well as form formats with clear key-value pair mappings.

[0067] Unstructured data includes streaming files (such as Word, Excel, and PPT), formatted files (such as PDF and ODF), and images (such as JPG, PNG, GIF, and BMP). These data do not have explicit key-value mapping relationships and have no restrictions on text content and format.

[0068] Because of the independent file parsing within this module, multiple files can be parsed in parallel, improving the overall processing efficiency of large batches of files.

[0069] Step 120: Calculate data fingerprints in parallel based on the text content data and the corresponding text type.

[0070] Among them, text content data is divided into structured file data and unstructured file data according to text type.

[0071] In this step, data fingerprints are calculated in parallel based on the text content data and the corresponding text type, specifically including:

[0072] When the text type corresponds to structured file data, a parallel computing approach is used to extract all keys from each text content data to form a set of keys, and the TF-IDF algorithm is used to calculate the term frequency vector corresponding to each key. The data fingerprint is then calculated using the set of keys and the term frequency vectors. Each key has the same weight in the TF-IDF algorithm (traditional TF-IDF algorithms consider the influence of IDF, i.e., the higher the frequency of a keyword in the entire corpus, the less important it is. In the context of structured data in this paper, each key should be considered to have the same weight, therefore, the interference of IDF needs to be excluded when calculating the term frequency vector).

[0073] When the text type corresponds to unstructured file data, parallel computing is used to calculate the text length of each unstructured file data.

[0074] When the text length exceeds a preset length threshold (e.g., text length > 300 is considered long text; the specific threshold can be adjusted according to the actual situation), extract the text summary of the unstructured file data (using a summary generation engine (based on a domain adaptive model with a BART-T5 hybrid architecture or the TextRank++ algorithm) to extract summary information), and use the text summary to calculate the data fingerprint; when the text length is less than or equal to the preset length threshold, directly use the unstructured file data to calculate the data fingerprint.

[0075] Step 130: Based on the text type corresponding to the text content data, determine whether the data fingerprint exists in the historical fingerprint similarity in the preset fingerprint cache area. When the similarity is greater than the preset similarity threshold, determine the classification and grading result of the historical fingerprint cache as the classification and grading result corresponding to the traffic data.

[0076] In this step: based on the text type corresponding to the text content data, determine whether the data fingerprint exists in the historical fingerprints in the preset fingerprint cache, specifically including:

[0077] When the text type corresponds to structured file data, the cosine similarity algorithm is used to determine the similarity between the data fingerprint and the historical fingerprint;

[0078] When the text type corresponds to unstructured file data, a hybrid similarity algorithm is used, which combines semantic-based vector similarity and structure-based set similarity to determine the similarity between the data fingerprint and the historical fingerprint.

[0079] The method employs a hybrid similarity algorithm, combining semantic-based vector similarity and structure-based set similarity to determine the similarity between data fingerprints and historical fingerprints. Specifically, this includes:

[0080] Using SBERT semantic technology, the first similarity between the data fingerprint and the historical fingerprint is calculated;

[0081] The second similarity between the data fingerprint and the historical fingerprint is calculated using the TF-IDF algorithm.

[0082] Obtain the text length of the unstructured file data, and determine the first similarity weight and the second similarity weight based on the preset relationship between the text length, the first similarity weight, and the second similarity weight;

[0083] According to the formula:

[0084] ;

[0085] Calculate the similarity between the data fingerprint and historical fingerprints;

[0086] in, Indicates the first similarity weight, Indicates the first similarity, Indicates the second similarity weight, This indicates the second similarity.

[0087] Step 140: When there are no historical fingerprints with a similarity greater than the preset threshold, determine the corresponding classification and grading results in parallel based on the text content data and the corresponding text type.

[0088] In some embodiments, the corresponding classification and grading results are determined in parallel based on the text content data and the corresponding text type, specifically including:

[0089] When the text type corresponds to structured file data, a parallel computing approach is used to match the key-value pairs and data tags of each structured file data using a rule engine to obtain the corresponding data tags. When a key-value pair and data tag do not match, named entity recognition technology is used to determine the key-value pair and obtain the corresponding data tag. All data tags are then aggregated and input into a trained classification program to obtain the classification and grading results.

[0090] When the text type corresponds to unstructured file data, a parallel computing approach is adopted. Based on the topic content described in the natural paragraphs within the unstructured file data corresponding to the current thread, BERTopic topic modeling technology is used to segment topics (i.e., a topic segment may contain multiple natural paragraphs). The content in the topic segments is preprocessed (including text cleaning, sentence segmentation, encoding standardization, etc.). The preprocessed files are scanned using a rule engine to obtain the corresponding data labels. For topic segments without data labels, NER identification is performed to obtain the corresponding data labels. After the data labels of all topic segments have been identified, the data labels are summarized and input into a trained classification program to obtain the classification and grading results. Different topic segments can be executed simultaneously to reduce the overall processing time.

[0091] Step 150: Cache the classification and grading results and the corresponding data fingerprints to the preset fingerprint cache area.

[0092] It should be noted that this step utilizes the concepts of data fingerprinting and result caching to eliminate the need for further classification and grading of similar data.

[0093] As described above, existing solutions do not address the handling of identical or similar content. Therefore, when duplicate data exists in the traffic, classification and grading are still re-executed, which can easily lead to data congestion and inability to process large amounts of data. This application introduces a data fingerprint comparison process, which eliminates the need for re-classification and grading of similar data, instead directly retrieving cached results. This improves the classification and grading processing speed under large data volumes and reduces the system's own performance consumption. Furthermore, large texts are segmented into thematic segments based on semantics for parallel computation, which also directly reduces the overall text processing time.

[0094] In addition, this application Figure 2 This application provides a rapid classification and grading system for traffic data. For example... Figure 2 As shown in the embodiments of this application, the system mainly includes:

[0095] The parsing module 210 is used to collect plaintext data corresponding to traffic data; based on the text type of the plaintext data, it parses in parallel to obtain the text content data corresponding to the plaintext data.

[0096] The calculation module 220 is used to calculate data fingerprints in parallel based on the text content data and the corresponding text type; wherein, the text content data is divided into structured file data and unstructured file data according to the text type.

[0097] The calculation module 220 includes a calculation unit, which, when the text type corresponds to structured file data, uses parallel computing to extract all keys from each text content data to form a set of keys, and uses the TF-IDF algorithm to calculate the word frequency vector corresponding to each key; using the set of keys and the word frequency vector, it calculates the data fingerprint; wherein, each key has the same weight in the TF-IDF algorithm;

[0098] When the text type corresponds to unstructured file data, parallel computing is used to calculate the text length of each unstructured file data.

[0099] When the text length exceeds a preset length threshold, extract the text summary of the unstructured file data and use the text summary to calculate the data fingerprint.

[0100] When the text length is less than or equal to a preset length threshold, the data fingerprint is calculated directly using unstructured file data.

[0101] The determination module 230 is used to determine whether the data fingerprint exists in the historical fingerprint in the preset fingerprint cache based on the text type corresponding to the text content data. When the similarity is greater than the preset similarity threshold, the classification and grading result of the historical fingerprint cache is determined as the classification and grading result corresponding to the traffic data. When there is no historical fingerprint greater than the preset similarity threshold, the corresponding classification and grading result is determined in parallel according to the text content data and the corresponding text type.

[0102] The determining module 230 includes a determining unit, used to determine, based on the text type corresponding to the text content data, whether the data fingerprint exists in the similarity of historical fingerprints in a preset fingerprint cache, specifically including:

[0103] When the text type corresponds to structured file data, the cosine similarity algorithm is used to determine the similarity between the data fingerprint and the historical fingerprint;

[0104] When the text type corresponds to unstructured file data, a hybrid similarity algorithm is used, which combines semantic-based vector similarity and structure-based set similarity to determine the similarity between the data fingerprint and the historical fingerprint.

[0105] The caching module 240 is used to cache the classification and grading results and the corresponding data fingerprints to a preset fingerprint cache area.

[0106] In addition, this application embodiment also provides a non-volatile computer storage medium storing executable instructions, which, when executed, implement a method for rapid classification and grading of traffic data as described above.

[0107] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of fast classification of traffic data, characterized in that, The method comprises: Collecting plaintext data corresponding to the traffic data; based on the text type of the plaintext data, obtaining text content data corresponding to the plaintext data through parallel analysis; According to the text content data and the corresponding text type, the data fingerprint is calculated in parallel; wherein the text content data is divided into structured file data and unstructured file data according to the text type; specifically including: When the text type corresponds to the structured file data, a parallel computing method is used to extract all keys in each text content data to form a key set, and a TF-IDF algorithm is used to calculate the word frequency vector corresponding to each key; the key set and the word frequency vector are used to calculate the data fingerprint; wherein the weight of each key in the TF-IDF algorithm is the same; when the text type corresponds to the unstructured file data, the text length of each unstructured file data is calculated by using the parallel computing method; when the text length is greater than the preset length threshold, the text abstract of the unstructured file data is extracted, and the text abstract is used to calculate the data fingerprint; when the text length is less than or equal to the preset length threshold, the data fingerprint is directly calculated using the unstructured file data; Based on the text type corresponding to the text content data, the similarity between the data fingerprint and the historical fingerprint of the preset fingerprint cache area is determined, and when the similarity is greater than the preset similarity threshold, the classification and grading result of the historical fingerprint cache is determined as the classification and grading result corresponding to the traffic data; When there is no historical fingerprint greater than the preset similarity threshold, the corresponding classification and grading result is determined in parallel according to the text content data and the corresponding text type; specifically including: When the text type corresponds to the structured file data, a parallel computing method is used to match the key value and data label of each structured file data using a rule engine to obtain the corresponding data label; when there is a key value that does not match the data label, the key value is determined using named entity recognition technology to obtain the corresponding data label; all data labels are summarized and input into a trained classification program to obtain the classification and grading result; when the text type corresponds to the unstructured file data, the BERTopic topic modeling technology is used to segment the topics according to the topic content described in the paragraphs in the unstructured file data corresponding to the current link; the contents in the topic segments are preprocessed, and the preprocessed files are scanned using a rule engine to obtain the corresponding data label; the topic segments that do not have data labels are identified using NER to obtain the corresponding data label; after all the data labels of the topic segments are identified, the data labels are summarized and input into a trained classification program to obtain the classification and grading result; The classification and grading result and the corresponding data fingerprint are cached in the preset fingerprint cache area.

2. The method of fast classification hierarchy of traffic data according to claim 1, characterized in that, Collecting plaintext data corresponding to the traffic data, specifically including: Through the decryption collection module deployed on the server corresponding to the encrypted traffic data, the ebpf technology is used to hook the core encryption and decryption functions of the user space SSL / TLS library to obtain the plaintext data corresponding to the encrypted traffic data; Or, the ebpf technology is used to directly obtain the plaintext data corresponding to the traffic data on the network card of the application server; Or, the docking deployment is arranged in the application server gateway or bypass third party message capture module, utilizes the communication protocol restoration technology to parse out the traffic data corresponding to the plaintext data transmitted in the network message.

3. The method of rapid triage of traffic data according to claim 1, wherein, Based on the text type corresponding to the text content data, the similarity of the data fingerprint and the historical fingerprint in the preset fingerprint cache area is determined, specifically including: When the text type corresponds to structured file data, the cosine similarity algorithm is used to determine the similarity of the data fingerprint and the historical fingerprint; When the text type corresponds to unstructured file data, a hybrid similarity algorithm is used to determine the similarity of the data fingerprint and the historical fingerprint by combining the semantic-based vector similarity and the structure-based set similarity.

4. The method of fast classification hierarchy of traffic data according to claim 3, characterized in that, The hybrid similarity algorithm is used to determine the similarity of the data fingerprint and the historical fingerprint by combining the semantic-based vector similarity and the structure-based set similarity, specifically including: The SBERT semantic technology is used to calculate the first similarity of the data fingerprint and the historical fingerprint; The TF-IDF algorithm is used to calculate the second similarity of the data fingerprint and the historical fingerprint; The text length of the unstructured file data is obtained, and the first similarity weight and the second similarity weight are determined according to the preset relationship between the text length, the first similarity weight and the second similarity weight; According to the formula: ； The similarity of the data fingerprint and the historical fingerprint is calculated; wherein, denotes a first similarity weight, denotes a first similarity, denotes a second similarity weight, denotes a second similarity.

5. A system for fast classification of traffic data, characterized by The system includes: The analysis module is used to collect the plaintext data corresponding to the traffic data; based on the text type of the plaintext data, the text content data corresponding to the plaintext data is obtained by parallel analysis; The calculation module is used to calculate the data fingerprint according to the text content data and the corresponding text type in parallel; wherein the text content data is divided into structured file data and unstructured file data according to the text type; The calculation module includes a calculation unit, which is used to extract all keys in each text content data to form a key set when the text type corresponds to structured file data, and calculate the word frequency vector corresponding to each key using the TF-IDF algorithm in a parallel calculation manner; the key set and the word frequency vector are used to calculate the data fingerprint; wherein the weight of each key in the TF-IDF algorithm is the same; when the text type corresponds to unstructured file data, the text length of each unstructured file data is calculated in a parallel calculation manner; when the text length is greater than a preset length threshold, the text abstract of the unstructured file data is extracted, and the data fingerprint is calculated using the text abstract; when the text length is less than or equal to the preset length threshold, the data fingerprint is directly calculated using the unstructured file data; The determination module is used to determine the similarity of the data fingerprint and the historical fingerprint in the preset fingerprint cache area based on the text type corresponding to the text content data, and when the similarity is greater than a preset similarity threshold, the classification and grading result of the historical fingerprint cache is determined as the classification and grading result corresponding to the traffic data; when there is no historical fingerprint greater than the preset similarity threshold, the corresponding classification and grading result is determined in parallel according to the text content data and the corresponding text type; specifically including: When the text type corresponds to structured file data, a parallel computing method is used to match the key values and data labels of each structured file data using a rule engine to obtain corresponding data labels; when there is a key value that does not match the data label, named entity recognition technology is used to determine the key value to obtain the corresponding data label; all data labels are summarized and input into a trained classification program to obtain a classification grading result; when the text type corresponds to unstructured file data, a parallel computing method is used to perform topic segmentation using the BERTopic topic modeling technology according to the topic content described in the natural paragraph of the unstructured file data corresponding to the current link; the content in the topic segmentation is preprocessed, and the preprocessed file is scanned using a rule engine to obtain corresponding data labels; NER recognition is performed on the topic segmentation that does not have a scanned data label to obtain the corresponding data label; after the data labels of all topic segmentations are identified, the data labels are summarized and input into a trained classification program to obtain a classification grading result; The cache module is configured to cache the classification grading result and the corresponding data fingerprint to a preset fingerprint cache area.

6. The fast classification hierarchy system of traffic data according to claim 5, characterized in that, The determination module includes a determination unit configured to determine the similarity between the data fingerprint and the historical fingerprint of the preset fingerprint cache area based on the text type corresponding to the text content data, specifically including: When the text type corresponds to structured file data, a cosine similarity algorithm is used to determine the similarity between the data fingerprint and the historical fingerprint; When the text type corresponds to unstructured file data, a hybrid similarity algorithm is used to determine the similarity between the data fingerprint and the historical fingerprint by combining the semantic-based vector similarity and the structure-based set similarity.

7. A non-transitory computer storage medium, comprising: The computer instructions stored thereon, when executed, implement a method for rapid classification and grading of traffic data according to any one of claims 1-4.

Images