Transportation data classification method and device based on vertical field large model

Abstract

The invention discloses a traffic and transportation data classification method and device based on a vertical field large model, and relates to the related field of traffic and transportation data processing and analysis, and the method comprises the steps: configuring a vertical field training data source link, the method comprises the following steps: collecting a vertical field pre-training data set, mixing the vertical field pre-training data set with a general pre-training data set with the data volume 5-10 times that of the vertical field pre-training data set to generate a pre-training data set, constructing a large model fine-tuning training data set containing various data sets, and generating a traffic and transportation data classification pre-training model based on the pre-training data set; a specific data set is used for fine tuning training to obtain a traffic transportation data classification large model, and finally the large model is used for traffic transportation data classification. The technical problems of low classification accuracy and reliability caused by difficulty in data acquisition, poor quality, non-fine and comprehensive classification standard, single model and insufficient adaptability in the existing traffic and transportation data classification technology are solved, and the technical effect of improving the accuracy and reliability of traffic and transportation data classification is achieved.

Description

Technical Field

[0001] The present application relates to fields related to transportation data processing and analysis, and in particular to a transportation data classification method and device based on a vertical field big model. Background Art

[0003] In the field of transportation, the importance of transportation data classification is becoming increasingly prominent. Especially when dealing with massive and complex data, the accuracy and efficiency of classification are becoming more and more urgent, and related contradictions are becoming more prominent. Achieving efficient and accurate transportation data classification to better meet the development needs of the transportation industry has become a key issue that needs to be solved. Traditional transportation data classification methods are usually relatively simple and one-sided, relying on limited data sources or simple classification methods. They do not have a good grasp of the overall characteristics of transportation data, lack in-depth mining and effective use of relevant data, and are difficult to comprehensively and accurately identify the characteristics and value of various types of data. There are unreasonable aspects in resource investment and use, resulting in a lack of resources in some key data classification links, while some links have excess resources. The formulation of classification strategies is relatively rigid and fixed, and cannot effectively respond to the ever-changing actual data situation in the transportation field.

[0004] Among the current relevant technologies, transportation data classification technology has technical problems such as difficulty in data collection, poor quality, incomplete and imprecise classification standards, and a single model with insufficient adaptability, resulting in low classification accuracy and reliability. Summary of the invention

[0005] The present application provides a transportation data classification method and device based on a vertical field big model, configures a vertical field training data source link, collects a vertical field pre-training data set and mixes it with a general pre-training data set with a data volume 5-10 times that of the vertical field to generate a pre-training data set, constructs a large model fine-tuning training data set containing multiple data sets, generates a transportation data classification pre-training model based on the pre-training data set, fine-tunes it with a specific data set to obtain a transportation data classification big model, and finally uses the big model to classify transportation data, thereby achieving the technical effect of improving the accuracy and reliability of transportation data classification by efficiently collecting high-quality data, making the classification standard more refined and comprehensive, enhancing the adaptability and diversity of the model, and achieving the high-quality data collection.

[0006] This application provides a transportation data classification method based on a vertical field big model, including: Configure a vertical field training data source link; according to the vertical field training data source link, collect a vertical field pre-training data set, mix it with a general pre-training data set, and generate a pre-training data set, wherein a first data volume of the general pre-training data set is 5 to 10 times the second data volume of the vertical field pre-training data set; construct a large model fine-tuning training data set, wherein the large model fine-tuning training data set includes a classification standard matching model construction data set, an interface classification model construction data set, and a table classification model construction data set; perform pre-training according to the pre-training data set to generate a transportation data classification pre-training model; perform fine-tuning training on the transportation data classification pre-training model according to the classification standard matching model construction data set, the interface classification model construction data set, and the table classification model construction data set to generate a transportation data classification large model; classify transportation data according to the transportation data classification large model.

[0007] The present application also provides a transportation data classification device based on a vertical domain large model, including: A configuration link module, the configuration link module is used to configure a vertical field training data source link; a pre-training data set generation module, the pre-training data set generation module is used to collect a vertical field pre-training data set according to the vertical field training data source link, and mix it with a general pre-training data set to generate a pre-training data set, wherein the first data volume of the general pre-training data set is 5 to 10 times the second data volume of the vertical field pre-training data set; a data set construction module, the data set construction module is used to construct a large model fine-tuning training data set, wherein the large model fine-tuning training data set includes a classification standard matching model construction data set, an interface classification model construction data set, and a table classification model construction data set; a pre-training model generation module, the pre-training model generation module is used to perform pre-training according to the pre-training data set to generate a transportation data classification pre-training model; a classification large model generation module, the classification large model generation module is used to fine-tune the transportation data classification pre-training model according to the classification standard matching model construction data set, the interface classification model construction data set, and the table classification model construction data set to generate a transportation data classification large model; a data classification module, the data classification module is used to classify transportation data according to the transportation data classification large model.

[0008] The method and device for transportation data classification based on a vertical field big model proposed in this application first configure a vertical field training data source link, collect a vertical field pre-training data set according to the vertical field training data source link, mix it with a general pre-training data set, and generate a pre-training data set, wherein the first data volume of the general pre-training data set is 5 to 10 times the second data volume of the vertical field pre-training data set, and construct a big model fine-tuning training data set, wherein the big model fine-tuning training data set includes a classification standard matching model construction data set, an interface classification model construction data set, and a table classification model construction data set, and pre-training is performed according to the pre-training data set to generate a transportation data classification pre-training model, and fine-tuning training is performed on the transportation data classification pre-training model according to the classification standard matching model construction data set, the interface classification model construction data set, and the table classification model construction data set to generate a transportation data classification big model, and transportation data is classified according to the transportation data classification big model, thereby achieving the technical effect of improving the accuracy, comprehensiveness and adaptability of transportation data classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the accompanying drawings of the embodiment of the present invention will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the device according to the embodiment of the present application. It should be understood that the previous or following operations are not necessarily performed precisely in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0010] Figure 1 A flow chart of a transportation data classification method based on a vertical domain big model provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a transportation data classification device based on a vertical domain big model provided in an embodiment of the present application.

[0011] Explanation of the accompanying drawings: configuration link module 10, pre-trained data set generation module 20, data set construction module 30, pre-trained model generation module 40, classification large model generation module 50, data classification module 60. DETAILED DESCRIPTION

[0012] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0013] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0014] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, device, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0015] The present application embodiment provides a transportation data classification method based on a vertical domain big model, such as Figure 1 As shown, the method includes: Step S100, configuring a vertical field training data source link.

[0016] Specifically, configuring vertical field training data source links refers to clarifying the specific vertical field category to which transportation data belongs, determining the category and scope of the required data, such as road transportation data, railway transportation data, air transportation data, etc., and finding reliable data sources through research and analysis. After the data source is clarified, its access link is obtained. The obtained links need to be carefully recorded and organized to ensure the accuracy and completeness of the links. The links need to be classified and labeled so that they can be quickly and accurately identified and used later. During the configuration process, factors such as data update frequency, data format, and data quality also need to be considered to ensure that the configured data source links can continuously and stably provide high-quality data support for subsequent data collection and processing.

[0017] Step S200: According to the vertical field training data source link, a vertical field pre-training data set is collected and mixed with a general pre-training data set to generate a pre-training data set, wherein a first data volume of the general pre-training data set is 5 to 10 times the second data volume of the vertical field pre-training data set.

[0018] Specifically, the process of collecting vertical field pre-training data sets and mixing to generate pre-training data sets refers to obtaining relevant data and integrating it with the general pre-training data set according to the configured vertical field training data source link. Select a specific vertical field data source, prepare a data collection tool, collect data according to the link, and obtain a vertical field pre-training data set, which covers the relevant information of a specific field. During the collection process, you may face data source access restrictions, data format differences, partial data missing and other conditions. The obtained vertical field pre-training data set is mixed with the general pre-training data set. The general pre-training data set is large in scale, and its data volume is about 5 to 10 times that of the vertical field pre-training data set. When mixing, it is necessary to deal with data compatibility and consistency issues. Data cleaning, standardization and other means may be used to eliminate noise, outliers and inconsistencies. Finally, a pre-training data set is generated, laying the foundation for subsequent model training.

[0019] In a possible implementation, step S200 further includes step S210, obtaining a first data source link of the vertical field training data source link. Specifically, the first data source link in the vertical field training data source link is obtained, and the determination process considers factors such as reliability of the data source, data update frequency, and data integrity.

[0020] Step S220: When the first data source link has a verification code link, access the verification code link to obtain a verification code image. Specifically, determine the first data source link, and if the first data source link has a verification code link, access the verification code link to obtain a verification code image.

[0021] Step S230, the verification code image is identified by the target detection model to obtain verification code information, and the first data source link is accessed through the verification code information to obtain the first data source initial collection data. Specifically, the verification code image is accurately identified by the target detection model to obtain the verification code information, and the first data source link is accessed again with the verification code information to obtain the first data source initial collection data.

[0022] Step S240: when the first data source link does not have a verification code link, access the first data source link to obtain the first data source initial collection data. Specifically, if the first data source link does not have a verification code link, directly access it to obtain the first data source initial collection data.

[0023] Step S250, perform redundant data cleaning on the data initially collected from the first data source, generate the data selected from the first data source, and add it to the vertical field pre-training data set, wherein the redundant data types include preset characters, semantic-free data, and chart data. Specifically, the obtained data initially collected from the first data source is cleaned for redundant data. During the cleaning process, redundant data types such as preset characters, semantic-free data, and chart data are removed. After the cleaning process, the data selected from the first data source is generated and added to the vertical field pre-training data set to prepare data for subsequent data analysis and model training. If problems such as inaccurate verification code recognition, incomplete initial collection data, and redundant data judgment errors are encountered, the target detection model needs to be continuously optimized to improve the recognition accuracy in the case of inaccurate verification code recognition. For incomplete initial collection data, it is necessary to check the validity of the data source link or re-collect. For redundant data judgment errors, it is necessary to improve the judgment rules and algorithms to ensure the accuracy and effectiveness of cleaning.

[0024] In a possible implementation, step S230 further includes step S231, configuring a verification code image data set and a verification code identification data set. Specifically, configuring a verification code image data set and a verification code identification data set, collecting a large number of verification code images, and marking each verification code image with a corresponding correct identification to form a complete data set.

[0025] Step S232, constructing a verification code recognition loss function based on the Hamming distance. Specifically, the Hamming distance is an indicator used to measure the difference between two equal-length character strings, specifically the number of different characters at corresponding positions. When constructing the verification code recognition loss function, the verification code is composed of several characters. The verification code result predicted by the model and the real verification code identifier are compared position by position. If the predicted character is different from the real character at a certain position, it is recorded as a difference. The number of positions of all different characters is added up to be the Hamming distance. When constructing the loss function, it is determined according to the calculated Hamming distance. If the Hamming distance is larger, the difference between the predicted result and the real result is larger, and the loss is greater. The smaller the Hamming distance, the smaller the loss. In the above manner, a verification code recognition loss function based on the Hamming distance is constructed.

[0026] Step S233, in combination with the verification code recognition loss threshold, the verification code identification data set is used as the supervision data, and the verification code image data set is used as the input data to train the target detection pre-training model. Specifically, the process of training the target detection pre-training model refers to combining the set verification code recognition loss threshold, using the verification code identification data set as the supervision data, and using the verification code image data set as the input data. Select a suitable target detection pre-training model, prepare the relevant training environment, set the training parameters, such as the number of training rounds, optimization algorithm, etc., and start training. The pictures in the verification code image data set are input into the model in sequence, and the model outputs the prediction results, which are compared with the correct identification in the verification code identification data set. If the prediction result is consistent with the correct identification, it indicates that the prediction is accurate; if there is a difference, the difference is analyzed, such as the number and position of character recognition errors. Continue to perform the above input, prediction, and comparison operations to process a large number of verification code images. After each round of training, the overall performance of the model in this round is comprehensively evaluated, including accuracy, error type and degree, etc. According to the evaluation results, if the model performance does not reach the set verification code recognition loss threshold, it means that the model needs to be improved. You may need to adjust the model parameters, such as network structure, learning rate, etc., or adopt a better training strategy to improve the model's recognition ability. Repeat the training and optimization process until the model performance reaches or exceeds the set threshold. Record the model optimization status and the time required to reach the threshold during the entire training process.

[0027] Step S234, when the verification code recognition loss function satisfies the verification code recognition loss threshold for a preset number of consecutive times, the target detection model is generated. Specifically, in the process of training the target detection pre-training model, the value of the verification code recognition loss function is calculated at the end of each training, the verification code recognition loss threshold and a certain number of consecutive satisfactions are preset, iterative training is performed, and the verification code recognition loss function value obtained from each training is continuously evaluated. If the loss function value is less than the set threshold in the first few trainings, the loss function value is also less than the threshold in the following multiple trainings. When this loss function value is less than the threshold for many consecutive times, it is considered that the model has good performance and stability, and the final target detection model is generated. The model has been fully trained and optimized, and can recognize and process the verification code more accurately, and is ready for subsequent practical applications. During the training process, it is ensured that the model has met the generation conditions. If there is a situation in which the loss threshold is not met in a continuous evaluation, the previous continuous count will be restarted until the number of times the set threshold is met is reached again, and then the target detection model will be generated.

[0028] Step S300, constructing a large model fine-tuning training data set, wherein the large model fine-tuning training data set includes a classification standard matching model construction data set, an interface classification model construction data set, and a table classification model construction data set.

[0029] Specifically, we construct a large model fine-tuning training dataset. In order to further fine-tune and optimize the pre-trained large model so that it can more accurately adapt to specific tasks and fields, the dataset is clearly divided into three main parts during the construction process. They are the classification standard matching model construction dataset, the interface classification model construction dataset, and the table classification model construction dataset. For the classification standard matching model construction dataset, it is necessary to collect and organize a large amount of data related to the classification standard, including various industry specifications, standard documents, existing classification cases, etc. Through the analysis and extraction of the data, a dataset that can be used to train the classification standard matching model is constructed so that it can understand and follow specific classification rules and standards. The construction of the interface classification model construction dataset focuses on the characteristics and attributes of different interfaces, and collects descriptive information of various interfaces. , parameter settings, usage scenarios and other data, and classify and label them so that the model can learn how to accurately classify and identify interfaces. The table classification model builds the data set, focusing on the processing of various types of tabular data, collecting tabular data of various types and structures, including but not limited to data tables, statistical tables, etc., and conducting detailed analysis and labeling of the data content, format, header information, etc. in the table so that the model can learn to accurately classify according to the characteristics of the table. During the entire construction process, it is necessary to ensure the accuracy, completeness and representativeness of the data, carefully screen, clean and preprocess the collected data, remove noise and invalid data, so as to improve the quality and availability of the data set, and reasonably adjust the scale and structure of the data set according to actual needs and the characteristics of the model to ensure that it can provide effective support for the fine-tuning training of large models.

[0030] In a possible implementation, step S300 further includes step S310, performing field frequency statistics on the vertical field pre-training data set to obtain a high-frequency field sequence. Specifically, statistical work is carried out on the vertical field pre-training data set. When performing field frequency statistics, the number of occurrences of each field in the data set is calculated and recorded. Through statistics, it can be determined which fields appear more frequently, thereby obtaining a high-frequency field sequence. Step S320, data interface statistics are performed on the vertical field pre-training data set to obtain a data interface set. Specifically, data interface statistics are performed to sort out the data interface situation in the data set, and all data interface information is summarized to form a data interface set. Step S330, table statistics are performed on the vertical field pre-training data set to obtain a data table set. Specifically, table statistics are performed to analyze various tables in the data set, including the structure, content, etc. of the table, thereby obtaining a data table set. Step S340, according to the high-frequency field sequence, the data table set, and in combination with the classification guide, the classification standard matching model is constructed to construct the data set. Specifically, when constructing a classification standard matching model to build a data set, based on the acquired high-frequency field sequence and data table set, and combined with the pre-established classification guidelines, fields closely related to the classification standard are screened out from the high-frequency field sequence, and tabular data that meets the classification requirements are selected from the data table set. According to the rules and requirements of the classification guidelines, these data are integrated and processed, and finally a classification standard matching model to build a data set is constructed. Step S350, construct the interface classification model to build a data set based on the data interface set and the high-frequency field sequence. Specifically, when constructing an interface classification model to build a data set, based on the data interface set and the high-frequency field sequence, representative and typical data interfaces are selected from the data interface set, and at the same time, according to the important fields related to the interface in the high-frequency field sequence, the interface and field information are associated and combined to form an interface classification model to build a data set. Step S360, construct the table classification model to build a data set based on the data table set and the high-frequency field sequence. Specifically, when constructing a table classification model to construct a data set, based on the data table set and the high-frequency field sequence, tabular data with different characteristics and types are selected from the data table set, combined with the fields in the high-frequency field sequence that have an important impact on table classification, through comprehensive analysis and collation of these tabular data and fields, a table classification model is constructed to construct various data sets that meet the needs, providing strong data support for subsequent model training and application.

[0031] In a possible implementation, step S310 further includes step S311, obtaining first vertical field pre-training data of the vertical field pre-training data set. Specifically, obtaining first vertical field pre-training data of the vertical field pre-training data set.

[0032] Step S312: extracting a plurality of field extraction results of the first vertical field pre-training data. Specifically, performing a field extraction operation on the first vertical field pre-training data to obtain a plurality of field extraction results.

[0033] Step S313, performing pairwise semantic similarity analysis on the plurality of field extraction results to obtain pairwise semantic similarity. Specifically, the first vertical field pre-training data is the basis for subsequent analysis and processing, performing pairwise semantic similarity analysis on the field extraction results, comparing the semantic similarity between each two field extraction results, and thus obtaining pairwise semantic similarity.

[0034] Step S314, based on the semantic consistency similarity, combined with the pairwise semantic similarity, cluster analysis is performed on the several field extraction results to obtain multiple cluster field extraction results, wherein the number of fields within multiple clusters of the multiple cluster field extraction results is the intra-cluster field frequency. Specifically, after obtaining the pairwise semantic similarity, based on the pre-set semantic consistency similarity standard, combined with the pairwise semantic similarity just obtained, cluster analysis is performed on the several field extraction results, through cluster analysis, semantically similar field extraction results are grouped into one cluster, and finally multiple cluster field extraction results are obtained, and the number of fields in each cluster is counted, and this number is called the intra-cluster field frequency.

[0035] Step S315, according to the field frequency within the cluster, delete the fields that are less than or equal to the preset field frequency from the several field extraction results, and sort them from large to small according to the field frequency, generate a first high-frequency field sequence, and add it to the high-frequency field sequence. Specifically, according to the field frequency within the cluster, delete the fields that are less than or equal to the preset field frequency in the several field extraction results, remove the fields with low frequency of occurrence and not very important for subsequent analysis, sort the remaining fields from large to small according to the field frequency, generate a first high-frequency field sequence, and add it to the high-frequency field sequence, so as to provide more valuable and representative data for subsequent data analysis and model training. The scheme screens out important and representative high-frequency field sequences from the original data, which helps to improve the efficiency of data processing and the accuracy of the model.

[0036] In a possible implementation, step S340 includes step S341, extracting a first-level constraint category set, a second-level constraint category set and a third-level constraint category set according to the classification guide, wherein the second-level constraint category set is a subcategory set of the first-level constraint category set, and the third-level constraint category set is a subcategory set of the second-level constraint category set. Specifically, a complete classification guide is obtained, which systematically plans and defines the categories in related fields. The classification guide is analyzed, and the division logic and relationship of constraint categories at different levels are clarified from the overall structure. In the specific extraction process, the first-level constraint category set is first screened out. The first-level category is a relatively macro and basic classification summary of the entire field. On the basis of the determined first-level constraint category set, the subdivided content is further explored to extract the second-level constraint category sets belonging to each first-level category. Each second-level category belongs to a specific first-level category, forming a hierarchical sub-category relationship. For the second-level constraint category set, the third-level constraint category set is extracted as a sub-category set of the second-level category. Through such a rigorous and orderly extraction process, it is ensured that the first-level constraint category set, the second-level constraint category set and the third-level constraint category set that are interrelated and hierarchical are accurately obtained, providing a clear and accurate classification basis for subsequent work based on these category sets.

[0037] Step S342, constructing a first-level classification standard matching model to construct a data set according to the data table set, the high-frequency field sequence and the first-level constraint category set. Specifically, obtain the existing data table set, high-frequency field sequence and first-level constraint category set, analyze the structure, content and data distribution of each data table, the field information, data type and the correlation between data contained in each table, conduct in-depth research on the high-frequency field sequence, clarify that the field appears frequently in the data, and the high-frequency field is often of great significance for the classification and feature extraction of data. Compare with the first-level constraint category set, understand the specific definition and scope of each first-level constraint category, build a data set, filter out the data table related to the first-level constraint category from the data table set, extract the high-frequency field data from the selected data table according to the high-frequency field sequence, in the process of extracting data, according to the requirements of the first-level constraint category, preliminarily classify and mark the data, organize and clean the extracted and marked data, remove duplicate, erroneous or irrelevant data, ensure the accuracy and consistency of the data, and combine the organized data into a complete data set, that is, the first-level classification standard matching model to build a data set. The data set contains the data related to the first-level constraint category that has been screened, extracted, classified and organized, providing a targeted and high-quality data foundation for subsequent model training.

[0038] Step S343, constructing a secondary classification standard matching model to construct a data set based on the data table set, the high-frequency field sequence and the secondary constraint category set. Specifically, in a similar manner, combining the data table set, the high-frequency field sequence and the secondary constraint category set, constructing a secondary classification standard matching model to construct a data set, extracting corresponding data for constructing a data set based on data characteristics and specific provisions of the secondary constraint category.

[0039] Step S344, constructing a three-level classification standard matching model construction data set based on the data table set, the high-frequency field sequence and the three-level constraint category set. Specifically, based on the data table set, the high-frequency field sequence and the three-level constraint category set, constructing a three-level classification standard matching model construction data set, fully considering the detailed requirements of the three-level constraint category and the relevance of the data.

[0040] Step S345, adding the first-level classification standard matching model construction data set, the second-level classification standard matching model construction data set and the third-level classification standard matching model construction data set into the classification standard matching model construction data set. Specifically, all the constructed first-level classification standard matching model construction data set, the second-level classification standard matching model construction data set and the third-level classification standard matching model construction data set are added into the classification standard matching model construction data set.

[0041] In a possible implementation, step S350 includes step S351, obtaining the first interface name of the data interface set. Specifically, the first interface name is obtained from the data interface set to determine a specific object for subsequent analysis and processing.

[0042] Step S352, extracting the first interface high-frequency field of the first interface name from the high-frequency field sequence. Specifically, in the high-frequency field sequence, according to the first interface name that has been obtained, extract the first interface high-frequency field related thereto, and screen out the high-frequency field closely related to the specific interface, which is of great significance for the classification and feature description of the interface.

[0043] Step S353, construct the first construction data according to the first interface name, the first interface high-frequency field, and the classification standard, and add it to the interface classification model construction data set, wherein the category set of the classification standard corresponds to the high-frequency field one by one. Specifically, the first construction data is constructed according to the first interface name, the first interface high-frequency field, and the classification standard, and the classification standard needs to be referred to. The classification standard is a systematic and well-organized framework, in which the category set and the high-frequency field have a one-to-one correspondence. The interface name and the high-frequency field are matched and associated with the corresponding category, and the specific category corresponding to the high-frequency field is determined. Combined with the determined category and the first interface name, the first construction data is constructed. The construction data not only includes the name of the interface, but also clearly marks the classification category to which it belongs and the detailed information of the relevant high-frequency field. After the construction of the first construction data is completed, it is added to the interface classification model construction data set. The adding operation must ensure that the data is accurately integrated into the data set, and will not destroy the original structure and logic of the data set. The first construction data that meets the requirements is constructed and made part of the interface classification model construction data set, providing strong support for the subsequent training and optimization of the interface classification model.

[0044] Step S354, wherein the classification standard includes a primary classification standard, a secondary classification standard and a tertiary classification standard. Specifically, in the whole scheme, the classification standards are divided into primary classification standards, secondary classification standards and tertiary classification standards, forming a hierarchical and gradually refined classification system. The primary classification standard provides the most macro and basic classification framework, which is a rough division of the overall category, covering a wide range of basic category definitions, including "data input interface", "data output interface", "data processing interface", etc. The secondary classification standard is a more detailed division based on the primary classification standard, further subdividing each category in the primary classification, including "manual input interface", "file import interface", "network data acquisition interface", etc. The tertiary classification standard is the most specific and detailed level in the classification system, and the categories under the secondary classification standard are further subdivided and clarified. In actual applications, when classifying an object, its approximate category will be determined according to the primary classification standard first, and then further refined according to the secondary classification standard. Finally, the most accurate and specific classification positioning is achieved through the tertiary classification standard, which meets the needs of accurate classification of objects at different precisions and granularities, and provides a clear and hierarchical basis for related analysis, processing and decision-making.

[0045] Step S400, performing pre-training according to the pre-training data set to generate a transportation data classification pre-training model.

[0046] Specifically, prepare a pre-training data set, which contains data related to transportation, such as operating data of different types of transportation tools, traffic flow data, traffic accident data, road condition data, etc. Select a deep learning model to process the data, and input the pre-training data set into the selected model for training. During the training process, the model will automatically learn the features and patterns in the data and determine the relationship between different data, such as the rules of operating data of different transportation tools, the trend of traffic flow over time, etc. According to the input data, the internal parameters are continuously adjusted to optimize the fitting and prediction capabilities of the data. The adjustment of parameters is achieved through stochastic gradient descent. In each round of training, the model will calculate the error between its predicted results and the real data, and update the parameters based on this error. When the performance of the model on the validation set reaches a certain standard, or the number of training rounds reaches the preset value, the pre-training is considered to be completed, and a pre-training model for transportation data classification is generated.

[0047] Step S500, fine-tuning the transportation data classification pre-training model according to the classification standard matching model construction data set, the interface classification model construction data set, and the table classification model construction data set to generate a transportation data classification large model.

[0048] Specifically, three important data sets are prepared: classification standard matching model construction data set, interface classification model construction data set and table classification model construction data set. A pre-trained model for transportation data classification is obtained. The classification standard matching model construction data set is input into the pre-trained model for training. According to the matching between the classification standard in the data set and the actual data, the parameters of the model are adjusted to better understand and adapt to the specific classification standard. The interface classification model construction data set is used to further fine-tune the model, classify various transportation data interfaces, and improve the recognition and classification capabilities of interface-related features. The table classification model construction data set is used for training to determine the characteristics of transportation data tables of different types and structures to improve the classification accuracy of table data. Each time the data set is used for training, the model will continuously calculate the difference between the predicted results and the actual data, and adjust the model parameters through the optimization algorithm according to the difference to gradually optimize the performance of the model. After multiple rounds of iterative training, when the model's performance on the validation set meets expectations or meets other set evaluation indicators, the fine-tuning training is completed, and a large transportation data classification model for classifying transportation data is generated, providing more reliable and effective support for practical applications.

[0049] In a possible implementation, step S500 includes step S510, fine-tuning the transportation data classification pre-training model according to the classification standard matching model construction data set to construct a classification standard matching model. Specifically, a classification standard matching model construction data set is obtained, the transportation data classification pre-training model is used as a basis, and the data in the classification standard matching model construction data set is input into the pre-training model to start fine-tuning training. During the training process, the model will continuously adjust its own parameters according to the relevant features and patterns in the data set that match the classification standard. For example, the model will learn how to identify and distinguish different types of transportation data according to specific standards. Through multiple iterations and optimizations, when the performance of the model on the data set reaches a certain excellent standard, the fine-tuning training is completed, thereby constructing a classification standard matching model that can accurately match the classification standard.

[0050] Step S520, fine-tune the transportation data classification pre-trained model according to the interface classification model construction data set and the table classification model construction data set to construct an interface / table classification model. Specifically, obtain the interface classification model construction data set and the table classification model construction data set, and again use the transportation data classification pre-trained model as a starting point, first use the interface classification model construction data set for fine-tuning training, focus on interface-related features and classification patterns, such as data formats and transmission methods of different interfaces, and continue training using the table classification model construction data set, so that the model can master the structure and content features of various transportation data tables, so as to accurately classify different types of table data, and after repeated training and optimization on the two data sets, successfully construct an interface / table classification model.

[0051] Step S530, the classification standard matching model and the interface / table classification model are integrated to generate the transportation data classification model. Specifically, the previously constructed classification standard matching model and the interface / table classification model are integrated. During the integration process, it is necessary to deal with the problems of parameter integration, structural connection and functional coordination of the two models, so that the two models can work together and complement each other, ensuring that in the classification process, the matching of classification standards and the classification of interfaces / tables can cooperate with each other, and jointly contribute to the accurate classification of transportation data, and successfully generate a transportation data classification model that can comprehensively and accurately classify transportation data.

[0052] Step S600: classify the transportation data according to the transportation data classification model.

[0053] Specifically, prepare the transportation data that needs to be classified. The data comes from traffic monitoring equipment, transportation management systems, vehicle sensors, etc., and its forms include structured data (such as tables), semi-structured data (such as XML or JSON format) and unstructured data (such as text descriptions, images). Input the prepared transportation data into the constructed transportation data classification model. Inside the model, the data will undergo a series of processing and calculations to extract the features in the input data. The features include numerical features, semantic features, spatial features, etc. The model analyzes and judges the extracted features based on the knowledge and patterns learned from previous training, matches and compares the features of the input data with the existing classification patterns, and calculates the probability or score that the input data belongs to each classification category. Based on the calculated probability or score, the model determines the category to which the input data belongs and outputs the classification result. After obtaining the classification result, it is further evaluated and analyzed. If the classification result is inaccurate or does not meet expectations, you can check the quality of the input data and the performance of the model, or consider further optimization and improvement of the model.

[0054] The embodiment of the present application configures a vertical field training data source link, collects a vertical field pre-training data set and mixes it with a general pre-training data set with a data volume 5-10 times that of the vertical field to generate a pre-training data set, constructs a large model fine-tuning training data set containing multiple data sets, generates a transportation data classification pre-training model based on the pre-training data set, fine-tunes it with a specific data set to obtain a transportation data classification large model, and finally uses the large model to classify transportation data, thereby achieving the technical effect of improving the accuracy, comprehensiveness and adaptability of transportation data classification.

[0055] In the above, refer to Figure 1 The transportation data classification method based on the vertical domain big model according to the embodiment of the present invention is described in detail. Figure 2 A transportation data classification device based on a vertical domain big model according to an embodiment of the present invention is described.

[0056] The transportation data classification device based on the vertical domain big model according to the embodiment of the present invention is used to solve the technical problems of low classification accuracy and reliability caused by the difficulty of data collection, poor quality, incomplete and incomplete classification standards, single model and insufficient adaptability in the existing transportation data classification technology, and achieve the technical effect of improving the accuracy and reliability of transportation data classification. The transportation data classification device based on the vertical domain big model includes: a configuration link module 10, a pre-trained data set generation module 20, a data set construction module 30, a pre-trained model generation module 40, a classification big model generation module 50, and a data classification module 60.

[0057] The configuration link module 10 is used to configure the vertical field training data source link; The pre-training data set generation module 20 is used to collect a vertical field pre-training data set according to the vertical field training data source link, mix it with the general pre-training data set, and generate a pre-training data set, wherein the first data volume of the general pre-training data set is 5 to 10 times the second data volume of the vertical field pre-training data set; The data set construction module 30 is used to construct a large model fine-tuning training data set, wherein the large model fine-tuning training data set includes a classification standard matching model construction data set, an interface classification model construction data set, and a table classification model construction data set; The pre-training model generation module 40 is used to perform pre-training according to the pre-training data set to generate a transportation data classification pre-training model; The classification large model generation module 50 is used to fine-tune the transportation data classification pre-training model according to the classification standard matching model construction data set, the interface classification model construction data set, and the table classification model construction data set to generate a transportation data classification large model; The data classification module 60 is used to classify the transportation data according to the transportation data classification model.

[0058] The specific configuration of the pre-training data set generation module 20 will be described in detail below. As described above, according to the vertical field training data source link, a vertical field pre-training data set is collected and mixed with a general pre-training data set to generate a pre-training data set, wherein the first data volume of the general pre-training data set is 5 to 10 times the second data volume of the vertical field pre-training data set, and the pre-training data set generation module 20 may further include: a first data source link unit is used to obtain a first data source link of the vertical field training data source link; a verification code image acquisition unit is used to access the verification code link when the first data source link has a verification code link to obtain a verification code image; an initial acquisition data unit is used to identify the verification code image through a target detection model, obtain verification code information, access the first data source link through the verification code information, and obtain the first data source initial acquisition data; the first data source initial acquisition data acquisition unit is used to access the first data source link when the first data source link does not have a verification code link to obtain the first data source initial acquisition data; a pre-training data set adding unit is used to perform redundant data cleaning on the first data source initial acquisition data, generate the first data source selected acquisition data, and add it to the vertical field pre-training data set, wherein the redundant data types include preset characters, semantic-free data, and chart data.

[0059] Among them, the verification code image is identified by the target detection model to obtain the verification code information, the first data source link is accessed by the verification code information, and the initial collection data of the first data source is obtained. The initial collection data unit may further include: a configuration subunit for configuring a verification code image data set and a verification code identification data set; a loss function construction subunit for constructing a verification code recognition loss function based on the Hamming distance; a target detection pre-trained model training subunit for combining the verification code recognition loss threshold, using the verification code identification data set as supervision data, and using the verification code image data set as input data to train the target detection pre-trained model; a target detection model generation subunit for generating the target detection model when the verification code recognition loss function meets the verification code recognition loss threshold for a preset number of consecutive times.

[0060] The specific configuration of the data set construction module 30 will be described in detail below. As described above, a large model fine-tuning training data set is constructed, wherein the large model fine-tuning training data set includes a classification standard matching model construction data set, an interface classification model construction data set, and a table classification model construction data set. The data set construction module 30 may further include: a high-frequency field sequence acquisition unit is used to perform field frequency statistics on the vertical field pre-training data set to obtain a high-frequency field sequence; a data interface set acquisition unit is used to perform data interface statistics on the vertical field pre-training data set to obtain a data interface set; a data table set acquisition unit is used to perform table statistics on the vertical field pre-training data set to obtain a data table set; a classification standard matching model construction data set construction unit is used to construct the classification standard matching model construction data set according to the high-frequency field sequence, the data table set, and the classification guide; an interface classification model construction data set construction unit is used to construct the interface classification model construction data set according to the data interface set and the high-frequency field sequence; a table classification model construction data set construction unit is used to construct the table classification model construction data set according to the data table set and the high-frequency field sequence.

[0061] Among them, the field frequency statistics of the vertical field pre-training data set are performed to obtain a high-frequency field sequence, and the high-frequency field sequence acquisition unit may further include: a first vertical field pre-training data acquisition subunit is used to obtain the first vertical field pre-training data of the vertical field pre-training data set; a field extraction result subunit is used to extract several field extraction results of the first vertical field pre-training data; a pairwise semantic similarity acquisition subunit is used to perform pairwise semantic similarity analysis on the several field extraction results to obtain pairwise semantic similarity; a multi-cluster field extraction result acquisition subunit is used to perform cluster analysis on the several field extraction results based on semantic consistency similarity and the pairwise semantic similarity to obtain multi-cluster field extraction results, wherein the number of fields within multiple clusters of the multi-cluster field extraction results is the field frequency within the cluster; a first high-frequency field sequence generation subunit is used to delete fields that are less than or equal to the preset field frequency from the several field extraction results according to the field frequency within the cluster, and sort them from large to small according to the field frequency to generate a first high-frequency field sequence, and add them to the high-frequency field sequence.

[0062] Wherein, according to the high-frequency field sequence, the data table set, and in combination with the classification guide, the classification standard matching model is constructed to construct a data set, and the classification standard matching model data set construction unit may further include: a category set extraction subunit is used to extract a first-level constraint category set, a second-level constraint category set, and a third-level constraint category set according to the classification guide, wherein the second-level constraint category set is a sub-category set of the first-level constraint category set, and the third-level constraint category set is a sub-category set of the second-level constraint category set; a first-level classification standard matching model data set construction subunit is used to construct a The first-level classification standard matching model constructs a data set; the second-level classification standard matching model constructs a data set construction subunit, which is used to construct a second-level classification standard matching model constructs a data set according to the data table set, the high-frequency field sequence and the second-level constraint category set; the third-level classification standard matching model constructs a data set construction subunit, which is used to construct a third-level classification standard matching model constructs a data set according to the data table set, the high-frequency field sequence and the third-level constraint category set; the adding subunit is used to add the first-level classification standard matching model constructs a data set, the second-level classification standard matching model constructs a data set and the third-level classification standard matching model constructs a data set into the classification standard matching model constructs a data set.

[0063] Among them, the interface classification model construction data set is constructed according to the data interface set and the high-frequency field sequence, and the interface classification model construction data set construction unit may further include: a first interface name acquisition subunit is used to obtain the first interface name of the data interface set; a first interface extraction subunit is used to extract the first interface high-frequency field of the first interface name from the high-frequency field sequence; a first construction data addition subunit is used to construct the first construction data according to the first interface name, the first interface high-frequency field, and the classification standard, and add it to the interface classification model construction data set, wherein the category set of the classification standard corresponds one-to-one to the high-frequency field; the classification standard subunit is used, and the classification standard includes a first-level classification standard, a second-level classification standard and a third-level classification standard.

[0064] The specific configuration of the classification large model generation module 50 will be described in detail below. As described above, the transportation data classification pre-training model is fine-tuned and trained according to the classification standard matching model construction data set, the interface classification model construction data set, and the table classification model construction data set to generate the transportation data classification large model. The classification large model generation module 50 may further include: a classification standard matching model construction unit is used to fine-tune and train the transportation data classification pre-training model according to the classification standard matching model construction data set to construct a classification standard matching model; an interface / table classification model construction unit is used to fine-tune and train the transportation data classification pre-training model according to the interface classification model construction data set and the table classification model construction data set to construct an interface / table classification model; and a transportation data classification large model generation unit is used to fuse the classification standard matching model with the interface / table classification model to generate the transportation data classification large model.

[0065] The transportation data classification method device based on the vertical field big model provided in the embodiment of the present invention can execute the transportation data classification method based on the vertical field big model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0066] Although the present application makes various references to certain modules in the device according to the embodiment of the present application, any number of different modules can be used and run on the user terminal and / or server, and the various units and modules included are only divided according to the functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing from each other, and are not used to limit the scope of protection of the present invention. The above specific implementation does not constitute a limitation on the scope of protection of the present application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiment and the desired results can still be achieved. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A transportation data classification method based on a vertical domain big model, characterized in that: include: Configure vertical field training data source links; According to the vertical field training data source link, a vertical field pre-training data set is collected and mixed with a general pre-training data set to generate a pre-training data set, wherein a first data volume of the general pre-training data set is 5 to 10 times the second data volume of the vertical field pre-training data set; Constructing a large model fine-tuning training data set, wherein the large model fine-tuning training data set includes a classification standard matching model construction data set, an interface classification model construction data set, and a table classification model construction data set; Perform pre-training based on the pre-training data set to generate a transportation data classification pre-training model; Fine-tune the transportation data classification pre-training model according to the classification standard matching model construction data set, the interface classification model construction data set, and the table classification model construction data set to generate a transportation data classification large model; The transportation data is classified according to the transportation data classification model.

2. The method for classifying transportation data based on a vertical domain big model as claimed in claim 1, characterized in that: According to the vertical field training data source link, a vertical field pre-training data set is collected, including: Obtain a first data source link of the vertical field training data source link; When the first data source link has a verification code link, access the verification code link to obtain a verification code image; The verification code image is identified by a target detection model to obtain verification code information, and the first data source link is accessed by the verification code information to obtain initial collection data of the first data source; When the first data source link does not have a verification code link, accessing the first data source link to obtain the first data source initial collection data; Redundant data is cleaned for the initial data collected from the first data source to generate selected data collected from the first data source, which is added to the vertical field pre-training data set, wherein the redundant data types include preset characters, semantic-free data, and chart data.

3. The method for classifying transportation data based on a vertical domain big model as claimed in claim 2 is characterized in that: The verification code image is identified by a target detection model to obtain verification code information, and the first data source link is accessed by the verification code information to obtain the initial collection data of the first data source, which includes: Configure the verification code image dataset and verification code logo dataset; Construct the verification code recognition loss function based on the Hamming distance; Combined with the verification code recognition loss threshold, the verification code identification dataset is used as supervision data, and the verification code image dataset is used as input data to train the target detection pre-training model; When the verification code recognition loss function satisfies the verification code recognition loss threshold for a preset number of consecutive times, the target detection model is generated.

4. The method for classifying transportation data based on a vertical domain big model as claimed in claim 1, characterized in that: Build a large model fine-tuning training dataset, including: Performing field frequency statistics on the vertical field pre-training data set to obtain a high-frequency field sequence; Performing data interface statistics on the vertical field pre-training data set to obtain a data interface set; Performing table statistics on the vertical field pre-training data set to obtain a data table set; According to the high-frequency field sequence, the data table set, and the classification guide, the classification standard matching model is constructed to construct a data set; Constructing the interface classification model to construct a data set according to the data interface set and the high-frequency field sequence; The table classification model is constructed according to the data table set and the high-frequency field sequence to construct a data set.

5. The method for classifying transportation data based on a vertical domain big model as claimed in claim 4 is characterized in that: Perform field frequency statistics on the vertical field pre-training data set to obtain a high-frequency field sequence, including: Obtain first vertical field pre-training data of the vertical field pre-training data set; Extracting several field extraction results of the first vertical field pre-training data; Performing pairwise semantic similarity analysis on the plurality of field extraction results to obtain pairwise semantic similarity; Based on the semantic consistency similarity, cluster analysis is performed on the plurality of field extraction results in combination with the pairwise semantic similarity to obtain a multi-cluster field extraction result, wherein the number of fields within the multiple clusters of the multi-cluster field extraction result is the frequency of the field within the cluster; According to the field frequency within the cluster, fields with a frequency less than or equal to a preset field frequency are deleted from the plurality of field extraction results, and the fields are sorted from large to small according to the field frequency to generate a first high-frequency field sequence, which is added to the high-frequency field sequence.

6. The method for classifying transportation data based on a vertical domain big model as claimed in claim 4, characterized in that: According to the high-frequency field sequence, the data table set, and the classification guide, the classification standard matching model is constructed to construct a data set, including: According to the classification guide, extracting a first-level constraint category set, a second-level constraint category set and a third-level constraint category set, wherein the second-level constraint category set is a subcategory set of the first-level constraint category set, and the third-level constraint category set is a subcategory set of the second-level constraint category set; Constructing a first-level classification standard matching model to construct a data set according to the data table set, the high-frequency field sequence and the first-level constraint category set; Constructing a secondary classification standard matching model to construct a data set according to the data table set, the high-frequency field sequence and the secondary constraint category set; Constructing a three-level classification standard matching model to construct a data set according to the data table set, the high-frequency field sequence and the three-level constraint category set; The first-level classification standard matching model construction data set, the second-level classification standard matching model construction data set and the third-level classification standard matching model construction data set are added into the classification standard matching model construction data set.

7. The method for classifying transportation data based on a vertical domain big model as claimed in claim 4, characterized in that: Constructing the interface classification model and constructing the data set according to the data interface set and the high-frequency field sequence includes: Obtaining a first interface name of the data interface set; Extracting a first interface high-frequency field of the first interface name from the high-frequency field sequence; Constructing first construction data according to the first interface name, the first interface high-frequency field, and the classification standard, and adding the first construction data into the interface classification model construction data set, wherein the category set of the classification standard corresponds to the high-frequency field one by one; Among them, the classification standards include primary classification standards, secondary classification standards and tertiary classification standards.

8. The method for classifying transportation data based on a vertical domain big model as claimed in claim 1, characterized in that: The transportation data classification pre-training model is fine-tuned and trained according to the classification standard matching model construction data set, the interface classification model construction data set, and the table classification model construction data set to generate a transportation data classification large model, including: Constructing a data set based on the classification standard matching model to fine-tune the transportation data classification pre-training model and construct a classification standard matching model; According to the interface classification model to construct a data set, and the table classification model to construct a data set, fine-tune the transportation data classification pre-training model to construct an interface / table classification model; The classification standard matching model and the interface / table classification model are integrated to generate the transportation data classification model.

9. A transportation data classification device based on a vertical domain big model, characterized in that: The device is used to implement the transportation data classification method based on the vertical field large model according to any one of claims 1 to 8, and the device comprises: A configuration link module, wherein the configuration link module is used to configure a vertical field training data source link; A pre-training data set generation module, the pre-training data set generation module is used to collect a vertical field pre-training data set according to the vertical field training data source link, mix it with the general pre-training data set, and generate a pre-training data set, wherein the first data volume of the general pre-training data set is 5 to 10 times the second data volume of the vertical field pre-training data set; A data set construction module, wherein the data set construction module is used to construct a large model fine-tuning training data set, wherein the large model fine-tuning training data set includes a classification standard matching model construction data set, an interface classification model construction data set, and a table classification model construction data set; A pre-training model generation module, wherein the pre-training model generation module is used to perform pre-training according to the pre-training data set to generate a pre-training model for transportation data classification; A classification large model generation module, the classification large model generation module is used to fine-tune the transportation data classification pre-training model according to the classification standard matching model construction data set, the interface classification model construction data set, and the table classification model construction data set to generate a transportation data classification large model; A data classification module is used to classify transportation data according to the transportation data classification model.

Images