Data industry classification method and device based on large model

Abstract

The invention provides a data industry classification method and device based on a large model. Formulating a third-level industry classification standard, and determining division boundaries and affiliation relationships of the first-level industry, the second-level industry and the third-level industry; the method comprises the following steps: collecting original industry data from a data source, and performing data cleaning, data standardization, word segmentation processing, vectorization conversion and data enhancement on the original industry data; selecting a large-scale deep learning model adaptive to an industry classification task, dividing the preprocessed original industry data into a training set, a verification set and a test set, performing targeted adjustment on a pre-training model, and completing model training in combination with optimization strategies such as data enhancement, an attention mechanism and regularization; performing feature extraction and classification decision on newly input industry data by using the trained optimization model, and outputting a three-level industry classification result; and verifying the reliability of the classification result through a confidence evaluation mechanism. According to the scheme, data industry classification can be realized more intelligently and efficiently.

Claims

1. A data industry classification method based on a large model, characterized in that, include: Establish a three-tier industry classification standard to determine the boundaries and affiliations of primary, secondary, and tertiary industries. The coverage of primary industries is greater than that of secondary industries, and the coverage of secondary industries is greater than that of tertiary industries. Raw industry data is collected from the data source, and data cleaning, data standardization, word segmentation, vectorization transformation and data enhancement are performed on the raw industry data. We selected a large-scale deep learning model suitable for industry classification tasks, divided the preprocessed raw industry data into training set, validation set and test set, made targeted adjustments to the pre-trained model, and completed the model training by combining optimization strategies such as data augmentation, attention mechanism and regularization. The trained and optimized model is used to extract features and make classification decisions on newly input industry data, and output three-level industry classification results; the reliability of the classification results is verified through a confidence evaluation mechanism.

2. The method according to claim 1, characterized in that, The formulation of the three-tier industry classification standard includes: A parallel labeling mode is adopted to simultaneously label the same batch of data with first-level, second-level, and third-level industry tags. The multi-bypass operation mechanism enables cross-validation of label results at different levels. If there is a mismatch between the labels at different levels, the automatic validation rules are triggered to recheck the fit between the data features and the classification standards until the label logic is consistent, thereby improving the accuracy and consistency of the classification results.

3. The method according to claim 1, characterized in that, The process of collecting raw industry data from data sources includes: Build a data source management system and store connection information for various data sources through JSON or YAML configuration files, including the database address, port, account password, API access URL, request parameters, target site address of web crawler, and file system storage path. For databases, connections are established via JDBC or dedicated drivers to extract data in batches. For API interfaces, Python's requests library is used to send requests, with rate limiting and error retry strategies implemented. When the request frequency exceeds the limit or an interface response is abnormal, the request interval is automatically adjusted or a retry process is triggered. For web platforms, the Scrapy framework or a custom crawler is used to scrape data. Combined with anti-crawler strategies, BeautifulSoup or lxml tools are used to parse HTML pages and extract structured data. For local files and cloud storage files, batch data import is achieved through file read / write interfaces. A data scheduling system is built based on Spark's distributed task queue. Task priorities are set according to the importance level of the data source and the real-time requirements of the data. The task monitoring module tracks and collects the running status of tasks in real time, triggers alarms for abnormal tasks, and automatically attempts to repair or restart them.

4. The method according to claim 1, characterized in that, The process of performing data cleaning, data standardization, word segmentation, vectorization transformation, and data enhancement on the original industry data includes: Remove duplicate data based on unique identifiers; for missing numerical values, fill with the mean, median, or linear interpolation method according to the data distribution characteristics; for missing text values, fill with the default value or delete the missing samples directly. Regular expressions combined with BeautifulSoup are used to process text data, unify character encoding, and remove irrelevant information; time data in different formats are uniformly converted into a standard format, and numerical data are normalized or standardized. For text data, word segmentation tools are used for word segmentation. After removing stop words, the segmentation results are converted into fixed-dimensional vectors using a pre-trained word vector model. For image data, image preprocessing tools are used to normalize the size and standardize the pixel values ​​before converting the data into vector data in tensor form. For text data, synonym replacement is performed, and training data is expanded by random insertion, random deletion, and sentence rearrangement; for image data, expanded samples are generated by random cropping, flipping, rotating, and brightness adjustment.

5. The method according to claim 1, characterized in that, The process involves selecting a large-scale deep learning model suitable for the industry classification task, dividing the preprocessed original industry data into training, validation, and test sets, and making targeted adjustments to the pre-trained model. This includes combining data augmentation, attention mechanisms, regularization, and other optimization strategies to complete model training, including: Choose the appropriate large-scale deep learning model based on the data type. For text-based industry data, use BERT, RoBERTa, or Transformer models and load pre-trained weights through Hugging Face's transformers library. For image-based industry data, use ResNet, EfficientNet, or Vision Transformer models and load pre-trained weights through PyTorch's torchvision library. The preprocessed augmented data was divided into training, validation and test sets according to a preset ratio using the train_test_split tool of Scikit-learn. The learning rate was set to 2e-5, the batch size to 32, and the number of training rounds to 3-5. The AdamW optimizer was used and a weight decay coefficient was configured to prevent the model from overfitting. The cross-entropy loss function was selected as the basis for loss calculation for multi-class classification tasks. A multi-head attention mechanism is introduced during model training; regularization is implemented using PyTorch's nn.Dropout or TensorFlow's tf.keras.layers.Dropout; and a Cosine Annealing learning rate scheduling strategy is adopted to dynamically adjust the learning rate during training, enabling the model to converge in the later stages of training. After each training round, the model performance is evaluated using a validation set. If the model performance does not reach the preset threshold, the model parameters, optimization strategies, or data augmentation methods are adjusted, and the training is repeated until the model's performance on the validation set is stable and meets the target.

6. The method according to claim 1, characterized in that, The optimized model, after training, is used to extract features and make classification decisions on newly input industry data, outputting three-level industry classification results, including: For text data, deep semantic feature vectors are extracted using a trained BERT or Transformer model; for image data, visual feature vectors are extracted using a ResNet or EfficientNet model; for structured data, key features are extracted using a fully connected neural network or XGBoost model; and the extracted high-dimensional feature vectors are reduced to 64 or 128 dimensions using PCA or t-SNE algorithms. The appropriate activation function is selected based on the type of classification task. For multi-class tasks, the Softmax function is used to output the probability distribution of each class, and for binary classification tasks, the Sigmoid function is used to output the classification probability. A classification threshold is set, and when the probability of a certain class is higher than the threshold, the data is determined to belong to that class. A multi-model fusion strategy is adopted, which combines the prediction results of multiple different models through weighted averaging or voting mechanisms. For real-time data stream scenarios, the computational process of feature extraction and classification decision is optimized, and two modes, batch processing and streaming processing, are set up. The batch processing mode dynamically adjusts the batch size according to hardware resources, while the streaming processing mode realizes real-time data input, real-time processing and real-time output.

7. The method according to claim 1, characterized in that, The verification of the reliability of the classification results through the confidence assessment mechanism includes: The confidence score is calculated by taking the class probability output during the classification decision process as the confidence score value and extracting the maximum value among the class probabilities as the confidence score of the classification result. Set a confidence threshold. If the confidence of the classification result is higher than the confidence threshold, the classification result is output directly; if the confidence is lower than the confidence threshold, it is marked as a low-confidence result and a manual review process is triggered. Monte Carlo Dropout or Bayesian methods are used to quantitatively evaluate the predictive uncertainty of the model. Anomaly detection algorithms are combined to analyze the data characteristics of low confidence results, identify special scenarios such as data noise, missing features, or classification criteria not covered, and feed back the identified anomalies to the data preprocessing module or classification criteria development stage.

8. A data industry classification device based on a large model, characterized in that, include: The standard-setting unit is used to develop three-level industry classification standards, determine the boundaries and affiliations of first-level, second-level, and third-level industries, with the first-level industry covering a wider range than the second-level industry, and the second-level industry covering a wider range than the third-level industry. The data processing unit is used to collect raw industry data from the data source and perform data cleaning, data standardization, word segmentation, vectorization conversion and data enhancement on the raw industry data. The model training unit is used to select a large-scale deep learning model that is suitable for the industry classification task. It divides the pre-processed raw industry data into training set, validation set and test set, makes targeted adjustments to the pre-trained model, and completes the model training by combining optimization strategies such as data augmentation, attention mechanism and regularization. The decision-making unit is used to extract features and make classification decisions on newly input industry data using the trained optimized model, and output three-level industry classification results; the reliability of the classification results is verified through a confidence evaluation mechanism.

9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data industry classification method based on a large model as described in any one of claims 1-7.

10. A computer-readable medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data industry classification method based on a large model as described in any one of claims 1-7.

Images