Intelligent customs commodity classification model based on feature engineering and its establishment method and application

Abstract

The present invention discloses a method for establishing an intelligent customs commodity classification model based on feature engineering, including combing the classification feature elements of the target tax number according to the commodity standard declaration catalog; using regular expressions to perform preliminary extraction of features; manually correcting and annotating the features extracted by the program; digitally annotating the features; combining Bert and CRF, constructing an artificial intelligence training model and using annotated data to carry out feature recognition training; evaluating the training results until the recognition accuracy reaches the standard and the training is stopped; using the Bert model as a base for training, selecting a suitable loss function, defining a gradient update strategy, designing a training cycle and a stopping strategy, and using early stopping and other technologies to prevent overfitting and improve the generalization ability of the model. The present invention can not only ensure a higher recall rate and precision rate, but its algorithm and process design maintain a lower requirement for computing power. The actual measurement shows that 8G video memory can meet the requirements, and the inference speed reaches 150,000-180,000 per hour.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent commodity classification, and relates to an intelligent customs commodity classification method and classification system based on feature engineering and applications thereof. Background Art

[0002] At present, when customs or enterprises are looking for customs codes corresponding to commodities, they mostly rely on experience to find the corresponding customs codes in the huge customs database for classification. This is entirely based on manual experience, which has a high error rate and great business risks, and is inefficient and cannot improve the declaration speed and review efficiency. There are also a few public intelligent customs commodity classification methods, but there are still some defects:

[0003] 1. Methods based on empirical databases, such as the patent with publication number CN109559029A. Its essence is to search for historical classification results of the same or similar products and give classification suggestions through data matching. This method still remains at the data comparison level and does not use artificial intelligence algorithms to analyze and classify the semantics of the declared content. The data comparison method relies on a known database and it is difficult to give accurate product classification for unknown or new products.

[0004] 2. Commodity classification methods based on machine learning, such as Ruan Qiming, Guo Yi, Zheng Nan, et al. Customs declaration commodity classification algorithm based on hierarchical multi-task BERT[J]. Computer Applications, 2022, 42(1): 71-77. DOI: 10.11772 / j.issn.1001-9081.2021010122.

[0005] This method uses the pre-trained BERT as the base and builds the HM-Bert architecture with an accuracy of 80.03%. This method uses a general language model to carry out commodity classification analysis and has achieved good results. However, this method does not take into account the complexity of customs declaration data and the quality of training samples, does not perform targeted noise reduction on the data, and does not capture commodity characteristics that are highly correlated with commodity classification for different commodity types. The accuracy rate cannot meet the needs of customs classification risk screening application scenarios. Summary of the invention

[0006] In order to solve the above-mentioned deficiencies of the prior art, the purpose of the present invention is to provide an intelligent customs commodity classification model based on feature engineering and its establishment method and application. The technical problems to be solved include:

[0007] 1. Data Sampling

[0008] This technology uses labeled data for training. Labeling data requires a lot of manual work. In order to control labor costs, there are high requirements for the diversity, coverage, and product distribution of data extraction.

[0009] 2. Rapid data annotation

[0010] The annotation data is calculated based on Chinese characters, and each product area requires hundreds of thousands of annotation points, so an efficient annotation method needs to be designed.

[0011] 3. Feature recognition training

[0012] Design a suitable artificial intelligence algorithm to carry out feature recognition training. Since the results of product feature recognition directly affect the subsequent classification training effect, the feature recognition training needs to achieve extremely high accuracy.

[0013] 4. Feature analysis

[0014] Analyze the recognition features, especially the conversion of digital features into text features.

[0015] 5. Classification prediction training

[0016] Design appropriate artificial intelligence algorithms to carry out classification and prediction training.

[0017] 6. Hardware device limitations

[0018] The current mainstream large language model (LLM) has high requirements for hardware equipment and capital investment. According to the current actual hardware equipment configuration of the customs, it is necessary to minimize the algorithm's requirements for hardware and computing power.

[0019] 7. Inference Speed

[0020] In view of the rapid growth of customs business volume, the algorithm must achieve a higher computing speed to meet the business needs of quickly reviewing the classification of large quantities of goods.

[0021] Based on the above, the present invention explores a path to implement artificial intelligence in this vertical application scenario by deeply analyzing the inherent requirements of the customs classification business scenario, including accuracy, breadth, and speed.

[0022] The present invention proposes a method for establishing an intelligent customs commodity classification model based on feature engineering, comprising the following steps:

[0023] Step 1: Feature sorting: sort out the classification feature elements of the target tariff number according to the "Standardized Declaration Catalogue of Import and Export Commodities of the Customs of the People's Republic of China";

[0024] Step 2, feature extraction: use regular expressions to perform preliminary feature extraction;

[0025] Step 3, feature annotation: manually correct and annotate the features extracted by the program; digitally annotate the features using the BIO method;

[0026] Step 4, feature training and evaluation: Combine Bert and CRF to build an artificial intelligence training model, and use labeled data to carry out feature recognition training; evaluate the training results until the recognition accuracy reaches more than 99%, and then stop training;

[0027] Step 5, classification training and evaluation: Use the Bert model as the base for training, select a suitable loss function, define a gradient update strategy, design a training loop and stopping strategy, and use early stopping, L2 regularization, and dropout regularization techniques to prevent overfitting and improve the generalization ability of the model.

[0028] The commodity code in the present invention is the first 8 digits of the customs commodity HS code.

[0029] The present invention mainly realizes the feature recognition, feature analysis and establishment of classification prediction model of declared data through a 9-step training method, which consists of data sampling, data labeling, feature extraction, feature recognition, feature analysis, feature training, feature evaluation, classification training and classification evaluation.

[0030] In a specific implementation scheme, the 9-step training method is used to implement feature recognition, feature analysis and establishment of a classification prediction model for declared data through 8 steps, including the following steps:

[0031] Step 1: Feature sorting: sort out the classification feature elements of the target tariff number (HS) according to the "Standardized Declaration Catalogue of Import and Export Commodities of the Customs of the People's Republic of China";

[0032] Step 2, sample extraction: based on the distribution of the number of commodity declaration records, the declaration data is extracted according to the data volume threshold, and the declaration data includes: commodity name, commodity specification and model; for example, as shown in Table 1:

[0033] Table 1 Example of an overview of commodity declaration records (HS code 8524)

[0034]

[0035] Step 3, feature extraction: use regular expressions to perform preliminary feature extraction;

[0036] Step 4, feature annotation: manually correct and annotate the features extracted by the program; digitally annotate the features using the BIO method;

[0037] Step 5, feature training and evaluation: Combine Bert and CRF to build an artificial intelligence training model, and use labeled data for training. Evaluate the training results until the recognition accuracy reaches more than 99%, then stop training;

[0038] Step 6, feature analysis: parse the digital features in the training results into text features for subsequent training;

[0039] Step 7, classification and manual annotation: classify the sample annotation as correct or incorrect;

[0040] Step 8, classification training and evaluation: Use the Bert model as the base for training, select a suitable loss function, define a gradient update strategy, design a training loop and stopping strategy, and use early stopping, L2 regularization, and dropout regularization techniques to prevent overfitting and improve the generalization ability of the model.

[0041] In step 1 of the present invention, not all declared elements are related to the final commodity classification, so it is necessary to extract the elements related to the classification, and merge the elements that have different expressions but actually point to the same attributes. Step 1 is judged manually.

[0042] The classification characteristic elements are the basis for determining the HS code of customs goods. When classifying, it is necessary to comprehensively consider different factors based on different goods to ensure that the goods are correctly classified into the corresponding HS code. The classification characteristic elements generally include the following aspects as shown in Table 2:

[0043] Table 2 Classification characteristic elements

[0044] Classification feature elements content Product Name The common name or description of the product. Product form Such as solid, liquid, powder, granule, etc. Manufacturing or processing method The manufacturing process or processing method of the product, such as casting, forging, welding, etc. Product Features Including the physical properties, chemical properties, uses, etc. of the product. Product Usage The main purpose or function of the product, such as food, medicine, mechanical parts, etc. Product Materials The main material or component of the product, such as metal, plastic, wood, etc. Product Specifications The product's dimensions, weight, volume and other specifications. Product Packaging The packaging method of the goods, such as bulk, boxed, bagged, etc. Added value of goods The degree of processing the product has undergone, such as simple processing, deep processing, etc. Origin of the product Where a product is produced or manufactured sometimes affects how it is classified.

[0045] Taking the classification reference elements of the four tax numbers under item 8413 as an example, the declaration elements of the three tax numbers can be combined to obtain: 1. Product name; 2. Whether manual; 3. Purpose; 4. Driving mode; 5. Principle; 6. Speed. As shown in Table 3.

[0046] Table 3 Examples of combined declaration elements

[0047]

[0048]

[0049] Step 2 of the present invention includes the following sub-steps:

[0050] (1) Data preprocessing: Samples are screened and deleted according to the set threshold. In order to improve accuracy, the program deletes incorrectly classified data and deletes samples with a record number lower than the required number.

[0051] (2) Threshold setting: This is determined based on a comprehensive consideration of the computing power of the training and the labor cost of subsequent labeling. The diversity of samples is ensured while a reasonable number of samples are extracted, and the number of samples after sampling is ensured to be distributed according to the regularity of the number of original declared records.

[0052] discard_threshold=1:Discard all samples where the number of occurrences of a single product code is less than or equal to 1. If there is only one record, it is impossible to divide the test set and the training set, so at least two are required.

[0053] name_limit=300: The upper limit of sampling of different commodity names in each group of commodity codes. A group of commodity codes refers to all records under one tax number as a group.

[0054] code_limit=100: The sampling upper limit of a single product code, determined based on computing power and labor costs.

[0055] (3) Sampling process: For samples that meet the discard_threshold condition, the following processing is performed:

[0056] First, extract at least one record from each product name.

[0057] If the number of records after extraction exceeds name_limit, random sampling is performed to the number of name_limit.

[0058] If the number of records is between code_limit and name_limit, then keep those records.

[0059] If the number of records is less than code_limit, try to randomly select from the unselected records to fill up to code_limit. If it is insufficient, select all.

[0060] Step 3 of the present invention includes the following sub-steps:

[0061] 3.1. Data and configuration reading:

[0062] Read the standard declaration documents (Standard Declaration Catalogue of Import and Export Commodities of the Customs of the People's Republic of China), including:

[0063] ① Commodity code (8 digits). The customs HS code is an internationally standardized commodity classification system developed by the World Customs Organization (WCO). It is used to unify the classification of import and export commodities in countries around the world to facilitate statistics, taxation and supervision in international trade. The correct HS code is crucial for determining the tariff rate, trade statistics, and regulatory requirements of commodities. ② Declaration element sequence number and declaration element name.

[0064] Feature list reading: Read the feature list (i.e. the manually specified declaration element name)

[0065] Data file reading: read the customs declaration number, commodity serial number, commodity name, commodity specification model and commodity code.

[0066] 3.2. Data preprocessing:

[0067] Optionally, check again to see if there are any samples that do not meet the requirements, including a small number of records, incorrect classification, etc.

[0068] 3.3. Feature extraction preparation:

[0069] Use the str.split method to split the product specifications and models, and merge the split data back into the original data set, as shown in Table 4.

[0070] Table 4 Extraction of the original data set

[0071]

[0072]

[0073] Declaration element mapping: Convert declaration element information (declaration element information includes declaration element sequence number and declaration element name) into a nested dictionary structure with commodity code as key and declaration element sequence number and declaration element name as value, so as to facilitate the subsequent extraction of specific declaration elements by commodity code and sequence number.

[0074] 3.4. Feature extraction:

[0075] By constructing the extract_features function, for each row of data, the corresponding declaration element name is matched according to its commodity code and declaration element sequence number, and the corresponding value is filled in. This step is to check the declaration element corresponding to each commodity code in the constructed elements_dict dictionary, and then map the feature column obtained after the product specification model is split (if it exists and is not empty) to the corresponding declaration element name.

[0076] 3.5. Results collation and export:

[0077] Column selection: define and select the required columns (including basic information columns and feature columns determined in step 3.4, the basic information columns refer to customs declaration number, commodity serial number, commodity name).

[0078] Export data: Export the final organized data set to and save it in the specified file path.

[0079] The features extracted in step 3 of the present invention use the idea of ​​position correspondence, but the enterprise may not declare the declared elements to the designated position as required when declaring, such as declaring "brand" to the position of "principle", declaring "purpose" to the position of "power", etc. This step requires manual correction of the enterprise's declaration errors.

[0080] The manual correction and annotation in step 4 of the present invention is to correct the incorrect classification in step 3 to ensure that each declaration element correctly corresponds to its content.

[0081] The BIO annotation method in step 5 of the present invention is used for sequence annotation tasks in order to annotate various features in the text as one of start (B), internal (I) or external (O), so that the subsequent machine learning model can identify and extract the key information in the declaration elements, wherein the key information refers to the features.

[0082] The BIO annotation not only indicates the type of the declared element, but also the location of the declared element in the text (i.e., whether it is the beginning of a feature, inside a feature, or does not belong to any feature).

[0083] The model in step 6 of the present invention comprises:

[0084] Structural improvements: BERT as a feature extractor: Use the pre-trained BERT model as a label encoder to extract deep semantic features directly from the original text. CRF as a sequence tagger: Add a CRF layer on the basis of BERT for sequence tagging. CRF is able to learn the dependencies between labels. For example, in the BIO tagging scheme, the I-label usually follows the B-label. This combination takes advantage of the powerful representation ability of BERT and the advantages of CRF in handling sequence dependencies.

[0085] Differences in the training process: Customized data loading and preprocessing: The BertCRFDataset class and preprocessing function preprocess_data are customized to process data in a specific format (including BIO annotations) as shown in Table 5, which ensures that the model can receive input in the correct format.

[0086] Table 5 Specific format data (1 item)

[0087]

[0088] Different learning rate optimizers: Set different learning rates for the BERT and CRF layers. This approach takes into account the different learning rate requirements of the pre-trained model (BERT) and the non-pre-trained model part (CRF), thereby controlling the training process more finely.

[0089] Learning rate scheduling: Use the ReduceLROnPlateau learning rate scheduler to automatically adjust the learning rate based on the F1 score on the validation set to avoid overfitting and improve the generalization ability of the model.

[0090] Early Stopping: Stop training by monitoring that the performance on the validation set has not improved for several consecutive cycles to prevent overfitting and save training time.

[0091] The model of the present invention has been specifically adjusted and optimized in the above aspects to make it more suitable for processing specific feature recognition tasks. In particular, the detailed management of data preprocessing, learning rate adjustment strategy and training process reflects the in-depth understanding and optimization of task characteristics. In addition, customized processing for feature extraction and labeling tasks, such as setting different learning rates and dynamic learning rate adjustment based on performance improvement, also shows the differences from some existing technologies.

[0092] In step 7 of the present invention, there is no need to parse the features (declaration elements) of the plain text, and BERT directly extracts deep semantic features from the original text during the training process.

[0093] Parse digital features, such as the features extracted by the intelligent program (BERT+CRF model) after feature learning. After multiple tests, Bert cannot well identify the association between these digital features and product classification. Therefore, the "feature parsing" processing step is added.

[0094] This step considers various influencing factors such as expression methods, units, Chinese and English, and intervals, uses regular expressions to extract digital features, and converts them into intervals represented by text that are compatible with the classification rules. The parsed results are then used as input elements to learn product classification using Bert.

[0095] In step 8 of the present invention, data preprocessing in a specific field:

[0096] The data was preprocessed specifically for the customs commodity classification task, and the data was verified and cleaned to ensure that there were sufficient samples for each category and the uniqueness of the feature combination, which helped to improve the quality and reliability of model training and demonstrated a deep understanding of the characteristics of commodity classification data.

[0097] Customization of model training:

[0098] Applying the BERT model to unimodal text data processing reflects adaptive adjustments to task requirements.

[0099] The specific settings of the optimizer and learning rate scheduler based on the results of a large number of tests and verifications reflect the fine management of the training process. For example, setting LEARNING_RATE = 1e-5 (learning rate, a hyperparameter for adjusting the step size of model parameters in the optimization algorithm), PATIENCE = 3 (the number of cycles to wait before taking action = 3), MAX_NO_REDUCE_EPOCHS = 5 (if the performance of the model does not improve within 3 consecutive cycles, the learning rate will be reduced).

[0100] Model preservation and information recording:

[0101] During the training process, the optimal model and related configuration information are automatically saved, such as model weights (the set of model parameters obtained through training in steps 5 and 8), BERT configuration, label encoder, etc., to facilitate subsequent model reuse and analysis.

[0102] Generate and record key indicators of the training process to TensorBoard and CSV files, which facilitates model evaluation and performance monitoring. The key indicators involved in the present invention include the following:

[0103] 1. Loss value:

[0104] ①Train Loss: The average loss value of the model on the training data, which is used to measure the difference between the model prediction and the actual label. The lower the training loss, the better the model performs on the training data.

[0105] ②Val Loss (validation loss): the average loss value of the model on the validation data set. The validation data set is data that the model has not seen during the training process and is used to evaluate the generalization ability of the model.

[0106] 2. Accuracy (Train Accuracy and Val Accuracy):

[0107] ①Train Accuracy: The ratio of the number of samples correctly predicted by the model on the training data to the total number of samples. A high accuracy indicates that the model has good classification performance on the training data.

[0108] ②Val Accuracy: The proportion of correct predictions made by the model on the validation data set. It is used to evaluate the model’s ability to predict new data.

[0109] 3. Train Recall:

[0110] Train Recall: The proportion of positive samples correctly predicted by the model among all actual positive samples. A high recall rate means that the model is able to capture more positive samples. The higher the recall rate, the better, but it also needs to be balanced with precision to avoid false positives.

[0111] 4. F1 score (Train F1 and Val F1):

[0112] ①Train F1: The F1 score is the harmonic mean of precision and recall, and is used to measure the overall performance of the model on the training data. The higher the F1 score, the better the balance and accuracy of the model on the training data.

[0113] ②Val F1 (Verification F1 score): The F1 score of the model on the validation dataset, which is used to measure the overall performance of the model on new data.

[0114] 5. Epoch (training round): An epoch represents the entire training data set being used to train the model once. In each epoch, the model will be exposed to every sample in the training data set and update the weights based on them.

[0115] The present invention is different from the known methods:

[0116] The present invention performs text processing and classification based on the BERT model, and makes targeted creative optimization and adjustment in the application of the algorithm to specific commodity classification tasks, including specific data preprocessing steps, feature recognition and analysis related to commodity classification, adaptive modification of the model structure and fine management of the training process.

[0117] In particular, the adjustment of hyperparameters has undergone multiple iterations, including LEARNING_RATE, BATCH_SIZE, MAX_LENGTH, EPOCHS, etc. This is not only a process of finding the optimal performance, but also reflects the exploration of a deep understanding of the model behavior.

[0118] Based on the above method, the present invention also proposes an intelligent customs commodity classification system based on feature engineering, including: a memory and a processor; a computer program is stored on the memory, and when the computer program is executed by the processor, the above method is implemented.

[0119] Based on the above method, the present invention also proposes an intelligent customs commodity classification device based on feature engineering, including a feature combing module, a sample extraction module, a feature extraction module, a feature manual labeling module, a feature BIO labeling module, a feature training and evaluation module, a feature analysis module, a classification manual labeling module, and a classification training and evaluation module.

[0120] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0121] The innovative features of the present invention include but are not limited to:

[0122] 1. Deeply customized data preprocessing and feature analysis methods for customs commodity classification scenarios:

[0123] The algorithm solves the problem of processing complex commodity standard declaration data through data preprocessing and feature analysis methods specific to customs commodity classification, improves data quality, and lays the foundation for efficient model training. In particular, based on the characteristics of commodity declaration information in the customs declaration form, the BIO annotation method is creatively applied to customs commodity feature annotation and recognition, greatly improving the recognition speed and accuracy. At the same time, combined with the use of regular expressions, the conversion of digital features to text tags is realized, providing a guarantee for improving the accuracy of the final commodity classification.

[0124] 2. Comprehensive use of deep learning for feature extraction and classification:

[0125] By leveraging the deep learning capabilities of the BERT model and the sequence modeling advantages of the CRF model, this method not only fully mines the deep semantic information in the text data, but also considers the dependencies between labels. This is an important technological breakthrough in the field of customs commodity classification.

[0126] 3. Multi-stage training strategy and optimization:

[0127] In a specific implementation scheme, a 9-step training method is adopted, including data sampling, data labeling, feature extraction, feature identification, feature analysis, feature training, feature evaluation, classification training, and classification evaluation steps, which demonstrates a deep understanding of the commodity classification task.

[0128] In addition, the application of strategies such as early stopping, L2 regularization, and dropout regularization further improves the generalization ability of the model.

[0129] 4. Automatically monitor the training process:

[0130] Through automated sample extraction and feature labeling processes, this method effectively improves training efficiency and visualization of model performance and reduces manual intervention.

[0131] 5. Model adjustment and optimization for commodity classification application scenarios:

[0132] Customized adjustments were made to the model structure and training process to meet the specific needs of customs commodity classification, including targeted hyperparameter adjustments and learning rate scheduling strategies. These adjustments are based on a deep understanding of task characteristics and data features.

[0133] In summary, this invention combines the latest deep learning technology to provide a new solution for intelligent customs commodity classification, solving the problems of low efficiency and low accuracy in traditional methods, and has significant practical value. Clear implementation steps and methods ensure that the algorithm is not only feasible in theory, but can also be replicated and applied in practice.

[0134] The beneficial effects of the present invention include but are not limited to:

[0135] Compared with existing technologies (including conventional data analysis, artificial intelligence without feature engineering, etc.), the present invention has made significant progress in detection rate, accuracy and practicality.

[0136] Customs officers can manually judge no more than 30 declaration records per hour, which is 240 per day if calculated based on 8 hours. Using the method of the present invention, if a single consumer-grade graphics card (16G video memory) is used, about 300,000 records can be calculated per hour, which is 10,000 times more than manual judgment on an hourly basis, and even higher on a daily basis (the system can run 24 hours a day). And using a graphics card with higher performance, the computing speed can be further improved.

[0137] The present invention realizes feature recognition, feature analysis and classification prediction of declared data through a 9-step training method, which greatly improves work efficiency and error detection rate (in the test field, after inspection by customs experts, the detection rate is 3.9 times the sum of other methods), and has a high precision rate (the accuracy of high-risk documents prompted by the model reaches more than 99%, and the overall accuracy rate is more than 88%).

[0138] In order to achieve the above technical effects, the present invention overcomes the following technical difficulties:

[0139] 1. Model base selection and technical framework determination: How to select a suitable model base from a large number of models and determine a feasible technical framework. After in-depth research and comparison, the present invention comprehensively considers factors such as the feasibility of private deployment, parameter scale, compatibility with application scenarios, reasoning speed, and hardware cost, and finally selects a strategy combining Bert-Base-Chinese with CRF (Conditional Random Fields) to achieve an effective combination of semantic understanding and sequence data prediction, thereby improving the model's ability to recognize customs declaration text patterns.

[0140] 2. Sample collection: The distribution of commodities in various customs classifications (commodity codes) is uneven. When sampling, both the sample coverage and the workload of manual labeling must be considered. The present invention adopts a nonlinear sampling strategy to automatically sample according to the original data volume and commodity complexity in each commodity code.

[0141] 3. Data labeling: Data labeling directly affects the recognition accuracy of the model. Based on the comparison of various labeling methods, the present invention selects BIO labeling as the labeling method for feature recognition, and constructs the mapping relationship between factor categories and BIO tags according to the characteristics of customs commodity descriptions to achieve semi-automatic labeling, thereby providing high-quality training data for the feature recognition of the model.

[0142] 4. Hyperparameter tuning and preventing overfitting: The selection of hyperparameters has an important impact on model performance, and the present invention faces challenges in hyperparameter tuning. Through a large number of experiments, the present invention optimizes the hyperparameter settings, including batch_size (batch size), learning_rate (learning rate), max_length (maximum text length), epochs (training rounds), etc., and uses L2 regularization, early stopping, dropout and other technologies to effectively avoid the overfitting problem. A variety of effect evaluation indicators and methods are used, such as accuracy, precision, recall rate, F1 score, early stopping, etc., to ensure accurate evaluation of model effects.

[0143] 5. Determination of the training implementation path and integration of expert knowledge: In the process of determining the training path, the present invention needs to solve the problem of how to efficiently annotate data and integrate expert knowledge. To this end, the present invention independently explored and established a 9-step training method (data sampling, data annotation, feature extraction, feature identification, feature analysis, feature training, feature evaluation, classification training, classification evaluation), realizing the deep integration of artificial intelligence technology and expert experience, improving the efficiency and quality of model training, and finding a feasible path for the application of cutting-edge technology in the vertical field of customs tax collection and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0145] Figure 1 It is a flow chart of the training of the intelligent customs commodity classification model based on feature engineering of the present invention.

[0146] Figure 2 It is a flow chart of training the intelligent customs commodity classification model based on feature engineering for commodities under item 8413 in an embodiment of the present invention.

[0147] Figure 3 This is a diagram of the intelligent customs commodity classification model based on feature engineering of the present invention.

[0148] Figure 4 Diagram of the classification training parameter tuning process.

[0149] Figure 5 It is a feature recognition result diagram in the present invention.

[0150] Figure 6 This is a partial sampling result of item 8524, and a quantity table of the same item data before and after sampling. DETAILED DESCRIPTION

[0151] The invention is further described in detail with reference to the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0152] The invention discloses a method for establishing an intelligent customs commodity classification model based on feature engineering, comprising the following steps: feature combing: combing the classification feature elements of the target tax number according to the commodity specification declaration catalog; feature extraction: using regular expressions to perform preliminary extraction of features; feature labeling: manually correcting and labeling the features extracted by the program; using the BIO method to digitally label the features; feature training and evaluation: combining Bert and CRF to build an artificial intelligence training model, and using the labeled data to carry out feature recognition training; evaluating the training results until the recognition accuracy reaches more than 99%, stopping the training; classification training and evaluation: using the Bert model as a base for training, selecting a suitable loss function, defining a gradient update strategy, designing a training cycle and a stopping strategy, and using early stopping, L2 regularization, and dropout regularization techniques to prevent overfitting, and improve the generalization ability of the model. This method can not only ensure a high recall rate and precision rate, but also its algorithm and process design maintain a low requirement for computing power. The actual measurement shows that 8G video memory can meet the requirements, and the inference speed reaches 150,000 to 180,000 per hour.

[0153] Example 1, taking 8413 (pump) as an example

[0154] This embodiment takes 8413 (pump) as an example to describe the specific implementation steps in detail:

[0155] Step 1, data sampling:

[0156] The first part is to sort out the classification characteristics corresponding to the HS of the product. There are 74 HSs (10-digit tax numbers) in total, and there are 19 declaration elements involved in the standardized declaration catalog:

[0157] Brand type, export preferential treatment, purpose, whether it is equipped or can be equipped with a metering device, brand (Chinese or foreign name), model, GTIN, CAS, other, whether it is manual, purpose (whether it is for a piston internal combustion engine), output power of applicable engines, driving mode (pneumatic, electric, hydraulic, etc.), principle (reciprocating), principle (rotary), principle (centrifugal), speed, principle (plunger, gear, screw, etc.), purpose (applicable to XX brand XX machine or universally applicable to XX machine, etc.).

[0158] Not all of the above declaration elements are related to the final commodity classification, so it is necessary to extract the elements related to the classification and manually merge the elements that have different expressions but actually point to the same attributes.

[0159] The specific results after extraction and merging are as follows: (a total of 13 declaration elements are involved)

[0160] Export preferential treatment, purpose, whether it is equipped or can be equipped with a metering device, brand (Chinese or foreign name), model, GTIN, CAS, other, whether it is manual, output power of the applicable engine, driving mode (pneumatic, electric, hydraulic, etc.), principle (reciprocating, rotary, centrifugal, plunger, gear, screw, etc.), speed.

[0161] The second part is to conduct sampling by stratification and classification. The sampling rules take into account factors such as the amount of original declared data, product diversity, and labeling workload. A balance is sought between sample coverage and manual labeling workload. The sampling algorithm uses a random combination of classification and specified threshold methods.

[0162] The input of this step is the original data, and the output is the data sampled according to the rules.

[0163] Step 2, data annotation: In the declaration results received by the customs, taking HS: 84132000 as an example, the following features are required to be declared: 1. Product name 2. Whether manual 3. Brand 4. Model. Some companies will misreport or omit relevant features when declaring due to unfamiliarity with the declaration requirements or other reasons. The declaration order is: 1. Product name 2. Whether manual 3. Model 4. Brand. The contents of feature 3 and feature 4 are reversed. In order to avoid affecting the subsequent training effect, manual annotation is required to adjust the position manually.

[0164] Step 3, feature extraction: extract each feature according to the “|” distribution, as shown in Table 6.

[0165] Table 6 Feature extraction results

[0166]

[0167] Step 4, feature recognition: The input of this step is the Chinese or English characters of the product specifications and models. After manual labeling and program recognition, the output is the labeling result of each character in the format of [X,X,X,X], where X is a 1 to 2-digit number, as shown in Table 7.

[0168] Table 7 Data annotation results (2 records)

[0169]

[0170]

[0171] Different numbers in the annotation are used to locate the position of the elements related to the classification. B represents the beginning of the relevant element entity, I represents the internal part of the entity (including characters), and 0 represents irrelevant elements (elements other than B, I and delimiters). Take the first item as an example:

[0172] 'O':0;

[0173] '|':1;

[0174] 'B-Purpose':2, 'I-Purpose':3;

[0175] 'B- Planar display module type': 4, 'I- Planar display module type': 5;

[0176] 'B-Whether it contains a driver or control circuit': 6, 'I-Whether it contains a driver or control circuit': 7;

[0177] 'B-Size':8,'I-Size':9;

[0178] 'B-Brand':10,'I-Brand':11;

[0179] 'B-model':12,'I-model':13;

[0180] 'B-Configuration':14,'I-Configuration':15;

[0181] 'B-single piece weight':16,'I-single piece weight':17;

[0182] 'B-Has it been cut':18,'I-Has it been cut':19.

[0183] Step 5, feature parsing: There is no need to parse the features of the plain text (declaration elements). During the training process, BERT directly extracts deep semantic features from the original text.

[0184] For the analysis of digital features, after feature learning, the power features extracted by the intelligent program (BERT+CRF model) have different expressions, as shown in Table 8.

[0185] Table 8 Digital features before feature analysis

[0186]

[0187]

[0188] After multiple tests, Bert was unable to identify the association between these digital features and product classifications, so a "feature analysis" processing step was added.

[0189] This step considers various influencing factors such as expression methods, units, Chinese and English, and intervals, uses regular expressions to extract digital features, and converts them into intervals represented by text that are compatible with the classification rules. The parsed results are then used as input elements to learn product classification using Bert, as shown in Table 9.

[0190] Table 9 Digital features after feature analysis

[0191]

[0192]

[0193] Step 6, feature training: training based on feature annotations. Divide the data into training set, validation set, and test set, which are non-intersecting, to objectively evaluate the training effect and generalization ability.

[0194] Write training programs, build NER recognition tasks, build training, validation, and test data sets, select appropriate loss functions, define gradient update strategies, involve training loops, and design stopping strategies.

[0195] Use regularization, early stopping and other techniques to prevent overfitting. Tune batch_size, learning_rate, max_lenth parameters, and use the learning rate scheduler to adjust the learning rate at different stages of learning. Use different learning rates for the semantic recognition layer and conditional random layer to increase the convergence speed.

[0196] After this layer of training, the program can automatically identify the preliminary features for classification. The feature training results are shown in Table 10.

[0197] Table 10 Feature recognition training results (2 records)

[0198]

[0199]

[0200] Step 7, feature evaluation: Evaluate the training results. Iterate the training based on the evaluation, perform accuracy verification, generalization ability verification, and iterate the training until the recognition accuracy of the intelligent model for the feature reaches the requirement for classification training. Taking into account the business tolerance and the cost-effectiveness of training, the accuracy of the feature is usually required to reach 99%.

[0201] Step 8, classification training: Classification training and evaluation are performed based on feature training and analysis results. Training is iterated based on the evaluation. During training, the data is divided into a training set, a validation set, and a test set, which are not intertwined, to objectively evaluate the training effect and generalization ability.

[0202] Write relevant training programs, including building multi-classification tasks, building training, validation, and test data sets, selecting appropriate loss functions, defining gradient update strategies, involving training loops, and designing stopping strategies.

[0203] Introduce regularization, early stopping and other techniques to prevent overfitting. Tune parameters such as batch_size, learning_rate, max_lenth, etc. And use the learning rate scheduler to adjust the learning rate at different stages of learning. Figure 4 For example, as the number of training times increases, the cross entropy loss value of the training set gradually decreases, which means that the difference between the model prediction value and the actual label gradually decreases.

[0204] The output of this subroutine is the final recognition result of the model, including whether the classification is correct, the tax number recommended by the model, as well as the confidence probability and risk level. Figure 5 shown.

[0205] Step 9, classification and evaluation:

[0206] Evaluate the training results. Based on the evaluation, iterate the training, verify the accuracy, generalization ability, and iterate the training until the F1 score (an indicator for evaluating the performance of the classification model, the harmonic mean of the model's recall and precision) of the classification prediction results of the intelligent model meets the requirements. Taking into account the business tolerance and the cost-effectiveness of the training, the F1 score usually needs to reach 80%.

[0207] Model Application:

[0208] Save the trained models and configuration files of each link in the whole process, carry out practical applications, and output the model's prediction and confidence probability of commodity classification, and mark the risk level judged by the model (automatically mark high, medium, and low risks according to the confidence probability threshold).

[0209] Test verification:

[0210] There are 20,970 test data in total, and the model indicates that 2,051 of them are at risk, and they are divided into high, medium, and low risks according to the confidence probability. Table 11 shows the results of the expert verification one by one.

[0211] Table 11 Test verification status table

[0212]

[0213] In Table 11, “Unable to judge” means that the enterprise’s declaration content is incomplete, and even classification experts cannot judge the classification based on the current information. “Decision efficiency” refers to the accuracy rate after eliminating these undecidable situations. “Overall efficiency” refers to the proportion of tax numbers predicted correctly (i.e. consistent with the actual results) by the model in all test data.

[0214] Comparative Example 1:

[0215] Comparison with the detection rate (detection error rate) of existing technologies:

[0216] The customs declaration forms declared under item 8413 from January to September 2023 were retrieved nationwide. Through manual review, data analysis and other screening methods, a total of 1,254 commodity code (HS) modification records were generated under item 8413, indicating that the existing customs screening methods detected a total of 1,254 erroneous data. The method of the present invention was used to review the customs declaration form data for the same period, and a total of 196,000 data were reviewed, indicating 3,611 HS erroneous data, which is 2.9 times the sum of all other screening methods.

[0217] Comparative Example 2:

[0218] Practicality comparison with existing technologies:

[0219] The output of the existing general artificial intelligence method without feature engineering is usually a set of candidate HSs, and the accuracy calculation is generally that as long as there is one correct HS in the set, the model prediction result is considered correct. In fact, common classification errors are usually concentrated in only a few HSs. After giving a few HSs, customs staff still need to make another judgment, which limits their practicality.

[0220] The method of the present invention directly gives a unique predicted HS, and the accuracy criterion is also to use this unique HS as the calculation target of the accuracy rate. This puts higher requirements on the accuracy of the artificial intelligence method, but is closer to the actual application scenario. Comparative Example 3:

[0221] Comparison of accuracy with existing technologies:

[0222] Compared with the existing general artificial intelligence methods without feature engineering, the accuracy rate has a significant advantage when the first HS is predicted as the result. 4419 identical customs declaration records are used for testing, as shown in Table 12.

[0223] Table 12 Comparison of accuracy with existing technologies

[0224]

[0225] Note: The range of precision rates is compared with the classification results obtained by testing the intelligent classification method developed by Cuicheng Technology using exactly the same 4419 data sets, and the overall accuracy and precision of the two models.

[0226] This method can achieve different precision rates under different recall requirements, as shown in Table 13.

[0227] Table 13. Precision of this method

[0228] Recall Precision 60％ 99％ 90％ 95％ >99％ 89％

[0229] Other information: Training process (24 rounds of training)

[0230]

[0231] The above shows that:

[0232] 1. Epoch (training round): The entire training dataset is used to train the model twenty-four times.

[0233] 2. Train_loss: The average loss value of the model on the training data has decreased from 1.68 to 0.03, indicating that the difference between the model prediction results and the actual labels has decreased.

[0234] 3. Val_loss: The loss value of the model on the validation set decreased from 0.84 to 0.50, indicating that the performance of the model on unseen data is improving and the generalization ability is enhanced. At the same time, the decreasing trend of Val_loss is similar to that of Train_loss, which usually means that the model is learning generalized features instead of just memorizing the training data, avoiding overfitting.

[0235] 4. Train_accuracy: The training accuracy of the model has increased from 0.61 to 0.99, indicating that after adjusting the hyperparameters such as learning rate, batch size, and optimizer, a more suitable parameter setting for the current model and data has been found. For unbalanced data sets, other indicators may need to be considered, such as Precision, Recall, F1 Score, etc.

[0236] 5.Val_accuracy: The validation accuracy of the model increased from 0.77 to 0.89, indicating that the model's performance on unseen validation data has improved and the overall performance of the model has improved.

[0237] 6.train_f1: The training F1 score takes into account both the training precision and the training recall. The improvement from 0.59 to 0.99 indicates that the model has improved in both aspects.

[0238] 7.Val_f1: The verification F1 score increased to 0.89, indicating that the model has been optimized in handling the balance between positive and negative samples. Through learning, it better distinguishes samples of different categories and reduces misclassification.

[0239] 8.Train_recall: The training recall rate measures the proportion of samples correctly identified as positive by the model to all actual positive samples. The increase of this indicator from 0.61 to 0.99 indicates that the model can now capture more samples that are actually positive and reduce missed detections.

[0240] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A method for establishing an intelligent customs commodity classification model based on feature engineering, characterized in that: The following steps are involved: Step 1: Feature sorting: sort out the classification feature elements of the target tax number according to the commodity standard declaration catalog; Step 2, feature extraction: use regular expressions to perform preliminary feature extraction; Step 3, feature annotation: manually correct and annotate the features extracted by the program; Use BIO method to digitally annotate features; Step 4: Feature training and evaluation: Combine BERT and CRF to build an artificial intelligence training model, and use labeled data to carry out feature recognition training; Evaluate the training results until the recognition accuracy reaches the target, and then stop the training; Step 5, classification training and evaluation: Use the Bert model as the base for training, select a suitable loss function, define a gradient update strategy, design a training loop and stopping strategy, and use early stopping, L2 regularization, and dropout regularization techniques to prevent overfitting and improve the generalization ability of the model.

2. The method of establishing as claimed in claim 1, characterized in that: In step 1, the elements related to classification in the declared elements are extracted, and the elements with different expressions but actually pointing to the same attributes are merged; and / or, The step 1 further includes a sample extraction step: extracting the declared data according to the distribution of the number of commodity declared records and the threshold of the data volume; Preferably, the sample extraction step includes the following sub-steps: (1) Data preprocessing: Delete samples with incorrectly classified data and samples with fewer records than required; (2) Threshold setting: discard_threshold=1: discard all samples where a single product code appears less than or equal to 1 times; name_limit=300: the upper limit of sampling of different product names within each group of product codes; code_limit=100: the upper limit of sampling for a single product code; (3) Sampling process: For samples that meet the discard_threshold condition, the following processing is performed: First, extract at least one record from each product name; If the number of records after extraction exceeds name_limit, random sampling is performed to the number of name_limit; If the number of records is between code_limit and name_limit, keep these records; If the number of records is less than code_limit, try to randomly select from the unselected records to fill up to code_limit. If it is insufficient, select all.

3. The establishment method according to claim 1, characterized in that: The step 2 includes the following sub-steps: Step 2.1, data and configuration reading: Read the standard declaration documents, including: commodity code, declaration element sequence number and declaration element name; Feature list read: read feature list; Data file reading: reading the customs declaration number, commodity serial number, commodity name, commodity specification model and commodity code; Step 2.2, feature extraction preparation: Use the str.split method to split the product specifications and models, and merge the split data back into the original data set; Declaration element mapping: convert the declaration element information into a nested dictionary structure with the commodity code as the key and the declaration element sequence number and declaration element name as the value, so as to facilitate the subsequent extraction of specific declaration elements by commodity code and sequence number; Step 2.3, feature extraction: By constructing the extract_features function, for each row of data, the corresponding declaration element name is matched according to its commodity code and declaration element sequence number, and the corresponding value is filled; by checking the declaration element corresponding to each commodity code in the constructed elements_dict dictionary, the feature column obtained after the commodity specification model is split is mapped to the corresponding declaration element name; Step 2.4, result collation and export: Column selection: define and select the required columns, including basic information columns and feature columns determined in step 2.3; the basic information columns include customs declaration number, commodity serial number, and commodity name; Export data: Save the final organized data set in the specified file path.

4. The establishment method according to claim 1, characterized in that: The BIO method in step 3 is used for sequence labeling tasks in order to label various features in the text as one of start (B), internal (I) or external (O), so that the subsequent machine learning model can identify and extract the key information in the declaration elements, where the key information refers to the features.

5. The establishment method according to claim 1, characterized in that: In step 4, The structure of the model includes: BERT as a feature extractor: Use the pre-trained BERT model as a label encoder to extract deep semantic features directly from the original text; CRF as a sequence tagger: Add a CRF layer on top of BERT for sequence tagging; CRF can learn dependencies between tags; and / or, The training process of the model: Custom data loading and preprocessing: Process data in a specific format by customizing the BertCRFDataset class and the preprocessing function preprocess_data; and / or, The model includes optimizers with different learning rates: different learning rates are set for the BERT and CRF layers respectively; and / or, The model includes learning rate scheduling: using ReduceLROnPlateau learning rate scheduler, the learning rate is automatically adjusted according to the F1 score on the validation set to avoid overfitting and improve the generalization ability of the model; and / or, The model includes early stopping: training is stopped by monitoring that the performance on the validation set has not improved for several consecutive epochs, preventing overfitting and saving training time.

6. The establishment method according to claim 1, characterized in that: After step 4, feature analysis is also included: The features of plain text are not parsed. During the training process, BERT directly extracts deep semantic features from the original text. For the analysis of numeric features, after feature learning, the features extracted by the BERT+CRF model are extracted using regular expressions and converted into intervals represented by text that are compatible with the classification rules; The parsed results are used as input elements to learn product classification using BERT.

7. The establishment method according to claim 1, characterized in that: Step 5 includes preprocessing the data specific to the customs commodity classification task, as well as validating and cleaning the data to ensure that there are sufficient samples for each category and the uniqueness of the feature combination.

8. An intelligent customs commodity classification model established by the method according to any one of claims 1 to 7, and its application in customs commodity classification.

9. A system for establishing an intelligent customs commodity classification model based on feature engineering, characterized in that: include: A memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images