Intelligent foreign trade data classification system based on natural language processing

Abstract

The application relates to the technical field of natural language processing, in particular to an intelligent foreign trade data classification system based on natural language processing, which comprises a semantic pattern recognition module, a classification task execution module, a classification effect monitoring module, a cross-domain data migration module and a rule optimization feedback module.In the application, semantic analysis is carried out based on language features and context association rules, the semantic structure of foreign trade data can be effectively recognized and classified, the dependence on manual rules in the classification process is reduced, the accuracy and automation degree of classification are improved, the task allocation path can be optimized according to actual requirements, abnormal data can be identified in time and classification rules can be adjusted by monitoring the classification effect, the overall classification quality is improved, the cross-domain data migration and rule optimization feedback mechanism ensure that the classification rules can be adapted in different fields, the flexibility and expansibility of the system are enhanced, and the overall efficiency and precision of foreign trade data processing are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a foreign trade data intelligent classification system based on natural language processing. BACKGROUND

[0002] The technical field of natural language processing refers to the research field of the intersection of computer science and linguistics, mainly studying how computers simulate, analyze, understand and generate natural language for effective language processing and intelligent operation. The technical field of natural language processing includes text analysis, information extraction, semantic understanding, machine translation, dialogue system, sentiment analysis and other core matters, and its goal is to enable computers to understand and process language like humans, complete tasks such as automatic summarization, document classification, named entity recognition, etc. The core tasks of natural language processing technology include language model, text classification, machine translation, sentiment analysis, etc., involving the combination of linguistic theory and computational model, and promoting the wide application of artificial intelligence technology in language processing. Among them, the traditional foreign trade data intelligent classification system refers to the intelligent analysis and classification of foreign trade data in the foreign trade field to improve the efficiency and accuracy of foreign trade data processing. The system relies on manual rule setting and manual classification method, and uses a pre-set rule base to classify and process foreign trade data. This method has problems such as high labor cost, low classification accuracy, and low efficiency. The traditional foreign trade data classification adopts a rule-based manual classification method, which realizes the classification task by manually labeling and rule matching of foreign trade data, but due to the diversity and complexity of data, the adaptability and accuracy of manual rules are limited.

[0003] The existing technology of foreign trade data classification relies on manual rule setting and manual labeling. This method has limited adaptability and accuracy of rules when facing a large amount of complex data, has high labor cost and low efficiency, and cannot quickly respond to data changes. Due to the diversity and complexity of foreign trade data, manual classification is difficult to accurately process all different types of records, resulting in inconsistent classification results and even incorrect labeling. The traditional rule base is updated slowly and lacks real-time optimization mechanism, which easily causes rule adaptation problems in cross-domain applications, which makes the system fail when processing complex and changeable data, affecting the effect and efficiency of foreign trade data classification. SUMMARY

[0004] The purpose of the present application is to solve the problems existing in the prior art, and to provide a foreign trade data intelligent classification system based on natural language processing.

[0005] In order to achieve the above purpose, the present application adopts the following technical scheme: the foreign trade data intelligent classification system based on natural language processing comprises:

[0006] The semantic pattern recognition module extracts the syntax structure, vocabulary distribution, sentence pattern change and semantic weight in the text based on the language features and context association rules of foreign trade data, analyzes the theme attribution of the data content by combining a theme clustering algorithm, and generates a semantic classification label set;

[0007] The classification task execution module calls the semantic classification label set, extracts label data features according to a preset classification rule base and a dynamic adjustment strategy, matches corresponding items in the rule base, filters a target data set and assigns a classification label, rearranges the task order according to the label priority, and obtains a classification task allocation path;

[0008] The classification effect monitoring module extracts the task completion time and error marking record in the classification process based on the classification task allocation path, judges the consistency of the classification result, identifies abnormal sections in the classification task, and generates a classification quality evaluation data set;

[0009] The cross-domain data migration module identifies the adaptability problem of the classification rule according to the abnormal section marked in the classification quality evaluation data set, analyzes the semantic features and classification requirements of the target domain data, and generates a task adjustment implementation list.

[0010] As a further scheme of the present application, the semantic classification label set includes a syntax complexity label, a vocabulary frequency distribution label, a theme weight distribution, a sentence structure identifier, and a semantic correlation degree label; the classification task allocation path includes a classification rule priority sequence, a task grouping list, a classification node mapping table, and a task execution order identifier; the classification quality evaluation data set includes a classification consistency score, an error marking distribution, a task abnormal section identifier, and a classification accuracy evaluation value; and the task adjustment implementation list includes a classification rule update item, a semantic feature adaptation identifier, and an optimization condition list.

[0011] As a further scheme of the present application, the semantic pattern recognition module includes:

[0012] The syntax structure analysis submodule extracts syntax structure information, part-of-speech tagging sequences and sentence pattern change rules based on the language features and context association rules of foreign trade data, identifies the syntax complexity distribution in the text, and generates a syntax complexity value;

[0013] The vocabulary distribution analysis submodule calls the syntax complexity value, analyzes the distribution characteristics of the vocabulary under the differentiated theme in combination with the vocabulary frequency distribution and the semantic weight, judges whether the theme clustering condition is met, and obtains a semantic weight value;

[0014] The theme clustering generation submodule extracts the theme distribution features in the text according to the semantic weight value, performs theme clustering in combination with the context association rules, and obtains a semantic classification label set.

[0015] As a further scheme of the present application, the classification task execution module comprises:

[0016] The rule matching sub-module calls the semantic classification label set, detects the matching degree of rules and semantic labels according to rule entries in a preset classification rule library, filters rules meeting conditions and sorts them according to priority to generate a classification rule priority sequence;

[0017] The task grouping optimization sub-module groups classification tasks based on the classification rule priority sequence in combination with a dynamic adjustment strategy, generates a task grouping list according to task complexity and resource occupation;

[0018] The node mapping generation sub-module calls the task grouping list, matches execution nodes corresponding to classification tasks, sorts them according to node load and task priority to obtain a classification task allocation path.

[0019] As a further scheme of the present application, the classification effect monitoring module comprises:

[0020] The task completion extraction sub-module extracts the start time, completion time and error marker record of a classification task based on the classification task allocation path, merges task execution sections and generates a task completion time sequence;

[0021] The error marker analysis sub-module calls the task completion time sequence, identifies the error marker distribution in a classification task, filters tasks with error marker frequency exceeding a threshold and performs classification judgment to generate an error marker distribution list;

[0022] The classification quality evaluation sub-module calculates a classification quality evaluation value according to the error marker distribution list in combination with classification consistency and accuracy indexes, marks abnormal sections in a classification task and generates a classification quality evaluation dataset.

[0023] As a further scheme of the present application, the cross-domain data migration module comprises:

[0024] The abnormal section extraction sub-module extracts classification task records corresponding to each abnormal section according to the classification quality evaluation dataset, analyzes the execution state of a task and the reason for an error marker and generates an abnormal task record set;

[0025] The rule adaptation analysis sub-module identifies the semantic features and classification requirements of target domain data based on the abnormal task record set, matches the adaptability of classification rules, filters rule update items meeting requirements and generates a rule adaptation result set;

[0026] The optimization condition screening submodule performs condition adaptability judgment on each rule update item according to the rule adaptation result set, combines the classification task demand and the resource state, calculates a rule optimization adaptability index, identifies rule update items meeting the optimization condition, and generates a task adjustment implementation list.

[0027] As a further scheme of the present application, the system further comprises a rule optimization feedback module:

[0028] The rule optimization feedback module calls the rule update items in the task adjustment implementation list, compares the execution effects of the original classification rules and the optimized rules, analyzes the accuracy rate improvement and error label reduction of the classification task, screens performance improvement rules, marks the optimization matching items, and generates a foreign trade data classification rule mapping table.

[0029] The foreign trade data classification rule mapping table comprises a rule optimization score result, a task execution improvement identifier, a rule matching degree, and a classification task optimization item.

[0030] As a further scheme of the present application, the rule optimization feedback module comprises:

[0031] The rule effect comparison submodule calls the rule update items in the task adjustment implementation list, identifies the execution logs of the original classification rules and the optimized rules, calculates a classification performance improvement value by comparing and analyzing the accuracy rate improvement and error label reduction of the classification task, and generates a rule effect comparison result.

[0032] The performance improvement screening submodule screens tasks with performance improvement and few error labels in the classification task based on the rule effect comparison result, records the screening standard, marks the tasks meeting the condition, and generates a performance improvement task set.

[0033] The optimization mapping generation submodule screens tasks matching optimization according to the performance improvement task set, determines the optimization priority of each task, and generates a foreign trade data classification rule mapping table.

[0034] Compared with the prior art, the present application has the following advantages and positive effects:

[0035] In the present application, by performing semantic analysis based on language features and context association rules, the semantic structure of foreign trade data can be effectively recognized and classified, and the data is subjected to theme attribution analysis by combining a theme clustering algorithm, thereby reducing the dependence on manual rules in the classification process and improving the accuracy and automation degree of classification. By dynamically adjusting the classification strategy, the task allocation path can be optimized according to actual needs to avoid the inefficiency problem in traditional manual classification, and by monitoring the classification effect, abnormal data can be identified in time and the classification rules can be adjusted, thereby improving the overall classification quality. The cross-domain data migration and rule optimization feedback mechanism ensures that the classification rules can be adapted in different fields, enhances the flexibility and expansibility of the system, and effectively improves the overall efficiency and precision of foreign trade data processing. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a system flowchart of the present application;

[0037] Figure 2 is a semantic pattern recognition module flowchart in the present application;

[0038] Figure 3 is a classification task execution module flowchart in the present application;

[0039] Figure 4 is a classification effect monitoring module flowchart in the present application;

[0040] Figure 5 is a cross-domain data migration module flowchart in the present application;

[0041] Figure 6 is a rule optimization feedback module flowchart in the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0043] In the description of the present application, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, in the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0044] Please refer toFigure 1 The foreign trade data intelligent classification system based on natural language processing comprises:

[0045] The semantic pattern recognition module extracts the grammatical structure, vocabulary distribution, sentence pattern change and semantic weight in the text based on the language features and context association rules of the foreign trade data, analyzes the theme attribution of the data content by combining a theme clustering algorithm, and generates a semantic classification label set;

[0046] The classification task execution module calls the semantic classification label set, extracts label data features according to a preset classification rule library and a dynamic adjustment strategy, matches corresponding items in the rule library, filters a target data set and assigns classification labels, rearranges the task order according to the label priority, and obtains a classification task allocation path;

[0047] The classification effect monitoring module extracts the task completion time and error marking record in the classification process based on the classification task allocation path, judges the consistency of the classification result, identifies abnormal sections in the classification task, and generates a classification quality evaluation data set;

[0048] The cross-domain data migration module identifies the adaptability problem of the classification rule according to the abnormal sections marked in the classification quality evaluation data set, analyzes the semantic features and classification requirements of the target domain data, and generates a task adjustment implementation list;

[0049] The rule optimization feedback module calls the rule update items in the task adjustment implementation list, compares the execution effects of the original classification rule and the optimized rule, analyzes the accuracy rate improvement and error marking reduction of the classification task, filters performance improvement rules, marks the optimized matching items, and generates a foreign trade data classification rule mapping table;

[0050] The semantic classification label set comprises a grammatical complexity label, a vocabulary frequency distribution label, a theme weight distribution, a sentence structure identifier, and a semantic correlation degree label. The classification task allocation path comprises a classification rule priority sequence, a task grouping list, a classification node mapping table, and a task execution order identifier. The classification quality evaluation data set comprises a classification consistency score, an error marking distribution, a task abnormal section identifier, and a classification accuracy evaluation value. The task adjustment implementation list comprises a classification rule update item, a semantic feature adaptation identifier, and an optimization condition list. The foreign trade data classification rule mapping table comprises a rule optimization score result, a task execution improvement identifier, a rule matching degree, and a classification task optimization item.

[0051] Please refer to Figure 2 The semantic pattern recognition module comprises:

[0052] The grammatical structure analysis submodule extracts syntactic structure information, part-of-speech tagging sequences and sentence pattern change rules based on the language features and context association rules of the foreign trade data, identifies the grammatical complexity distribution in the text, and generates a grammatical complexity value;

[0053] The language features of the cleaned foreign trade data collected from foreign trade business emails, customs documents, contract texts, etc., such as vocabulary richness, average sentence length, frequency of subordinate clauses, and context association rules obtained by analysis, are used to extract syntactic structure information in foreign trade data, such as subject-predicate-object structure, modifier components, identify part-of-speech tagging sequences, such as the arrangement of nouns, verbs, and adjectives, analyze sentence pattern variation rules, such as the frequency of passive voice and inverted sentences, and calculate a value representing the complexity of the text syntax structure, such as setting the recognition frequency of subject-predicate-object structure to a weight of 0.3, setting the frequency of complex clauses to a weight of 0.5, and setting the frequency of passive voice to a weight of 0.2. The weighted sum of the syntax complexity value is obtained, for example, the syntactic structure information of a certain email identifies subject-predicate-object structure 5 times, the frequency of complex clauses is 0.1 (the total number of sentences in the text is 10, and complex clauses appear 1 time), and the frequency of passive voice is 0.05 (the total number of sentences in the text is 10, and passive voice appears 0.5 times, i.e., 0.05 times per sentence on average), then the syntax complexity value is The syntax complexity value of this email is 1.56.

[0054] The vocabulary distribution analysis submodule calls the syntax complexity value, combines the vocabulary frequency distribution and semantic weight, analyzes the distribution characteristics of the vocabulary under the differentiated theme, and determines whether it meets the theme clustering conditions. The formula is:

[0055] ;

[0056] Get the semantic weight value;

[0057] Wherein, represents the semantic weight value, represents the weight of the th vocabulary, represents the frequency of the th vocabulary, represents the variation of the theme distribution, represents the average value of the theme distribution, represents the total number of words;

[0058] The call syntax complexity value 1.56 is combined with the word frequency distribution in foreign trade data and the preset word semantic weight to analyze the distribution characteristics of words under different themes. For example, under the theme "payment method", the word "letter of credit" has a high frequency, and the semantic weight is set to 0.8, the word "telegraphic transfer" has a lower frequency, and the semantic weight is set to 0.6. When the frequency of a certain word exceeds the set frequency threshold, for example, the frequency threshold is set to 1.5 times the average frequency of the word in the theme, and its semantic weight is higher than the set semantic weight threshold, for example, the semantic weight threshold is set to 0.7, it is judged that the word meets the theme clustering condition, and the semantic weight value is calculated by the formula;

[0059] The overall semantic weight value of the text is used to quantify the importance and relevance of the text under a certain theme;

[0060] The weight of the first word in the preset word semantic weight library reflects the importance of the word in the field. The weight is set according to expert experience and the relevance of the word and co-occurrence words in a large number of foreign trade texts. For example, the word "Invoice" is important in trade contracts, and the weight can be set to 0.9. The word "Attachment" is relatively less important, and the weight can be set to 0.5. The weight value is set in the range of 0 to 1, for example, for the word "letter of credit", ;

[0061] The frequency of the first word in the current text to be analyzed is obtained by counting the number of words in the text. For example, in a certain email, the word "letter of credit" appears times, the word "telegraphic transfer" appears times, the word "shipment" appears times, and the word "bill of lading" appears times;

[0062] The change amount of the current text relative to the average value of the theme distribution is calculated as the Euclidean distance between the theme vector of the current text and the average theme vector, which reflects the degree of deviation of the text in the theme. The average theme vector is obtained by averaging the theme vectors of a large number of similar foreign trade texts. For example, the Euclidean distance between the theme vector of the current email and the average theme vector is calculated as ;

[0063] The average value of the theme distribution of a large number of similar foreign trade texts is obtained by averaging the theme vectors of 10,000 historical foreign trade emails, and the length of the average theme vector is Here, the length of the module is used as a representation of the average value;

[0064] represents the total number of words in the current text to be analyzed, for example, after the word segmentation and stop word filtering of the email, the total number of words is ;

[0065] The core logic of the formula is to measure the semantic importance of the text by accumulating the weight and frequency of the words, combined with the deviation of the text theme distribution. The parameter values are substituted into the formula for calculation, for example, the weighted frequency of the first four words is calculated: , where the weight of the word "shipment" is set to 0.75, and the weight of the word "bill of lading" is set to 0.7. For the remaining 196 words, assume that the sum of the weighted frequencies of the remaining 196 words is 60, then , the calculated sum of the weighted frequencies of the words is 86.9, the theme distribution change , the average value of the theme distribution Substitute the semantic weight value calculated by the formula :

[0066] ;

[0067] The benefit of the formula is that by introducing the change of the theme distribution, it can correct the semantic weight calculated by simply relying on the weighted frequency of the words, so that it can better reflect the deviation of the text from the typical theme, and thus more accurately evaluate its semantic importance under the differentiated theme. The result value 87.2 indicates that the text has a high semantic weight under the current analyzed theme, which meets the conditions for further theme clustering. The calculated semantic weight value 87.2 is used as the semantic weight value of the text, and the semantic weight value is obtained, for example, the semantic weight value of the email is 87.2.

[0068] The theme clustering generation submodule extracts the theme distribution features in the text according to the semantic weight value, and performs theme clustering combined with the context association rules to obtain a set of semantic classification tags;

[0069] According to the text semantic weight value 87.2, the theme distribution features in the text are extracted, for example, the high-frequency theme words and phrases in the text are identified, and the context association rules obtained from the foreign trade data are combined, for example, the word "letter of credit" often appears with "issuing bank" and "negotiation", and theme clustering is performed, for example, the clustering algorithm is used to cluster the texts with similar semantic weight values and similar theme features into the same theme category, for example, all emails about "letter of credit payment" are clustered into a theme, and a set of semantic classification tags is obtained, for example, a set containing "payment method: letter of credit", "cargo transportation: sea transportation", "document processing: customs declaration" and other tags is generated.

[0070] Please refer toFigure 3 The classification task execution module comprises:

[0071] The rule matching submodule calls the semantic classification label set, detects the matching degree of the rules and the semantic labels according to the rule entries in the preset classification rule library, filters the rules meeting the conditions and sorts them according to the priority, and generates a classification rule priority sequence;

[0072] The semantic classification label set is called, such as the semantic classification label “payment method: letter of credit” and the semantic classification label “cargo transportation: sea transportation”, the matching degree of the semantic labels and the rule entries is detected according to the rule entries in the preset classification rule library, for example, there is a rule in the rule library: “if the text contains the label ‘payment method: letter of credit’ and contains the label ‘cargo transportation: sea transportation’, then classify as ‘letter of credit sea transportation transaction’”, for example, it is judged whether the semantic label set of the text completely contains or partially contains the label specified in the rule entry, the rules meeting the conditions are filtered, for example, the rule “letter of credit sea transportation transaction” is filtered, and the rules are sorted according to the preset priority, for example, the rules can be set according to the proportion of the text covered, the historical classification accuracy rate and the like, and the high-priority rules are matched first, for example, the priority of the rule “letter of credit sea transportation transaction” is set to 10, and the priority of a more general “sea transportation transaction” rule is set to 5, the higher the priority value, the higher the priority, and the rules are sorted according to the size of the priority value, and a classification rule priority sequence is generated, for example, a sequence containing the rules “letter of credit sea transportation transaction (priority 10)” and “sea transportation transaction (priority 5)” is generated.

[0073] The task grouping optimization submodule groups the classification tasks based on the classification rule priority sequence and in combination with the dynamic adjustment strategy, generates a task grouping list according to the task complexity and resource occupation;

[0074] Based on the classification rule priority sequence, for example, according to the rule matching submodule output “letter of credit sea transportation transaction (priority 10)” rule priority sequence, the classification tasks of foreign trade data are grouped in combination with the dynamic adjustment strategy, for example, according to the processing capacity of the current system and the number of tasks to be classified, the tasks with similar priority or similar processing requirements are grouped into the same group, according to the task complexity, such as the length of the text, the number of rules contained, and the resource occupation, such as the CPU and memory usage estimation, for example, all the letter of credit related mail classification tasks with a priority greater than or equal to 8 are grouped into the first group and assigned to a high-performance processing node, and other sea transportation transaction mail classification tasks are grouped into the second group and assigned to a normal processing node, and a task grouping list is generated.

[0075] The node mapping generation submodule calls the task grouping list, matches the execution nodes corresponding to the classification tasks, sorts them according to the node load and task priority, and obtains a classification task distribution path;

[0076] The first group of credit-related classification tasks in the task grouping list is called, and the execution nodes corresponding to the classification tasks are matched, for example, according to the task type in the task grouping list and the preset node function correspondence relationship, the credit-related tasks are mapped to node A with encryption processing capability, and the ordinary sea transportation tasks are mapped to node B, according to the current load of the nodes, for example, the CPU utilization of node A is 70%, the CPU utilization of node B is 30%, and the task priority, for example, the priority of the first group of tasks is 10, and the priority of the second group of tasks is 5, sorting, for example, preferentially assigning tasks with high priority and low node load to the node, calculating the predicted completion time of node A processing the first group of tasks, calculating the predicted completion time of node B processing the second group of tasks, comparing the predicted time, adjusting the task allocation order between nodes, obtaining the classification task allocation path, for example, determining the specific execution order of the first group of tasks on node A, and the specific execution order of the second group of tasks on node B.

[0077] Please refer to Figure 4 The classification effect monitoring module comprises:

[0078] The task completion extraction submodule extracts the start time, completion time and error mark record of the classification task based on the classification task allocation path, merges the task execution section, and generates a task completion time sequence;

[0079] Based on the classification task allocation path, for example, according to the specific execution order of the classification task on node A and node B and the corresponding timestamp plan, the start time, completion time and error mark record of each classification task in the actual execution process are extracted, for example, the system time stamp of the start and end of each task is obtained through the monitoring node execution log, and the error mark record, for example, whether there is "classification error", "rule mismatch" and other error mark information is extracted from the task execution result, the task execution section is merged, for example, the same type of tasks executed continuously or the tasks executed in the same time period are merged into a section, and a task completion time sequence is generated, for example, a state sequence containing task ID, start time, completion time and error mark is generated.

[0080] The error mark analysis submodule calls the task completion time sequence, identifies the error mark distribution in the classification task, filters tasks with error mark frequency exceeding a threshold and performs classification judgment, and generates an error mark distribution list;

[0081] The task completion time sequence, such as sequence data containing task execution time and error label information, is called to identify the error label distribution in the classification task, the number of occurrences of different types of error labels is counted, tasks with error label frequency exceeding the threshold are screened, and the error label frequency threshold is set to 5%. That is, if the number of error labels exceeds 5 times in 100 consecutive tasks of the same type, it is considered that the error label frequency of the type task exceeds the threshold, a classification judgment is made, such as judging whether the error label is due to data quality problems or rule mismatch problems, and an error label distribution list is generated, such as generating a list containing task type, error label type, error label quantity, and whether it exceeds the threshold.

[0082] The classification quality evaluation submodule generates an error label distribution list, combines classification consistency and accuracy indicators, and uses the formula:

[0083] ;

[0084] The classification quality evaluation value is calculated, the abnormal section in the classification task is marked, and the classification quality evaluation dataset is generated.

[0085] Wherein, represents the classification quality evaluation value, represents the total number of classification tasks, represents the number of error labels, represents the maximum value of task completion time, represents the average value of task completion time.

[0086] According to the error label distribution list, such as a list containing the number of error labels, combined with the classification consistency and accuracy indicators, such as comparing the classification results of manual sampling and checking with the system classification results, the classification consistency and accuracy are calculated, and the classification quality evaluation value is calculated using the formula:

[0087] represents the overall classification quality evaluation value of the current classification task execution section, and the higher the value, the better the classification quality, and the evaluation value is set in the range of 0 to 1.

[0088] represents the total number of classification tasks in the current evaluation section, which is obtained by counting the number of task records in the task completion time sequence in the section, such as the section containing classification tasks;

[0089] represents the number of error labels in the current evaluation section, which is obtained by counting the number of error label tasks in the error label distribution list in the section, such as error label tasks in the section;

[0090] This represents the maximum completion time of a single task within the current evaluation segment, obtained by analyzing the task completion time series. For example, the maximum task completion time within this segment is... Second;

[0091] This represents the average completion time of a single task within the current evaluation segment. It is obtained by analyzing the task completion time series, calculating the sum of all task completion times, and dividing by the total number of tasks. For example, the average task completion time within this segment is... Second;

[0092] The core logic of the formula lies in simultaneously considering the accuracy of classification and processing efficiency, substituting the values ​​of each parameter into the formula for calculation, for example, ... , , , Substitute into the formula to calculate the classification quality assessment value :

[0093] ;

[0094] The advantage of the formula is that it focuses not only on the error rate ( (Partially), and also introduced the factor of task completion time ( The ratio of maximum completion time to average completion time (partial) reflects the uniformity of task processing time distribution. If a few tasks take exceptionally long, this ratio will increase, lowering the overall quality assessment value and prompting the system to focus on and optimize abnormal tasks that cause process blockages. A result of 1.979 exceeds the preset classification quality benchmark value (e.g., a benchmark value of 0.8), indicating that the classification quality of this segment is acceptable. Abnormal segments in the classification tasks are then marked. If the quality threshold is less than 0.7, the segment is marked as an anomaly, and a classification quality assessment dataset is generated. For example, a dataset containing segment identifier, total number of tasks, number of errors, maximum completion time, average completion time, evaluation value, and whether it is marked as an anomaly is generated.

[0095] Please see Figure 5 The cross-domain data migration module includes:

[0096] The abnormal segment extraction submodule extracts the classification task record corresponding to each abnormal segment based on the classification quality assessment dataset, analyzes the execution status of the task and the reason for the error labeling, and generates an abnormal task record set.

[0097] According to the classification quality evaluation data set, the data set containing abnormal marking information, the classification task record corresponding to each abnormal section is extracted, the detailed record of all tasks contained in the section marked as abnormal is screened, the execution state of the task is analyzed, for example, is it successfully completed but marked incorrectly, or is it failed to execute, and the error marking reason, by analyzing the error log, it is determined whether it is caused by non-standard data format, rule conflict or resource shortage, an abnormal task record set is generated, for example, a set containing abnormal task ID, error type, error reason and related data segment is generated.

[0098] The rule adaptation analysis submodule identifies the semantic features of the target domain data and the classification requirements based on the abnormal task record set, matches the adaptability of the classification rules, screens the rule update items that meet the requirements, and generates a rule adaptation result set;

[0099] According to the abnormal task record set, the semantic features of the target domain data are identified, for example, the difference between the word usage habits, sentence structure in the target domain (for example, the documents from new cooperative customers) and the existing rule training data, and the classification requirements, for example, the newly added document type needs a new classification label, the adaptability of the existing classification rules is matched, for example, the classification effect and error situation of the existing rules on the foreign trade document data are evaluated, the rule update items that meet the requirements are screened, for example, which rules need to modify parameters, add new judgment conditions or create new rules, and the rule adaptation result set is generated, for example, a set containing suggested modified rule ID, suggested modification content, suggested new rule content and adaptability score is generated.

[0100] The optimization condition screening submodule judges the condition adaptability of each rule update item according to the rule adaptation result set, combines the classification task requirements and resource state, and uses the formula:

[0101] ;

[0102] The rule optimization adaptability index is calculated, the rule update items that meet the optimization conditions are identified, and a task adjustment implementation list is generated;

[0103] Among them, represents the rule optimization adaptability index, represents the number of rule update items, represents the weight coefficient of the th rule update item, represents the condition compliance degree of the th rule update item, represents the resource occupation degree of the th rule update item, represents the differentiation weight of the th rule update item;

[0104] According to the rule adaptation result set, the conditional adaptability of each rule update item is judged, whether the modified rule can correctly process the data in the abnormal task set, and at the same time, the accuracy of the existing normal classification task is not affected, combined with the classification task demand, the error type with the widest influence needs to be solved first, and the resource state, such as whether the current system computing resource and storage resource are sufficient to support batch verification and deployment of the rule, is used to calculate the rule optimization adaptability index;

[0105] The overall optimization adaptability index of the rule update item in the current evaluation batch is represented, and the smaller the value is, the better the overall adaptability of the rule update item is, and the more it meets the optimization implementation condition;

[0106] The number of rule update items in the current evaluation batch is represented, which is obtained by counting the number of items in the task adjustment implementation list, for example, there are rule update items in this batch;

[0107] The weight coefficient of the rule update item is represented, which reflects the importance of the update item, the weight of the rule update item solving the serious error type is high, and the weight of the rule update item solving the slight error is low, the weight is set according to the influence range, frequency of the error and the priority set by the artificial, and the setting range is between 0 and 1, for example, the first update item solves the high-frequency error, ;

[0108] The condition compliance degree of the rule update item is represented, which evaluates the effectiveness of the update item in solving the abnormal task it aims at and the non-interference to the normal task, the value range is between 0 and 1, the update item is evaluated by running on the test set containing abnormal data and normal data, for example, a rule update item makes 95% of the abnormal data correctly classified on the test set, and does not introduce new errors, then ;

[0109] The resource occupation degree of the rule update item is represented, which measures the consumption of the update item to computing and storage resources in verification, deployment and running, the value range is greater than 0, for example, the resource occupation degree is obtained by normalizing the CPU and memory occupation peak value of the update item running in the standard test environment, for example, the resource occupation of a certain update item is low, ;

[0110] The resource occupation degree of the a differentiated weight of each rule update item, used to punish or reward specific types of rule updates, for example, setting a higher differentiated weight for rule modifications involving sensitive information to increase the difficulty of review, the value range is greater than or equal to 0, for example, some update items involve core classification logic modification, ;

[0111] The core logic of the formula is to comprehensively evaluate the importance, effectiveness, resource consumption and specific differentiated factors of the rule update items, and to amplify the impact of important update items by weight coefficients , partially balance effectiveness and resource consumption, and differentiated weight introduce specific considerations, and finally measure the deviation of this batch of update items from the ideal state by summing and absolute value, calculate the optimization adaptability index of the batch containing 5 update items by substituting the parameter values into the formula, for example , assuming that the parameter values are as shown in Table 1;

[0112]

[0113] Referring to Table 1, the parameters of each rule update item are substituted into the formula for calculation:

[0114] Item 1: ;

[0115] Item 2: ;

[0116] Item 3: ;

[0117] Item 4: ;

[0118] Item 5: ;

[0119] The calculation results of each item are summed to obtain the overall optimization adaptability index :

[0120] ;

[0121] The formula is beneficial in that it constructs a comprehensive index to quantitatively evaluate the overall pros and cons of a batch of rule update items, especially balancing the improvement effect brought by the update items, their importance and the required resource input, thereby providing a quantitative basis for selecting and implementing rule updates. The result value 15.836 is compared with the preset optimization threshold, for example, if the threshold is set to 20, if is less than or equal to the threshold, it is considered that the batch of rule update items meets the optimization condition, and the rule update items meeting the optimization condition are identified, for example, the calculated The five rule update items are determined to meet the optimization condition, and a task adjustment implementation list is generated, for example, a list containing the five rule update items meeting the condition and corresponding parameters is generated.

[0122] Referring to Figure 6 The rule optimization feedback module includes:

[0123] The rule effect comparison submodule calls the rule update items in the task adjustment implementation list, identifies the execution logs of the original classification rules and the optimized rules, compares and analyzes the accuracy rate improvement and error label reduction of the classification task, calculates the classification performance improvement value, and generates a rule effect comparison result;

[0124] The rule update items in the task adjustment implementation list are called, including the list of rules suggested for modification and new rules, the execution logs of the original classification rules and the optimized rules on foreign trade data are identified, for example, the original rules and the optimized rules are respectively run on the historical data set, the execution process and result are recorded, the accuracy rate improvement of the classification task is compared and analyzed, the number of correctly classified documents of the optimized rules relative to the original rules is counted, and the error label reduction is analyzed, for example, the number of error labels generated by the optimized rules is counted relative to the reduction of the original rules, the classification performance improvement value is calculated, for example, the percentage improvement of accuracy rate or the percentage reduction of error label number, and a rule effect comparison result is generated, for example, a comparison result list containing rule ID, original accuracy rate, optimized accuracy rate, accuracy rate improvement percentage, original error number, optimized error number, and error number reduction percentage.

[0125] The performance improvement filtering submodule filters the tasks with performance improvement and few error labels in the classification task based on the rule effect comparison result, records the filtering criteria, marks the tasks meeting the criteria, and generates a performance improvement task set;

[0126] Based on the rule effect comparison result, the rule update items corresponding to the tasks with performance improvement and few error labels in the classification task are filtered, for example, the filtering criteria is set as accuracy rate improvement greater than or equal to 2% and error label reduction greater than or equal to 10%, the filtering criteria is recorded, for example, “accuracy rate improvement >= 2% AND error label reduction >= 10%” is recorded, the tasks meeting the criteria are marked, for example, the rule effect comparison result list is traversed, and the rule update items meeting the filtering criteria are marked, and a performance improvement task set is generated, for example, a set containing the rule update item IDs meeting the filtering criteria is generated.

[0127] The optimization mapping generation submodule filters the tasks matching the optimization according to the performance improvement task set, determines the optimization priority of each task, and generates a foreign trade data classification rule mapping table;

[0128] According to the performance improvement task set, the rule update item set meeting the performance improvement standard is updated, the tasks matched with optimization are screened, the tasks that will benefit from the optimization rules are found from the original tasks to be classified, the optimization priority of each task is determined, the highest priority is set for the task using the highest accuracy improvement rule, the higher priority is set for the task using the rule of the most reduction of error labels, the priority value is calculated according to the performance improvement value and the error reduction value, and the foreign trade data classification rule mapping table is generated.

[0129] The above is only the preferred embodiment of the present application, and does not limit the present application in other forms. Any skilled person in the art can use the disclosed technical content to make changes or modifications into equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments without departing from the technical solution content of the present application still belongs to the protection scope of the technical solution of the present application.

Claims

1. A foreign trade data intelligent classification system based on natural language processing, characterized in that: The system includes: The semantic pattern recognition module extracts grammatical structure, vocabulary distribution, sentence variation and semantic weight from the text based on the language features and context association rules of foreign trade data. It combines topic clustering algorithm to analyze the topic attribution of data content and generate a set of semantic classification labels. The classification task execution module includes a rule matching submodule, a task grouping optimization submodule, and a node mapping generation submodule; The rule matching submodule calls the semantic classification tag set, detects the degree of matching between the rules and semantic tags according to the rule entries in the preset classification rule library, filters the rules that meet the conditions and sorts them by priority to generate a classification rule priority sequence; The task grouping optimization submodule groups the classified tasks based on the priority sequence of the classification rules and combines a dynamic adjustment strategy, and generates a task grouping list according to the task complexity and resource consumption. The node mapping generation submodule calls the task grouping list, matches the execution nodes corresponding to the categorized tasks, sorts them according to node load and task priority, and obtains the categorized task allocation path; The classification performance monitoring module extracts the task completion time and error mark records during the classification process based on the classification task allocation path, judges the consistency of the classification results, identifies abnormal segments in the classification task, and generates a classification quality assessment dataset. The cross-domain data migration module identifies the adaptability issues of classification rules based on the abnormal segments marked in the classification quality assessment dataset, analyzes the semantic features and classification requirements of the target domain data, and generates a task adjustment implementation list.

2. The intelligent classification system for foreign trade data based on natural language processing according to claim 1, characterized in that, The semantic classification label set includes grammatical complexity labels, vocabulary frequency distribution labels, topic weight distribution, sentence structure identifiers, and semantic relevance labels. The classification task allocation path includes a classification rule priority sequence, a task grouping list, a classification node mapping table, and a task execution order identifier. The classification quality evaluation dataset includes a classification consistency score, error label distribution, task abnormal segment identifiers, and classification accuracy evaluation values. The task adjustment implementation list includes classification rule update items, semantic feature adaptation identifiers, and an optimization condition list.

3. The intelligent classification system for foreign trade data based on natural language processing according to claim 1, characterized in that, The semantic pattern recognition module includes: The grammar structure parsing submodule extracts syntactic structure information, part-of-speech tagging sequences and sentence variation patterns based on the language features and context association rules of foreign trade data, identifies the distribution of grammatical complexity in the text, and generates grammatical complexity values. The vocabulary distribution analysis submodule calls the grammatical complexity value, combines the vocabulary frequency distribution and semantic weight, analyzes the distribution characteristics of vocabulary under differentiated topics, determines whether it meets the topic clustering conditions, and obtains the semantic weight value. The topic clustering generation submodule extracts topic distribution features from the text based on the semantic weight values, and performs topic clustering in combination with context association rules to obtain a set of semantic classification labels.

4. The intelligent classification system for foreign trade data based on natural language processing according to claim 1, characterized in that, The classification effect monitoring module includes: The task completion extraction submodule extracts the start time, completion time and error mark records of the classified tasks based on the classified task allocation path, merges the task execution segments, and generates a task completion time sequence. The error labeling analysis submodule calls the task completion time series, identifies the error labeling distribution in the classified tasks, filters tasks whose error labeling frequency exceeds the threshold and performs classification judgment, and generates an error labeling distribution list. The classification quality assessment submodule calculates the classification quality assessment value based on the error label distribution list and the classification consistency and accuracy indicators, marks the abnormal segments in the classification task, and generates the classification quality assessment dataset.

5. The intelligent classification system for foreign trade data based on natural language processing according to claim 4, characterized in that, The cross-domain data migration module includes: The abnormal segment extraction submodule extracts the classification task record corresponding to each abnormal segment based on the classification quality assessment dataset, analyzes the execution status and error marking reasons of the task, and generates an abnormal task record set. The rule adaptation analysis submodule identifies the semantic features and classification requirements of the target domain data based on the abnormal task record set, matches the adaptability of classification rules, filters out rule update items that meet the requirements, and generates a rule adaptation result set. The optimization condition filtering submodule performs condition adaptability judgment on each rule update item based on the rule adaptation result set, calculates the rule optimization adaptability index by combining the classification task requirements and resource status, identifies the rule update items that meet the optimization conditions, and generates a task adjustment implementation list.

6. The intelligent classification system for foreign trade data based on natural language processing according to claim 1, characterized in that, The system also includes a rule optimization feedback module: The rule optimization feedback module calls the rule update items in the task adjustment implementation list, compares the execution effect of the original classification rules with the optimized rules, analyzes the improvement in the accuracy of classification tasks and the reduction in error marking, filters the performance improvement rules, marks the optimization matching items, and generates a foreign trade data classification rule mapping table. The foreign trade data classification rule mapping table includes rule optimization scoring results, task execution improvement indicators, rule matching degree, and classification task optimization items.

7. The intelligent classification system for foreign trade data based on natural language processing according to claim 6, characterized in that, The rule optimization feedback module includes: The rule effect comparison submodule calls the rule update item in the task adjustment implementation list, identifies the execution logs of the original classification rules and the optimized rules, and calculates the classification performance improvement value by comparing and analyzing the improvement in the accuracy of the classification task and the reduction in error markings, and generates rule effect comparison results. Based on the results of the comparison of the effects of the rules, the performance improvement filtering submodule filters out tasks with improved performance and fewer error markings in the classified tasks, records the filtering criteria, marks tasks that meet the conditions, and generates a set of performance improvement tasks. The optimization mapping generation submodule filters and optimizes matching tasks based on the performance improvement task set, determines the optimization priority of each task, and generates a foreign trade data classification rule mapping table.

Images