Natural language processing system and method based on data driving

Abstract

The invention discloses a natural language processing system and method based on data driving, and relates to the technical field of natural language processing.The method comprises the steps that response time and data features in each time of natural language processing in the history are extracted, the correlation degree between the response time and the data features is calculated, and complex features influencing original data complexity are screened out; calculating the complexity of the real-time original data by using the feature functions of all the complex features; the model calculation amount is analyzed to obtain division nodes, multi-stage processing paths are constructed, and natural language processing models are constructed for the multi-stage processing paths respectively; calculating a mapping relation between the complexity and the calculation amount, setting a complexity threshold value, performing training optimization on the complexity threshold value through the verification set by utilizing reinforcement learning, and setting a training target; constructing a dynamic routing gating function, and selecting a real-time processing path; and constructing a gradient cooperative training mechanism, and combining a plurality of processing paths to train and optimize a complexity threshold.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a data-driven natural language processing system and method. Background Art

[0002] In the early days of natural language processing, rule-based approaches were primarily used. For example, based on Chomsky's generative grammar theory, formal rules were used to describe human language. Parsing and generating sentences involved manually compiling numerous grammatical rules and dictionaries. This approach was somewhat effective for processing simple language fragments with clear rules. However, given the complexity, diversity, and ambiguity of natural language, such as varying linguistic conventions, emerging vocabulary, and expressions, rule writing and maintenance were extremely costly, and rules struggled to cover all situations, resulting in poor adaptability.

[0003] With the widespread use of the internet and the rapid development of information technology, the amount of data has exploded, providing a rich source of material for data-driven natural language processing. Vast amounts of text data, such as web pages, social media content, news reports, and academic literature, have become a valuable resource for training natural language processing models. This data covers a wide range of fields and topics, enabling models to learn rich linguistic expressions and semantic information. However, in current data-driven natural language processing, fixed-structure models are often used for different processing tasks, significantly reducing the system's reasoning efficiency. Therefore, the ability to flexibly select different processing paths based on the difficulty of the task is crucial for processing efficiency. Summary of the Invention

[0004] The purpose of the present invention is to provide a data-driven natural language processing system and method to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solutions: A data-driven natural language processing method, comprising the following steps: S100: Collect all raw data from historical natural language processing, extract the response time and data features of each natural language processing in history, calculate the correlation between the response time and data features, and filter out complex features that affect the complexity of the raw data; Furthermore, the specific steps to filter out the complex features that affect the complexity of the original data are: S101. Extract all raw data from the natural language processing system log during historical natural language processing, remove duplicate data, and standardize the raw data. Extract the start and end timestamps of each natural language processing task, and calculate the response time of each natural language processing task. The formula is: ; In the formula, T i represents the response time of the i-th natural language processing in history, t end，i Indicates the end timestamp of the i-th natural language processing task in history, t start，i Indicates the start timestamp of the i-th natural language processing task in history; S102. Extract data features of the original data in each natural language processing task from the system log. The data features include but are not limited to data length and data semantics. Calculate the response time of each natural language processing task in history and the correlation between different data features. Standardize the response time and data features using the formula: ; In the formula, ρ(X j , T) represents the correlation between the jth data feature and the response time, X j，i represents the j-th data feature value in the i-th natural language processing task, Xp j represents the average value of the j-th data feature in n natural language processing tasks, T i represents the response time of the i-th natural language processing task, Tp represents the average response time of n natural language processing tasks, and n represents the total number of historical natural language processing tasks collected; S103. Sort the calculated correlations between all data features and response time from smallest to largest, calculate the differences between adjacent correlations, select the maximum difference as the boundary, set the maximum value of the two correlations corresponding to the boundary as the correlation threshold, use the correlation threshold to judge all correlations, and when the correlation is greater than the correlation threshold, treat the corresponding data feature as a complex feature to obtain a complex feature set.

[0006] Extracting response times and data features and calculating correlations accurately locates key features that significantly impact the complexity of the raw data, avoiding interference from irrelevant features and improving the efficiency and relevance of subsequent processing. Deduplication and standardization of the raw data ensures data quality and consistency, laying a solid foundation for subsequent analysis and modeling.

[0007] S200, extracting corresponding feature functions based on the screened complex features, and calculating the complexity of the real-time raw data using the feature functions of all the complex features; Furthermore, the specific steps for calculating the complexity of real-time raw data using the characteristic functions of all complex features are as follows: S201. Extract corresponding feature functions based on the selected complex features, set fusion weights for each feature function, and use the fusion weights to fuse the feature functions to build a complexity calculation model. The model is: ; In the formula, Com represents the complexity of the original data, w h Represents the fusion weight of the h-th feature function, g(s) h represents the hth characteristic function, and H represents the total number of characteristic functions; S202. Utilize a deep learning algorithm to train, optimize, and update the fusion weights of each feature function in a natural language processing system.

[0008] Extracting characteristic functions based on the selected complex features and constructing a complexity calculation model accurately quantifies the complexity of real-time raw data, providing an important basis for selecting subsequent processing paths. Using deep learning algorithms to train and optimize fusion weights enables the complexity calculation model to continuously adapt to data changes, improving the model's accuracy and generalization capabilities.

[0009] S300: Collect model computational loads from all historical natural language processing tasks, analyze the model computational loads to obtain partition nodes, divide the total computational load of the natural language processing model, construct a multi-level processing path, and construct a natural language processing model for each of the multi-level processing paths; Furthermore, the specific steps for constructing a multi-level processing path are: S301. Collect the natural language processing system computational loads from all historical natural language processing tasks, sort the natural language processing system computational loads from small to large, and draw a computational load curve. Derivative the computational load curve to obtain a change curve. Calculate the standard deviation and average value of the change curve. Add the average value and the standard deviation to set a change threshold. In the change curve, define the computational load curve points that are greater than the change threshold as partition nodes. Use the partition nodes to obtain an m-level processing path. S302. Set a natural language processing model for each level of processing path, and assume that the natural language processing model structure is {G: f(s) = Q layers of transformer, computational load}; where G represents the processing path number, transformer represents the natural language processing algorithm, and the value of Q is set by combining professional knowledge and computational load.

[0010] By analyzing the computational load of historical models and constructing multi-level processing paths, corresponding processing paths can be provided according to different computational load requirements, thus achieving reasonable allocation and utilization of resources.

[0011] A natural language processing model is constructed for each level of processing path, so that the model can better adapt to data processing requirements of different complexities and improve processing efficiency and effectiveness.

[0012] S400, calculating the mapping relationship between complexity and computational effort, setting a complexity threshold, using reinforcement learning to train and optimize the complexity threshold using a validation set, and setting a training target; Furthermore, the specific steps for setting training goals are: S401. Collect the historical computational load and raw data complexity of the natural language processing system when processing tasks, generate a mapping function between computational load and complexity using a linear regression algorithm, input the computational load of the partitioning nodes of the m-level processing path into the mapping function, and calculate the complexity threshold corresponding to each level of the processing path; S402: Utilize reinforcement learning to train and optimize the complexity threshold using the validation set, setting the training objective to: ; In the formula, Total represents the total loss, Total task represents the task loss, FLOPs represents the system computing efficiency, and λ represents the computing efficiency weight; the result with the minimum total loss is selected during training.

[0013] The mapping relationship between computational complexity and computational effort provides a scientific basis for setting the complexity threshold, so that the threshold can accurately reflect the relationship between data complexity and computational effort.

[0014] By using reinforcement learning to train and optimize the complexity threshold, the threshold can be continuously adjusted according to the actual processing effect, thereby improving the system's adaptability and performance.

[0015] S500: Build a dynamic routing gating function to judge the complexity of real-time raw data and select a real-time processing path; Furthermore, the specific steps for selecting the real-time processing path are: S501. Set the routing gating function using the calculated complexity threshold. The formula is: ; In the formula, Gate(s) represents the routing gating function, f1(s), f2(s), ..., f m-1 (s) represents the natural language processing model of the 1st, 2nd, ..., mth level processing path, Com(s) represents the complexity of the real-time raw data, α1, α2, ..., α m-1 represents the 1st, 2nd, ..., m-1th complexity threshold; S502: Input the real-time original data into the routing threshold function, determine the complexity threshold interval where the complexity of the real-time original data lies, and select the natural language model corresponding to the processing path to perform real-time task processing.

[0016] By building a dynamic routing gating function, it can automatically select the appropriate processing path according to the complexity of the real-time raw data, realize the intelligent and automated processing process, and improve the efficiency and accuracy of processing.

[0017] By judging the threshold range of the complexity of real-time data, the processing path is selected to ensure that the selection of the processing path is targeted and reasonable, and can better meet the processing requirements of different data.

[0018] S600: Construct a gradient collaborative training mechanism to optimize the complexity threshold by combining multiple processing paths for training when training the complexity threshold.

[0019] Furthermore, the specific steps for optimizing the complexity threshold by combining multiple processing paths with training are as follows: S601. When training the complexity threshold, obtain the deviation parameter of each processing path through reinforcement learning, extract the temperature coefficient during the training process, and calculate the gating probability of each processing path in the routing gating function during training. The formula is: ; In the formula, p k represents the gating probability of the kth processing path, b k represents the deviation parameter of the kth processing path, and τ represents the temperature coefficient; S602: Utilize the probability of each processing path to combine all processing paths during training to process the natural language processing task during training, and build a gradient collaborative training mechanism. The formula is: ; In the formula, y represents the natural language processing result during training, f k (s) represents the natural language processing model of the k-th processing path; it is trained using the gradient collaborative training mechanism, and the complexity threshold is trained, optimized and updated.

[0020] Constructing a gradient collaborative training mechanism and utilizing multiple processing paths to combine training to optimize the complexity threshold can fully utilize the advantages of different processing paths and improve the efficiency and accuracy of threshold training.

[0021] By calculating the gating probability of each processing path and combining it with processing tasks, the training process becomes more scientific and reasonable, and can better adapt to data changes and processing requirements.

[0022] A data-driven natural language processing system includes a data acquisition module, a complex feature module, a complexity calculation module, a processing path classification module, a complexity threshold module, a routing gating judgment module, and a training and updating module; The data collection module is used to collect all original data, response time and data features during natural language processing in history; The complex feature module is used to calculate the correlation between response time and data features, and filter out complex features that affect the complexity of original data; The complexity calculation module is used to extract corresponding feature functions based on the screened complex features, and calculate the complexity of the real-time raw data using the feature functions of all complex features; The processing path grading module is used to analyze the model calculation amount to obtain division nodes, divide the total calculation amount of the natural language processing model, and construct a multi-level processing path; The complexity threshold module is used to calculate the mapping relationship between complexity and computational effort, set the complexity threshold, use reinforcement learning to train and optimize the complexity threshold through a validation set, and set a training target; The routing gating judgment module is used to construct a dynamic routing gating function, judge the complexity of real-time raw data, and select a real-time processing path; The training update module is used to construct a gradient collaborative training mechanism, and when training the complexity threshold, multiple processing paths are used to combine training to optimize the complexity threshold.

[0023] The routing gating judgment module includes a routing gating function unit and a judgment unit; The routing gating function unit is used to set the routing gating function using the calculation complexity threshold; The judgment unit is used to input the real-time original data into the routing threshold function, judge the complexity threshold interval where the complexity of the real-time original data lies, and select the natural language model of the corresponding processing path to perform real-time task processing.

[0024] The training update module includes a gated probability unit and a gradient collaborative training mechanism unit; The gating probability unit is used to calculate the gating probability of each processing path using the deviation parameter and the temperature coefficient; The gradient collaborative training mechanism unit is used to utilize the probability of each processing path to combine all processing paths during training to process the natural language processing task during training, and to construct a gradient collaborative training mechanism to update the complexity threshold.

[0025] Compared with the prior art, the present invention has the following beneficial effects: 1. Through accurate assessment of data complexity and selection of dynamic routing gating functions, the present invention enables the system to automatically match the optimal processing path for data of different complexities, avoiding waste of resources and unnecessary calculations, thereby significantly improving the efficiency of natural language processing.

[0026] 2. Through the screening of complex features, precise calculation of complexity, and optimization of the gradient collaborative training mechanism, the system can process natural language data more accurately, improving the quality and accuracy of natural language processing.

[0027] 3. The present invention trains and updates the complexity threshold through a gradient collaborative training mechanism, solves the problem of non-differentiable processing paths, and ensures end-to-end training. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a module distribution diagram of a data-driven natural language processing system according to the present invention; Figure 2 This is a schematic diagram of the steps of a data-driven natural language processing method of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution. A data-driven natural language processing method, comprising the following steps: S100: Collect all raw data from historical natural language processing, extract the response time and data features of each natural language processing in history, calculate the correlation between the response time and data features, and filter out complex features that affect the complexity of the raw data; The specific steps to filter out the complex features that affect the complexity of the original data are: S101. Extract all raw data from the natural language processing system log during historical natural language processing, remove duplicate data, and standardize the raw data. Extract the start and end timestamps of each natural language processing task, and calculate the response time of each natural language processing task. The formula is: ; In the formula, T i represents the response time of the i-th natural language processing in history, t end，i Indicates the end timestamp of the i-th natural language processing task in history, t start，i Indicates the start timestamp of the i-th natural language processing task in history; S102. Extract data features of the original data in each natural language processing task from the system log. The data features include but are not limited to data length and data semantics. Calculate the response time of each natural language processing task in history and the correlation between different data features. Standardize the response time and data features using the formula: ; In the formula, ρ(X j , T) represents the correlation between the jth data feature and the response time, X j，i represents the j-th data feature value in the i-th natural language processing task, Xp j represents the average value of the j-th data feature in n natural language processing tasks, T i represents the response time of the i-th natural language processing task, Tp represents the average response time of n natural language processing tasks, and n represents the total number of historical natural language processing tasks collected; S103. Sort the calculated correlations between all data features and response time from smallest to largest, calculate the differences between adjacent correlations, select the maximum difference as the boundary, set the maximum value of the two correlations corresponding to the boundary as the correlation threshold, use the correlation threshold to judge all correlations, and when the correlation is greater than the correlation threshold, treat the corresponding data feature as a complex feature to obtain a complex feature set.

[0031] Extracting response times and data features and calculating correlations accurately locates key features that significantly impact the complexity of the raw data, avoiding interference from irrelevant features and improving the efficiency and relevance of subsequent processing. Deduplication and standardization of the raw data ensures data quality and consistency, laying a solid foundation for subsequent analysis and modeling.

[0032] S200, extracting corresponding feature functions based on the screened complex features, and calculating the complexity of the real-time raw data using the feature functions of all the complex features; The specific steps for calculating the complexity of real-time raw data using the characteristic functions of all complex features are: S201. Extract corresponding feature functions based on the selected complex features, set fusion weights for each feature function, and use the fusion weights to fuse the feature functions to build a complexity calculation model. The model is: ; In the formula, Com represents the complexity of the original data, w h Represents the fusion weight of the h-th feature function, g(s) h represents the hth characteristic function, and H represents the total number of characteristic functions; S202. Utilize a deep learning algorithm to train, optimize, and update the fusion weights of each feature function in a natural language processing system.

[0033] Extracting characteristic functions based on the selected complex features and constructing a complexity calculation model accurately quantifies the complexity of real-time raw data, providing an important basis for selecting subsequent processing paths. Using deep learning algorithms to train and optimize fusion weights enables the complexity calculation model to continuously adapt to data changes, improving the model's accuracy and generalization capabilities.

[0034] S300: Collect model computational loads from all historical natural language processing tasks, analyze the model computational loads to obtain partition nodes, divide the total computational load of the natural language processing model, construct a multi-level processing path, and construct a natural language processing model for each of the multi-level processing paths; The specific steps to build a multi-level processing path are: S301. Collect the natural language processing system computational loads from all historical natural language processing tasks, sort the natural language processing system computational loads from small to large, and draw a computational load curve. Derivative the computational load curve to obtain a change curve. Calculate the standard deviation and average value of the change curve. Add the average value and the standard deviation to set a change threshold. In the change curve, define the computational load curve points that are greater than the change threshold as partition nodes. Use the partition nodes to obtain an m-level processing path. S302. Set a natural language processing model for each level of processing path, and assume that the natural language processing model structure is {G: f(s) = Q layers of transformer, computational load}; where G represents the processing path number, transformer represents the natural language processing algorithm, and the value of Q is set by combining professional knowledge and computational load.

[0035] By analyzing the computational load of historical models and constructing multi-level processing paths, corresponding processing paths can be provided according to different computational load requirements, thus achieving reasonable allocation and utilization of resources.

[0036] A natural language processing model is constructed for each level of processing path, so that the model can better adapt to data processing requirements of different complexities and improve processing efficiency and effectiveness.

[0037] S400, calculating the mapping relationship between complexity and computational effort, setting a complexity threshold, using reinforcement learning to train and optimize the complexity threshold using a validation set, and setting a training target; The specific steps for setting a training target are: S401. Collect the historical computational load and raw data complexity of the natural language processing system when processing tasks, generate a mapping function between computational load and complexity using a linear regression algorithm, input the computational load of the partitioning nodes of the m-level processing path into the mapping function, and calculate the complexity threshold corresponding to each level of the processing path; S402: Utilize reinforcement learning to train and optimize the complexity threshold using the validation set, setting the training objective to: ; In the formula, Total represents the total loss, Total task represents the task loss, FLOPs represents the system computing efficiency, and λ represents the computing efficiency weight; the result with the minimum total loss is selected during training.

[0038] The mapping relationship between computational complexity and computational effort provides a scientific basis for setting the complexity threshold, so that the threshold can accurately reflect the relationship between data complexity and computational effort.

[0039] By using reinforcement learning to train and optimize the complexity threshold, the threshold can be continuously adjusted according to the actual processing effect, thereby improving the system's adaptability and performance.

[0040] S500: Build a dynamic routing gating function to judge the complexity of real-time raw data and select a real-time processing path; The specific steps for selecting the real-time processing path are: S501. Set the routing gating function using the calculated complexity threshold. The formula is: ; In the formula, Gate(s) represents the routing gating function, f1(s), f2(s), ..., f m-1 (s) represents the natural language processing model of the 1st, 2nd, ..., mth level processing path, Com(s) represents the complexity of the real-time raw data, α1, α2, ..., α m-1 represents the 1st, 2nd, ..., m-1th complexity threshold; S502: Input the real-time original data into the routing threshold function, determine the complexity threshold interval where the complexity of the real-time original data lies, and select the natural language model corresponding to the processing path to perform real-time task processing.

[0041] By building a dynamic routing gating function, it can automatically select the appropriate processing path according to the complexity of the real-time raw data, realize the intelligent and automated processing process, and improve the efficiency and accuracy of processing.

[0042] By judging the threshold range of the complexity of real-time data, the processing path is selected to ensure that the selection of the processing path is targeted and reasonable, and can better meet the processing requirements of different data.

[0043] S600: Construct a gradient collaborative training mechanism to optimize the complexity threshold by combining multiple processing paths for training when training the complexity threshold.

[0044] The specific steps for optimizing the complexity threshold using multiple processing paths combined with training are: S601. When training the complexity threshold, obtain the deviation parameter of each processing path through reinforcement learning, extract the temperature coefficient during the training process, and calculate the gating probability of each processing path in the routing gating function during training. The formula is: ; In the formula, p k represents the gating probability of the kth processing path, b k represents the deviation parameter of the kth processing path, and τ represents the temperature coefficient; S602: Utilize the probability of each processing path to combine all processing paths during training to process the natural language processing task during training, and build a gradient collaborative training mechanism. The formula is: ; In the formula, y represents the natural language processing result during training, f k (s) represents the natural language processing model of the k-th processing path; it is trained using the gradient collaborative training mechanism, and the complexity threshold is trained, optimized and updated.

[0045] Constructing a gradient collaborative training mechanism and utilizing multiple processing paths to combine training to optimize the complexity threshold can fully utilize the advantages of different processing paths and improve the efficiency and accuracy of threshold training.

[0046] By calculating the gating probability of each processing path and combining it with processing tasks, the training process becomes more scientific and reasonable, and can better adapt to data changes and processing requirements.

[0047] A data-driven natural language processing system includes a data acquisition module, a complex feature module, a complexity calculation module, a processing path classification module, a complexity threshold module, a routing gating judgment module, and a training and updating module; The data collection module is used to collect all original data, response time and data features during natural language processing in history; The complex feature module is used to calculate the correlation between response time and data features, and filter out complex features that affect the complexity of original data; The complexity calculation module is used to extract corresponding feature functions based on the screened complex features, and calculate the complexity of the real-time raw data using the feature functions of all complex features; The processing path grading module is used to analyze the model calculation amount to obtain division nodes, divide the total calculation amount of the natural language processing model, and construct a multi-level processing path; The complexity threshold module is used to calculate the mapping relationship between complexity and computational effort, set the complexity threshold, use reinforcement learning to train and optimize the complexity threshold through a validation set, and set a training target; The routing gating judgment module is used to construct a dynamic routing gating function, judge the complexity of real-time raw data, and select a real-time processing path; The training update module is used to construct a gradient collaborative training mechanism, and when training the complexity threshold, multiple processing paths are used to combine training to optimize the complexity threshold.

[0048] The routing gating judgment module includes a routing gating function unit and a judgment unit; The routing gating function unit is used to set the routing gating function using the calculation complexity threshold; The judgment unit is used to input the real-time original data into the routing threshold function, judge the complexity threshold interval where the complexity of the real-time original data lies, and select the natural language model of the corresponding processing path to perform real-time task processing.

[0049] The training update module includes a gated probability unit and a gradient collaborative training mechanism unit; The gating probability unit is used to calculate the gating probability of each processing path using the deviation parameter and the temperature coefficient; The gradient collaborative training mechanism unit is used to utilize the probability of each processing path to combine all processing paths during training to process the natural language processing task during training, and to construct a gradient collaborative training mechanism to update the complexity threshold.

[0050] Example 1: Taking the short text sentiment analysis task as an example, we implement sentiment polarity classification (positive / negative / neutral) of social media comments. The processed data includes historical user comments, response time, and model computational complexity. We extracted 100,000 historical comments from social media logs, removed duplicate text, and standardized them into a unified format. We extracted the start and end timestamps of each comment's processing. Data features included text length (X1), number of negative words (X2), sentiment word density (X3), and syntactic complexity (X4). The correlations between these features and response time were calculated to be 0.72, 0.58, 0.65, and 0.81, respectively. Sort the correlations into [0.58, 0.65, 0.72, 0.81] and calculate the adjacent differences (0.07, 0.07, 0.09). The maximum difference of 0.09 corresponds to a correlation threshold of 0.72. Select features with a correlation greater than 0.72: text length (X1) and syntactic complexity (X4), forming a complex feature set.

[0051] Example 2: Collect the FLOPs of the model in historical tasks, sort them, draw a curve and take the derivative, set the mean + standard deviation as the change threshold, determine three partition nodes, and build a three-level processing path (m=3): Level 1 path (lightweight): 2-layer Transformer, 1.2GFLOPs computation, for simple text.

[0052] Level 2 path (medium): 4-layer Transformer, 3.5GFLOPs of computation, for medium-complexity text.

[0053] Level 3 path (heavy): 6-layer Transformer, 7.8GFLOPs computation, for complex text.

[0054] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A data-driven natural language processing method, characterized by: The method comprises the following steps: S100: Collect all raw data from historical natural language processing, extract the response time and data features of each natural language processing in history, calculate the correlation between the response time and data features, and filter out complex features that affect the complexity of the raw data; S200, extracting corresponding feature functions based on the screened complex features, and calculating the complexity of the real-time raw data using the feature functions of all the complex features; S300: Collect model computational loads from all historical natural language processing tasks, analyze the model computational loads to obtain partition nodes, divide the total computational load of the natural language processing model, construct a multi-level processing path, and construct a natural language processing model for each of the multi-level processing paths; S400, calculating the mapping relationship between complexity and computational effort, setting a complexity threshold, using reinforcement learning to train and optimize the complexity threshold using a validation set, and setting a training target; S500: Build a dynamic routing gating function to judge the complexity of real-time raw data and select a real-time processing path; S600: Construct a gradient collaborative training mechanism to optimize the complexity threshold by combining multiple processing paths for training when training the complexity threshold.

2. The data-driven natural language processing method according to claim 1, characterized in that: The specific steps of screening out the complex features that affect the complexity of the original data in S100 are: S101. Extract all raw data from the natural language processing system log during historical natural language processing, remove duplicate data, and standardize the raw data. Extract the start and end timestamps of each natural language processing task, and calculate the response time of each natural language processing task. The formula is: ； In the formula, T i represents the response time of the i-th natural language processing in history, t end，i Indicates the end timestamp of the i-th natural language processing task in history, t start，i Indicates the start timestamp of the i-th natural language processing task in history; S102. Extract data features of the original data in each natural language processing task from the system log. The data features include but are not limited to data length and data semantics. Calculate the response time of each natural language processing task in history and the correlation between different data features. Standardize the response time and data features using the formula: ； In the formula, ρ(X j , T) represents the correlation between the jth data feature and the response time, X j，i represents the j-th data feature value in the i-th natural language processing task, Xp j represents the average value of the j-th data feature in n natural language processing tasks, T i represents the response time of the i-th natural language processing task, Tp represents the average response time of n natural language processing tasks, and n represents the total number of historical natural language processing tasks collected; S103. Sort the calculated correlations between all data features and response time from smallest to largest, calculate the differences between adjacent correlations, select the maximum difference as the boundary, set the maximum value of the two correlations corresponding to the boundary as the correlation threshold, use the correlation threshold to judge all correlations, and when the correlation is greater than the correlation threshold, treat the corresponding data feature as a complex feature to obtain a complex feature set.

3. The data-driven natural language processing method according to claim 2, characterized in that: The specific steps of calculating the complexity of the real-time raw data using the characteristic functions of all complex features in S200 are: S201. Extract corresponding feature functions based on the selected complex features, set fusion weights for each feature function, and use the fusion weights to fuse the feature functions to build a complexity calculation model. The model is: ； In the formula, Com represents the complexity of the original data, w h Represents the fusion weight of the h-th feature function, g(s) h represents the hth characteristic function, and H represents the total number of characteristic functions; S202. Utilize a deep learning algorithm to train, optimize, and update the fusion weights of each feature function in a natural language processing system.

4. The data-driven natural language processing method according to claim 1, characterized in that: The specific steps of constructing the multi-level processing path in S300 are: S301. Collect the natural language processing system computational loads from all historical natural language processing tasks, sort the natural language processing system computational loads from small to large, and draw a computational load curve. Derivative the computational load curve to obtain a change curve. Calculate the standard deviation and average value of the change curve. Add the average value and the standard deviation to set a change threshold. In the change curve, define the computational load curve points that are greater than the change threshold as partition nodes. Use the partition nodes to obtain an m-level processing path. S302. Set a natural language processing model for each level of processing path, and assume that the natural language processing model structure is {G: f(s) = Q layers of transformer, computational load}; where G represents the processing path number, transformer represents the natural language processing algorithm, and the value of Q is set by combining professional knowledge and computational load.

5. The data-driven natural language processing method according to claim 1, characterized in that: The specific steps of setting the training target in S400 are: S401. Collect the historical computational load and raw data complexity of the natural language processing system when processing tasks, generate a mapping function between computational load and complexity using a linear regression algorithm, input the computational load of the partitioning nodes of the m-level processing path into the mapping function, and calculate the complexity threshold corresponding to each level of the processing path; S402: Utilize reinforcement learning to train and optimize the complexity threshold using the validation set, setting the training objective to: ； In the formula, Total represents the total loss, Total task represents task loss, FLOPs represents system computational efficiency, and λ represents computational efficiency weight; During training, the result with the minimum total loss is selected.

6. The data-driven natural language processing method according to claim 5, characterized in that: The specific steps of selecting the real-time processing path in S500 are: S501. Set the routing gating function using the calculated complexity threshold. The formula is: ； In the formula, Gate(s) represents the routing gating function, f1(s), f2(s), ..., f m-1 (s) represents the natural language processing model of the 1st, 2nd, ..., mth level processing path, Com(s) represents the complexity of the real-time raw data, α1, α2, ..., α m-1 represents the 1st, 2nd, ..., m-1th complexity threshold; S502: Input the real-time original data into the routing threshold function, determine the complexity threshold interval where the complexity of the real-time original data lies, and select the natural language model corresponding to the processing path to perform real-time task processing.

7. The data-driven natural language processing method according to claim 6, characterized in that: The specific steps of optimizing the complexity threshold by using multiple processing paths in combination with training in S600 are: S601. When training the complexity threshold, obtain the deviation parameter of each processing path through reinforcement learning, extract the temperature coefficient during the training process, and calculate the gating probability of each processing path in the routing gating function during training. The formula is: ； In the formula, p k represents the gating probability of the kth processing path, b k represents the deviation parameter of the kth processing path, and τ represents the temperature coefficient; S602: Utilize the probability of each processing path to combine all processing paths during training to process the natural language processing task during training, and build a gradient collaborative training mechanism. The formula is: ； In the formula, y represents the natural language processing result during training, f k (s) represents the natural language processing model of the k-th processing path; it is trained using the gradient collaborative training mechanism, and the complexity threshold is trained, optimized and updated.

8. A data-driven natural language processing system, characterized by: The natural language processing system includes a data acquisition module, a complex feature module, a complexity calculation module, a processing path classification module, a complexity threshold module, a routing gating judgment module and a training update module; The data collection module is used to collect all original data, response time and data features during natural language processing in history; The complex feature module is used to calculate the correlation between response time and data features, and filter out complex features that affect the complexity of original data; The complexity calculation module is used to extract corresponding feature functions based on the screened complex features, and calculate the complexity of the real-time raw data using the feature functions of all complex features; The processing path grading module is used to analyze the model calculation amount to obtain division nodes, divide the total calculation amount of the natural language processing model, and construct a multi-level processing path; The complexity threshold module is used to calculate the mapping relationship between complexity and computational effort, set the complexity threshold, use reinforcement learning to train and optimize the complexity threshold through a validation set, and set a training target; The routing gating judgment module is used to construct a dynamic routing gating function, judge the complexity of real-time raw data, and select a real-time processing path; The training update module is used to construct a gradient collaborative training mechanism, and when training the complexity threshold, multiple processing paths are used to combine training to optimize the complexity threshold.

9. The data-driven natural language processing system according to claim 8, characterized in that: The routing gating judgment module includes a routing gating function unit and a judgment unit; The routing gating function unit is used to set the routing gating function using the calculation complexity threshold; The judgment unit is used to input the real-time original data into the routing threshold function, judge the complexity threshold interval where the complexity of the real-time original data lies, and select the natural language model of the corresponding processing path to perform real-time task processing.

10. The data-driven natural language processing system according to claim 8, characterized in that: The training update module includes a gated probability unit and a gradient collaborative training mechanism unit; The gating probability unit is used to calculate the gating probability of each processing path using the deviation parameter and the temperature coefficient; The gradient collaborative training mechanism unit is used to utilize the probability of each processing path to combine all processing paths during training to process the natural language processing task during training, and to construct a gradient collaborative training mechanism to update the complexity threshold.

Images