A high-quality corpus data screening method, platform and system based on big data

Abstract

The application provides a high-quality corpus screening method, platform and system based on big data, relating to the technical fields of big data processing and natural language processing, which comprises the following steps: constructing a four-dimensional weight matrix and building a global and node double-layer quality distribution, generating a hierarchical screening weight through two-by-two block fusion of dimensions, and completing initial boundary generation combined with training requirements; obtaining a feedback correction signal by matching experience rules, generating a trend compensation signal with sensitivity correction relying on local distribution prediction quality wave, and completing two-stage steady-state verification by independent weighting iteration of dimensions to obtain a hierarchical screening decision vector; and scheduling the end based on node deduplication capability and real-time load to dynamically adjust the collection frequency, and outputting qualified corpora through hierarchical sorting. The application solves the problem of single weight one-size-fits-all and threshold adjustment lag distortion, takes into account corpus hierarchical adaptation and distributed scheduling stability, and is suitable for large model multi-scene corpus collection and screening scenarios.

Description

Technical Field

[0001] This invention relates to the fields of big data processing and natural language processing, and in particular to a method, platform and system for screening high-quality corpus data based on big data. Background Technology

[0002] The training performance of large language models is highly sensitive to the quality of the corpus. Selecting a high-quality subset of corpus from massive amounts of multi-source raw data is a crucial preliminary step in the model training process that determines the final performance. In distributed multi-node acquisition scenarios, the complexity of corpus selection increases significantly because the quality of corpus from different data sources fluctuates dynamically, and downstream training tasks have significant differences in their preferences for corpus type and quality requirements.

[0003] The mainstream technical solutions in the industry can be divided into two categories. The first category is the fixed threshold screening scheme, which pre-sets a uniform set of quality thresholds and dimensional weights to perform standardized filtering on the corpus of all collection nodes. This type of scheme is simple to implement and has low computational overhead, but it cannot adapt to the dynamic changes in data source quality. In addition, the use of globally uniform dimensional weights cannot adapt to the differences in dimensional importance of data sources of different quality levels, which can easily lead to an imbalance problem of over-screening of high-quality data sources and under-screening of low-quality data sources.

[0004] The second type is the feedback-based dynamic adjustment scheme, which involves fine-tuning the screening threshold based on the actual output corpus quality after filtering. This type of scheme partially solves the adaptability problem of fixed thresholds, but it is essentially a post-hoc remedial adjustment with significant lag. Furthermore, the threshold adjustment uses a fixed proportional coefficient, without considering the local characteristics of the quality distribution, making the correction accuracy uncontrollable. At the same time, the granularity of adaptation to downstream training needs is relatively coarse, usually only adjusting the overall screening strictness by switching gears, and failing to achieve fine-grained weight adjustment between dimensions and quality levels.

[0005] At the distributed data acquisition and scheduling level, existing solutions mostly adopt simple polling or random allocation strategies, which do not fully consider the differences in capabilities of different nodes, overlapping acquisition ranges, and real-time load constraints. This can easily lead to problems such as data source resource contention, mismatch between node capabilities and task types, and insufficient system stability under high load, resulting in low overall data acquisition efficiency.

[0006] In summary, the existing technology has three core limitations: First, the dynamic adaptation capability of the screening boundary is insufficient, the post-adjustment is lagging, and the correction accuracy is uncontrollable due to the influence of the distribution location; second, the demand adaptation granularity is coarse, and the global single weight cannot take into account the dimensional interaction effect and the quality stratification difference; third, the distributed scheduling efficiency is low, and resource conflicts, capacity mismatch and lack of load constraints coexist.

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a method, platform, and system for filtering high-quality corpus data based on big data. This solution addresses the problems of lagging dynamic adaptation, coarse granularity of demand adaptation, and low efficiency of distributed scheduling, achieving a comprehensive technical effect of stable corpus quality, accurate hierarchical adaptation, and improved collection efficiency. Summary of the Invention

[0008] This invention provides a method, platform, and system for filtering high-quality corpus data based on big data, in order to solve the aforementioned technical problems.

[0009] This invention provides a method for filtering high-quality corpus data based on big data, including: S1: Perform statistical calculations on the feature distribution of the full corpus data sources of all distributed corpus collection nodes to generate a global quality distribution. At the same time, generate a local quality distribution based on the subset of data sources bound to each collection node. Construct a global unified weight matrix covering four preset dimensions, wherein the preset dimensions include field completeness dimension, semantic self-consistency dimension, source credibility dimension, and redundancy noise control dimension. S2: Traverse all preset corpus quality screening thresholds. Under each corpus quality screening threshold, perform a dual fusion of linear interaction and non-linear correlation between the downstream large language model training requirements and the weight matrix to generate the dimensional weight coefficients of the quality level. Combine the threshold vector of the level to generate the initial corpus screening judgment boundary of the corresponding level. S3: Analyze each initial corpus in the initial corpus set of each distributed corpus collection node according to the four preset dimensions to obtain the actual quality profile of the corresponding corpus, and compare it with the expected quality profile to identify the deviation dimension, deviation direction, deviation degree and deviation change trend, and generate the corpus quality deviation pattern of the corresponding node. The expected quality profile includes the quality target level and allowable quality fluctuation range of each of the four preset dimensions. S4: Match the corpus quality deviation pattern with the preset corpus screening boundary control experience rules, and after correction by the distribution sensitivity coefficient, output the independent screening boundary feedback correction signal for each dimension of the corresponding node. At the same time, based on the mapping relationship between the historical change pattern of the local quality distribution of the node and the boundary prediction compensation amount, perform trend analysis on the local quality distribution of each node by combining the distribution sensitivity coefficient, and output the independent screening boundary trend prediction compensation signal for each dimension of each node. S5: Independently merge the screening boundary feedback correction signal and the screening boundary trend prediction compensation signal for each dimension, and perform adaptive processing on the initial corpus screening decision boundary for each level to obtain the optimal corpus screening decision vector for the corresponding node in each level. S6: Based on the optimal corpus selection decision vector for each level of the corresponding node, generate a multi-dimensional corpus collection control instruction set and send it to the corresponding node. Under the node load constraint, dynamically adjust the corpus collection frequency, trigger heterogeneous corpus cleaning operation and block low-quality data sources. Combine the dimension weight coefficients of the quality level to perform hierarchical selection and output the selected high-quality target corpus subset.

[0010] Preferably, a weight matrix covering four preset dimensions is constructed, including: Based on the difference between the field missing rate of the corpus data source and the preset field completeness benchmark threshold, the preset completeness benchmark weight value is scaled inversely, and the field completeness dimension weight value is obtained after value range truncation constraint. Extract the semantic contradiction degree and logical discontinuity ratio parameters from the corpus data source, input the preset semantic quality segmentation decay function to obtain the first initial weight value of the semantic self-consistency dimension, and modify the first initial weight value in combination with the domain matching degree of the corpus data source to obtain the weight value of the semantic self-consistency dimension. The credibility index is obtained by weighting the main body qualification level, historical compliance record and information traceability level of the corpus data source. The second initial weight value is obtained by range normalization mapping. The second weighting correction is carried out by combining the information release link length of the corpus data source to obtain the source credibility dimension weight value. The weighted sum of the proportion of duplicate data and the proportion of invalid noise in the statistical corpus data source is used to calculate the comprehensive redundancy. After normalizing the comprehensive redundancy, the third initial weight value is obtained by subtracting the normalization result from 1. The weight value of the redundancy noise control dimension is then obtained. The weight values ​​for field completeness, semantic consistency, source credibility, and redundancy noise control are mapped to the corresponding dimension channels of the weight matrix to construct a globally unified weight matrix.

[0011] Preferably, adaptive processing is performed on the initial corpus selection decision boundaries for each level to obtain the optimal corpus selection decision vector for the corresponding node at each level, including: S51: Configure a first-level allowable amplitude coefficient and a second-level allowable amplitude coefficient for each preset dimension. Multiply the boundary adjustment benchmark value of each dimension with the corresponding first-level allowable amplitude coefficient to generate a first-level boundary migration allowable interval. Multiply the preset dimension quality deviation benchmark value with the corresponding second-level allowable amplitude coefficient to generate a second-level boundary migration allowable interval. The boundary adjustment benchmark value of each dimension is the boundary value of the corresponding dimension in the initial corpus screening and judgment boundary of the current level. The dimension quality deviation benchmark value is the average absolute deviation value of the actual quality profile and the expected quality profile of the corresponding node in each dimension within the most recent K evaluation periods. S52: Configure the feedback signal weight and prediction signal weight independently for each dimension, perform weighted fusion of the screening boundary feedback correction signal after the distribution sensitivity coefficient correction and the screening boundary trend prediction compensation signal for the corresponding dimension to generate the boundary migration correction amount for that dimension, and concatenate all dimensions to form a first-level boundary migration correction vector. The first-level boundary migration correction vector is superimposed onto the initial corpus screening and judgment boundary of the current level using a relative percentage multiplication method to obtain the first-level migration boundary parameter set. Each dimension parameter in the first-level migration boundary parameter set is compared with the first-level boundary migration allowable interval of the corresponding dimension. If all dimension parameters fall within the corresponding first-level boundary migration allowable interval, the first-level migration boundary parameter set is marked as a candidate migration boundary set and retained. Otherwise, the weight ratio of the two signals of the out-of-bounds dimension is adjusted independently. If it is still impossible to make all dimension parameters fall within the allowable interval after the adjustment of the preset maximum number of iterations, the first-level migration boundary parameter set corresponding to the weight ratio that minimizes the total deviation of the out-of-bounds dimension parameters is taken as the candidate migration boundary set. S53: Based on the empirical distribution of node local quality distribution, Monte Carlo sampling is performed to generate simulated corpus samples. A preset corpus quality fluctuation perturbation factor is introduced into the four-dimensional quality score of the simulated corpus to simulate the random fluctuation of corpus quality in real scenarios. Then, the candidate migration boundary set is used as the screening threshold to perform simulated screening, simulate the full corpus screening process, and output the simulated value of corpus quality deviation and the simulated value of screening pass rate corresponding to the secondary verification boundary set to obtain the steady-state effective boundary set. S54: Map each dimension parameter of the steady-state effective boundary set into an executable corpus screening judgment threshold, preset dimension weight coefficient and cleaning trigger condition parameter, and concatenate them to generate the optimal corpus screening decision vector for the current level of the corresponding node.

[0012] Preferably, the initial corpus selection and judgment boundaries for the corresponding grade include: S21: Extract the training demand intensity features and training demand accuracy features from the training requirements of downstream large language models; Based on the first preset mapping rule, under the current corpus quality screening threshold, calculate the quota of nonlinear correlation fusion component and the quota of linear interaction fusion component; S22: For the four preset dimensions covered by the nonlinear correlation fusion component, calculate the preset dimension activation coefficient of each dimension according to the second preset mapping rule and the dimension importance level. S23: Decompose the weight matrix into 6 groups of two-dimensional mass weight sub-planes by combining them pairwise according to the dimensions; For each two-dimensional quality weight subplane, all data sources are first sorted in ascending order by weighted comprehensive score according to the preset dimension weight coefficients of the two dimensions corresponding to the subplane. Then, equidistant gridding is performed. The grid block size is dynamically determined based on the row and column dimensions of the subplane. The greatest common divisor or the closest integer divisor of the row and column dimensions is taken as the block scale parameter to form several quality weight block units with consistent size. For each block unit, calculate the mean weight of each of the two dimensions within the unit to obtain the block dimension weight vector; The downstream large language model training requirement feature vector is split according to the dimension level. Based on the dimension combination mapped by each quality weight block unit, the feature values ​​of the corresponding dimension are extracted from the requirement feature vector, normalized and quantized to form requirement feature sub-blocks, so that each group of requirement feature sub-blocks establishes a spatial mapping relationship with all quality weight block units in the corresponding two-dimensional quality weight sub-plane. S24: Based on the spatial mapping relationship, using the allocated linear interaction fusion component quota and nonlinear correlation fusion component quota as block fusion coefficients, calculate the fusion result of the corresponding quality weight block unit and the demand feature sub-block block block by block to generate a block-level two-dimensional importance coefficient vector; aggregate all block-level two-dimensional importance coefficient vectors by dimension to obtain the dimension weight coefficients of the quality stratification level; multiply the dimension weight coefficients of the quality stratification level with the tier benchmark threshold vector by dimension to generate the initial screening threshold corresponding to each quality stratification level, and concatenate them to obtain the initial corpus screening judgment boundary corresponding to the current corpus quality screening threshold tier.

[0013] The preferred subset of high-quality target corpus after output filtering includes: S61: Sort all idle distributed corpus collection nodes according to the preset node polling scheduling sequence, and take the idle corpus collection node at the top of the sequence as the current task execution node. Based on the corpus quality screening threshold required by the corpus collection task to be performed, the matching threshold decision vector is retrieved from the optimal corpus screening decision vector of each threshold of the corresponding node, and the redundancy noise control dimension threshold is corrected by combining the deduplication processing capability coefficient of the current node to obtain the actual redundancy threshold. Corpus collection tasks whose corpus collection coverage and corpus data type attributes both fall within the job adaptation domain of the current task execution node are included in the candidate collection task subset corresponding to the current node. S62: Verify one by one whether the data source range of each data collection task to be executed in the candidate data collection task subset overlaps with the current data source range of the other data collection nodes that are in the data collection work state. If the proportion of data sources in the overlapping areas of the collection range exceeds the preset collection range conflict threshold, the corresponding corpus collection task to be executed will be removed from the candidate collection task subset. S63: If the subset of candidate collection tasks after secondary filtering is not empty, first perform a corpus task type matching judgment on each candidate corpus collection task and divide them into two categories: standard structured corpus collection tasks and unstructured generalized corpus collection tasks. Retrieve the historical collection context data of the current task execution node, and calculate the corpus collection execution efficiency score for the two types of corpus collection tasks respectively; The candidate corpus collection task with the highest performance score is selected as the optimal collection task and assigned to the current task execution node. If the candidate collection task subset is empty, the current task execution node is moved to the end of the idle node polling sequence, and the number of idle rounds of the node is accumulated. If the number of consecutive idle rounds of the node exceeds the preset limit, the node is temporarily suspended to wait for new tasks to be added to the database, and the next idle corpus collection node is selected in order to repeat S61. S64: Remove the assigned optimal collection task from the global set of pending corpus collection tasks, change the working status of the current task execution node from idle to collecting data, and remove it from the idle node polling sequence; Take the next idle corpus collection node in the idle node polling sequence as the new current task execution node, and repeat S61 to S63 until the global set of pending corpus collection tasks is empty or all nodes have been traversed and there are no tasks to be assigned. S65: For the corpus collection tasks assigned to each corpus collection node, match the corresponding corpus collection control parameter template according to the task type, and calculate the optimal collection frequency in combination with the real-time load constraints of the node with the goal of maximizing the output efficiency of qualified corpus per unit time. By combining the cleaning trigger condition parameters and low-quality blocking rules in the optimal corpus selection decision vector, a multi-dimensional corpus acquisition control command is generated and sent to the corresponding nodes. Each node performs corpus collection, heterogeneous corpus cleaning, and low-quality data source blocking operations according to instructions. It also performs hierarchical filtering and comprehensive sorting based on the dimensional weight coefficients of the quality stratification level, and outputs a unified subset of high-quality target corpus.

[0014] Preferably, the performance score of the corpus collection is calculated using a normalized weighted method.

[0015] Preferably, generating a block-level two-dimensional importance coefficient vector includes: S241: Based on the corpus preset dimension attribute level mapped by each quality weight block unit, all quality weight block units are divided into two categories: core preset dimension blocks and auxiliary preset dimension blocks. Among them, quality weight block units containing semantic self-consistency dimension or source credibility dimension are marked as core preset dimension blocks, and quality weight block units containing only field completeness dimension and redundant noise control dimension are marked as auxiliary preset dimension blocks. S242: Divide the core preset dimensions into blocks, and use the linear interaction fusion component quota as the coefficient to perform linear weighting on the mean weight of the corresponding dimension in the block dimension weight vector to obtain the linear components of each dimension. Using the product of the nonlinear correlation fusion component quota and the corresponding dimension preset dimension activation coefficient as the coefficient, nonlinear weighting is applied to the feature quantization values ​​of each dimension in the demand feature sub-block to obtain the nonlinear components of each dimension. The linear and nonlinear components are summed according to their dimensions and then superimposed with a preset dimension baseline offset correction amount to obtain the initial two-dimensional importance coefficient vector of the block. The feature quantization value is the standardized value of each dimension feature value in the required feature sub-block after being normalized to the [0,1] interval by min-max. The preset dimension baseline offset correction amount is a constant correction term that has been normalized and scaled to the same dimension as the weighted summation result. For the auxiliary preset dimension blocks, the linear interactive fusion component quota is used as the dominant linear mapping coefficient. The average weight of each dimension in the block dimension weight vector is multiplied element by element with the feature quantization value of the corresponding dimension in the demand feature sub-block to obtain a linear mapping two-dimensional vector. The product of the nonlinear correlation fusion component quota and the corresponding dimension's preset dimension activation coefficient is used as the amplitude fine-tuning coefficient. Multiplication amplitude correction is performed on the linear mapping two-dimensional vector according to the dimension to obtain the block's initial two-dimensional importance coefficient vector. S243: For each block, the initial two-dimensional importance coefficient vector is selected, and its relative deviation from the global average importance coefficient in the corresponding two-dimensional quality weight subplane is calculated. If the relative deviation exceeds the reasonableness threshold, it is marked as an abnormal block and interpolation correction is performed using the subplane global mean combined with the block attribute weights; If the relative deviation does not exceed the threshold, the initial two-dimensional importance coefficient vector of the block selection is directly retained; S244: After processing the value range truncation constraint, the block-level two-dimensional importance coefficient vector corresponding to each block is obtained by taking the block-corrected importance coefficient vector corresponding to all quality weight block units.

[0016] Preferably, a high-quality corpus data screening platform based on big data is deployed and executed using any of the aforementioned high-quality corpus data screening methods based on big data, to achieve distributed node dynamic collection and hierarchical high-quality corpus output.

[0017] Preferably, a high-quality corpus data filtering system based on big data includes a scheduling service, a cluster of data collection nodes, and a storage module. The scheduling service runs and executes any of the high-quality corpus data filtering methods based on big data described above.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. Dual-loop control combined with distributed sensitivity correction significantly improves dynamic adaptation accuracy. A dual-loop control mechanism of feedback correction and trend prediction compensation is employed. The feedback loop corrects existing deviations, while the prediction loop anticipates fluctuation trends and compensates in advance. Simultaneously, a distributed sensitivity coefficient is introduced to correct the two types of correction signals, matching the distribution characteristics at different threshold positions. This solves the problem of uncontrollable accuracy in fixed-ratio correction and effectively reduces the fluctuation range of output quality.

[0019] 2. Two-dimensional segmentation and fusion achieves refined adaptation of both quality stratification and dimensional interaction. By constructing two-dimensional sub-planes through pairwise combinations of dimensions, the interaction effects under different dimensional combinations are accurately captured. Data sources are sorted and segmented according to quality level, and hierarchical dimensional weight coefficients are output. This allows data sources of different quality levels to correspond to differentiated weight strategies, avoiding the one-size-fits-all problem of a single global weight, and effectively improving corpus utilization under the same quality level.

[0020] 3. Load-aware multi-level scheduling optimizes distributed data acquisition performance. A three-level scheduling strategy is adopted: round-robin for fairness, conflict prevention to avoid contention, and efficiency enhancement. Combined with a load adjustment coefficient, the data acquisition frequency is dynamically optimized, balancing output efficiency and system stability. Simultaneously, precise matching of task types and node capabilities is achieved, significantly improving overall data acquisition performance.

[0021] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a high-quality corpus data screening method based on big data in an embodiment of the present invention. Detailed Implementation

[0024] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0025] Example 1: This invention provides a method for filtering high-quality corpus data based on big data, such as... Figure 1 As shown, it includes: S1: Perform statistical calculations on the feature distribution of the full corpus data sources of all distributed corpus collection nodes to generate a global quality distribution. At the same time, generate a local quality distribution based on the subset of data sources bound to each collection node. Construct a global unified weight matrix covering four preset dimensions, wherein the preset dimensions include field completeness dimension, semantic self-consistency dimension, source credibility dimension, and redundancy noise control dimension. S2: Traverse all preset corpus quality screening thresholds. Under each corpus quality screening threshold, perform a dual fusion of linear interaction and non-linear correlation between the downstream large language model training requirements and the weight matrix to generate the dimensional weight coefficients of the quality level. Combine the threshold vector of the level to generate the initial corpus screening judgment boundary of the corresponding level. S3: Analyze each initial corpus in the initial corpus set of each distributed corpus collection node according to the four preset dimensions to obtain the actual quality profile of the corresponding corpus, and compare it with the expected quality profile to identify the deviation dimension, deviation direction, deviation degree and deviation change trend, and generate the corpus quality deviation pattern of the corresponding node. The expected quality profile includes the quality target level and allowable quality fluctuation range of each of the four preset dimensions. S4: Match the corpus quality deviation pattern with the preset corpus screening boundary control experience rules, and after correction by the distribution sensitivity coefficient, output the independent screening boundary feedback correction signal for each dimension of the corresponding node. At the same time, based on the mapping relationship between the historical change pattern of the local quality distribution of the node and the boundary prediction compensation amount, perform trend analysis on the local quality distribution of each node by combining the distribution sensitivity coefficient, and output the independent screening boundary trend prediction compensation signal for each dimension of each node. S5: Independently merge the screening boundary feedback correction signal and the screening boundary trend prediction compensation signal for each dimension, and perform adaptive processing on the initial corpus screening decision boundary for each level to obtain the optimal corpus screening decision vector for the corresponding node in each level. S6: Based on the optimal corpus selection decision vector for each level of the corresponding node, generate a multi-dimensional corpus collection control instruction set and send it to the corresponding node. Under the node load constraint, dynamically adjust the corpus collection frequency, trigger heterogeneous corpus cleaning operation and block low-quality data sources. Combine the dimension weight coefficients of the quality level to perform hierarchical selection and output the selected high-quality target corpus subset.

[0026] In this embodiment, the full corpus data source is the sum of all corpus data sources that have been registered and accessed the collection system. It is managed uniformly through the data source registration list, which records the address, content type, authorization information, access protocol and last active time of each data source. This system covers six types of data sources: news, academic, social, industry forums, technical blogs and government information disclosure.

[0027] Distributed corpus collection nodes are cloud containers, edge proxies, or standardized collection service units with independent capture and preprocessing capabilities. Each node is bound to a dedicated set of data sources and collection permissions, and independently completes the corpus collection, initial screening, and cleaning process.

[0028] The four preset dimensions are the four evaluation dimensions of the corpus quality, which are divided into core dimensions (semantic consistency and source credibility) and auxiliary dimensions (field completeness and redundancy noise control). All of them adopt a positive percentage score of 0-100, and the higher the score, the better the corpus quality.

[0029] The corpus qualification determination is a four-dimensional independent threshold concatenation determination mechanism. A single corpus is considered to be basically qualified only if the scores of all four dimensions reach the corresponding lower bound threshold. After the basic qualification, the comprehensive score is calculated by combining the weights of the hierarchical dimensions to complete the corpus priority ranking.

[0030] The global quality distribution is a two-layer empirical distribution constructed based on sampling from all data sources. The first layer is the statistical distribution of the data source dimension, and the second layer is the empirical quantile of the corpus quality within a single data source. It does not enforce the assumption of normal distribution, supports incremental updates, and uses historical statistical results when no new data source is added.

[0031] The local quality distribution is a dedicated quality distribution for a single data acquisition node bound to a data source; when the number of data sources bound to a node is ≥30, an empirical distribution is constructed independently; when there are fewer than 30 data sources, a hybrid correction mode of global distribution plus node mean deviation is adopted.

[0032] The weight matrix is ​​an N-row, 4-column two-dimensional matrix, where N is the total number of data sources. Each element of the matrix takes a value from 0 to 1, representing the inherent basic quality of the corresponding data source in a single dimension. The matrix is ​​globally unique and only represents the attributes of the data source, and is not affected by the deduplication performance of nodes.

[0033] The corpus quality screening threshold has five levels of strictness, with each level tightening progressively. Each level is configured with a dedicated baseline threshold vector. The nominal target pass rate for each level is only used as a design reference, and the system operation prioritizes quality deviation constraints as the control condition.

[0034] The gear benchmark threshold vector is a four-dimensional standard lower bound threshold calibrated offline based on a general corpus. It is the basis for calculating the initial boundary of each gear and has a value range of 0-100 points.

[0035] The dimensional weight coefficients for quality tiers are a set of exclusive four-dimensional weights after dividing the data source into tiers based on its quality level. The weight values ​​range from 0.8 to 1.2, with different dimensional importance configured for different quality tiers to achieve fine-grained adaptation of the tiers.

[0036] The initial corpus screening and judgment boundary is a set of hierarchical four-dimensional screening thresholds, which is obtained by multiplying the base threshold of each level and the weight of each hierarchical dimension, including hierarchical differentiated screening standards and global basic screening standards.

[0037] The actual quality profile is a vector of the arithmetic mean of the four-dimensional scores of all data collected in a single node in a single session. It is used to compare the deviation with the preset expected standard and takes a value of 0-100.

[0038] The expected quality profile is a four-dimensional quality target value preset for downstream training tasks. Each dimension is associated with a fixed allowable fluctuation range. The higher the dimension has an impact on model accuracy, the smaller the allowable fluctuation range.

[0039] The corpus quality deviation pattern is a structured deviation record, which includes four types of information: the dimension corresponding to the deviation, the positive or negative direction, the deviation value, and the fluctuation trend. The direction is -1, which is below the standard, and +1, which is above the standard. The trend is +1, which indicates that the deviation is expanding, -1, which indicates that the deviation is converging, and 0, which is stable and unchanged.

[0040] Node load is a weighted comprehensive indicator of CPU, memory, and task queue length, with a value ranging from 0 to 1; the load threshold is 0.7. If the threshold is exceeded, the sampling frequency will be forcibly reduced to avoid system overload.

[0041] In this embodiment, a total of 12 priority-ranking empirical rules for corpus screening boundary control are used. The original threshold adjustment ratio is output based on the corpus deviation pattern. The original adjustment magnitude is calculated independently for each dimension of deviation and then weighted and summarized. This is the core basis for generating the feedback correction signal. These 12 rules are arranged from highest to lowest priority and are executed one by one during matching. The output of the high-priority rule serves as the calculation benchmark for the low-priority rule. When there are simultaneous deviations in multiple dimensions, each dimension is matched independently, and the final adjustment amount is obtained by combining the distribution sensitivity coefficient, as shown in Table 1. Table 1. Empirical Rules for Corpus Screening Boundary Control

[0042] In this embodiment, a large language model fine-tuning task is used as the downstream scenario, with a training requirement intensity s=0.8 and a training requirement accuracy a=0.9. Selection is performed using level 3 (standard level, with a target pass rate of approximately 50%). The system deployment environment consists of: one central scheduling server, eight distributed data collection nodes, 120 registered data sources, and five preset selection levels. The specific process is as follows: S1: Perform statistical calculations on the feature distribution of the full corpus data sources of all distributed corpus collection nodes to generate a global quality distribution. At the same time, generate a local quality distribution based on the subset of data sources bound to each collection node. Construct a global unified weight matrix covering four preset dimensions, wherein the preset dimensions include field completeness dimension, semantic self-consistency dimension, source credibility dimension, and redundancy noise control dimension. In this embodiment, the central scheduling server collects the raw quality metrics from all 120 data sources in the previous evaluation period and constructs a global quality distribution: Data source level distribution: Statistical analysis of frequency distribution and empirical quantiles for each dimension, extracting 16 statistical features including mean, standard deviation, skewness, and kurtosis; Corpus-level distribution: Each data source retains five empirical quantiles (10th, 25th, 50th, 75th, and 90th percentiles) of the four-dimensional quality of its internal corpus for subsequent sampling.

[0043] Simultaneously, a local quality distribution is generated for each node: when a node is bound to ≥30 data sources, an empirical distribution is constructed independently; when there are fewer than 30 data sources, a hybrid mode is used: local distribution = global distribution + node bias correction amount, where the correction amount is the difference between the mean of the node's local samples and the global distribution multiplied by a correction coefficient of 0.5, and the correction coefficient gradually increases as the sample size accumulates. In this embodiment, node N7 is bound to 18 data sources, and a hybrid mode is used to generate the local distribution.

[0044] Subsequently, following the procedure in Example 2, a global weight matrix W of 120×4 was constructed.

[0045] S2: Traverse all preset corpus quality screening thresholds. Under each corpus quality screening threshold, perform a dual fusion of linear interaction and non-linear correlation between the downstream large language model training requirements and the weight matrix to generate the dimensional weight coefficients of the quality level. Combine the threshold vector of the level to generate the initial corpus screening judgment boundary of the corresponding level. Taking gear 3 as an example, the fusion component quota is first calculated based on the training requirement characteristics: nonlinear fusion quota. =0.7, linear fusion quota =0.3. Subsequently, a block-level weighted fusion method (see Example 4) is used to fuse the demand features and weight matrix block by block in six groups of two-dimensional sub-planes to obtain the dimensional weight coefficients of the quality tier level. These coefficients are then multiplied by the benchmark threshold vector for tier 3 to obtain the initial screening boundary for tier 3. The global benchmark threshold is... Different quality strata correspond to threshold fluctuations within ±5%.

[0046] S3: Analyze each initial corpus in the initial corpus set of each distributed corpus collection node according to the four preset dimensions to obtain the actual quality profile of the corresponding corpus, and compare it with the expected quality profile to identify the deviation dimension, deviation direction, deviation degree and deviation change trend, and generate the corpus quality deviation pattern of the corresponding node. The expected quality profile includes the quality target level and allowable quality fluctuation range of each of the four preset dimensions. Taking node N7 as an example, the node quality deviation pattern recognition is bound to 18 news and forum data sources in a certain region. The previous round of data collection yielded approximately 50,000 original corpora. After calculating the four-dimensional quality score for each corpus and taking the average, the actual quality profile is as follows: field completeness 85 points, semantic consistency 78 points, source credibility 72 points, and redundancy noise control 78 points.

[0047] The expected quality profile for this medical fine-tuning task is as follows: field completeness 90 points (±5), semantic consistency 85 points (±5), source credibility 80 points (±5), and redundancy and noise control 85 points (±5). The deviations for each dimension were compared: Field completeness: Deviation 5 points, negative, within the acceptable range; Semantic consistency: Deviation 7 points, negative, exceeding the allowable range; Source credibility: Deviation of 8 points, negative, exceeding the allowable range; Redundancy noise control: Deviation of 7 points, negative (score is lower than the target, i.e., redundancy is higher than the target), exceeding the allowable range.

[0048] To determine the trend, retrieve the deviation records from the most recent three rounds: take the deviation values ​​d(t-2), d(t-1), and d(t) of the same dimension for the most recent three periods, and calculate the first difference. ; like Marked as an expanding trend (+1); like , marked as a converging trend (-1); the rest are stable trends (0).

[0049] Credibility deviation sequence [3,5,8]: =2, =3, 3>1.2×2=2.4, marked as an expanding trend (+1); Semantic deviation sequence [2,4,7]: Same as above, marked as expanding trend (+1); Redundant noise bias sequence [2,4,7]: Same as above, marked as increasing trend (+1); Field completeness deviation sequence [3,4,5]: =1, =1, 0.8 < 1 < 1.2, marked as a stable trend (0).

[0050] The corpus quality deviation patterns of the final output node N7 are: credibility dimension [3,-1,8,+1], semantic dimension [2,-1,7,+1], redundancy dimension [4,-1,7,+1], and completeness dimension [1,-1,5,0].

[0051] S4: Match the corpus quality deviation pattern with the preset corpus screening boundary control experience rules, and after correction by the distribution sensitivity coefficient, output the independent screening boundary feedback correction signal for each dimension of the corresponding node. At the same time, based on the mapping relationship between the historical change pattern of the local quality distribution of the node and the boundary prediction compensation amount, perform trend analysis on the local quality distribution of each node by combining the distribution sensitivity coefficient, and output the independent screening boundary trend prediction compensation signal for each dimension of each node. The final calculation formula for the feedback correction signal is as follows: ,in, Let be the distribution sensitivity coefficient of the j-th dimension.

[0052] In this embodiment, the sensitivity coefficients of each dimension are close to 1.0, and the final feedback correction signal... .

[0053] Trend prediction compensation signal generation: Extract the 8-dimensional core statistical features (4-dimensional mean + 4-dimensional standard deviation) of the local quality distribution of node N7 and the change features of the last two rounds, totaling 24-dimensional input features. Input these features into a pre-trained random forest regression model, and output the original prediction offset of the quality scores of each dimension in the next round. Combine the distribution sensitivity coefficient to calculate the final compensation signal: Finally, the trend prediction compensation signal is obtained. The model predicts that credibility and redundant noise quality may continue to decline in the next round, while semantic quality will slightly improve. Positive compensation (raising the threshold to tighten in advance) and negative compensation (lowering the threshold to avoid over-screening) are given to the corresponding dimensions respectively.

[0054] S5: Independently merge the screening boundary feedback correction signal and the screening boundary trend prediction compensation signal for each dimension, and perform adaptive processing on the initial corpus screening decision boundary for each level to obtain the optimal corpus screening decision vector for the corresponding node in each level. After two levels of verification (see Example 3 for details), the optimal corpus screening decision vector for node N7 in level 3 is finally obtained: global threshold [81,77,73,76], three groups of hierarchical weight coefficients, the cleaning trigger condition is that the redundant noise score <76 points triggers deep cleaning, and the blocking rule is that data sources with a field completeness score <60 points for three consecutive periods are marked as low quality and blocked.

[0055] S6: Based on the optimal corpus selection decision vector for each level of the corresponding node, generate a multi-dimensional corpus collection control instruction set and send it to the corresponding node. Under the node load constraint, dynamically adjust the corpus collection frequency, trigger heterogeneous corpus cleaning operation and block low-quality data sources. Combine the dimension weight coefficients of the quality level to perform hierarchical selection and output the selected high-quality target corpus subset.

[0056] If the decision vector is distributed to node N7, and the node deduplication capability coefficient k=1.0 (standard configuration) is used, the redundancy threshold remains unchanged at 76 points. Data acquisition frequency calculation: The baseline frequency for structured tasks is 1.0 times / hour. Based on Monte Carlo simulation, the current boundary pass rate is estimated to be approximately 48%. The current node load is 0.55 (below the threshold of 0.7), and the load adjustment coefficient is taken as 1.0. Therefore, the actual frequency = 1.0 × (50% / 48%) × 1.0 ≈ 1.04 times / hour, which is basically the same as the baseline.

[0057] Three C-level data sources with consecutive missing values ​​exceeding 40% were identified, marked as low-quality, and their access was suspended. After basic qualification was determined based on four-dimensional thresholds, a comprehensive quality score for each corpus was calculated by combining the weights of the hierarchical dimensions. The high-quality corpus with higher priority was output according to the scores, and finally, about 22,000 corpora were included in the high-quality target corpus subset, which was then aggregated and stored in the database.

[0058] In this embodiment, the feedback correction signal addresses the problem of correcting existing deviations, while the trend prediction compensation signal, combined with the distribution sensitivity, addresses the problem of accurately hedging future fluctuations. The combination of the two forms a closed-loop + feedforward dual control mechanism, which results in lower quality fluctuation amplitude than the traditional single feedback scheme. The hierarchical weights of the two-dimensional block output achieve quality differentiation adaptation and improve the efficiency of corpus utilization.

[0059] Example 2: This invention provides a method for filtering high-quality corpus data based on big data, constructing a weight matrix covering four preset dimensions, including: Based on the difference between the field missing rate of the corpus data source and the preset field completeness benchmark threshold, the preset completeness benchmark weight value is scaled inversely, and the field completeness dimension weight value is obtained after value range truncation constraint. Extract the semantic contradiction degree and logical discontinuity ratio parameters from the corpus data source, input the preset semantic quality segmentation decay function to obtain the first initial weight value of the semantic self-consistency dimension, and modify the first initial weight value in combination with the domain matching degree of the corpus data source to obtain the weight value of the semantic self-consistency dimension. The credibility index is obtained by weighting the main body qualification level, historical compliance records and information traceability level of the corpus data source. The second initial weight value is obtained by range normalization mapping. The second weighting correction is performed by combining the information release link length of the corpus data source to obtain the source credibility dimension weight value. The weighted sum of the proportion of duplicate data and the proportion of invalid noise in the statistical corpus data source is used to calculate the comprehensive redundancy. After normalizing the comprehensive redundancy, the third initial weight value is obtained by subtracting the normalization result from 1. The weight value of the redundancy noise control dimension is then obtained. The weight values ​​for field completeness, semantic consistency, source credibility, and redundancy noise control are mapped to the corresponding dimension channels of the weight matrix to construct a globally unified weight matrix.

[0060] Differentiated quantification strategies are employed across the four dimensions, with all weight values ​​confined to the [0,1] range. The weight matrix represents only the inherent quality attributes of the data source, is independent of the capabilities of the acquisition nodes, and is globally unique. The node deduplication capability is reflected through threshold correction during task execution, without affecting the weight matrix itself.

[0061] In this embodiment, the weight value of the field completeness dimension : ; in, This is the industry benchmark missing rate based on a general corpus dataset, which is a preset threshold for field completeness. A missing rate below this value is considered normal, and the value is 0.1. Based on corpus usability statistics, for every 1 percentage point increase in the field missing rate, the actual usability of the corpus decreases by approximately 2%. Therefore, a scaling factor is set. It is 2.0; The standard weight value corresponding to the baseline missing rate is the preset completeness baseline weight value, which is 0.8. The original weights of the i-th data source; This represents the weight value for the field completeness dimension of the i-th data source; Let be the field missing rate of the i-th data source, with a value range of [0,1], representing the completeness of the fields in the data source.

[0062] Semantic self-consistency dimension weight value : Extracting semantic contradictions from data sources (Probability of contradictory assertions output by the semantic implication model) and the proportion of logical breaks (The percentage of sentences with a text coherence score < 0.4), each input a uniform piecewise decay function h(x): ; The segmentation thresholds are set based on the quality annotation statistics of the CC100 public corpus dataset. 0.05, 0.10, 0.20, 0.35, and 0.50 correspond to the dividing points of five quality levels: excellent, good, qualified, poor, and inferior, respectively, and have clear business semantics.

[0063] The smaller value between the two indicator decay results is taken as the first initial weight. : The smaller value is taken according to the principle of the weakest link: if either of the two indicators falls into the poor level, the poorer level shall be used as the standard to avoid the problem of a single indicator being too high and covering up quality issues.

[0064] Introducing domain matching degree (Calculated by the RoBERTa domain classification model, with a classification granularity of first-level industry and a total of 12 categories) This data is then corrected, with correction coefficients ranging from [0.7, 1.0]: Even if the semantic quality of the corpus itself is good, if the content is irrelevant to the target domain, the actual training value will be reduced. There is no reduction when the domain is completely matched, and the weight is reduced by 30% when it is completely mismatched.

[0065] In this embodiment, the weight value of the source credibility dimension : Calculating the credibility index It includes three evaluation factors: Main body qualification level Grade A (Official Media / Government Agencies) 1.0, Grade B (Legitimate Registered Corporate Media) 0.7, Grade C (Individual Self-Media / Unverified) 0.4; Historical compliance record The range is [0,1]. Information traceability level score :in, Let be the number of information propagation levels in the i-th data source. =1 indicates that the information comes from the original publisher, and each additional 1 indicates that the information has gone through one more layer of reposting and dissemination.

[0066] .

[0067] Weighting was determined by pairwise comparisons of factors influencing corpus quality using the Analytic Hierarchy Process (AHP). Subject qualification, being the foundation of trust, had the highest weight (0.40), followed by historical compliance, which is behavioral evidence (0.35), and source tracing level, being the reference factor, had the lowest weight (0.25). The sum of the three was 1.

[0068] Second initial weights ; Introducing the information publishing link length (Including the number of parallel retransmissions at the same level) undergoes exponential decay correction, with a baseline link length. =2, attenuation coefficient =0.05, at this time, The longer the information transmission link, the higher the risk of distortion. Therefore, the weight decays exponentially. When the link length is ≤2, there is no decay, which is consistent with the distortion law of conventional information transmission.

[0069] Redundancy noise control dimension weight values : The percentage of repeated data in statistical corpora (SimHash Hamming distance ≤ 3 is considered a duplicate) and the proportion of invalid noise (For corpus texts containing more than 30% noisy content), calculate the overall redundancy. : The negative impact of repetitive content on training effect is slightly greater than that of invalid noise. Therefore, the weight of repetition is 0.55 and the weight of noise is 0.45.

[0070] Perform range normalization on the overall redundancy of the entire data source: ; ; The 120-dimensional weight vectors in four dimensions are concatenated column-wise to form a 120×4 global weight matrix: Each row of the matrix corresponds to the four-dimensional quality weight of a data source, and each column corresponds to the weight of all data sources in one dimension.

[0071] Example 3: This invention provides a method for filtering high-quality corpus data based on big data. Adaptive processing is performed on the initial corpus filtering decision boundaries for each level to obtain the optimal corpus filtering decision vector for the corresponding node at each level, including: S51: Configure a primary tolerance coefficient and a secondary tolerance coefficient for each preset dimension. Multiply the boundary adjustment benchmark value of each dimension with the corresponding primary tolerance coefficient to generate a primary boundary migration tolerance interval. Multiply the preset dimension quality deviation benchmark value with the corresponding secondary tolerance coefficient to generate a secondary boundary migration tolerance interval. The boundary adjustment benchmark value of each dimension is the boundary value of the corresponding dimension in the initial corpus screening and judgment boundary of the current level. The dimension quality deviation benchmark value is the average absolute deviation value of the actual quality profile and the expected quality profile of the corresponding node in each dimension within the most recent K evaluation periods. K is the backtracking window length of the quality deviation statistics, which is a positive integer and is recommended to be in the range of 3 to 10. The larger the value of K, the higher the smoothness of the deviation benchmark and the stronger the resistance to random fluctuations.

[0072] S52: Configure the feedback signal weight and prediction signal weight independently for each dimension, perform weighted fusion of the screening boundary feedback correction signal after the distribution sensitivity coefficient correction and the screening boundary trend prediction compensation signal for the corresponding dimension to generate the boundary migration correction amount for that dimension, and concatenate all dimensions to form a first-level boundary migration correction vector. The first-level boundary migration correction vector is superimposed onto the initial corpus screening and judgment boundary of the current level using a relative percentage multiplication method to obtain the first-level migration boundary parameter set. Each dimension parameter in the first-level migration boundary parameter set is compared with the first-level boundary migration allowable interval of the corresponding dimension. If all dimension parameters fall within the corresponding first-level boundary migration allowable interval, the first-level migration boundary parameter set is marked as a candidate migration boundary set and retained. Otherwise, the weight ratio of the two signals of the out-of-bounds dimension is adjusted independently. If it is still impossible to make all dimension parameters fall within the allowable interval after the adjustment of the preset maximum number of iterations, the first-level migration boundary parameter set corresponding to the weight ratio that minimizes the total deviation of the out-of-bounds dimension parameters is taken as the candidate migration boundary set. S53: Based on the empirical distribution of node local quality distribution, Monte Carlo sampling is performed to generate simulated corpus samples. A preset corpus quality fluctuation perturbation factor is introduced into the four-dimensional quality score of the simulated corpus to simulate the random fluctuation of corpus quality in real scenarios. Then, the candidate migration boundary set is used as the screening threshold to perform simulated screening, simulate the full corpus screening process, and output the simulated value of corpus quality deviation and the simulated value of screening pass rate corresponding to the secondary verification boundary set to obtain the steady-state effective boundary set. S54: Map each dimension parameter of the steady-state effective boundary set into an executable corpus screening judgment threshold, preset dimension weight coefficient and cleaning trigger condition parameter, and concatenate them to generate the optimal corpus screening decision vector for the current level of the corresponding node.

[0073] The calibration interval uses a globally preset default deviation value during the system cold start phase.

[0074] Monte Carlo empirical sampling relies on multiple empirical quantiles of data sources to randomly generate simulated corpora, without employing the normal distribution assumption. It is used to verify the actual quality deviation and screening pass rate at various thresholds, and includes: Gradient iteration ratio adjustment: Independently adjust the fusion weight of the feedback and prediction signals of a single dimension when it exceeds the limit. The step size of a single round is 0.05, the maximum number of iterations is 20, and the iteration termination condition is that all dimensions fall within the first-level allowable range or the total deviation is minimized. Two-level boundary allowable range: The first-level amplitude range limits the maximum adjustment range of a single threshold to avoid violent oscillations; the second-level fluctuation range limits the acceptable quality deviation range after disturbance, and the verification takes the second-level range as a hard catch-all condition. Data source change and reset rules: If the percentage of data source additions or removals to a node exceeds 30%, the local quality distribution and prediction model will be completely reset, and three rounds of local cold starts will be initiated; if the percentage of changes is less than 30%, only new data sources will be added and globally initialized. System initialization rules: When there is no collected data for the first time, the weight matrix and level thresholds are all calibrated offline; for the first 3 periods, the hierarchical weights and trend predictions are turned off, and only the fixed global parameters are used to perform basic screening.

[0075] In this embodiment, the cold start phase is the first three evaluation cycles of the system. There is insufficient historical data, so the trend prediction module is disabled, the hierarchical dynamic weight is invalid, only the feedback single-loop control is enabled, and all statistical parameters use global default values.

[0076] The distribution sensitivity coefficient characterizes the degree of influence of a small change in the threshold on the average quality of the overall corpus at the current threshold position, and its value is limited to 0.5 to 2.0. It is calculated as the reciprocal of the probability density at the threshold and is used to correct the original adjustment amount and eliminate the correction error caused by the nonlinearity of the distribution.

[0077] The filtering boundary feedback correction signal is obtained by matching the empirical rules to get the original adjustment range, multiplying it by the distribution sensitivity coefficient to get the final adjustment ratio of each dimension. Positive numbers represent tightening the threshold, and negative numbers represent relaxing the threshold, which is used to correct the corpus quality deviation that has appeared.

[0078] The screening boundary trend prediction compensation signal is the proportion of corpus quality shift predicted by the random forest model for the next period. Combined with the distribution sensitivity coefficient, the pre-adjustment amount is obtained to hedge against future quality fluctuations in advance. It belongs to the feedforward control signal.

[0079] The optimal corpus selection decision vector is the output of complete execution parameters after adaptive verification, including four-dimensional selection threshold, global normalized dimension weight, hierarchical weight, cleaning trigger conditions, and low-quality data source blocking rules.

[0080] The preset dimension weight coefficients are globally unified four-dimensional normalized weights, with the sum of the four weights being 1, and the values ​​ranging from 0 to 1; they are used for calculating the comprehensive quality score and for multi-dimensional comprehensive deviation weighted statistics.

[0081] Heterogeneous corpus cleaning is a standardized processing flow for multi-format text, which includes tag stripping, text extraction, and multiple SimHash-MinHash joint denoising, and outputs UTF-8 standard text with metadata.

[0082] The initial boundary of node N7 in gear 3 , For example: Preset Level 1 Allowable Range Coefficient =0.10 (the adjustment range of a single wheel boundary shall not exceed ±10% of the benchmark value), secondary allowable range coefficient =0.08 (the permissible fluctuation of the quality deviation after disturbance is ±8% of the reference deviation), satisfying 0 < < <0.5. Parameter setting basis: =0.10 is a commonly used safety limit for single-wheel regulation in the field of industrial control, which can effectively avoid regulation oscillation; =0.08 corresponds to the acceptable range of quality fluctuations, which is set based on historical statistics of corpus quality fluctuations.

[0083] First-level boundary migration allowable range: based on the initial boundary values ​​of each dimension. Based on, the interval is The calculation results for each dimension are shown in Table 2: Table 2 Calculation Results for Each Dimension

[0084] Dimensional quality deviation benchmark: The mean absolute deviation of node N7 over the most recent K=5 periods. Assume the deviation records for the 5 periods are: Credibility [3,5,6,7,8] → mean 5.8; Semantics [2,3,5,6,7] → mean 4.6; Completeness [3,4,5,5,5] → mean 4.4; Redundancy noise [2,4,5,6,7] → mean 4.8. That is, the benchmark deviation vector. .

[0085] Secondary boundary migration allowable range: based on the benchmark deviation value, the range is... The calculation results for each dimension are shown in Table 3: Table 3 Calculation Results for Each Dimension

[0086] Cold start phase alternative rules: When the system has run for less than 3 evaluation cycles, the dimensional quality deviation benchmark value adopts the preset default value: field completeness 5 points, semantic self-consistency 6 points, source credibility 7 points, and redundancy noise control 6 points; the trend prediction compensation signal is 0 by default, and only the feedback correction signal is used for adjustment.

[0087] In this embodiment, the feedback signal weight Predicted signal weights ,satisfy Weighting is based on the following criteria: feedback signals, based on actual deviations, have higher reliability and therefore higher weights; prediction signals, based on trend forecasts, serve as auxiliary compensation and therefore have lower weights. Each dimension is independently weighted and fused to generate the boundary migration correction for that dimension. The fusion correction vector can then be obtained. .

[0088] The correction vectors are superimposed onto the initial boundary as a relative percentage to obtain the first-level migration boundary parameter set: At this point, i.e., the first-level migration boundary By comparing each dimension against the first-level allowable interval, all dimensions fall within the interval range; therefore, Marked as a set of candidate migration boundaries.

[0089] Out-of-bounds adjustment rules: If a dimension exceeds the bounds, iterative adjustment shall be performed according to the following rules: Single-dimensional boundary crossing: Identify the direction of the boundary crossing, reduce the weight of the signal that caused the boundary crossing in a step of 0.05, and increase the weight of another signal to ensure that the sum of the weights is 1; Multi-dimensional same-direction boundary crossing: Adjust all weight allocation ratios uniformly according to the overall boundary crossing direction; Multi-dimensional out-of-bounds: The weight ratio of each out-of-bounds dimension is adjusted independently, and the dimensions do not interfere with each other; If all iterations fail after the maximum number of iterations (20), the group with the smallest total relative deviation from the boundary among all iteration results is selected as the candidate boundary.

[0090] In this embodiment, the secondary verification adopts a dual-objective hierarchical verification mechanism. The primary objective is to ensure that the quality deviation meets the standard and does not exceed the limit throughout the process. The secondary objective is for the pass rate to approach the target value within the quality constraints. Introducing a corpus quality fluctuation disturbance factor The data follows a uniform distribution to simulate the random fluctuations in corpus quality in real-world scenarios. Perturbation values ​​are independently sampled for each dimension of the candidate boundaries to generate secondary validation boundaries. .

[0091] Monte Carlo sampling is performed based on an empirical distribution of node-local quality distribution: First layer: Based on the empirical distribution at the data source level, sample 10,000 data source samples; The second layer: For each sampled data source, based on 5 empirical quantiles of its internal corpus, linear interpolation is used to sample the four-dimensional quality score of a single corpus. use The 10,000 simulated texts were used as the screening boundary for a series of judgments, and the mean quality and pass rate after screening were statistically analyzed.

[0092] The simulated value of the corpus quality deviation was calculated. The simulated pass rate was 47%.

[0093] Dual-target verification: Quality deviation verification: The simulated deviations in all dimensions all fall within the secondary tolerance range and do not touch the range boundary, thus the primary target is met; Pass rate verification: The target pass rate for gear 3 is 50%, with an allowable range of ±10%, i.e., [45%, 55%]. If 47% falls within the range, the secondary target is met.

[0094] If both conditions are met, determine the candidate boundary. This is the set of steady-state effective boundaries. If the quality meets the standard but the pass rate does not, the threshold is fine-tuned within the boundary of the secondary allowable interval (step size 0.5 points) to make the pass rate as close to the target value as possible without exceeding the quality deviation constraint. If fine-tuning to the boundary of the quality deviation interval still cannot meet the pass rate requirement, fine-tuning is stopped, and the quality deviation meeting the standard is used as the fallback standard.

[0095] The two-stage verification process has a clear division of labor: the adjustment range of the first-stage verification tube must not be excessive to prevent oscillation caused by excessive adjustment in a single stage; the adjustment effect of the second-stage verification tube must be stable and the gear position must be accurate to verify the consistency between actual performance and pass rate under quality fluctuation scenarios. The two checkpoints together ensure the safety and effectiveness of adaptive adjustment.

[0096] In this embodiment, the parameters of each dimension of the steady-state effective boundary are rounded down. The rule is: all dimensions are lower bound thresholds (score ≥ threshold qualified), and rounding up is uniformly used to ensure that the screening rigor is not reduced—rounding up increases the threshold standard without relaxing the screening requirements, as shown in Table 4: Table 4. Rounding Results

[0097] An executable threshold vector [81, 77, 73, 78] is obtained. Global dimension weight coefficients are set to [0.20, 0.30, 0.30, 0.20] according to preset importance, with core dimensions (semantics, credibility) having higher weights, used for comprehensive quality score calculation and multi-dimensional bias weighting. The cleaning trigger condition is that deep cleaning is triggered when the redundant noise score is <76; the low-quality blocking rule is that data sources with a field completeness score <60 for three consecutive periods are automatically blocked. The optimal corpus selection decision vector is generated by concatenating these vectors. .

[0098] Example 4: This invention provides a method for filtering high-quality corpus data based on big data, generating initial corpus filtering judgment boundaries for corresponding levels, including: S21: Extract the training demand intensity features and training demand accuracy features from the training requirements of downstream large language models; Based on the first preset mapping rule, under the current corpus quality screening threshold, calculate the quota of nonlinear correlation fusion component and the quota of linear interaction fusion component; S22: For the four preset dimensions covered by the nonlinear correlation fusion component, calculate the preset dimension activation coefficient of each dimension according to the second preset mapping rule and the dimension importance level. S23: Decompose the weight matrix into 6 groups of two-dimensional mass weight sub-planes by combining them pairwise according to the dimensions; For each two-dimensional quality weight subplane, all data sources are first sorted in ascending order by weighted comprehensive score according to the preset dimension weight coefficients of the two dimensions corresponding to the subplane. Then, equidistant gridding is performed. The grid block size is dynamically determined based on the row and column dimensions of the subplane. The greatest common divisor or the closest integer divisor of the row and column dimensions is taken as the block scale parameter to form several quality weight block units with consistent size. For each block unit, calculate the mean weight of each of the two dimensions within the unit to obtain the block dimension weight vector; The downstream large language model training requirement feature vector is split according to the dimension level. Based on the dimension combination mapped by each quality weight block unit, the feature values ​​of the corresponding dimension are extracted from the requirement feature vector, normalized and quantized to form requirement feature sub-blocks, so that each group of requirement feature sub-blocks establishes a spatial mapping relationship with all quality weight block units in the corresponding two-dimensional quality weight sub-plane. S24: Based on the spatial mapping relationship, using the allocated linear interactive fusion component quota and nonlinear correlation fusion component quota as block fusion coefficients, calculate the fusion result of the corresponding quality weight block unit and the demand feature sub-block block block by block to generate a block-level two-dimensional importance coefficient vector; aggregate all block-level two-dimensional importance coefficient vectors by dimension to obtain the dimension weight coefficients of the quality hierarchy. Multiply the dimensional weight coefficients of the quality stratification level with the benchmark threshold vector of the tier by dimension to generate the initial screening threshold corresponding to each quality stratification level. Concatenate the results to obtain the initial corpus screening judgment boundary corresponding to the current corpus quality screening threshold tier.

[0099] In this embodiment, the data source is arranged in ascending order according to the weighted comprehensive score of each two-dimensional combined subplane. The greatest common divisor of rows and columns is used as the block size to divide the data source into blocks of the same size and distinguish between three quality levels: low, medium and high.

[0100] The training demand intensity feature s ranges from 0 to 1, representing the stringent requirements of model training on the overall quality of the corpus. The higher the value, the stricter the overall selection criteria. The value is lower for pre-training tasks and higher for vertical fine-tuning tasks.

[0101] The value range of the training precision feature 'a' is 0 to 1, which controls the weight difference between the core dimension and the auxiliary dimension. The higher the value, the greater the weight difference between the two dimensions, thus adapting to high-precision and detailed training needs.

[0102] Linear interaction is an element-wise Hadamard product operation between demand features and dimensional weights. It is used to capture the basic linear relationship of dimensions, has low computational cost, and is suitable for basic weight fusion.

[0103] Nonlinear correlation: A differentiated nonlinear mapping mechanism is adopted for core dimension blocks and auxiliary dimension blocks. The core block adopts a weighted nonlinear mapping of nonlinear fusion quota × dimension activation coefficient × demand feature quantification value, superimposed with dimension benchmark offset correction amount; at the same time, two layers of ReLU fully connected perceptron can be configured to supplement and correct the high-order coupling relationship between dimensions. The input of the perceptron is the two-dimensional weight vector of the corresponding block and the two-dimensional demand feature sub-block, and the output is the two-dimensional nonlinear correction coefficient.

[0104] Taking the medical fine-tuning task with s=0.8 and a=0.9 at level 3 as an example.

[0105] The first pre-defined mapping rule is that the quota for non-linear fusion components increases linearly with the intensity of training demand, as shown in the formula: ; in, These represent the minimum and maximum values ​​of the nonlinear fusion quota.

[0106] The higher the training intensity (such as fine-tuning tasks), the more complex the nonlinear coupling between corpus dimensions becomes, requiring a larger proportion of nonlinear associations to capture deep relationships; low-intensity tasks (such as general pre-training) are mainly linear interactions, while also taking into account computational efficiency.

[0107] The second preset mapping rule is that the activation coefficient of the core dimension increases with the increase of the precision feature, while the auxiliary dimension decreases with the increase of the precision feature, thus amplifying the dimensional differences under the requirement of high precision.

[0108] Core dimensions (semantic consistency, source credibility) activation coefficients: ,in, , ; The baseline activation coefficient represents the basic activation level of a dimension when the required training precision is 0. The activation coefficient adjustment range represents the total change in the activation coefficient as the precision feature changes from 0 to 1.

[0109] Activation coefficients for auxiliary dimensions (field completeness, redundancy and noise control): ,in, =0.55, =0.3.

[0110] In this embodiment, the focus is on the gridded partitioning of the weight matrix and feature mapping: The first step is to decompose the subplanes by pairwise combinations of dimensions. By randomly selecting two dimensions from the four dimensions, a total of C(4,2)=6 sets of two-dimensional mass weight subplanes are obtained: Plane (1,2): Field completeness × semantic consistency; Plane (1,3): Field completeness × Source credibility; Plane (1,4): Field completeness × Redundancy noise control; Plane (2,3): Semantic consistency × Source credibility; Plane (2,4): Semantic consistency × Redundancy noise control; Plane (3,4): Source credibility × Redundancy noise control; Each subplane corresponds to a 120×2 matrix, with rows corresponding to 120 data sources and columns corresponding to the weights of two dimensions.

[0111] The second step is data source sorting and equidistant segmentation. For each sub-plane, a weighted composite score is calculated based on the preset dimension weight coefficients for the two dimensions corresponding to that sub-plane. All data source rows are then sorted in ascending order of score to ensure a continuous transition in quality levels between adjacent rows within the two-dimensional space. The preset dimension weight coefficients are globally uniform dimension importance weights, consistent with the overall dimension hierarchy settings.

[0112] Then perform equidistant meshing, with the meshing scale parameter k set to... The largest approximation in the vicinity. In this embodiment, N=120, so k=10 is taken. Each subplane is divided into 12 10×2 mass weighted block units, corresponding to the three mass levels of low, medium and high.

[0113] The third step is to calculate the block-level dimensional weights. For each block unit, the average weights within each of the two dimensions are calculated separately to form a block-level dimensional weight vector. This avoids mixing and calculating a single average weight, ensuring dimensional independence. For example, if the p-th block in plane (2,3) contains 10 data sources, the average of the 10 semantic weights and the average of the 10 credibility weights are calculated separately to obtain a two-dimensional weight vector.

[0114] The fourth step is to map the demand features. This involves mapping the four-dimensional demand feature vector... The system splits the data by dimension, generating requirement feature sub-blocks for each two-dimensional sub-plane. For example, plane (2,3) corresponds to the requirement feature sub-block. ,in, These are the min-max normalized values ​​for the corresponding dimensional features. All blocks within the same subplane share the same set of requirement feature sub-blocks, establishing a one-to-one spatial mapping relationship.

[0115] It should be noted that each original dimension appears in 3 sets of subplanes, which is not redundant calculation. Different combinations of dimensions form different interaction spaces, and the importance of the same dimension varies when paired with different dimensions. Calculating them separately can capture the differentiated interaction effects between dimensions.

[0116] In this embodiment, block-level weight fusion and hierarchical boundary generation are performed as follows: The first step is block-by-block fusion computation. Based on spatial mapping relationships, using... For each block, a differential fusion calculation is performed to generate a two-dimensional importance coefficient vector for the initial selection of blocks (see Example 7 for details).

[0117] The second step is statistical consistency verification. The importance coefficient of each block is checked for deviation from the global mean of the subplane, and interpolation corrections are performed on outlier blocks.

[0118] The third step is value range truncation and coefficient mapping. The dimensions of the importance coefficients are unified, and the value range is truncated to the interval [0.8, 1.2] to ensure that the weight adjustment range is within a reasonable range.

[0119] The fourth step is quality stratification and aggregation. All data sources within the two-dimensional subplane, sorted in ascending order of weighted composite score, are divided into blocks. These blocks are then divided into three quality levels: low, medium, and high, with each level containing an equal number of blocks. Alternatively, unequal-frequency division based on quality score thresholds can be used according to business needs. The default number of levels is 3, which can be adjusted from 2 to 5 levels depending on the data source size and filtering granularity requirements. For example, 12 blocks can be merged into three levels (low, medium, and high) based on quality level. The importance coefficient of the same dimension within each level is averaged to obtain the dimension weight for that level. Then, the level weights of the same dimension across the three subplanes are averaged to obtain the final quality stratification dimension weight coefficient. Taking field completeness as an example, the weights for the three levels are 0.92, 1.0, and 1.05, respectively. Lower quality levels have higher completeness weights, reinforcing the basic quality threshold.

[0120] Each two-dimensional subplane gives the dimensional importance coefficient of each layer under the interaction scenario. The mean of the three independent scenarios integrates the dimensional importance under different interaction constraints. The final layer weight can reflect the comprehensive dimensional importance under multiple interaction scenarios.

[0121] Step 5: Generation of initial boundaries for each layer. This involves combining the four-dimensional dynamic weight coefficients of each layer with the threshold vector of the gear level. Multiply by the dimension to obtain the initial screening threshold for each quality stratum, and then splice them together to form a complete initial screening decision boundary.

[0122] In this embodiment, during the fusion process of S24, the linear interaction component is directly achieved by multiplying and accumulating the element-wise weights of the block dimension and the demand features; the nonlinear correlation component is divided into blocks for the core dimension, and the demand features are first modulated by the dimension activation coefficient, and then nonlinearly weighted and mapped with the block dimension weights. Finally, the linear component and the nonlinear component are superimposed according to the quota ratio to obtain the block-level two-dimensional importance coefficient.

[0123] It should be noted that the overall coarse weight adjustment is broken down into multiple local fine-grained blocks. Each block corresponds to a set of data sources with similar quality levels and a set of specific dimensional interactions. The final output is a hierarchical dimensional weight, which makes the adjustment of demand features more precise and has a higher adaptability to data sources of different quality levels. This avoids the problem of over-screening of high-quality data sources and under-screening of low-quality data sources caused by a single global weight.

[0124] Example 5: This invention provides a method for filtering high-quality corpus data based on big data, outputting a filtered subset of high-quality target corpus, including: S61: Sort all idle distributed corpus collection nodes according to the preset node polling scheduling sequence, and take the idle corpus collection node at the top of the sequence as the current task execution node. Based on the corpus quality screening threshold required by the corpus collection task to be performed, the matching threshold decision vector is retrieved from the optimal corpus screening decision vector of each threshold of the corresponding node, and the redundancy noise control dimension threshold is corrected by combining the deduplication processing capability coefficient of the current node to obtain the actual redundancy threshold. Corpus collection tasks whose corpus collection coverage and corpus data type attributes both fall within the job adaptation domain of the current task execution node are included in the candidate collection task subset corresponding to the current node. S62: Verify one by one whether the data source range of each data collection task to be executed in the candidate data collection task subset overlaps with the current data source range of the other data collection nodes that are in the data collection work state. If the proportion of data sources in the overlapping areas of the collection range exceeds the preset collection range conflict threshold, the corresponding corpus collection task to be executed will be removed from the candidate collection task subset. S63: If the subset of candidate collection tasks after secondary filtering is not empty, first perform a corpus task type matching judgment on each candidate corpus collection task and divide them into two categories: standard structured corpus collection tasks and unstructured generalized corpus collection tasks. Retrieve the historical data of the current task execution node and calculate the corpus collection performance score for the two types of corpus collection tasks respectively; Select the candidate corpus collection task with the highest performance score as the optimal collection task and assign it to the current task execution node; If the candidate collection task subset is empty, the current task execution node is moved to the end of the idle node polling sequence, and the number of idle rounds of the node is accumulated. If the number of consecutive idle rounds of the node exceeds the preset limit, the node is temporarily suspended to wait for new tasks to be added to the database, and the next idle corpus collection node is selected in order to repeat S61. S64: Remove the assigned optimal collection task from the global set of pending corpus collection tasks, change the working status of the current task execution node from idle to collecting data, and remove it from the idle node polling sequence; Take the next idle corpus collection node in the idle node polling sequence as the new current task execution node, and repeat S61 to S63 until the global set of pending corpus collection tasks is empty or all nodes have been traversed and there are no tasks to be assigned. S65: For the corpus collection tasks assigned to each corpus collection node, match the corresponding corpus collection control parameter template according to the task type, and calculate the optimal collection frequency in combination with the real-time load constraints of the node with the goal of maximizing the output efficiency of qualified corpus per unit time. By combining the cleaning trigger condition parameters and low-quality blocking rules in the optimal corpus selection decision vector, multi-dimensional corpus acquisition control instructions are generated and sent to the corresponding nodes; Each node performs corpus collection, heterogeneous corpus cleaning, and low-quality data source blocking operations according to instructions. It also performs hierarchical filtering and comprehensive sorting based on the dimensional weight coefficients of the quality stratification level, and outputs a unified subset of high-quality target corpus.

[0125] The job adaptation domain is the boundary of the tasks that can be processed, which is jointly determined by the node hardware, computing power, collection permissions, and data source type. Only tasks within the adaptation domain can be included in the candidate allocation queue.

[0126] The node historical collection context data is stored according to task type, and the node's historical operation indicators are retained. Data on collection time, redundancy rate, and pass rate from the last 20 rounds are used for performance scoring and prediction. If there are fewer than 5 historical records for a single task type, global reference is enabled.

[0127] The deduplication capability coefficient characterizes the performance of the node text denoising algorithm, with a value ranging from 0.8 to 1.2; the high-performance dual-path deduplication configuration uses 1.2, the standard configuration uses 1.0, and the simple configuration uses 0.8; the higher the coefficient, the more lenient the redundancy entry threshold.

[0128] Assume the current system state is as follows: nodes N1, N3, and N5 are in the data collection process, while nodes N2, N4, N6, N7, and N8 are in an idle state; there are a total of 12 tasks to be executed globally, each task is labeled with the required level, corpus type, and data source range.

[0129] Idle nodes are sorted in ascending order by ID as [N2, N4, N6, N7, N8], with the first node, N2, selected as the current task execution node. First, the optimal decision vector for N2 at the target level is retrieved, with a baseline redundancy threshold of 78 points. Combining this with N2's deduplication capability coefficient k=1.2 (advanced deduplication configuration, supporting SimHash+MinHash dual-path deduplication), the redundancy noise threshold is corrected. Actual redundancy threshold = baseline redundancy threshold / k.

[0130] Next, tasks to be executed were screened: tasks whose corpus collection scope and data types fell within the N2 job adaptation domain (N2 excels at structured corpora and is compatible with 22 news and document data sources in the Northwest region) were included in the candidate subset. Four candidate tasks were obtained from the initial screening.

[0131] Each candidate task's data source range is checked to see if it overlaps with the current collection range of the active node. The proportion of overlapping data sources to the total number of candidate task data sources is calculated. If it exceeds a preset conflict threshold, the conflict is resolved. If the percentage is 25%, remove it.

[0132] Task T3: The data source set {source A, B, C, D, E} overlaps with the collection range {source B, F, G, H} of active node N3. Source B accounts for 20% < 25%, so it is retained. Task T7: The data source set {source J, K, L, M} overlaps with the collection range {source J, K, N, O} of active node N5. Sources J and K account for 50% > 25%, so they are removed.

[0133] After filtering, the candidate subset has 3 tasks remaining.

[0134] The three candidate tasks were divided into two categories: structured corpus collection tasks (T3, T9) and unstructured generalized corpus collection tasks (T11). Historical collection context data from N2 was retrieved, and the performance scores for both types of tasks were calculated using the formula in Example 6. Structured task performance score: 0.634; Unstructured task performance score: 0.604; The structured task received a higher score, so T3 with the smallest ID was selected as the optimal data collection task and assigned to N2.

[0135] If the candidate subset is empty, move the current node to the end of the idle sequence and increment the idle round count by 1; if there are 3 consecutive idle rounds, mark the node as temporarily suspended, wait for a new task to be added to the database, and then take the next node in order to continue.

[0136] In this embodiment, T3 is removed from the global set of tasks to be executed (leaving 11), and the state of N2 changes to "in operation" and it is removed from the idle sequence. The idle sequence is updated to [N4, N6, N7, N8], and the next node N4 is taken and the above process is repeated until all tasks are assigned or no nodes are available for assignment.

[0137] In this embodiment, for task T3 assigned to N2, the optimal acquisition frequency is calculated by matching the control parameter template of the structured task: Baseline frequency: 1.0 times / hour for structured tasks; Pass rate correction: The Monte Carlo simulation based on the candidate boundary estimates a pass rate of approximately 52%, and the target pass rate for gear 3 is 50%. The pass rate correction coefficient is approximately 50% / 52% ≈ 0.96. Load Adjustment: N2's current CPU utilization is 45%, memory usage is 40%, and task queue length is 2. The weighted calculated load value is 0.42, which is lower than the threshold of 0.7. The load adjustment coefficient is 1.0, and the load value = w_cpu × CPU utilization + w_mem × memory usage + w_queue × (current task queue length / maximum queue length), where w_cpu = 0.4, w_mem = 0.4, and w_queue = 0.2. All indicators are normalized to the [0,1] interval. When the load value ≤ 0.7, the adjustment coefficient is 1.0; when 0.7 < load value ≤ 1.0, the adjustment coefficient decreases linearly from 1.0 to 0.5, that is, the adjustment coefficient = 1.0 - (load value - 0.7) × (0.5 / 0.3).

[0138] Optimal acquisition frequency = reference frequency × (target pass rate / estimated pass rate) × load adjustment coefficient.

[0139] With the goal of maximizing the output efficiency of qualified data per unit time, the frequency is appropriately increased when the pass rate is low to ensure output, and the frequency is reduced when the load is high to avoid system overload, thus achieving a balance between output and resources. If the load exceeds 0.7, the load adjustment coefficient is linearly reduced to 0.5, forcibly reducing the acquisition frequency to ensure system stability.

[0140] Other control commands: Data collection priority: For data with a high credibility threshold, data will be collected primarily from official data sources of level A, with at least 70% of the data collected from high-credibility data sources. Cleaning trigger: Deep cleaning is triggered when the redundancy score is <65, and SimHash+MinHash dual-path deduplication is performed; Low-quality blocking: Data sources with a field completeness score of <60 for 3 consecutive periods are automatically blocked.

[0141] Control commands are packaged into JSON format and sent to the N2 node. After parsing, the node performs collection, cleaning, filtering, and blocking operations. After basic qualification judgment, the comprehensive quality score of each corpus is calculated based on the weight of the hierarchical dimensions. The high-quality corpus with higher priority is output according to the score and finally sent back to the central server for storage.

[0142] Polling ensures fair task allocation among nodes, conflict checking avoids data source resource contention, performance optimization achieves precise matching of tasks and capabilities, and load-aware frequency optimization ensures stable system operation. The combination of these four factors can significantly improve overall data acquisition efficiency.

[0143] Example 6: This invention provides a method for filtering high-quality corpus data based on big data, wherein the corpus collection performance score is calculated using a normalized weighted method.

[0144] ,in, This is the estimated acquisition time after min-max normalization, with a value range of [0,1]. The shorter the acquisition time, the smaller the value. The larger the value, the higher the score.

[0145] This is a predicted redundancy rate of the corpus after min-max normalization, with a value range of [0,1]. Lower redundancy indicates higher redundancy. The smaller, The larger the value, the higher the score.

[0146] This is a min-max normalized estimate of the post-cleaning pass rate, with a value range of [0,1]. A higher pass rate generally indicates a better pass rate. The larger the value, the higher the score.

[0147] These are the acquisition efficiency weight, redundancy control weight, and pass rate weight, respectively, all of which are positive numbers and satisfy the following conditions: =1. In this scheme, we take... =0.30、 =0.30、 =0.40, with the pass rate having the highest weight, because the proportion of the final usable corpus is the most critical output indicator of the data collection task. Efficiency and redundancy are equally important, each accounting for 30%.

[0148] Normalized reference system rules: When there are ≥5 historical records of the same type of task at a node, the reference for min-max normalization is the historical maximum and minimum values ​​(historical minimum and maximum values) of the same type of task at that node. When there are fewer than 5 historical records of the same type of task on a node (cold start phase), a global universal reference system (the historical maximum and minimum values ​​of the same type of task on all nodes) is adopted.

[0149] Performance scoring is used for internal task selection within the same node, rather than cross-node comparison. Therefore, using the node's own history as a reference is more reasonable and can avoid scoring bias caused by differences in basic capabilities between new and old nodes. During the cold start phase when data is insufficient, a global reference system is used as a transition, and it automatically switches after sufficient data is accumulated.

[0150] Example 7: This invention provides a method for filtering high-quality corpus data based on big data, generating a block-level two-dimensional importance coefficient vector, including: S241: Based on the corpus preset dimension attribute level mapped by each quality weight block unit, all quality weight block units are divided into two categories: core preset dimension blocks and auxiliary preset dimension blocks. Among them, quality weight block units containing semantic self-consistency dimension or source credibility dimension are marked as core preset dimension blocks, and quality weight block units containing only field completeness dimension and redundant noise control dimension are marked as auxiliary preset dimension blocks. S242: Divide the core preset dimensions into blocks, and use the linear interaction fusion component quota as the coefficient to perform linear weighting on the mean weight of the corresponding dimension in the block dimension weight vector to obtain the linear components of each dimension. Using the product of the nonlinear correlation fusion component quota and the corresponding dimension preset dimension activation coefficient as the coefficient, nonlinear weighting is applied to the feature quantization values ​​of each dimension in the demand feature sub-block to obtain the nonlinear components of each dimension. The linear and nonlinear components are summed according to their dimensions and then superimposed with a preset dimension baseline offset correction amount to obtain the initial two-dimensional importance coefficient vector of the block. The feature quantization value is the standardized value of each dimension feature value in the required feature sub-block after being normalized to the [0,1] interval by min-max. The preset dimension baseline offset correction amount is a constant correction term that has been normalized and scaled to the same dimension as the weighted summation result. For the auxiliary preset dimension blocks, the linear interactive fusion component quota is used as the dominant linear mapping coefficient. The average weight of each dimension in the block dimension weight vector is multiplied element by element with the feature quantization value of the corresponding dimension in the demand feature sub-block to obtain a linear mapping two-dimensional vector. The product of the nonlinear correlation fusion component quota and the corresponding dimension's preset dimension activation coefficient is used as the amplitude fine-tuning coefficient. Multiplication amplitude correction is performed on the linear mapping two-dimensional vector according to the dimension to obtain the block's initial two-dimensional importance coefficient vector. S243: For each block, the initial two-dimensional importance coefficient vector is selected, and its relative deviation from the global average importance coefficient in the corresponding two-dimensional quality weight subplane is calculated. If the relative deviation exceeds the reasonableness threshold, it is marked as an abnormal block and interpolation correction is performed using the subplane global mean combined with the block attribute weights; If the relative deviation does not exceed the threshold, the initial two-dimensional importance coefficient vector of the block selection is directly retained; S244: After processing the value range truncation constraint, the block-level two-dimensional importance coefficient vector corresponding to each block is obtained by taking the block-corrected importance coefficient vector corresponding to all quality weight block units.

[0151] In this embodiment, the feature quantization value is the original demand feature normalized to the 0-1 range by min-max, which unifies the numerical scale of different indicators and is used for block fusion calculation.

[0152] The block attribute weights are the weights for correcting block anomalies through interpolation. The weight for core dimension blocks is 0.7, and the weight for auxiliary dimension blocks is 0.3. The weights are set based on the statistical stability of the block data source samples, with highly stable blocks retaining a higher proportion of their own calculation results.

[0153] The preset dimension baseline offset correction amount is a fixed constant that compensates for the systematically low scores of the core dimension. The normalized dimensionless value ranges from 0 to 0.03, with a semantic dimension offset of 0.03 and a credibility dimension offset of 0.02.

[0154] Using plane (2,3) (semantic × credibility, core dimension combination) and plane (1,4) (completeness × redundancy, auxiliary dimension combination) as examples, the calculation process of block-level importance coefficient is fully demonstrated.

[0155] Based on the dimensional types contained in the subplane, all blocks are divided into two categories: core blocks and auxiliary blocks. Core blocks contain only semantic or credibility dimensions; auxiliary blocks contain only completeness and redundancy dimensions. The classification results are shown in Table 5. Table 5 Classification Results

[0156] Semantics and credibility have a greater impact on the actual usability of the corpus. The corresponding blocks are mostly collections of medium-to-high-quality data sources with sufficient sample size and high statistical stability. Therefore, their own results have a higher weight when correcting anomalies. Completeness and redundancy are basic quality dimensions. The corresponding blocks are mostly collections of low-quality data sources with large data fluctuations and poor statistical stability. Therefore, the global mean has a higher reference weight. The parameter values ​​are directly matched with the statistical stability of the blocks.

[0157] Different fusion strategies are adopted for the core blocks and auxiliary blocks. Each block outputs a two-dimensional importance coefficient vector, which contains the initial selection coefficient values ​​of each of the two dimensions.

[0158] Core dimension block example (plane 2,3 p-th block): Given: the block dimension weight vector is Demand feature sub-block semantic activation coefficient =0.915, credibility activation coefficient =0.915, Dimensional baseline offset correction amount =0.03、 =0.02.

[0159] Fusion Formula: Calculate the linear and nonlinear components for each of the two dimensions separately, sum them, and then superimpose the offset correction to obtain a two-dimensional initial importance coefficient vector. .

[0160] , j=2,3, that is, the initial two-dimensional importance coefficient vector of this block is [0.780,0.726] (normalized space).

[0161] Example of auxiliary dimension partitioning (the q-th block in plane 1,4): Given: the block dimension weight vector is Demand feature sub-block Completeness activation coefficient =0.28, Redundancy activation coefficient =0.28.

[0162] Fusion strategy: First calculate the linearly mapped two-dimensional vector, then use nonlinear coefficients for amplitude fine-tuning. Linearly mapped components: j=1,4; The amplitude fine-tuning yields the initial importance coefficient vector: , j=1,4, meaning the initial two-dimensional importance coefficient vector of this block is [0.183,0.134] (normalized space). The coefficient values ​​of the auxiliary blocks are significantly lower than those of the core blocks, which is consistent with the design intention of lower weights for auxiliary dimensions in high-precision tasks.

[0163] In this embodiment, the data source is sorted and divided into blocks according to a two-dimensional weighted score. The overall importance coefficient should follow a continuous statistical distribution. Block discretization may produce individual outliers. Consistency checks are used to identify and correct these outliers, rather than forcibly smoothing out all block differences.

[0164] For each block, the relative deviation between its initial coefficient vector and the subplane's global average coefficient vector is calculated. A reasonableness threshold is set at 1.5 times the standard deviation of the importance coefficients of all blocks within the subplane, based on a statistical outlier criterion (a simplified version of the 3σ principle), used to identify anomalous blocks that significantly deviate from the overall distribution. If the maximum relative deviation exceeds the reasonableness threshold, it is marked as an anomalous block and interpolation correction is performed. Where β is the weight of the block attribute, with the core block having a β value of 0.7 and the auxiliary block having a β value of 0.3.

[0165] The adjusted importance coefficients of all blocks are truncated within the range of [0.8, 1.2] to ensure that the weight adjustment range does not exceed ±20% and to avoid over-adjustment. The mapping is used to obtain the final block-level two-dimensional importance coefficient vector, which is then used for subsequent hierarchical aggregation.

[0166] After all the two-dimensional coefficient vectors of the blocks are generated, the final quality hierarchical dimension weight coefficients are calculated according to the aggregation method in Example 4.

[0167] In this embodiment, trend prediction uses a random forest regression model to predict the relative offset of quality scores for each dimension in the next round. After distribution sensitivity correction, this offset serves as a compensation signal for boundary pre-adjustment. The complete parameters are as follows: Input features (24 dimensions in total): The average quality across the four dimensions of the current round (4 dimensions); The current round's standard deviation of quality across four dimensions (4 dimensions); The most recent round of mean change (4 dimensions). The mean change in the last two rounds (4 dimensions); The most recent standard deviation change (4 dimensions); The most recent two rounds of standard deviation changes (4 dimensions).

[0168] Output labels (4 dimensions): The relative offset of each dimension's quality score in the next round (dimensionless), with positive values ​​indicating an increase in quality and negative values ​​indicating a decrease in quality. The labels are calculated using the actual quality differences between adjacent rounds in historical data.

[0169] Distribution sensitivity correction: Original prediction offset multiplied by sensitivity coefficient The final compensation signal is obtained, where, This is the kernel density estimate for the current threshold point. Based on the empirical quantile data of the local quality distribution of nodes in the j-th dimension, Gaussian kernel density estimation is used to obtain the probability density value at the current threshold point. The sensitivity coefficient is limited to the range of [0.5, 2.0] to prevent extreme values ​​from causing compensation to runaway.

[0170] Model hyperparameters: 100 decision trees, maximum depth 8, minimum number of leaf node samples 5, feature sampling ratio 0.8, sample sampling ratio 0.8, splitting criterion is mean squared error (MSE), 5-fold cross-validation is selected.

[0171] Training and update rules: When a node has ≥30 local data sources and ≥50 rounds of historical data, an independent local model is trained; if insufficient, a global model is used with node bias corrections; the model is updated incrementally every 100 rounds.

[0172] Cold start rule: Trend prediction is turned off for the first 3 evaluation cycles, and the compensation signal is set to 0 by default.

[0173] In this embodiment, a two-layer fully connected perceptron is used to implement nonlinear correlation operations, and the complete parameters are as follows: Network structure: 8 neurons in the input layer (4-dimensional demand features + 4-dimensional weight vector concatenation), 16 neurons in the hidden layer, and 4 neurons in the output layer; the activation function of the hidden layer is ReLU, and the output layer has no activation (linear output).

[0174] Training samples: 50 sets of labeled samples, which are high-quality boundary parameter sets manually selected during historical operation. The labeling standard is comprehensive quality utility = quality compliance × 0.6 + corpus utilization rate × 0.4. The boundary parameters with the top 20% utility are selected as positive samples.

[0175] Training configuration: loss function is mean squared error, optimizer is Adam, learning rate is 0.001, training epochs are 200, parameters are fixed after training and are not updated during online operation.

[0176] In this embodiment, the node data source change rules are as follows: when the change ratio of the node's bound data source exceeds 30%, the local quality distribution and trend prediction model is reset, and a three-round local cold start is initiated; when the change ratio is less than 30%, historical data of overlapping data sources are retained, and new data sources are initialized with global distribution and updated iteratively step by step.

[0177] System initialization rules: Before the first round of data collection, the weight matrix and the benchmark threshold for each level are calibrated offline based on a general public corpus dataset; the initial quality distribution adopts a global general empirical distribution; during the cold start phase, fixed global weights and benchmark thresholds are used, only feedback correction is performed, and trend prediction and hierarchical dynamic weights are not enabled; after running for 3 rounds, the system is gradually switched to full-function mode.

[0178] This invention provides a high-quality corpus data screening platform based on big data, which deploys and executes any of the high-quality corpus data screening methods based on big data, and realizes distributed node dynamic collection and hierarchical high-quality corpus output.

[0179] This invention provides a high-quality corpus data filtering system based on big data, comprising a scheduling service, a cluster of data collection nodes, and a storage module. The scheduling service runs and executes any of the high-quality corpus data filtering methods based on big data described above.

[0180] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for filtering high-quality corpus data based on big data, characterized in that, include: S1: Perform statistical calculations on the feature distribution of the full corpus data sources of all distributed corpus collection nodes to generate a global quality distribution. At the same time, generate a local quality distribution based on the subset of data sources bound to each collection node. Construct a global unified weight matrix covering four preset dimensions, wherein the preset dimensions include field completeness dimension, semantic self-consistency dimension, source credibility dimension, and redundancy noise control dimension. S2: Traverse all preset corpus quality screening thresholds. Under each corpus quality screening threshold, perform a dual fusion of linear interaction and non-linear correlation between the downstream large language model training requirements and the weight matrix to generate the dimensional weight coefficients of the quality level. Combine the threshold vector of the level to generate the initial corpus screening judgment boundary of the corresponding level. S3: Analyze each initial corpus in the initial corpus set of each distributed corpus collection node according to the four preset dimensions to obtain the actual quality profile of the corresponding corpus, and compare it with the expected quality profile to identify the deviation dimension, deviation direction, deviation degree and deviation change trend, and generate the corpus quality deviation pattern of the corresponding node. The expected quality profile includes the quality target level and allowable quality fluctuation range of each of the four preset dimensions. S4: Match the corpus quality deviation pattern with the preset corpus screening boundary control experience rules, and after correction by the distribution sensitivity coefficient, output the independent screening boundary feedback correction signal for each dimension of the corresponding node. At the same time, based on the mapping relationship between the historical change pattern of the local quality distribution of the node and the boundary prediction compensation amount, perform trend analysis on the local quality distribution of each node by combining the distribution sensitivity coefficient, and output the independent screening boundary trend prediction compensation signal for each dimension of each node. S5: Independently merge the screening boundary feedback correction signal and the screening boundary trend prediction compensation signal for each dimension, and perform adaptive processing on the initial corpus screening decision boundary for each level to obtain the optimal corpus screening decision vector for the corresponding node in each level. S6: Based on the optimal corpus selection decision vector for each level of the corresponding node, generate a multi-dimensional corpus collection control instruction set and send it to the corresponding node. Under the node load constraint, dynamically adjust the corpus collection frequency, trigger heterogeneous corpus cleaning operation and block low-quality data sources. Combine the dimension weight coefficients of the quality level to perform hierarchical selection and output the selected high-quality target corpus subset.

2. The method for filtering high-quality corpus data based on big data according to claim 1, characterized in that, Construct a weight matrix covering four preset dimensions, including: Based on the difference between the field missing rate of the corpus data source and the preset field completeness benchmark threshold, the preset completeness benchmark weight value is scaled inversely, and the field completeness dimension weight value is obtained after value range truncation constraint. Extract the semantic contradiction degree and logical discontinuity ratio parameters from the corpus data source, input the preset semantic quality segmentation decay function to obtain the first initial weight value of the semantic self-consistency dimension, and modify the first initial weight value in combination with the domain matching degree of the corpus data source to obtain the weight value of the semantic self-consistency dimension. The credibility index is obtained by weighting the main body qualification level, historical compliance records and information traceability level of the corpus data source. The second initial weight value is obtained by range normalization mapping. The second weighting correction is performed by combining the information release link length of the corpus data source to obtain the source credibility dimension weight value. The weighted sum of the proportion of duplicate data and the proportion of invalid noise in the statistical corpus data source is used to calculate the comprehensive redundancy. After normalizing the comprehensive redundancy, the third initial weight value is obtained by subtracting the normalization result from 1. The weight value of the redundancy noise control dimension is then obtained. The weight values ​​for field completeness, semantic consistency, source credibility, and redundancy noise control are mapped to the corresponding dimension channels of the weight matrix to construct a globally unified weight matrix.

3. The method for filtering high-quality corpus data based on big data according to claim 1, characterized in that, Adaptive processing is performed on the initial corpus selection decision boundaries for each level to obtain the optimal corpus selection decision vector for the corresponding node at each level, including: S51: Configure a first-level allowable amplitude coefficient and a second-level allowable amplitude coefficient for each preset dimension. Multiply the boundary adjustment benchmark value of each dimension with the corresponding first-level allowable amplitude coefficient to generate a first-level boundary migration allowable interval. Multiply the preset dimension quality deviation benchmark value with the corresponding second-level allowable amplitude coefficient to generate a second-level boundary migration allowable interval. The boundary adjustment benchmark value of each dimension is the boundary value of the corresponding dimension in the initial corpus screening and judgment boundary of the current level. The dimension quality deviation benchmark value is the average absolute deviation value of the actual quality profile and the expected quality profile of the corresponding node in each dimension within the most recent K evaluation periods. S52: Configure the feedback signal weight and prediction signal weight independently for each dimension, perform weighted fusion of the screening boundary feedback correction signal after the distribution sensitivity coefficient correction and the screening boundary trend prediction compensation signal for the corresponding dimension to generate the boundary migration correction amount for that dimension, and concatenate all dimensions to form a first-level boundary migration correction vector. The first-level boundary migration correction vector is superimposed onto the initial corpus screening and judgment boundary of the current level using a relative percentage multiplication method to obtain the first-level migration boundary parameter set. Each dimension parameter in the first-level migration boundary parameter set is compared with the first-level boundary migration allowable interval of the corresponding dimension. If all dimension parameters fall within the corresponding first-level boundary migration allowable interval, the first-level migration boundary parameter set is marked as a candidate migration boundary set and retained. Otherwise, the weight ratio of the two signals of the out-of-bounds dimension is adjusted independently. If it is still impossible to make all dimension parameters fall within the allowable interval after the adjustment of the preset maximum number of iterations, the first-level migration boundary parameter set corresponding to the weight ratio that minimizes the total deviation of the out-of-bounds dimension parameters is taken as the candidate migration boundary set. S53: Based on the empirical distribution of node local quality distribution, Monte Carlo sampling is performed to generate simulated corpus samples. A preset corpus quality fluctuation perturbation factor is introduced into the four-dimensional quality score of the simulated corpus to simulate the random fluctuation of corpus quality in real scenarios. Then, the candidate migration boundary set is used as the screening threshold to perform simulated screening, simulate the full corpus screening process, and output the simulated value of corpus quality deviation and the simulated value of screening pass rate corresponding to the secondary verification boundary set to obtain the steady-state effective boundary set. S54: Map each dimension parameter of the steady-state effective boundary set into an executable corpus screening judgment threshold, preset dimension weight coefficient and cleaning trigger condition parameter, and concatenate them to generate the optimal corpus screening decision vector for the current level of the corresponding node.

4. The method for filtering high-quality corpus data based on big data according to claim 1, characterized in that, Generate the initial corpus filtering and judgment boundaries for the corresponding grade, including: S21: Extract the training demand intensity features and training demand accuracy features from the training requirements of downstream large language models; Based on the first preset mapping rule, under the current corpus quality screening threshold, calculate the quota of nonlinear correlation fusion component and the quota of linear interaction fusion component; S22: For the four preset dimensions covered by the nonlinear correlation fusion component, calculate the preset dimension activation coefficient of each dimension according to the second preset mapping rule and the dimension importance level. S23: Decompose the weight matrix into 6 groups of two-dimensional mass weight sub-planes by combining them pairwise according to the dimensions; For each two-dimensional quality weight subplane, all data sources are first sorted in ascending order by weighted comprehensive score according to the preset dimension weight coefficients of the two dimensions corresponding to the subplane. Then, equidistant gridding is performed. The grid block size is dynamically determined based on the row and column dimensions of the subplane. The greatest common divisor or the closest integer divisor of the row and column dimensions is taken as the block scale parameter to form several quality weight block units with consistent size. For each block unit, calculate the mean weight of each of the two dimensions within the unit to obtain the block dimension weight vector; The downstream large language model training requirement feature vector is split according to the dimension level. Based on the dimension combination mapped by each quality weight block unit, the feature values ​​of the corresponding dimension are extracted from the requirement feature vector, normalized and quantized to form requirement feature sub-blocks, so that each group of requirement feature sub-blocks establishes a spatial mapping relationship with all quality weight block units in the corresponding two-dimensional quality weight sub-plane. S24: Based on the spatial mapping relationship, using the allocated linear interaction fusion component quota and nonlinear correlation fusion component quota as block fusion coefficients, calculate the fusion result of the corresponding quality weight block unit and the demand feature sub-block block block by block to generate a block-level two-dimensional importance coefficient vector; aggregate all block-level two-dimensional importance coefficient vectors by dimension to obtain the dimension weight coefficients of the quality stratification level; multiply the dimension weight coefficients of the quality stratification level with the tier benchmark threshold vector by dimension to generate the initial screening threshold corresponding to each quality stratification level, and concatenate them to obtain the initial corpus screening judgment boundary corresponding to the current corpus quality screening threshold tier.

5. The method for filtering high-quality corpus data based on big data according to claim 1, characterized in that, Output a filtered subset of high-quality target corpus, including: S61: Sort all idle distributed corpus collection nodes according to the preset node polling scheduling sequence, and take the idle corpus collection node at the top of the sequence as the current task execution node. Based on the corpus quality screening threshold required by the corpus collection task to be performed, the matching threshold decision vector is retrieved from the optimal corpus screening decision vector of each threshold of the corresponding node, and the redundancy noise control dimension threshold is corrected by combining the deduplication processing capability coefficient of the current node to obtain the actual redundancy threshold. Corpus collection tasks whose corpus collection coverage and corpus data type attributes both fall within the job adaptation domain of the current task execution node are included in the candidate collection task subset corresponding to the current node. S62: Verify one by one whether the data source range of each data collection task to be executed in the candidate data collection task subset overlaps with the current data source range of the other data collection nodes that are in the data collection work state. If the proportion of data sources in the overlapping areas of the collection range exceeds the preset collection range conflict threshold, the corresponding corpus collection task to be executed will be removed from the candidate collection task subset. S63: If the subset of candidate collection tasks after secondary filtering is not empty, first perform a corpus task type matching judgment on each candidate corpus collection task and divide them into two categories: standard structured corpus collection tasks and unstructured generalized corpus collection tasks. Retrieve the historical collection context data of the current task execution node, and calculate the corpus collection execution efficiency score for the two types of corpus collection tasks respectively; The candidate corpus collection task with the highest performance score is selected as the optimal collection task and assigned to the current task execution node. If the candidate collection task subset is empty, the current task execution node is moved to the end of the idle node polling sequence, and the number of idle rounds of the node is accumulated. If the number of consecutive idle rounds of the node exceeds the preset limit, the node is temporarily suspended to wait for new tasks to be added to the database, and the next idle corpus collection node is selected in order to repeat S61. S64: Remove the assigned optimal collection task from the global set of pending corpus collection tasks, change the working status of the current task execution node from idle to collecting data, and remove it from the idle node polling sequence; Take the next idle corpus collection node in the idle node polling sequence as the new current task execution node, and repeat S61 to S63 until the global set of pending corpus collection tasks is empty or all nodes have been traversed and there are no tasks to be assigned. S65: For the corpus collection tasks assigned to each corpus collection node, match the corresponding corpus collection control parameter template according to the task type, and calculate the optimal collection frequency in combination with the real-time load constraints of the node with the goal of maximizing the output efficiency of qualified corpus per unit time. By combining the cleaning trigger condition parameters and low-quality blocking rules in the optimal corpus selection decision vector, multi-dimensional corpus acquisition control instructions are generated and sent to the corresponding nodes; Each node performs corpus collection, heterogeneous corpus cleaning, and low-quality data source blocking operations according to instructions. It also performs hierarchical filtering and comprehensive sorting based on the dimensional weight coefficients of the quality stratification level, and outputs a unified subset of high-quality target corpus.

6. The method for filtering high-quality corpus data based on big data according to claim 5, characterized in that, The performance score of the corpus collection was calculated using a normalized weighted method.

7. The method for filtering high-quality corpus data based on big data according to claim 4, characterized in that, Generate a block-level two-dimensional importance coefficient vector, including: S241: Based on the corpus preset dimension attribute level mapped by each quality weight block unit, all quality weight block units are divided into two categories: core preset dimension blocks and auxiliary preset dimension blocks. Among them, quality weight block units containing semantic self-consistency dimension or source credibility dimension are marked as core preset dimension blocks, and quality weight block units containing only field completeness dimension and redundant noise control dimension are marked as auxiliary preset dimension blocks. S242: Divide the core preset dimensions into blocks, and use the linear interaction fusion component quota as the coefficient to perform linear weighting on the mean weight of the corresponding dimension in the block dimension weight vector to obtain the linear components of each dimension. Using the product of the nonlinear correlation fusion component quota and the corresponding dimension preset dimension activation coefficient as the coefficient, nonlinear weighting is applied to the feature quantization values ​​of each dimension in the demand feature sub-block to obtain the nonlinear components of each dimension. The linear and nonlinear components are summed according to their dimensions and then superimposed with a preset dimension baseline offset correction amount to obtain the initial two-dimensional importance coefficient vector of the block. The feature quantization value is the standardized value of each dimension feature value in the required feature sub-block after being normalized to the [0,1] interval by min-max. The preset dimension baseline offset correction amount is a constant correction term that has been normalized and scaled to the same dimension as the weighted summation result. For the auxiliary preset dimension blocks, the linear interactive fusion component quota is used as the dominant linear mapping coefficient. The average weight of each dimension in the block dimension weight vector is multiplied element by element with the feature quantization value of the corresponding dimension in the demand feature sub-block to obtain a linear mapping two-dimensional vector. The product of the nonlinear correlation fusion component quota and the corresponding dimension's preset dimension activation coefficient is used as the amplitude fine-tuning coefficient. Multiplication amplitude correction is performed on the linear mapping two-dimensional vector according to the dimension to obtain the block's initial two-dimensional importance coefficient vector. S243: For each block, the initial two-dimensional importance coefficient vector is selected, and its relative deviation from the global average importance coefficient in the corresponding two-dimensional quality weight subplane is calculated. If the relative deviation exceeds the reasonableness threshold, it is marked as an abnormal block and interpolation correction is performed using the subplane global mean combined with the block attribute weights; If the relative deviation does not exceed the threshold, the initial two-dimensional importance coefficient vector of the block selection is directly retained; S244: Block-level coefficient mapping output: After processing the value range truncation constraint, the block-level two-dimensional importance coefficient vector corresponding to each block is obtained by taking the block-corrected importance coefficient vector corresponding to all quality weight block units.

8. A high-quality corpus data filtering platform based on big data, characterized in that, Deploy and implement the high-quality corpus data screening method based on big data as described in any one of claims 1-7 to achieve distributed node dynamic collection and hierarchical high-quality corpus output.

9. A high-quality corpus data filtering system based on big data, characterized in that, It includes a scheduling service, a cluster of data collection nodes, and a storage module. The scheduling service runs and executes the high-quality corpus data screening method based on big data as described in any one of claims 1 to 7.

Images