AI corpus data automatic screening and obtaining method based on big data analysis

Abstract

The invention relates to the technical field of data processing, in particular to an AI corpus data automatic screening and obtaining method based on big data analysis, which comprises the following steps of: monitoring the updating frequency, the content evolution path and the abnormal fluctuation characteristic of each data source in real time, and generating a dynamic credibility parameter; constructing a space-time weighted screening model based on the dynamic credibility parameters, synchronously fusing real-time training demand features, and generating a screening decision vector; generating a multi-dimensional acquisition instruction set according to the dynamic credibility parameters and the screening decision vectors, and controlling the distributed acquisition nodes to execute the following steps: dynamically adjusting the acquisition frequency, triggering heterogeneous data cleaning, and blocking low-quality data streams; and obtaining the screened corpus data after executing the above steps. According to the method, quantitative evaluation of data source reliability, content consistency and abnormal diffusion risk is realized, low-quality, shoddy or abnormal propagation data are prevented from entering a training set, and the overall quality stability of the data is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for automatically screening and acquiring AI corpus data based on big data analysis. Background Art

[0002] With the rapid development of artificial intelligence technology, the dependence on high-quality corpus data is increasing. The core lies in the quality, timeliness and multimodal adaptability of the data. Therefore, how to efficiently acquire, screen and optimize large-scale corpus data has become the key to improving AI training results. However, the current AI corpus data acquisition and screening methods still have many problems and cannot meet the needs of large-scale AI training tasks.

[0003] First, most existing data screening methods rely on fixed rules and lack dynamic adaptation capabilities. Traditional data screening is usually based on preset quality thresholds or simple keyword matching rules, and cannot be adaptively adjusted according to changes in data sources, fluctuations in their own quality, and changes in AI training needs, resulting in low-quality, redundant or irrelevant data being incorrectly adopted or high-quality data being mistakenly eliminated. In addition, the data source credibility assessment method is relatively single and fails to fully consider the spatiotemporal dynamic characteristics of the data, making it difficult to maintain data quality stability in the long term. Summary of the invention

[0004] The present invention provides a method for automatically screening and acquiring AI corpus data based on big data analysis.

[0005] The method for automatically screening and acquiring AI corpus data based on big data analysis includes the following steps: S1. Dynamic evaluation of source credibility: real-time monitoring of the update frequency, content evolution path and abnormal fluctuation characteristics of each data source, generating dynamic credibility parameters including credibility decay rate, semantic deviation and abnormal propagation coefficient; S2. Multi-dimensional screening modeling: Based on dynamic credibility parameters, a spatiotemporal weighted screening model is constructed, which simultaneously integrates the real-time training demand characteristics to generate a screening decision vector including quality threshold, modal weight, and time sensitivity; S3, Adaptive collection execution: Generate a multi-dimensional collection instruction set based on dynamic credibility parameters and screening decision vectors, and control the execution of distributed collection nodes: Dynamically adjust the acquisition frequency; Trigger heterogeneous data cleaning; Block low-quality data streams; After executing the above, the filtered corpus data is obtained.

[0006] Optionally, the S1 specifically includes: S11, update frequency monitoring: statistics the release interval time series of each data source through a sliding time window, calculate the update interval (frequency) fluctuation coefficient, and generate the credibility decay rate based on the interval fluctuation coefficient, where the fluctuation coefficient and the decay rate are exponentially positively correlated; S12, content evolution tracking: using dynamic semantic baseline comparison technology, LSTM modeling is performed on the historical content of the data source to generate a semantic evolution baseline, and the cosine similarity deviation between the current content and the semantic evolution baseline is calculated in real time to generate a semantic deviation degree; S13, abnormal fluctuation detection: construct a data source association graph, analyze the abnormal propagation path across data sources in real time through the graph neural network, count the abnormal input edge weight and output edge diffusion speed of the target data source, and generate the abnormal propagation coefficient; S14, parameter fusion calculation: normalize the credibility decay rate, semantic deviation, and anomaly propagation coefficient to construct a three-dimensional feature vector of dynamic credibility parameters.

[0007] The S2 specifically includes: S21, spatiotemporal weight allocation: construct a three-dimensional weight matrix based on dynamic credibility parameters, and allocate weights for the three dimensions of time weight, semantic weight, and topological weight respectively; S22, training requirements integration: Double fusion of real-time training demand features and spatiotemporal weight matrices: Linear feature interaction: multiply the model training requirement features by the weights of each dimension diagonally; Nonlinear association capture: extract the deep association between weight matrix and demand features through activation function; Fusion weight generation: weighted superposition of linear transformation results and nonlinear interaction results to form dynamic weights that adapt to model requirements; S23, decision vector generation: converting fusion weights into actionable screening decision parameters.

[0008] Optionally, the time weight is reversely adjusted according to the credibility decay rate, and a data source with a higher decay rate is assigned a lower time weight; The semantic weight uses an exponential decay function to process the semantic deviation, and a data source with a large deviation degree has its semantic weight reduced; The topological weight is reversely normalized based on the anomaly propagation coefficient, and the topological weight of a data source with strong anomaly propagation capability is reduced.

[0009] Optionally, the decision vector generation in S23 specifically includes: Dynamic setting of quality threshold: combining time and topological weights, the adaptive standard deviation method is used to generate dynamic quality thresholds that change with data quality distribution; Modal weight balance allocation: Based on semantic weight and modal requirements, the collection ratio of data sources is generated through normalization processing; Time sensitivity control: Use the Sigmoid function to convert the time parameter into data update frequency control instructions.

[0010] Optionally, the S3 specifically includes: S31, dynamically adjust the acquisition frequency, and dynamically generate data acquisition frequency instructions based on the time sensitivity parameters in the screening decision vector; S32, triggering heterogeneous data cleaning, jointly using the modality weight and semantic deviation parameter in the screening decision vector, executing a hierarchical cleaning strategy, allocating different cleaning resources according to the modality weight, and giving priority to ensuring the data quality of high-weight modalities; S33, blocking low-quality data streams, setting dynamic blocking rules based on dynamic quality thresholds: Combine the dynamic quality threshold with the safety redundancy factor to generate a dynamic blocking threshold; The abnormal propagation degree of the data flow is calculated in real time, and it is blocked immediately when the abnormal propagation degree exceeds the blocking threshold.

[0011] Optionally, the graded cleaning strategy adopts three-stage progressive cleaning: Basic cleaning: standardization format and redundancy removal; Deep cleaning: cross-modal alignment and contradiction resolution; Reconstructing Cleaning: Adversarial Example Detection and Semantic Reconstruction.

[0012] Optionally, the dynamically generating data acquisition frequency instruction in S31 specifically includes: According to the size of the time sensitivity value, the basic collection frequency is adjusted proportionally; Fine-tune the credibility attenuation rate and implement acquisition frequency reduction compensation for data sources with high attenuation rates; When the time sensitivity reaches a critical value, the pulse enhancement mode of burst data acquisition is started.

[0013] Optionally, S3 further includes distributed node collaboration, dynamically selecting the optimal node type according to the maximum value of each screening decision vector component, specifically including: According to the quality assessment, modal weight and time sensitivity in the screening decision vector, the data processing nodes are classified and allocated. By comparing the values ​​of the screening decision vector, the dominant factors are selected and the processing node types are allocated; if the quality assessment factor is dominant, it is allocated to the quality priority node to ensure the rigor of data screening; if the modal weight is dominant, it is allocated to the modal priority node so that the data meets the modal requirements of different AI tasks; if the time sensitivity is dominant, it is allocated to the time priority node to increase the data update speed; if the three values ​​are similar, the comprehensive balance node is used for processing to ensure that multiple needs are taken into account. In addition, in the case of high risk of data anomaly propagation, the data will be directed to the anomaly filtering node to perform additional detection or blocking measures.

[0014] Beneficial effects of the present invention: 1. The present invention constructs a dynamic source credibility evaluation mechanism, comprehensively monitors the update frequency, content evolution trajectory and abnormal propagation characteristics of the data source, generates dynamic credibility parameters including credibility decay rate, semantic deviation and abnormal propagation coefficient, and uses the parameters to construct a spatiotemporal weighted screening model, and combines the characteristics of AI training requirements to generate accurate screening decision vectors. This mechanism realizes the quantitative evaluation of data source reliability, content consistency and abnormal propagation risk, avoids low-quality, counterfeit or abnormally propagated data from entering the training set, significantly improves the overall quality stability of the data, reduces the impact of data noise on the AI ​​training process, and improves the generalization ability and stability of the model.

[0015] 2. By constructing a training demand vector and integrating it with the data screening weight matrix, the data collection, cleaning and blocking strategies are dynamically optimized. The screening decision vector is used to guide the frequency regulation, modal weight allocation and abnormal data blocking of the data collection nodes, so that the data flow can be adaptively adjusted according to the changes in the AI ​​training tasks, thereby realizing the intelligent evolution of the data supply chain. This method ensures that the optimal data can be continuously obtained, avoids redundant and inefficient data occupying computing resources, and at the same time improves the pertinence and effectiveness of AI training data and reduces data mismatch problems during training.

[0016] 3. Adopting the data node collaborative optimization strategy driven by intelligent decision vectors, automatically matching the optimal data processing node according to the real-time calculation results of data quality, modal requirements and time sensitivity, and realizing accurate data allocation and processing through intelligent classification of node types, including quality priority, modal priority, time priority, comprehensive balance and abnormal filtering nodes, different types of data can be processed according to the optimal strategy, optimizing the allocation mode of data flow in computing resources, and reducing data redundancy and computing waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0020] like Figure 1 As shown, the method for automatically screening and acquiring AI corpus data based on big data analysis includes the following steps: S1. Dynamic evaluation of source credibility: real-time monitoring of the update frequency, content evolution path and abnormal fluctuation characteristics of each data source, generating dynamic credibility parameters including credibility decay rate, semantic deviation and abnormal propagation coefficient; S2. Multi-dimensional screening modeling: Based on dynamic credibility parameters, a spatiotemporal weighted screening model is constructed, which simultaneously integrates the real-time training demand characteristics to generate a screening decision vector including quality threshold, modal weight, and time sensitivity; S3, Adaptive collection execution: Generate a multi-dimensional collection instruction set based on dynamic credibility parameters and screening decision vectors, and control the execution of distributed collection nodes: Dynamically adjust the acquisition frequency; Trigger heterogeneous data cleaning; Block low-quality data streams; After executing the above, the filtered corpus data is obtained.

[0021] S1 specifically includes: S11, update frequency monitoring: statistics the release interval time series of each data source through a sliding time window, calculate the update interval fluctuation coefficient, and generate the credibility decay rate based on the interval fluctuation coefficient, where the interval fluctuation coefficient is exponentially positively correlated with the decay rate; S111, set the time window of the data source , calculate the release interval time series: , Indicates the first The data release time, that is, the latest release time point, Indicates the first Data release time, Indicates A point in time, ; S112, calculate interval fluctuation coefficient : ,in, represents the standard deviation of the time interval, Represents the mean of the time interval.

[0022] S113, calculate the credibility decay rate : ,in, is the attenuation sensitivity adjustment factor (setting range ).

[0023] S12, content evolution tracking: using dynamic semantic baseline comparison technology, LSTM modeling is performed on the historical content of the data source to generate a semantic evolution baseline, and the cosine similarity deviation between the current content and the semantic evolution baseline is calculated in real time to generate a semantic deviation degree; S121, assuming that the historical content sequence of the data source is , Indicates Data content, use LSTM to calculate the hidden state: ,in, is the text embedding function, represents the data content at the 𝑡th time point, Indicates time The hidden state of LSTM is Indicates time The hidden state of LSTM is Indicates data content Text embedding vector of ; S122, Calculate dynamic semantic baseline : ; S123, calculate the semantic deviation between the current content and the baseline : ,in, is the embedding vector of the current content, and the cosine similarity is used to measure the semantic change.

[0024] S13, abnormal fluctuation detection: construct a data source association graph, analyze the abnormal propagation path across data sources in real time through the graph neural network, count the abnormal input edge weight and output edge diffusion speed of the target data source, and generate the abnormal propagation coefficient; S131, set data source association map ,in, is a collection of data sources, is the edge set of data reference relations, is the edge weight, indicating the number of historical anomaly propagation times; S132, calculate the anomaly propagation coefficient : ,in, denote the input edge set and the output edge set respectively, is the time decay factor, Represent the input edge weight and output edge weight respectively: ; is the normalization coefficient: , is the current time, is the time when the abnormality occurred, Represents the input edge entering the target data source node, Represents the output edge from the target data source node; S14, parameter fusion calculation: perform spatiotemporal normalization on the credibility decay rate, semantic deviation, and anomaly propagation coefficient to generate a three-dimensional feature vector of dynamic credibility parameters .

[0025] S2 specifically includes: S21, spatiotemporal weight allocation: construct a three-dimensional weight matrix based on dynamic credibility parameters: ; in, is the time weight, is the semantic weight, is the topological weight, is the domain adaptation weight (the initial value is set by expert experience), is the smoothing factor, , to prevent the denominator from being zero, is the semantic shift sensitivity adjustment parameter, , This is the largest abnormal propagation coefficient in history.

[0026] S22, training requirement fusion: real-time acquisition of the training requirement feature vector of the target AI model : ; in, is the gradient modulus of the loss function, , is the modality attention weight norm, , is the modality attention matrix, which represents the distribution of attention weights of different modalities in multimodal input. is the rate of change of the accuracy of the validation set over time, , It represents the rate of change of the validation set accuracy over time, measuring the performance improvement or decline trend over time. is the increment of training time, used to calculate the performance improvement or decline trend; Execution Requirements - Weight Fusion: ,in, is a diagonal matrix, ensuring that each training requirement component acts on the corresponding weight. is the fusion strength coefficient, which controls the degree of fusion. , is the fused screening weight matrix, is the activation function, is the transpose of the training requirement feature vector; S23, decision vector generation: Generate a screening decision vector through normalization mapping: ; in, is the quality threshold, is the modal weight, It is time sensitivity; S231, quality threshold calculation: ; Dynamically set quality thresholds: ,in, are the historical quality mean and standard deviation, ,in is the inverse cumulative distribution function of the standard normal distribution, is the quality decision factor, which is a normalized weight factor used to measure the relative quality of the data. is an actual numerical threshold used to decide whether data passes the screening. Influence; S232, modal weight allocation: ,in, For the The intensity of real-time demand for each mode; S233, Time Sensitivity Control ,in, is the slope factor, controlling the rate of change, , is the offset, , adjust the benchmark point of time sensitivity; They are the temporal, topological and semantic weights after fusion respectively.

[0027] S3 specifically includes: S31, Dynamic acquisition frequency control: based on the time sensitivity in the screening decision vector Generate acquisition frequency instructions: ,in, is the adjusted sampling frequency, is the reference acquisition frequency, is the time sensitivity, is the credibility decay rate, which is used as a correction factor here. is the adjustment factor, ,control The impact on the acquisition frequency regulation, is the standard Sigmoid function: .

[0028] S32, Intelligent Data Cleaning Trigger: Joint Use of Modal Weights in Decision Vectors Semantic deviation Calculate the data cleaning level: ,in, is the data cleaning level, is the modal weight, which comes from the decision vector of S2, is the semantic deviation, is the semantic offset benchmark value, used for normalization processing, To round down the function, make sure the cleaning level is an integer.

[0029] S33, real-time blocking of data flow: based on quality threshold in decision vector Calculate the data flow blocking conditions: ; in, Data blocking flag, 0 means no blocking, 1 means blocking. is the anomaly propagation coefficient, is the dynamic quality threshold, is the safety redundancy factor, , used to adjust the threshold of the blocking condition.

[0030] S34, node coordination: mapping the three elements of the decision vector to node allocation: ,in, Indicates the data processing node type, Returns the index corresponding to the maximum value. , , They represent data quality, modality weight and time sensitivity respectively.

[0031] NodeType's decision logic: if Max: Prioritizes quality-first nodes, ensuring that the filtered data meets high quality standards.

[0032] if Maximum: Assigned to the modality priority node to match the modality requirements of the AI ​​task.

[0033] if Maximum: Use time-priority nodes to ensure real-time data.

[0034] If the three values ​​are similar: assign them to the comprehensive balance node to optimize data selection among multiple factors.

[0035] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0036] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for automatically screening and acquiring AI corpus data based on big data analysis, characterized in that: The following steps are involved: S1. Dynamic evaluation of source credibility: real-time monitoring of the update frequency, content evolution path and abnormal fluctuation characteristics of each data source, generating dynamic credibility parameters including credibility decay rate, semantic deviation and abnormal propagation coefficient; S2. Multi-dimensional screening modeling: Based on dynamic credibility parameters, a spatiotemporal weighted screening model is constructed, which simultaneously integrates the real-time training demand characteristics to generate a screening decision vector including quality threshold, modal weight, and time sensitivity; S3, Adaptive collection execution: Generate a multi-dimensional collection instruction set based on dynamic credibility parameters and screening decision vectors, and control the execution of distributed collection nodes: Dynamically adjust the acquisition frequency; Trigger heterogeneous data cleaning; Block low-quality data streams; After executing the above, the filtered corpus data is obtained.

2. The method for automatically screening and acquiring AI corpus data based on big data analysis according to claim 1 is characterized in that: The S1 specifically includes: S11, update frequency monitoring: statistics of the release interval time series of each data source through a sliding time window, calculate the update interval fluctuation coefficient, and generate the credibility decay rate based on the interval fluctuation coefficient. The interval fluctuation coefficient is exponentially positively correlated with the decay rate; S12, content evolution tracking: using dynamic semantic baseline comparison technology, LSTM modeling is performed on the historical content of the data source to generate a semantic evolution baseline, and the cosine similarity deviation between the current content and the semantic evolution baseline is calculated in real time to generate a semantic deviation degree; S13, abnormal fluctuation detection: construct a data source association graph, analyze the abnormal propagation path across data sources in real time through the graph neural network, count the abnormal input edge weight and output edge diffusion speed of the target data source, and generate the abnormal propagation coefficient; S14, parameter fusion calculation: normalize the credibility decay rate, semantic deviation, and anomaly propagation coefficient to construct a three-dimensional feature vector of dynamic credibility parameters.

3. The method for automatically screening and acquiring AI corpus data based on big data analysis according to claim 1 is characterized in that: The S2 specifically includes: S21, spatiotemporal weight allocation: construct a three-dimensional weight matrix based on dynamic credibility parameters, and allocate weights for the three dimensions of time weight, semantic weight, and topological weight respectively; S22, training requirements integration: Double fusion of real-time training demand features and spatiotemporal weight matrices: Linear feature interaction: multiply the model training requirement features by the weights of each dimension diagonally; Nonlinear association capture: extract the deep association between weight matrix and demand features through activation function; Fusion weight generation: weighted superposition of linear transformation results and nonlinear interaction results to form dynamic weights that adapt to model requirements; S23, decision vector generation: converting fusion weights into actionable screening decision parameters.

4. The method for automatically screening and acquiring AI corpus data based on big data analysis according to claim 3 is characterized in that: The time weight is reversely adjusted according to the credibility decay rate, and the data source with a higher decay rate is assigned a lower time weight; The semantic weight uses an exponential decay function to process the semantic deviation, and a data source with a large deviation degree has its semantic weight reduced; The topological weight is reversely normalized based on the anomaly propagation coefficient, and the topological weight of a data source with a strong anomaly propagation capability is reduced.

5. The method for automatically screening and acquiring AI corpus data based on big data analysis according to claim 3 is characterized in that: The decision vector generation in S23 specifically includes: Dynamic setting of quality threshold: combining time and topological weights, the adaptive standard deviation method is used to generate dynamic quality thresholds that change with data quality distribution; Modal weight balance allocation: Based on semantic weight and modal requirements, the collection ratio of data sources is generated through normalization processing; Time sensitivity control: Use the Sigmoid function to convert the time parameter into data update frequency control instructions.

6. The method for automatic screening and acquisition of AI corpus data based on big data analysis according to claim 1 is characterized in that: The S3 specifically includes: S31, dynamically adjust the acquisition frequency, and dynamically generate data acquisition frequency instructions based on the time sensitivity parameters in the screening decision vector; S32, triggering heterogeneous data cleaning, jointly using the modality weight and semantic deviation parameter in the screening decision vector, executing a hierarchical cleaning strategy, allocating different cleaning resources according to the modality weight, and giving priority to ensuring the data quality of high-weight modalities; S33, blocking low-quality data streams, setting dynamic blocking rules based on dynamic quality thresholds: Combine the dynamic quality threshold with the safety redundancy factor to generate a dynamic blocking threshold; The abnormal propagation degree of the data flow is calculated in real time, and it is blocked immediately when the abnormal propagation degree exceeds the blocking threshold.

7. The method for automatically screening and acquiring AI corpus data based on big data analysis according to claim 6 is characterized in that: The graded cleaning strategy adopts three-stage progressive cleaning: Basic cleaning: standardization of formats and removal of redundancy; Deep cleaning: cross-modal alignment and contradiction resolution; Reconstructing Cleaning: Adversarial Example Detection and Semantic Reconstruction.

8. The method for automatically screening and acquiring AI corpus data based on big data analysis according to claim 6 is characterized in that: The dynamic generation of data acquisition frequency instruction in S31 specifically includes: According to the size of the time sensitivity value, the basic collection frequency is adjusted proportionally; Fine-tune the credibility attenuation rate and implement acquisition frequency reduction compensation for data sources with high attenuation rates; When the time sensitivity reaches a critical value, the pulse enhancement mode of burst data acquisition is started.

9. The method for automatically screening and acquiring AI corpus data based on big data analysis according to claim 6 is characterized in that: The S3 also includes distributed node collaboration, dynamically selecting the optimal node type according to the maximum value of each screening decision vector component, specifically including: According to the quality evaluation, modal weight and time sensitivity in the screening decision vector, the data processing nodes are classified and allocated. By comparing the values ​​of the screening decision vector, the dominant factors are selected and the processing node types are allocated. If the quality evaluation factor is dominant, it is allocated to the quality priority node to ensure the rigor of data screening; if the modal weight is dominant, it is allocated to the modal priority node to make the data meet the modal requirements of different AI tasks; if the time sensitivity is dominant, it is allocated to the time priority node to increase the data update speed; if the three values ​​are similar, the comprehensive balance node is used for processing to ensure that multiple needs are taken into account.

Images