Big data screening method and system

Abstract

The invention discloses a big data screening method and system, and relates to the technical field of data processing, and the method comprises the following steps: S1, user demand analysis and confirmation: obtaining and analyzing user data demands, determining key features and conditions of data screening, formulating and confirming a screening standard, obtaining user feedback, and carrying out adjustment; s2, demand priority determination: carrying out comprehensive analysis by utilizing the optimal parameters, determining the relative importance of each parameter by applying an analytic hierarchy process, allocating weights, calculating weighted scores, and determining demand priorities according to score sorting; according to the method, the efficiency and accuracy of big data screening are remarkably improved by comprehensively analyzing the user requirements, formulating the screening standard and adopting measures such as algorithm optimization, feature engineering and parameter adjustment, the requirement priority is determined through the analytic hierarchy process, the screening strategy is dynamically adjusted, and it is ensured that the screening result is highly matched with the user requirements.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a big data screening method and system. Background Art

[0002] In the current data processing technology field, the rapid growth of big data has brought unprecedented opportunities and challenges. With the popularization of the Internet, the Internet of Things, and social networks, the amount of data has shown explosive growth, which provides rich resources for in-depth analysis and mining of data value. However, the huge volume, diverse types, and fast speed of big data also bring processing difficulties.

[0003] Traditional data screening methods usually rely on static rule bases or simple algorithms. These methods often have problems such as low efficiency, low accuracy, and difficulty in adapting to data diversity when dealing with large-scale, high-dimensional, and rapidly changing data sets. In addition, with the continuous accumulation of data, data quality control has also become a prominent issue. Noise, outliers, and inconsistencies in the data will seriously affect the results of data analysis.

[0004] In order to improve the efficiency and accuracy of data screening, researchers and engineers have developed a variety of data screening technologies, including keyword-based screening, content-based screening, pattern-based screening, etc. These technologies have improved the performance of screening to a certain extent, but there are still some limitations. For example, keyword-based screening may ignore contextual information and lead to misjudgment; content-based screening may require a complex feature extraction process with high computational cost; pattern-based screening may require a large amount of training data and is difficult to adapt to new data patterns.

[0005] In addition, existing data screening systems often lack flexibility and scalability, and are difficult to cope with the ever-changing business needs and data environment. Users often need to manually adjust the screening rules, which is not only time-consuming and labor-intensive, but also difficult to ensure the consistency and accuracy of the screening results. At the same time, as the demand for screening increases, how to reasonably allocate resources, optimize the screening process, and improve screening efficiency has also become an urgent problem to be solved.

[0006] In order to solve the above defects, a technical solution is now provided. Summary of the invention

[0007] The purpose of the present invention is to solve the problems of low efficiency, low accuracy, difficulty in adapting to data diversity, lack of flexibility and scalability in existing big data screening methods when processing large-scale, high-dimensional, and rapidly changing data sets, and to propose a big data screening method and system.

[0008] The purpose of the present invention can be achieved through the following technical solutions: A method for screening big data includes the following steps: S1: User demand analysis and confirmation: Obtain and analyze user data needs, determine the key features and conditions for data screening, formulate and confirm screening criteria, obtain user feedback and make adjustments; S2: Determine the priority of requirements: Use the optimal parameters for comprehensive analysis, apply the analytic hierarchy process to determine the relative importance of each parameter and assign weights, calculate the weighted scores, and determine the priority of requirements based on the score ranking; S3: Screening strategy development and testing: Develop a detailed data screening plan, perform prototype testing, select representative data samples, prepare the environment, design a screening prototype, collect test results, and evaluate the effectiveness of the screening method; S4: Optimization and selection of screening methods: Identify problems and deficiencies in testing, propose adjustment and optimization plans, including algorithm optimization, feature engineering and parameter adjustment optimization measures, evaluate screening methods through performance, efficiency, cost-effectiveness and user experience parameters, calculate comprehensive evaluation values, and select the best screening plan; S5: Screening execution and performance monitoring: Apply the optimal screening method to perform data screening and monitor the screening process, set up and train prediction models, evaluate and apply models for data analysis and prediction, monitor prediction model performance, solicit feedback, and regularly optimize screening and prediction strategies.

[0009] Furthermore, a big data screening system includes: The demand determination module is used to analyze the user's data screening needs and determine the priorities of different screening needs; The test module is used to conduct small-scale prototype tests to verify the effectiveness of the screening method and adjust the screening strategy based on the test results to determine the final optimal screening method; The screening and prediction module is used to screen big data using the optimal screening method and make predictions based on the screening results.

[0010] Furthermore, the specific operation steps of the demand determination module for analyzing the user's data screening demand and determining the priority of different screening demands are as follows: First, obtain the user's data demand expression and understand the user's specific needs, including the purpose of the data, the type of data to be screened, and the expected results; Analyze the collected demand information to determine the key features and conditions that users need to filter data, including the time range of the data, the value of specific fields, and the format of the data; Based on user needs, formulate screening criteria, which are specific numerical ranges, keywords, data attributes and business rules; Then confirm the screening criteria with users, obtain user feedback, and adjust the screening criteria based on the feedback to ensure that the screening criteria can meet the actual needs of users; When a user has multiple screening requirements, the priorities of the user's multiple screening requirements are analyzed. Specifically, the optimization parameters are collected for comprehensive analysis, and the relative importance of each optimization parameter is determined using the hierarchical analysis method multi-criteria decision-making tool. The corresponding weight factor is assigned according to the relative importance of the optimization parameter, and each optimization parameter is scored according to the weight factor determined for each optimization parameter, and finally the weighted score is calculated; Sort each user's screening requirements according to the calculated weighted score. The higher the priority of the screening requirements, the easier it is to allocate resources and manage time in the screening process. Then, based on the identified needs and priorities, develop a data screening plan, including screening methods, tools, schedule, and responsibility allocation.

[0011] Furthermore, the optimal parameters in the demand determination module include: Requirements Clarity: Assess the clarity and specificity of each requirement; Business value: Analyze the contribution of each requirement to business goals; Urgency: Understand the urgency of the demand and whether it needs to be handled immediately; Resource availability: assessing whether the resources required to implement the requirements are available; User influence: Analyze the impact of requirements on user satisfaction and user experience; Risk assessment: Analyze the risks and uncertainties that may arise from implementing each requirement; Dependencies: Identify dependencies between requirements; Return on investment: Evaluate the cost of implementing each requirement versus the expected benefits; Technical feasibility: assess the feasibility of technical implementation; Data availability: assess the availability of the required data; Historical data: View the past performance and user feedback of this requirement.

[0012] Furthermore, the specific operation steps of the test module for conducting a small-scale prototype test are as follows: First, clarify the purpose of the prototype test, determine the key aspects of the data screening method that need to be verified, and develop a test plan, including the selection of test data, the construction of the test environment, the specific steps of the test, and the expected test results; Then select representative data samples for testing, where the selected data samples cover different data types and characteristics to determine the comprehensiveness of the test results, prepare the required hardware and software environment, and ensure that the test is carried out in a stable and controllable environment; Design a preliminary screening prototype based on the screening requirements. The preliminary screening prototype contains the core functions of the screening method. Run the preliminary screening prototype, perform the screening operation using test data, and collect test results. Analyze test results, evaluate the effectiveness of screening methods, including accuracy, efficiency and stability indicators, and identify problems and deficiencies in the testing process, including low accuracy of data screening and long response times.

[0013] Furthermore, the specific operation steps of the test module to adjust the screening strategy according to the test results and determine the final optimal screening method are as follows: According to the test results and problem identification, the screening method is adjusted and optimized, and several adjustment and optimization combinations are provided for the adjustment and optimization parts to form several optimized screening methods; Then select an optimal screening method for several optimization screening methods. The specific steps are as follows: Data screening tests were conducted on different optimization screening methods, and evaluation parameters were collected through test results, including performance parameters, efficiency parameters, cost-effectiveness parameters and user experience parameters. The evaluation parameters were analyzed to obtain the performance evaluation value, effectiveness evaluation value, cost evaluation value and user value, and after normalization, they were entered into the following formula: To get the comprehensive evaluation value ZPZ, where NPZ, XPZ, CXZ and YTZ are the sex evaluation value, effect evaluation value, cost evaluation value and usage value respectively. The preset weight coefficients are respectively the sex evaluation value, the efficacy evaluation value, the composition evaluation value and the user value, and the obtained comprehensive evaluation value ZPZ is used as the evaluation standard of the test results of the screening method; Then, the comprehensive evaluation values ​​corresponding to different optimization screening methods are sorted according to size, and the optimization screening method with the largest comprehensive evaluation value is selected as the optimal screening method.

[0014] Furthermore, the adjustment and optimization part of the optimization screening method specifically includes the following process: Algorithm optimization: Improve algorithm efficiency, including upgrading from a simple keyword matching algorithm to a more complex natural language processing algorithm; Feature engineering: adding or adjusting features used for screening, including introducing more text analysis features or user behavior features; Parameter adjustment: Optimize the parameter settings in the screening algorithm, including adjusting the threshold of the classification algorithm and the number of centers of the clustering algorithm; Data preprocessing: Improve data cleaning and preprocessing steps, including removing more noisy data and standardizing data formats; Model training: using more or less training data or trying different machine learning models; Resource allocation: Adjust the use of computing resources, including parallel processing and adding computing nodes.

[0015] Furthermore, the specific steps of the test module to analyze the evaluation parameters to obtain the sex evaluation value, the effectiveness evaluation value, the achievement evaluation value and the usage value are as follows: The performance parameters include: Accuracy: the degree to which the screening results match the actual needs; Recall: the proportion of relevant data that the screening method can find; Precision: The selected data are displayed as relevant accuracy; F1 score: the harmonic mean of precision and recall; Robustness: The stability of the screening method when facing data of different types or qualities, and collect the robustness loss, that is, the difference between the worst-case performance and the average performance, to measure the robustness of the screening method; After normalizing the obtained accuracy, recall rate, precision, F1 score and robustness loss, the product of accuracy and precision plus F1 score is calculated as the base circle radius, and the product of robustness loss and correction factor is used as the height to establish a cylindrical model, and the mass point of the cylindrical model is selected as the center of the sphere. The spherical model is established with the sum of the recall rate and the base circle radius as the sphere diameter, and the volume of the heteromorphic body formed by the spherical body and the cylindrical model is calculated and recorded as the performance evaluation value, and this performance evaluation value is used as a reference standard for measuring the performance of the screening method; Efficiency parameters include: Processing time: the time required for the screening operation to complete; Resource consumption: the computing resources and storage resources required to perform the screening; The obtained processing time and resource effect are denoted as sr and zx respectively, and then normalized and entered into the following formula: To get the effectiveness evaluation value XPZ, where are weight factors for processing time and resource effects respectively, and the obtained effectiveness evaluation value is used as a standard to measure the efficiency of the screening method; Cost-effectiveness parameters include: Cost-effectiveness ratio: the ratio between the cost of a screening method and the value it brings; Maintenance costs: long-term maintenance and update costs of screening methods; After normalizing the obtained cost-effectiveness ratio and maintenance cost, the cost-effectiveness and maintenance cost are used as the two sides of the triangle, and the included angle is preset. The remaining side of the triangle is connected, and the area of ​​the triangle is calculated and recorded as the evaluation value. This evaluation value is used to measure the cost-effectiveness standard of the screening method. User experience parameters include: User satisfaction: the user's satisfaction with the screening results; Flexibility: the ability of the screening method to adapt to different screening needs; Real-time: whether the screening method can meet the needs of real-time data processing; Different user experience parameters are scored and weighted scores are calculated, recorded as user experience values, and the user experience values ​​are used as a standard to measure the user experience of this screening method.

[0016] Furthermore, the specific steps of the screening prediction module for prediction based on the screening results are as follows: Screen the data according to the optimal screening method. During the screening process, monitor the screened data, set up a prediction model based on the screening results, and determine the corresponding prediction algorithm, including regression, classification or time series prediction; Use the test and formally screened data to train the prediction model, and evaluate the performance of the prediction model through a test set or cross-validation; Apply predictive models to perform predictive analysis on the screened data, generate prediction results, interpret the generated prediction results, provide insights and suggestions, and apply the prediction results to the decision-making process; Then monitor the performance of the screening prediction model, track prediction accuracy and response time, collect feedback from users and business teams, adjust screening and prediction strategies based on the feedback, set up a review and optimization cycle, regularly review and optimize the prediction model according to the review and optimization cycle, and update the prediction model when new data and business changes appear.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention, by comprehensively analyzing user needs and formulating screening criteria, adopts measures such as algorithm optimization, feature engineering and parameter adjustment, significantly improves the efficiency and accuracy of big data screening, determines the priority of needs through hierarchical analysis, and dynamically adjusts the screening strategy to ensure that the screening results are highly matched with user needs; The present invention can adapt to the ever-changing business needs and data environment. Through modular design and multi-standard decision-making tools, it allows users to quickly adjust screening parameters and strategies according to different application scenarios, thereby achieving flexible processing and analysis of big data. The present invention selects the optimal screening scheme by evaluating the cost-effectiveness ratio, maintenance cost and user experience parameters of the screening method, thereby effectively balancing the screening cost and benefit. At the same time, through real-time performance monitoring and user feedback mechanism, the user interaction experience is continuously optimized to ensure the transparency of the screening process and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings; Figure 1is a flow chart of the method of the present invention; Figure 2 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0022] like Figure 1 As shown, a big data screening method includes the following steps: User demand determination: obtain user data demand expressions, analyze demand information, determine key features and conditions, formulate screening criteria, confirm with users, obtain feedback and make adjustments; Priority analysis: Comprehensively analyze the optimal parameters, use the analytic hierarchy process to determine the relative importance of the optimal parameters and assign weight factors, calculate the weighted score of each optimal parameter and sort them, and determine the priority of multiple screening requirements of users; Data screening plan development: Based on the identified needs and priorities, develop a screening plan, including screening methods, tools, schedules, and responsibility allocation; Screening test: clarify the purpose of prototype testing, develop a test plan, select representative data samples, prepare the test environment, design a preliminary screening prototype, perform screening operations and collect test results; Adjustment and optimization of screening methods: Analyze test results, identify problems and deficiencies, and provide several adjustment and optimization combinations to form an optimized screening method. Adjustments include algorithm optimization, feature engineering, and parameter adjustment. Selection of the optimal screening method: Conduct data screening tests on the optimized screening method, collect evaluation parameters, including performance parameters, efficiency parameters, cost-effectiveness parameters and user experience parameters, computational evaluation values, effectiveness evaluation values, cost evaluation values ​​and user value, obtain a comprehensive evaluation value, and select the optimized screening scheme with the largest comprehensive evaluation value as the final optimal screening scheme; Screening and prediction: Apply the best screening method to screen data, monitor the screening process, set up the prediction model, determine the prediction algorithm, train the prediction model, evaluate performance, and apply the model for analysis and prediction; Performance monitoring and optimization: Monitor the performance of screening and prediction models, track accuracy and response time, collect feedback from users and business teams, adjust screening and prediction strategies, set review and optimization cycles, and regularly review and optimize prediction models.

[0023] like Figure 2 As shown, a big data screening system includes a demand determination module, a testing module and a screening prediction module; The demand determination module is used to determine the user's data screening needs. The specific steps are as follows: First, obtain the user's data demand expression and understand the user's specific needs, including the purpose of the data, the type of data to be filtered and the expected results; analyze the collected demand information to determine the key features and conditions for the user to filter the data, including the time range of the data, the value of a specific field and the format of the data; formulate filtering criteria based on user needs, which can be a specific numerical range, keywords, data attributes and business rules; then confirm the filtering criteria with the user, obtain user feedback, and adjust the filtering criteria based on the feedback to ensure that the filtering criteria can meet the user's actual needs. When the user has multiple filtering requirements, analyze the priority of the user's multiple filtering requirements, and conduct a comprehensive analysis by collecting the optimal parameters, where the optimal parameters include: Requirement clarity: Evaluate the clarity and specificity of each requirement. The clearer the requirement description, the higher the feasibility of implementation and the higher the priority may be. Business value: Analyze the contribution of each requirement to the business goal. The greater the contribution to the business goal, the higher the priority. Urgency: Understand the urgency of the requirement and whether it needs to be handled immediately. Requirements with high urgency usually have a higher priority. Resource availability: Evaluate whether the resources required to implement the requirement (such as time, manpower, technology, etc.) are available. Requirements with easier access to resources may have a higher priority. User impact: Consider the impact of the requirement on user satisfaction and user experience. Requirements with greater impact on user satisfaction have a higher priority. Risk assessment: Analyze implementation Each requirement may bring risks and uncertainties. Requirements with lower risks can be given higher priority. Dependencies: determine the dependencies between requirements. Some requirements may be the prerequisites for other requirements, and these requirements may require higher priority. ROI (return on investment): evaluate the cost and expected benefits of implementing each requirement. Requirements with high ROI have higher priority. Technical feasibility: evaluate the feasibility of technical implementation. Requirements with low technical implementation difficulty may have higher priority. Data availability: evaluate the availability of required data. Requirements with easy data acquisition can be given higher priority. Historical data: check the performance and user feedback of similar requirements in the past. Requirements with good historical performance can be given priority. Then use the hierarchical analysis method multi-criteria decision-making tool to determine the relative importance of each optimization parameter, and assign corresponding weight factors according to the relative importance of the optimization parameters. According to the weight factor determined for each optimization parameter, each optimization parameter is scored, and finally the weighted score is calculated; each user's screening requirement is sorted according to the calculated weighted score. The higher the priority of the screening requirement in the front ranking, the easier it is to make reasonable resource allocation and time management in the screening process; then, according to the determined needs and priorities, formulate a data screening plan, including screening methods, tools, schedules and responsibility allocation.

[0024] The test module is used to conduct small-scale prototype tests before actual large-scale screening to verify the effectiveness of the screening method, and further adjust the screening strategy and determine the final optimal screening method based on the test results; First, clarify the purpose of the prototype test, determine the key aspects of the data screening method that needs to be verified, and develop a test plan, including the selection of test data, the construction of the test environment, the specific steps of the test and the expected test results; then select representative data samples for testing, where the selected data samples cover different data types and characteristics to determine the comprehensiveness of the test results, prepare the required hardware and software environment, and ensure that the test is carried out in a stable and controllable environment; design a preliminary screening prototype based on the screening requirements, the preliminary screening prototype contains the core functions of the screening method, run the preliminary screening prototype, perform the screening operation using the test data, and collect the test results; analyze the test results, evaluate the effectiveness of the screening method, including accuracy, efficiency and stability indicators, and identify problems and deficiencies in the testing process, including low accuracy of data screening and long response time; Then, according to the test results and problem identification, the screening method is adjusted and optimized, and several adjustment and optimization combinations are provided for the adjustment and optimization part, thereby forming several optimized screening methods. The specific adjustment and optimization part includes the following processes: Algorithm optimization: improve algorithm efficiency, including upgrading from a simple keyword matching algorithm to a more complex natural language processing algorithm; feature engineering: add or adjust the features used for screening, including introducing more text analysis features or user behavior features; parameter adjustment: optimize the parameter settings in the screening algorithm, including adjusting the threshold of the classification algorithm and the number of centers of the clustering algorithm; data preprocessing: improve data cleaning and preprocessing steps, including removing more noise data and standardizing data formats; model training: use more or less training data or try different machine learning models; resource allocation: adjust the use of computing resources, including parallel processing and adding computing nodes; and then select an optimal screening method for several optimization screening methods. The specific steps are as follows: Conduct data screening tests on different optimization screening methods, and collect evaluation parameters through test results, where the evaluation parameters include performance parameters, efficiency parameters, cost-effectiveness parameters and user experience parameters; performance parameters include: Accuracy: the degree to which the screening results match the actual needs; Recall: the proportion of relevant data that the screening method can find; Precision: the accuracy rate at which the screened data is displayed as relevant; F1 score: the harmonic mean of precision and recall, which is a balance indicator of accuracy and completeness; Robustness: the stability of the screening method when facing data of different types or qualities, and the robustness loss, that is, the difference between the performance in the worst case and the average performance, is collected to measure the robustness of the screening method; After normalizing the obtained accuracy, recall rate, precision, F1 score and robustness loss, the product of accuracy and precision plus F1 score is calculated as the base circle radius, and the product of robustness loss and correction factor is used as the height to establish a cylindrical model, and the mass point of the cylindrical model is selected as the center of the sphere. The spherical model is established with the sum of the recall rate and the base circle radius as the sphere diameter, and the volume of the heteromorphic body formed by the spherical body and the cylindrical model is calculated and recorded as the performance evaluation value, and this performance evaluation value is used as a reference standard for measuring the performance of the screening method; Efficiency parameters include: Processing Time: the time required to complete the screening operation; Resource Consumption: the computing resources and storage resources required to perform the screening; the obtained processing time and resource effect are denoted as sr and zx respectively, and after normalization, they are entered into the following formula: To get the effectiveness evaluation value XPZ, where are weight factors for processing time and resource effect, and their values ​​are set between 1-1.56 and 0.6-0.97 respectively; and the obtained effectiveness evaluation value is used as a standard to measure the efficiency of the screening method; Cost-benefit parameters include: Cost-Benefit Ratio: the ratio between the cost of the screening method and the value it brings; Maintenance Cost: the long-term maintenance and update cost of the screening method; After normalizing the obtained cost-benefit ratio and maintenance cost, use cost-benefit and maintenance cost as the two sides of the triangle, and preset the angle, connect the remaining side of the triangle, calculate the area of ​​the triangle, record it as the evaluation value, and use this evaluation value to measure the cost-benefit standard of the screening method; User experience parameters include: User Satisfaction: the user's satisfaction with the screening results; Flexibility: the ability of the screening method to adapt to different screening needs; Real-time Performance: whether the screening method can meet the needs of real-time data processing; Score the above user experience parameters and calculate the weighted score, which is recorded as the user value, and this user value is used as the standard for measuring the user experience of this screening method; Then normalize the obtained sex evaluation value, efficacy evaluation value, composition evaluation value and body value and substitute them into the following formula: To get the comprehensive evaluation value ZPZ, where NPZ, XPZ, CXZ and YTZ are the sex evaluation value, effect evaluation value, cost evaluation value and usage value respectively. The preset weight coefficients for the sex evaluation value, efficacy evaluation value, achievement evaluation value and usage value are 1.22, 0.97, 1.43 and 0.83 respectively, and the comprehensive evaluation value ZPZ is used as the evaluation standard for the test results of the screening method; the comprehensive evaluation values ​​corresponding to different optimized screening methods are then sorted according to size, and the optimized screening method with the largest comprehensive evaluation value is selected as the optimal screening method.

[0025] The screening prediction module is used to screen big data using the optimal screening method and make predictions based on the screening results; Screen the data according to the optimal screening method. During the screening process, monitor the screened data and set up a prediction model based on the screening results to determine the corresponding prediction algorithm, including regression, classification or time series prediction. Use the test and formally screened data to train the prediction model and evaluate the performance of the prediction model through test sets or cross-validation. Apply the prediction model to perform predictive analysis on the screened data and generate prediction results. Explain the generated prediction results, provide insights and suggestions, and apply the prediction results to the decision-making process. Then monitor the performance of the screening prediction model, track the prediction accuracy and response time, collect feedback from users and business teams, adjust the screening and prediction strategies based on the feedback, set up a review and optimization cycle, regularly review and optimize the prediction model according to the review and optimization cycle, and update the prediction model based on new data and business changes.

[0026] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for screening big data, characterized in that: The following steps are involved: S1: User demand analysis and confirmation: Obtain and analyze user data needs, determine the key features and conditions for data screening, formulate and confirm screening criteria, obtain user feedback and make adjustments; S2: Determine the priority of requirements: Use the optimal parameters for comprehensive analysis, apply the analytic hierarchy process to determine the relative importance of each parameter and assign weights, calculate the weighted scores, and determine the priority of requirements based on the score ranking; S3: Screening strategy development and testing: Develop a detailed data screening plan, perform prototype testing, select representative data samples, prepare the environment, design a screening prototype, collect test results, and evaluate the effectiveness of the screening method; S4: Optimization and selection of screening methods: Identify problems and deficiencies in testing, propose adjustment and optimization plans, including algorithm optimization, feature engineering and parameter adjustment optimization measures, evaluate screening methods through performance, efficiency, cost-effectiveness and user experience parameters, calculate comprehensive evaluation values, and select the best screening plan; S5: Screening execution and performance monitoring: Apply the optimal screening method to perform data screening and monitor the screening process, set up and train prediction models, evaluate and apply models for data analysis and prediction, monitor prediction model performance, solicit feedback, and regularly optimize screening and prediction strategies.

2. A big data screening system, characterized in that: include: The demand determination module is used to analyze the user's data screening needs and determine the priorities of different screening needs; The test module is used to conduct small-scale prototype tests to verify the effectiveness of the screening method and adjust the screening strategy based on the test results to determine the final optimal screening method; The screening and prediction module is used to screen big data using the optimal screening method and make predictions based on the screening results.

3. A big data screening system according to claim 2, characterized in that: The specific operation steps of the demand determination module to analyze the user's data screening requirements and determine the priorities of different screening requirements are as follows: First, obtain the user's data demand expression and understand the user's specific needs, including the purpose of the data, the type of data to be screened, and the expected results; Analyze the collected demand information to determine the key features and conditions that users need to filter data, including the time range of the data, the value of specific fields, and the format of the data; Based on user needs, formulate screening criteria, which are specific numerical ranges, keywords, data attributes and business rules; Then confirm the screening criteria with users, obtain user feedback, and adjust the screening criteria based on the feedback to ensure that the screening criteria can meet the actual needs of users; When a user has multiple screening requirements, the priorities of the user's multiple screening requirements are analyzed. Specifically, the optimization parameters are collected for comprehensive analysis, and the relative importance of each optimization parameter is determined using the hierarchical analysis method multi-criteria decision-making tool. The corresponding weight factor is assigned according to the relative importance of the optimization parameter, and each optimization parameter is scored according to the weight factor determined for each optimization parameter, and finally the weighted score is calculated; Sort each user's screening requirements according to the calculated weighted score. The higher the priority of the screening requirements, the easier it is to allocate resources and manage time in the screening process. Then, based on the identified needs and priorities, develop a data screening plan, including screening methods, tools, schedule, and responsibility allocation.

4. A big data screening system according to claim 3, characterized in that: The optimal parameters in the demand determination module include: Requirements Clarity: Assess the clarity and specificity of each requirement; Business value: Analyze the contribution of each requirement to business goals; Urgency: Understand the urgency of the demand and whether it needs to be handled immediately; Resource availability: assessing whether the resources required to implement the requirements are available; User influence: Analyze the impact of requirements on user satisfaction and user experience; Risk assessment: Analyze the risks and uncertainties of implementing each requirement; Dependencies: Identify dependencies between requirements; Return on investment: Evaluate the cost of implementing each requirement versus the expected benefits; Technical feasibility: assess the feasibility of technical implementation; Data availability: assess the availability of the required data; Historical data: View the past performance and user feedback of this requirement.

5. A big data screening system according to claim 2, characterized in that: The specific operation steps of the test module for small-scale prototype testing are as follows: First, clarify the purpose of the prototype test, determine the key aspects of the data screening method that need to be verified, and develop a test plan, including the selection of test data, the construction of the test environment, the specific steps of the test, and the expected test results; Then select representative data samples for testing, where the selected data samples cover different data types and characteristics to determine the comprehensiveness of the test results, prepare the required hardware and software environment, and ensure that the test is carried out in a stable and controllable environment; Design a preliminary screening prototype based on the screening requirements. The preliminary screening prototype contains the core functions of the screening method. Run the preliminary screening prototype, perform the screening operation using test data, and collect test results. Analyze test results, evaluate the effectiveness of screening methods, including accuracy, efficiency and stability indicators, and identify problems and deficiencies in the testing process, including low accuracy of data screening and long response times.

6. A big data screening system according to claim 2, characterized in that: The specific operation steps of the test module to adjust the screening strategy according to the test results and determine the final optimal screening method are as follows: According to the test results and problem identification, the screening method is adjusted and optimized, and several adjustment and optimization combinations are provided for the adjustment and optimization parts to form several optimized screening methods; Then select an optimal screening method for several optimization screening methods. The specific steps are as follows: Data screening tests were conducted on different optimization screening methods, and evaluation parameters were collected through test results, including performance parameters, efficiency parameters, cost-effectiveness parameters and user experience parameters. The evaluation parameters were analyzed to obtain the performance evaluation value, effectiveness evaluation value, cost evaluation value and user value, and after normalization, they were entered into the following formula: To get the comprehensive evaluation value ZPZ, where NPZ, XPZ, CXZ and YTZ are the sex evaluation value, effect evaluation value, cost evaluation value and usage value respectively. The preset weight coefficients are respectively the sex evaluation value, the efficacy evaluation value, the composition evaluation value and the user value, and the obtained comprehensive evaluation value ZPZ is used as the evaluation standard of the test results of the screening method; Then, the comprehensive evaluation values ​​corresponding to different optimization screening methods are sorted according to size, and the optimization screening method with the largest comprehensive evaluation value is selected as the optimal screening method.

7. A big data screening system according to claim 6, characterized in that: The adjustment and optimization part of the optimization screening method specifically includes the following processes: Algorithm optimization: Improve algorithm efficiency, including upgrading from a simple keyword matching algorithm to a more complex natural language processing algorithm; Feature engineering: adding or adjusting features used for screening, including introducing more text analysis features or user behavior features; Parameter adjustment: Optimize the parameter settings in the screening algorithm, including adjusting the threshold of the classification algorithm and the number of centers of the clustering algorithm; Data preprocessing: Improve data cleaning and preprocessing steps, including removing more noisy data and standardizing data formats; Model training: using more or less training data or trying different machine learning models; Resource allocation: Adjust the use of computing resources, including parallel processing and adding computing nodes.

8. A big data screening system according to claim 6, characterized in that: The specific steps of the test module to analyze the evaluation parameters to obtain the sex evaluation value, the effectiveness evaluation value, the cost evaluation value and the user value are as follows: The performance parameters include: Accuracy: the degree to which the screening results match the actual needs; Recall: the proportion of relevant data that the screening method can find; Precision: The selected data are displayed as relevant accuracy; F1 score: the harmonic mean of precision and recall; Robustness: The stability of the screening method when facing data of different types or qualities, and collect the robustness loss, that is, the difference between the worst-case performance and the average performance, to measure the robustness of the screening method; After normalizing the obtained accuracy, recall rate, precision, F1 score and robustness loss, the product of accuracy and precision plus F1 score is calculated as the base circle radius, and the product of robustness loss and correction factor is used as the height to establish a cylindrical model, and the mass point of the cylindrical model is selected as the center of the sphere. The spherical model is established with the sum of the recall rate and the base circle radius as the sphere diameter, and the volume of the heteromorphic body formed by the spherical body and the cylindrical model is calculated and recorded as the performance evaluation value, and this performance evaluation value is used as a reference standard for measuring the performance of the screening method; Efficiency parameters include: Processing time: the time required for the screening operation to complete; Resource consumption: the computing resources and storage resources required to perform the screening; The obtained processing time and resource effect are denoted as sr and zx respectively, and then normalized and entered into the following formula: To get the effectiveness evaluation value XPZ, where are weight factors for processing time and resource effects respectively, and the obtained effectiveness evaluation value is used as a standard to measure the efficiency of the screening method; Cost-effectiveness parameters include: Cost-effectiveness ratio: the ratio between the cost of a screening method and the value it brings; Maintenance costs: long-term maintenance and update costs of screening methods; After normalizing the obtained cost-effectiveness ratio and maintenance cost, the cost-effectiveness and maintenance cost are used as the two sides of the triangle, and the included angle is preset. The remaining side of the triangle is connected, and the area of ​​the triangle is calculated and recorded as the evaluation value. This evaluation value is used to measure the cost-effectiveness standard of the screening method. User experience parameters include: User satisfaction: the user's satisfaction with the screening results; Flexibility: the ability of the screening method to adapt to different screening needs; Real-time: whether the screening method can meet the needs of real-time data processing; Different user experience parameters are scored and weighted scores are calculated, recorded as user experience values, and the user experience values ​​are used as a standard to measure the user experience of this screening method.

9. A big data screening system according to claim 2, characterized in that: The specific steps of the screening prediction module for prediction according to the screening results are as follows: Screen the data according to the optimal screening method. During the screening process, monitor the screened data, set up a prediction model based on the screening results, and determine the corresponding prediction algorithm, including regression, classification or time series prediction; Use the test and formally screened data to train the prediction model, and evaluate the performance of the prediction model through a test set or cross-validation; Apply predictive models to perform predictive analysis on the screened data, generate prediction results, interpret the generated prediction results, provide insights and suggestions, and apply the prediction results to the decision-making process; Then monitor the performance of the screening prediction model, track prediction accuracy and response time, collect feedback from users and business teams, adjust screening and prediction strategies based on the feedback, set up a review and optimization cycle, regularly review and optimize the prediction model according to the review and optimization cycle, and update the prediction model when new data and business changes appear.

Images