Data acquisition method based on big data

Abstract

The invention discloses a data acquisition method based on big data, and relates to the technical field of big data, and the data acquisition method comprises the following steps: obtaining to-be-acquired data; obtaining target collection data; marking target acquisition abnormal data; evaluating the influence degree of the target acquisition abnormal data; and judging whether the target acquisition abnormal data meets a preset data acquisition standard or not. Based on the anomaly recognition algorithm, anomaly detection and marking can be quickly performed on the target data, accurate positioning of potential problem data is realized, the data anomaly processing efficiency can be improved, the abnormal data is compared with the existing database through the evaluation algorithm, the influence degree of the abnormal data can be accurately evaluated, and the accuracy of data anomaly processing is improved. The method shortens the manual judgment time, judges whether the data meets the collection standard or not according to the evaluation result, can effectively eliminate irrelevant or low-value data, guarantees the high correlation and application value of the collected data, and reduces the processing cost of invalid data.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data collection method based on big data. Background Art

[0002] With the advent of the big data era, the amount of data faced by all walks of life has exploded. From traditional structured data to today's unstructured data (such as images, videos, and social media content), the types and complexity of data are increasing. This trend has put forward higher requirements for data collection, storage, and processing technologies. Traditional data collection methods can no longer meet the current needs of high-frequency and multi-dimensional data generation, while data processing needs to cope with larger scales, faster speeds, and more complex analysis goals. At the same time, data quality assurance, privacy protection, and real-time processing have also become difficult problems that must be solved. In response to these challenges, more and more companies and research institutions have invested in the development of efficient algorithms, big data storage technologies (such as distributed storage systems), and artificial intelligence-driven real-time analysis tools. It can be said that big data is not only a resource, but also an important driving force for decision-making and innovation, and the innovation of data collection and processing technology will become the key to the development of the future digital economy.

[0003] However, the existing technology is not convenient for quickly detecting and marking anomalies in target data, and is not convenient for accurately locating potential problem data, which reduces the efficiency of data anomaly processing and cannot accurately assess the impact of abnormal data, thereby increasing manual judgment time. At the same time, it is not convenient to judge whether the data meets the collection standards, and cannot effectively eliminate irrelevant or low-value data, and cannot ensure the high relevance and application value of the collected data, thereby increasing the processing cost of invalid data.

[0004] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention proposes a data collection method based on big data, which solves the problems mentioned in the above background technology, such as it is inconvenient to quickly detect and mark anomalies in target data, it is inconvenient to accurately locate potential problem data, the efficiency of data anomaly processing is reduced, and it is impossible to accurately assess the impact of abnormal data, thereby increasing manual judgment time. At the same time, it is inconvenient to judge whether the data meets the collection standards, it is impossible to effectively eliminate irrelevant or low-value data, and it is impossible to ensure the high relevance and application value of the collected data, thereby increasing the processing cost of invalid data.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] A data collection method based on big data, the data collection method comprising the following steps:

[0008] S1. Obtain the data source to be collected, and pre-process the data source to be collected to obtain the data to be collected;

[0009] S2. Based on the feature extraction algorithm, feature extraction is performed on the obtained data to be collected to obtain target collected data;

[0010] S3. Based on the abnormality recognition algorithm, the obtained target collection data is identified and analyzed, and the abnormal target collection data is marked;

[0011] S4. Based on the evaluation algorithm, the identified and marked target collection abnormal data is compared with the known abnormal data in the database to evaluate the impact of the target collection abnormal data;

[0012] S5. Based on the result of evaluating the impact of the target collection abnormal data, determine whether the target collection abnormal data meets the preset data collection standard.

[0013] Further, based on the feature extraction algorithm, feature extraction is performed on the obtained data to be collected, and obtaining the target collected data includes the following steps:

[0014] S21, initializing the data set to be collected, determining the data set to be extracted, and adjusting the data to be collected;

[0015] S22, clustering the adjusted data to be collected, and determining a combination of data to be collected that has the greatest impact on the target collected data;

[0016] S23, using the feature extraction algorithm to adjust the data to be collected to perform a new classification, determine the category to which each data to be collected belongs, and find the optimal similarity measurement between different classifications;

[0017] S24, comparing the data to be collected obtained through the adjustment and classification operations with the standard requirements of the target collection data, and selecting the combination of data to be collected with the smallest deviation from the target collection data as the optimal data set to be collected;

[0018] S25, determine whether the adjustment and screening process of the current data to be collected meets the preset termination condition. If so, terminate the adjustment and screening, take the current optimal data set to be collected as the target collection data and output it; otherwise, go to step S21 to continue processing the data to be collected.

[0019] Further, the data to be collected adjusted by the feature extraction algorithm is newly classified, the category to which each data to be collected belongs is determined, and the optimal similarity measurement between different classifications is found, including the following steps:

[0020] S231, randomly selecting a number of adjusted data to be collected as initial classification clustering points, calculating the similarity measure between each unclassified data to be collected and all classification clustering points, and assigning the data to be collected to the classification clustering point with the highest similarity measure, thereby forming a preliminary classification result;

[0021] S232, randomly selecting data points that are not used as classification clustering points from the adjusted data to be collected as potential new classification clustering point candidates;

[0022] S233, using the randomly selected classification cluster point candidates, evaluating the overall error change caused by replacing the current classification cluster point, and updating the classification cluster point set according to the evaluation result;

[0023] S234. When the classification clustering point no longer changes, determine the final category to which each data to be collected belongs, and calculate the optimal similarity measure between different categories.

[0024] Further, based on the abnormality recognition algorithm, the obtained target collection data is identified and analyzed, and the target collection abnormal data is marked, including the following steps:

[0025] S31, obtaining the historical status record of the target collected data and counting the number of abnormalities, and calculating the abnormal distribution ratio of the target collected data;

[0026] S32. Analyze the abnormal distribution of the target collected data based on data mining technology, and draw a distribution frequency diagram of the abnormal proportion of the target collected data;

[0027] S33, based on the distribution frequency diagram of the abnormal proportion of the target collected data, construct an abnormal data recognition model, and deploy the constructed abnormal data recognition model to the area with high incidence of abnormal data;

[0028] S34. If the data in the high-incidence area of ​​abnormal data detected in the target collected data exceeds the threshold, the abnormal data recognition algorithm is used to identify and analyze the current target collected data, and the abnormal data points existing in the current target collected data are marked.

[0029] Furthermore, if the data in the high-incidence area of ​​abnormal data detected in the target collected data exceeds the threshold, the abnormal data points existing in the current target collected data are marked by using the abnormal recognition algorithm, including the following steps:

[0030] S341, discretizing the target collected data, and simplifying the attributes of the target collected data dimensions using an attribute importance reduction algorithm;

[0031] S342, initializing a pool including a plurality of analysis nodes, and setting a recognition threshold range for each analysis node;

[0032] S343, selecting a target collection data record from the current target collection data, and selecting a number of analysis nodes that best match the current target collection data through a node optimization algorithm, and performing sampling analysis on the target collection data;

[0033] S344, generating an abnormal identification signal from the target acquisition data according to the simplified attributes, extracting relevant signal values, calculating signal concentrations, and updating signal accumulation values ​​in the extracted analysis nodes;

[0034] S345, for each analysis node, determine whether its signal strength is less than a preset recognition threshold, if less than the recognition threshold, return to S343 to continue sampling;

[0035] S346, comparing the semi-mature signal and the mature signal of each analysis node, if the semi-mature signal is larger than the mature signal, marking the analysis node as an abnormal state node; if the semi-mature signal is smaller than the mature signal, marking the analysis node as a normal state node;

[0036] S347. Determine whether the analysis of several analysis nodes has been completed in the target collection data. If the analysis is completed, calculate the abnormal evaluation value of the target collection data record; if the abnormal evaluation value is greater than the preset abnormal threshold, mark the target collection data as an abnormal data point, otherwise mark it as a normal data point.

[0037] Further, a target collection data record is selected from the current target collection data, and a number of analysis nodes that best match the current target collection data are selected through a node optimization algorithm. Sampling and analyzing the target collection data includes the following steps:

[0038] S3431. Setting the initial node pool size, number of optimization variables and termination conditions of the optimization process for target data collection and analysis;

[0039] S3432, performing fitness evaluation on the set of analysis nodes currently selected from the node pool, and identifying the node combination that best matches the target collection data and the node combination that does not match the target collection data the most;

[0040] S3433, based on the feature difference between the most matching node combination and the least matching node combination, using the updating rule of the node optimization algorithm to generate an updated analysis node set;

[0041] S3434, comparing the fitness of the updated analysis node set with the previous analysis node set, if the updated node set has the best fitness, then replacing the old analysis node set, otherwise retaining the old analysis node set;

[0042] S3435. If the termination condition of the optimization process is reached, the best analysis node set is output as the analysis nodes that best match the current target acquisition data; otherwise, return to step S3432 to continue optimizing the analysis node set.

[0043] Furthermore, the update rule formula is:

[0044]

[0045] In the formula, Indicates J a,b,c The updated value of the node; J a,b,c Indicates the value of the ath node in the bth node pool size in the cth iteration; Indicates the value of the a-th node in the corresponding best matching node combination in the c-th iteration; It represents the value of the a-th node in the c-th iteration corresponding to the least matching node combination; o1 and o2 both represent random numbers.

[0046] Furthermore, based on the evaluation algorithm, the identified and marked target collection abnormal data is compared with the known abnormal data in the database, and the impact of the target collection abnormal data is evaluated, including the following steps:

[0047] S41, initializing the size of the candidate evaluation set of target collection abnormal data, setting the dynamic comparison probability and the maximum number of iterations, and selecting the initial best abnormal impact evaluation result, and calculating its impact score;

[0048] S42. Based on the dynamic comparison probability, compare the size of the dynamic comparison probability and the random number. If the dynamic comparison probability is greater than the random number, generate a new abnormal impact assessment candidate solution through a cross-category integration formula; if the dynamic comparison probability is less than the random number, generate a new abnormal impact assessment candidate solution through a same-category integration formula;

[0049] S43, applying the strategy optimization algorithm to the newly generated abnormal impact assessment candidate solution, and recalculating its impact score;

[0050] S44, comparing the impact scores of the newly generated abnormal impact assessment candidate solution and the current optimal abnormal impact assessment result. If the impact score of the current optimal abnormal impact assessment result is less than the impact score of the newly generated assessment result, then replacing the impact score of the current optimal abnormal impact assessment result with the new solution; otherwise, keeping the impact score of the current optimal abnormal impact assessment result unchanged, and proceeding to the next round of iteration;

[0051] S45. Check whether the maximum number of iterations has been reached. If so, output the impact score of the global current optimal abnormal impact assessment result as the impact degree of the abnormal data collected by the assessment target. If not, return to step S42 to continue optimization.

[0052] Furthermore, applying the strategy optimization algorithm to the newly generated abnormal impact assessment candidate solution and recalculating its impact score includes the following steps:

[0053] S431, randomly generate an initial abnormal impact assessment candidate solution set, calculate the corresponding impact score of each candidate solution in the initial abnormal impact assessment candidate solution set, and perform fast non-dominated sorting on the candidate solutions, calculate the congestion degree of each solution, and form a sorted preferred candidate solution set;

[0054] S432, based on the roulette wheel selection method, selecting the candidate solution with the highest influence score from the preferred candidate solution set, and duplicating it to form a new candidate solution set;

[0055] S433, performing combination and perturbation operations on the new candidate solution set, eliminating the candidate solutions that do not meet the constraints, retaining the candidate solutions that meet the constraints, and regenerating the candidate solution data that meets the constraints using the initialization function;

[0056] S434, merging the basic candidate solution set and the extended candidate solution set to form a new optimal candidate solution set, eliminating duplicate candidate solutions, and retaining only unique candidate solution data to form a non-duplicate candidate solution set;

[0057] S435, performing fast non-dominated sorting on the non-repetitive candidate solution set, and calculating the congestion degree of each solution to obtain a new non-repetitive candidate solution data set;

[0058] S436: Determine whether the current number of iterations reaches the threshold. If not, return to step S432 to continue iterating. If reached, output the final optimal candidate solution set as the optimization strategy result of abnormal impact assessment, and recalculate its impact score.

[0059] Furthermore, the formula for the roulette wheel selection method is:

[0060]

[0061] Where P i represents the cumulative probability of the ith alternative solution; m i represents the influence score of the ith alternative solution; m j represents the influence score of the jth alternative solution; n represents the total number of alternative solutions.

[0062] The beneficial effects of the present invention are:

[0063] 1. The present invention can effectively filter redundant or noisy data and extract high-quality, target-oriented data through preprocessing of data sources and application of feature extraction algorithms, thereby improving the accuracy and effectiveness of collected data. Based on the anomaly recognition algorithm, it can quickly detect and mark anomalies in target data, accurately locate potential problem data, and help improve the efficiency of data anomaly processing. By comparing abnormal data with existing databases through evaluation algorithms, the impact of abnormal data can be accurately evaluated, shortening the time for manual judgment. At the same time, whether the data meets the collection standards can be judged based on the evaluation results, and irrelevant or low-value data can be effectively eliminated, ensuring the high relevance and application value of the collected data, thereby reducing the processing cost of invalid data.

[0064] 2. The present invention can dynamically identify and focus on the feature combination that has the greatest impact on the target collected data through adjustment, classification, screening and evaluation of the collected data, thereby effectively reducing the interference of irrelevant data and improving the relevance and accuracy of the collected data. In addition, by initializing cluster points, calculating similarity metrics and dynamically updating the classified cluster point sets, efficient clustering analysis can be achieved. At the same time, the adjusted data is compared with the standard requirements, and only the data combination with the smallest deviation is retained as the optimal solution, which not only reduces unnecessary data storage and transmission, but also effectively improves the overall efficiency of data collection.

[0065] 3. The present invention uses an anomaly recognition algorithm to deploy models in areas with high incidence of anomalies, making anomaly detection more accurate and reducing resource waste in global scanning. Through the optimization of the node pool and the dynamic selection of the most matching node, the analysis node set can be adjusted in real time according to the current data characteristics, enhancing the adaptability of the model in different data environments. The node optimization algorithm uses the difference between the most matching and the least matching features for iterative updates, which not only improves the efficiency of node selection, but also makes the model more capable of capturing data features. By calculating the signal concentration, cumulative value and signal strength, and marking the node status as abnormal, the intelligent labeling of abnormal points in the target acquisition data is realized, reducing the need for human intervention, thereby greatly improving the quality and efficiency of the acquired data.

[0066] 4. The present invention can compare the identified abnormal data with the known abnormal data in the database through the evaluation algorithm, so as to more accurately evaluate the impact of the abnormal data, which is helpful to identify the severity and potential risks of the abnormal data. Through the dynamic comparison probability mechanism, the comparison strategy can be adjusted according to the actual situation to improve the flexibility and adaptability of the decision-making process. By using fast non-dominated sorting and congestion calculation, high-quality alternative solutions can be quickly screened out to improve the efficiency of the evaluation process. Based on the roulette selection method, through the combination of cumulative probability and influence score, it can ensure that the influence of alternative solutions is taken into account in the selection process. At the same time, diversity is introduced through randomness to avoid premature convergence to the local optimal solution, thereby improving the accuracy of abnormal data evaluation and enhancing the efficiency and robustness of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 It is a flow chart of a data collection method based on big data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0070] In the description of the present invention, unless otherwise specified, the meaning of "plurality" is two or more. In addition, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0071] According to an embodiment of the present invention, a data collection method based on big data is provided.

[0072] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to the data collection method based on big data in an embodiment of the present invention, the data collection method includes the following steps:

[0073] S1. Obtain the data source to be collected, and pre-process the data source to be collected to obtain the data to be collected;

[0074] Specifically, data source refers to the source of raw data, including:

[0075] Physical sensor data: real-time monitoring data generated by temperature sensors, pressure sensors, etc.

[0076] Network data: such as website traffic, social media content, log data, etc.

[0077] Database: such as the company's historical records, inventory system data, customer information, etc.

[0078] External data interface: such as weather information, financial data, etc. provided by open APIs.

[0079] Unstructured data: multimedia data such as images, audio, and video.

[0080] Experimental data: such as raw measurements produced by laboratory testing equipment.

[0081] Specifically, the data to be collected refers to a valid data subset that meets the collection target requirements after preprocessing from the data source, including:

[0082] 1) Structured Data

[0083] Numerical data: such as sales, inventory levels, temperature readings, etc.

[0084] Categorical data: such as product type, geographic location, customer category, etc.

[0085] Time series data: such as historical status of equipment operation, real-time sampling data, etc.

[0086] 2) Unstructured Data

[0087] Text data: such as customer reviews, social media comments, and email content.

[0088] Image data: such as images from surveillance cameras and sample images under a microscope.

[0089] Audio data: such as phone recordings and environmental noise samples.

[0090] Video data: such as traffic surveillance videos and advertising materials.

[0091] 3) Metadata

[0092] Source information of the data: such as timestamp, location, and identification of the device that generated the data.

[0093] Description of data attributes: such as sampling frequency, data accuracy, etc.

[0094] S2. Based on the feature extraction algorithm, feature extraction is performed on the obtained data to be collected to obtain target collected data;

[0095] Specifically, target collection data refers to a set of key feature data that meets specific collection goals or requirements after feature extraction, screening and optimization from the data to be collected.

[0096] S3. Based on the abnormality recognition algorithm, the obtained target collection data is identified and analyzed, and the abnormal target collection data is marked;

[0097] Specifically, abnormal data refers to data points or data sets that are inconsistent with the normal pattern or expected behavior of the data set, including:

[0098] 1) Outliers: These data points differ significantly in value from most other data points and may be caused by measurement error, data entry error, or true anomalous events.

[0099] 2) Outliers: These data points do not fit the regular pattern of the data and may be caused by system failures, environmental changes, or other unusual events.

[0100] 3) Fraudulent behavior: In finance, cybersecurity, and other fields, anomalous data may include fraudulent behavior such as credit card fraud, identity theft, or other illegal activities.

[0101] 4) Failure or Error: In industrial or system monitoring, abnormal data may indicate equipment failure, software error or other technical problems.

[0102] 5) Cyberattacks: In the field of cybersecurity, abnormal data may include DDoS attacks, malware activities, or other network intrusions.

[0103] 6) Pattern Change: In time series analysis, abnormal data may indicate a sudden change in the data pattern, such as a sudden change in market trends or a significant change in consumer behavior.

[0104] 7) Sensor Failure: In IoT and industrial monitoring, abnormal data may be erroneous readings caused by sensor failure or damage.

[0105] S4. Based on the evaluation algorithm, the identified and marked target collection abnormal data is compared with the known abnormal data in the database to evaluate the impact of the target collection abnormal data;

[0106] Specifically, the impact of abnormal data refers to the severity of the impact or damage caused by abnormal data on the system, business or analysis results, including:

[0107] 1) Severity of abnormal data: Assess the impact of abnormal data on the system or business, for example, whether the abnormal data has caused system failure, economic loss or data leakage.

[0108] 2) Scope of the abnormal data: Assess the scope of the impact of the abnormal data, for example, whether the abnormal data affects a single data point, part of a dataset, or the entire system.

[0109] 3) Frequency of abnormal data: Assess how often the abnormal data occurs, for example, whether the abnormal data is a one-time event or an ongoing problem.

[0110] 4) Difference between abnormal data and normal data: Evaluate the degree of difference between abnormal data and normal data, for example, whether the abnormal data deviates significantly from the normal data distribution or pattern.

[0111] 5) Potential impact of abnormal data: Assess the potential impact of abnormal data on the future, for example, whether the abnormal data indicates greater risks or problems.

[0112] 6) Recoverability of abnormal data: Assess the recoverability of the impact of abnormal data, for example, whether the abnormal data can be recovered through simple intervention or repair.

[0113] 7) Predictability of abnormal data: Evaluate the predictability of abnormal data, for example, whether the abnormal data can be predicted by existing models or algorithms.

[0114] S5. Based on the result of evaluating the impact of the target collection abnormal data, determine whether the target collection abnormal data meets the preset data collection standard.

[0115] It should be explained that the corresponding decision is made based on the comparison results. If the impact of abnormal data is within an acceptable range, the data may be considered to meet the standards and can continue to be used or only slightly adjusted. If the impact exceeds the preset standards, further investigation or processing may be required, such as data cleaning, removal of abnormal data, or other measures; the evaluation results and decisions are recorded and fed back to the relevant data management personnel or decision makers for subsequent monitoring or adjustment; based on the evaluation results and actual application feedback, the data collection standards are regularly updated and optimized to adapt to the changing data environment and business needs.

[0116] Preferably, based on a feature extraction algorithm, feature extraction is performed on the obtained data to be collected, and obtaining the target collected data includes the following steps:

[0117] S21, initializing the data set to be collected, determining the data set to be extracted, and adjusting the data to be collected;

[0118] S22, clustering the adjusted data to be collected, and determining a combination of data to be collected that has the greatest impact on the target collected data;

[0119] S23, using the feature extraction algorithm to adjust the data to be collected to perform a new classification, determine the category to which each data to be collected belongs, and find the optimal similarity measurement between different classifications;

[0120] S24, comparing the data to be collected obtained through the adjustment and classification operations with the standard requirements of the target collection data, and selecting the combination of data to be collected with the smallest deviation from the target collection data as the optimal data set to be collected;

[0121] S25, determine whether the adjustment and screening process of the current data to be collected meets the preset termination condition. If so, terminate the adjustment and screening, take the current optimal data set to be collected as the target collection data and output it; otherwise, go to step S21 to continue processing the data to be collected.

[0122] It needs to be explained that, for example, taking vehicle data collection in intelligent transportation systems as an example:

[0123] 1) Initialize the data set to be collected

[0124] Input: Raw traffic monitoring data (including vehicle speed, vehicle type, traffic flow, accident reports, etc.).

[0125] Action: Identify features that need to be extracted, such as speed, flow, and accident rate, and perform preliminary cleaning and formatting of the data.

[0126] 2) Aggregate the adjusted data to be collected

[0127] Input: Cleaned traffic monitoring data.

[0128] Action: Aggregate data by time period (such as hourly) to analyze traffic patterns.

[0129] 3) Classification using feature extraction algorithm

[0130] Input: Aggregated traffic monitoring data.

[0131] Operation: Apply clustering algorithms (such as K-means clustering algorithm) to classify the data and determine the category to which each data point belongs (such as peak period, off-peak period).

[0132] 4) Filter the best data combination to be collected

[0133] Input: Classified traffic monitoring data.

[0134] Operation: Compare with the standard traffic flow model and select the data combination with the smallest deviation from the standard, such as the characteristic combination of normal traffic flow.

[0135] 5) Determine the termination condition

[0136] Input: Filtered traffic monitoring data.

[0137] Action: Check whether the preset termination conditions are met, such as whether the data quality reaches the preset standard or the number of iterations reaches the preset limit.

[0138] Output: If the conditions are met, output the optimal data set to be collected; otherwise, return to step S21 to continue processing.

[0139] Specifically, the feature extraction algorithm is a K-medoids clustering algorithm based on differential evolution. The idea is: first, a group of a certain size is randomly selected from the population, and the size of the group is set to a fixed value. At the same time, the maximum size limit of the population is set. Then, the differential evolution algorithm is used to perform mutation and crossover operations on this population to determine the optimal mutant individuals and experimental individuals in the population, thereby avoiding the population from falling into the local optimum and enhancing the algorithm's local search ability in the cluster space area. Next, a new clustering analysis is performed on these optimized experimental individuals according to the K-medoids clustering algorithm to determine the cluster to which each individual belongs and the optimal distance between them.

[0140] Specifically, the differential evolution algorithm is an algorithm based on population evolution. It has the characteristics of memorizing individual optimal solutions and sharing information within the population. That is, it solves the optimization problem through cooperation and competition among individuals in the population. Its essence is a greedy genetic algorithm based on real number coding with the idea of ​​preserving optimality.

[0141] Preferably, the process of reclassifying the data to be collected after being adjusted by the feature extraction algorithm, determining the category to which each data to be collected belongs, and finding the optimal similarity measure between different classifications includes the following steps:

[0142] S231, randomly selecting a number of adjusted data to be collected as initial classification clustering points, calculating the similarity measure between each unclassified data to be collected and all classification clustering points, and assigning the data to be collected to the classification clustering point with the highest similarity measure, thereby forming a preliminary classification result;

[0143] S232, randomly selecting data points that are not used as classification clustering points from the adjusted data to be collected as potential new classification clustering point candidates;

[0144] S233, using the randomly selected classification cluster point candidates, evaluating the overall error change caused by replacing the current classification cluster point, and updating the classification cluster point set according to the evaluation result;

[0145] S234. When the classification clustering point no longer changes, determine the final category to which each data to be collected belongs, and calculate the optimal similarity measure between different categories.

[0146] It needs to be explained that, for example, taking vehicle data collection in intelligent transportation systems as an example:

[0147] 1) Randomly select the initial classification cluster point

[0148] Input: Adjusted vehicle dataset including vehicle speed, vehicle type, load, driving route and road occupancy time.

[0149] Operation: Randomly select some vehicle data as initial classification points. For example:

[0150] Select three vehicle data points as the initial clustering points of "small vehicles", "medium vehicles" and "large vehicles". Calculate the similarity measure (such as Euclidean distance, Manhattan distance) between each unclassified vehicle data and these clustering points. Assign each vehicle to the clustering point with the highest similarity measure to form a preliminary classification result.

[0151] Output: Preliminary classification results, for example:

[0152] Category 1 (small vehicles): Vehicle A, Vehicle D.

[0153] Category 2 (medium-sized vehicles): Vehicle B, Vehicle E.

[0154] Category 3 (large vehicles): Vehicle C, Vehicle F.

[0155] 2) Randomly select potential new gathering points

[0156] Input: Vehicle data after preliminary classification.

[0157] Operation: Randomly select potential new clustering point candidates from the vehicle data that has not been used as a classification clustering point. For example, select vehicle G and vehicle H from the remaining vehicles. Use these potential clustering points as candidate points to optimize the existing clustering point set.

[0158] 3) Evaluate and update the classification cluster set

[0159] Input: Current classified clusters and potential clusters.

[0160] Action: Evaluate the effect of replacing existing cluster points on the overall error. For example:

[0161] Calculate the error (such as the intra-class sum of squares) of the current cluster point set.

[0162] Recalculate the error after replacing a cluster point.

[0163] Compare the error changes before and after replacement and select the replacement that can reduce the error.

[0164] If the error is reduced after the latent clustering point H replaces the clustering point B, H is updated to the new clustering point.

[0165] Output: Updated set of classified cluster points.

[0166] 4) Determine the final classification and similarity measurement

[0167] Input: A stable set of cluster points.

[0168] Operation: When the clustering point set no longer changes, determine the final category to which each vehicle data belongs.

[0169] Compute the optimal similarity measure between different categories. For example:

[0170] The similarity measure between category 1 and category 2 is 0.8.

[0171] The similarity measure between category 1 and category 3 is 0.5.

[0172] The similarity measure between category 2 and category 3 is 0.6.

[0173] Output: Final vehicle classification results and similarity measures between categories.

[0174] 5) Implementation results

[0175] (1) Vehicle classification results:

[0176] Category 1 (small vehicles): Vehicle A, Vehicle D, Vehicle G.

[0177] Category 2 (medium-sized vehicles): Vehicle B, Vehicle E.

[0178] Category 3 (large vehicles): Vehicle C, Vehicle F, Vehicle H.

[0179] (2) Similarity measurement between categories:

[0180] Similarity measure between category 1 and category 2: 0.8 (e.g., the velocity distributions are close).

[0181] Similarity measure between category 2 and category 3: 0.6 (e.g., road occupancy time is similar).

[0182] Similarity measure between category 1 and category 3: 0.5 (if the load difference is large).

[0183] Preferably, based on the abnormality recognition algorithm, identifying and analyzing the obtained target acquisition data, and marking the target acquisition abnormal data includes the following steps:

[0184] S31, obtaining the historical status record of the target collected data and counting the number of abnormalities, and calculating the abnormal distribution ratio of the target collected data;

[0185] S32. Analyze the abnormal distribution of the target collected data based on data mining technology, and draw a distribution frequency diagram of the abnormal proportion of the target collected data;

[0186] S33, based on the distribution frequency diagram of the abnormal proportion of the target collected data, construct an abnormal data recognition model, and deploy the constructed abnormal data recognition model to the area with high incidence of abnormal data;

[0187] S34. If the data in the high-incidence area of ​​abnormal data detected in the target collected data exceeds the threshold, the abnormal data recognition algorithm is used to identify and analyze the current target collected data, and the abnormal data points existing in the current target collected data are marked.

[0188] Specifically, the anomaly recognition algorithm is an improved dendritic cell algorithm DCA. By improving the data preprocessing process of DCA, the rough set attribute importance is used to obtain the simplest attribute set, and better classification ability is retained with fewer attributes. Attribute simplification and signal presentation in the DCA data preprocessing process are achieved. The number of features to be detected is increased, and the algorithm operation speed is accelerated, with good comprehensive performance. The DCA algorithm is a group algorithm, and each DC cell has the ability to collect antigen and signal data and process them.

[0189] In order to facilitate understanding of the above technical solution of the present invention, the present invention will be described in detail below in the actual process based on the abnormality recognition algorithm to identify and analyze the obtained target collection data and mark the target collection abnormal data.

[0190] It needs to be explained that, for example, taking the vehicle speed anomaly detection in the intelligent transportation system as an example:

[0191] Step 1: Obtain the historical status records of the target collection data and count the number of abnormalities

[0192] Input data: vehicle speed data collected on a city's main roads, with 1,000 historical sampling records, including the following ranges:

[0193] Normal speed range: 30-70km / h.

[0194] Abnormal speed range: >80km / h (speeding).

[0195] Processing: In the historical data, there are 50 records with speed exceeding 80 km / h. Calculate the abnormal distribution ratio.

[0196]

[0197] Step 2: Analyze abnormal distribution based on data mining technology and draw distribution frequency graph

[0198] Input data: The current target collection data contains 1050 records, including:

[0199] Normal speed: 1000 items.

[0200] Abnormal speed: 50.

[0201] deal with:

[0202] Use data mining techniques (such as frequency statistics) to divide the speed data into 10 intervals:

[0203] 30-40km / h: 250.

[0204] 40-50km / h: 300.

[0205] 50-60km / h: 200.

[0206] 60-70km / h: 250.

[0207] 70-80km / h: 30 lines.

[0208] 80-90km / h: 15 lines.

[0209] 90-100km / h: 5 lines.

[0210] Plot a frequency distribution:

[0211] Horizontal axis: speed range (km / h).

[0212] Vertical axis: frequency.

[0213] result:

[0214] The high anomaly areas are concentrated in the speed range of 80-100km / h.

[0215] The distribution map shows that 90% of the data is between 30-70 km / h, and abnormal data accounts for 10%.

[0216] Step 3: Build an abnormal data recognition model

[0217] Input data: Abnormal distribution frequency diagram of target acquisition data.

[0218] deal with:

[0219] 1) Using the improved dendritic cell algorithm to build an abnormal data recognition model, the core idea is:

[0220] Feature selection: speed mean μ = 55 km / h, standard deviation σ = 10 km / h.

[0221] Set the abnormal threshold: abnormal detection threshold = μ + 2σ = 55 + 2 × 10 = 75 km / h.

[0222] 2) Deploy the recognition model to abnormally high-incidence areas (speed > 75km / h).

[0223] Results: The abnormal data identification threshold was determined to be 75km / h. The model was deployed in the speed range of 75-100km / h.

[0224] Step 4: Detect and mark abnormal data points

[0225] Input data: target collection data (1050 records).

[0226] Processing: Apply the anomaly recognition model: Detect records with speeds greater than 75km / h, and identify 60 abnormal data points in total. The abnormal points are mainly concentrated in the 80-100km / h range. Mark the abnormal data points and output the marking results.

[0227] result:

[0228] Detailed marking of abnormal data points:

[0229] 80-90km / h: 50.

[0230] 90-100km / h: 10.

[0231] Step 5: Results summary

[0232] Detection and marking results: Total data points: 1050. Abnormal data points: 60. Abnormal distribution ratio: 60 / 1050≈5.71%. Abnormal high-incidence area: 80-100km / h.

[0233] Preferably, if the data in the high-incidence area of ​​abnormal data exceeds the threshold value in the target collected data, the abnormal data points existing in the current target collected data are marked by using an abnormal recognition algorithm, including the following steps:

[0234] S341, discretizing the target collected data, and simplifying the attributes of the target collected data dimensions using an attribute importance reduction algorithm;

[0235] S342, initializing a pool including a plurality of analysis nodes, and setting a recognition threshold range for each analysis node;

[0236] S343, selecting a target collection data record from the current target collection data, and selecting a number of analysis nodes that best match the current target collection data through a node optimization algorithm, and performing sampling analysis on the target collection data;

[0237] S344, generating an abnormal identification signal from the target acquisition data according to the simplified attributes, extracting relevant signal values, calculating signal concentrations, and updating signal accumulation values ​​in the extracted analysis nodes;

[0238] S345, for each analysis node, determine whether its signal strength is less than a preset recognition threshold, if less than the recognition threshold, return to S343 to continue sampling;

[0239] S346, comparing the semi-mature signal and the mature signal of each analysis node, if the semi-mature signal is larger than the mature signal, marking the analysis node as an abnormal state node; if the semi-mature signal is smaller than the mature signal, marking the analysis node as a normal state node;

[0240] S347. Determine whether the analysis of several analysis nodes has been completed in the target collection data. If the analysis is completed, calculate the abnormal evaluation value of the target collection data record; if the abnormal evaluation value is greater than the preset abnormal threshold, mark the target collection data as an abnormal data point, otherwise mark it as a normal data point.

[0241] In order to facilitate understanding of the above technical solution of the present invention, the following is a detailed description of the present invention in the actual process. If it is detected that the data in the high-incidence area of ​​abnormal data exceeds the threshold in the target collected data, the abnormal recognition algorithm is used to identify and analyze the current target collected data, and the abnormal data points existing in the current target collected data are marked.

[0242] It needs to be explained that, for example, taking the abnormal vehicle speed analysis in the intelligent transportation system as an example:

[0243] Step 1: Discretize the target data and simplify its attributes

[0244] Input data: Target data collection records: vehicle speed (km / h), road occupancy time (s), road type (highway / urban road), traffic density (vehicles / km). Total data points: 1050.

[0245] deal with:

[0246] 1) Discretization: The speed is divided into 6 intervals: [30-40], [40-50], [50-60], [60-70], [70-80], [80-100]. The road occupancy time is divided into: [1-5], [5-10], [10-15] seconds. Road type: urban road = 0, expressway = 1. The traffic density is divided into: [10-30], [30-50], [50-80] vehicles / km.

[0247] 2) Attribute importance reduction: Calculate attribute weights based on the information gain method:

[0248] Vehicle speed weight: 0.5.

[0249] Road occupancy time weight: 0.3.

[0250] Road type weight: 0.15.

[0251] Traffic density weight: 0.05.

[0252] Remove low-weight attributes (traffic density) and keep speed, occupancy time and road type.

[0253] Step 2: Initialize the analysis node pool and set the recognition threshold range

[0254] Input: Node pool size: 5 analysis nodes. Each node sets the recognition threshold range: [60-75], [75-85], [85-95], [95-105] (km / h).

[0255] Step 3: Select the analysis node and perform sampling analysis

[0256] deal with:

[0257] 1) Select a record from the target data: vehicle speed = 85km / h, occupancy time = 7 seconds, road type = highway.

[0258] 2) Through the node optimization algorithm, match the analysis node closest to the speed range (range [75-85], [85-95]).

[0259] 3) Perform sampling analysis in nodes [85-95].

[0260] Step 4: Generate anomaly identification signal and update signal accumulation value

[0261] deal with:

[0262] Generate a signal based on attribute-reduced data:

[0263] Speed ​​signal value: 85 / 100=0.85.

[0264] Road occupancy time signal value: 7 / 15≈0.47.

[0265] Road type signal value: 1 / 1=1.0.

[0266] Integrated signal concentration:

[0267] Signal concentration = 0.5 × 0.85 + 0.3 × 0.47 + 0.15 × 1.0 = 0.672. Update the signal accumulation value of node [85-95]:

[0268] Initial cumulative value = 3.5.

[0269] The updated cumulative value = 3.5 + 0.672 = 4.172.

[0270] Step 5: Determine whether the signal strength is less than the preset recognition threshold

[0271] deal with:

[0272] The recognition threshold was set to 4.0.

[0273] The current signal cumulative value 4.172>4.0, skip sampling and go to the next step.

[0274] Step 6: Compare semi-mature signals with mature signals

[0275] deal with:

[0276] The semi-mature signal value of node [85-95] is: 0.5×0.85=0.425.

[0277] The node's maturity signal value: 0.5.

[0278] Comparison result: 0.425<0.5, marked as a normal state node.

[0279] Step 7: Complete all node analysis and calculate abnormal evaluation values

[0280] 1) All analysis nodes of the current data record complete the analysis.

[0281] 2) Calculate the abnormal evaluation value:

[0282]

[0283] 3) Compare the abnormal evaluation value with the abnormal threshold (0.8): The current evaluation value 0.8344>0.8. The data record is marked as an abnormal data point.

[0284] Step 8. Detection and marking results:

[0285] Total data points: 1050.

[0286] After completing the node analysis: Normal data points: 990; Abnormal data points: 60.

[0287] Preferably, a target acquisition data record is selected from the current target acquisition data, and a number of analysis nodes that best match the current target acquisition data are selected through a node optimization algorithm. The sampling and analysis of the target acquisition data includes the following steps:

[0288] S3431. Setting the initial node pool size, number of optimization variables and termination conditions of the optimization process for target data collection and analysis;

[0289] S3432, performing fitness evaluation on the set of analysis nodes currently selected from the node pool, and identifying the node combination that best matches the target collection data and the node combination that does not match the target collection data the most;

[0290] S3433, based on the feature difference between the most matching node combination and the least matching node combination, using the updating rule of the node optimization algorithm to generate an updated analysis node set;

[0291] S3434, comparing the fitness of the updated analysis node set with the previous analysis node set, if the updated node set has the best fitness, then replacing the old analysis node set, otherwise retaining the old analysis node set;

[0292] S3435. If the termination condition of the optimization process is reached, the best analysis node set is output as the analysis nodes that best match the current target acquisition data; otherwise, return to step S3432 to continue optimizing the analysis node set.

[0293] It needs to be explained that, for example, taking the abnormal vehicle speed analysis in the intelligent transportation system as an example:

[0294] 1) Set the initial node pool size, number of optimization variables and termination conditions

[0295] enter:

[0296] Initial node pool size: Select a number of analysis nodes, each of which contains a specific range of characteristic attributes (such as speed, occupancy time, etc.).

[0297] Optimization variables: attribute weights adjusted during the optimization process (such as speed weight, time weight).

[0298] Termination conditions:

[0299] Reach the maximum number of iterations (e.g. 50).

[0300] The overall fitness of the node set does not change significantly after several consecutive iterations.

[0301] 2) Evaluate the fitness of the analysis node set

[0302] enter:

[0303] The current target collects data records (such as vehicle speed, occupancy time, etc.).

[0304] Attribute weights and feature ranges for all nodes in the node pool.

[0305] deal with:

[0306] Calculate the fitness of each node with respect to the target data.

[0307] Sort by fitness and identify:

[0308] The best matching node combination: the nodes that are closest to the target data characteristics.

[0309] The least matching node combination: the nodes that are farthest away from the target data characteristics.

[0310] Output:

[0311] The best matching node combination (e.g., nodes whose speed range matches the target data).

[0312] The least matching node combination (such as nodes whose speed range is significantly different from the target data).

[0313] 3) Generate an updated set of analysis nodes based on feature differences

[0314] enter:

[0315] The differences in features and weights between the best-matching node combination and the worst-matching node combination.

[0316] Update rules for the node optimization algorithm.

[0317] deal with:

[0318] Compare the differences in feature weights between the most and least matching nodes.

[0319] Update rules for applying the node optimization algorithm:

[0320] The node weights are adjusted according to the differences so that the weights are closer to the best matching node combination.

[0321] Keep some randomness to explore possible optimization directions.

[0322] Output:

[0323] The updated node set contains the new weight distribution and feature range.

[0324] 4) Compare the updated node set with the previous node set

[0325] enter:

[0326] The updated node set and the previous node set.

[0327] deal with:

[0328] Recalculate the fitness of the updated node set.

[0329] Compare the fitness of the node set before and after the update:

[0330] If the updated node set has the best fitness, the old node set is replaced.

[0331] Otherwise, keep the old node set.

[0332] Output:

[0333] The set of nodes that are updated or retained for the next round of optimization.

[0334] 5) Determine whether the optimization termination condition is met

[0335] enter:

[0336] The current number of iterations and the fitness changes of the node set.

[0337] deal with:

[0338] Check whether the termination condition is met:

[0339] If the maximum number of iterations is reached or the fitness does not change significantly, the optimization is terminated.

[0340] Otherwise, return to step S3432 to continue optimization.

[0341] Output:

[0342] Best analysis node set: several nodes that best match the current target collection data for subsequent sampling and analysis.

[0343] Specifically, the node optimization algorithm is the Jaya algorithm, which is a new meta-heuristic algorithm. The working principle of the Jaya algorithm is continuous improvement, that is, the solution tends to move towards the optimal solution and away from the worst solution. The Jaya algorithm is a single-stage algorithm, and each iteration only needs to evaluate one equation to obtain the value of the new solution. It also has the advantage that it is a parameter-free algorithm that only requires two control parameters: population size and number of iterations. Therefore, the additional computational workload required to optimize the parameters to obtain the optimal solution is eliminated.

[0344] Preferably, the update rule formula is:

[0345]

[0346] In the formula, Indicates J a,b,c The updated value of the node; J a,b,c Indicates the value of the ath node in the bth node pool size in the cth iteration; Indicates the value of the a-th node in the corresponding best matching node combination in the c-th iteration; It represents the value of the a-th node in the c-th iteration corresponding to the least matching node combination; o1 and o2 both represent random numbers.

[0347] Specifically, o1 and o2 are random numbers between (0, 1). is to move the current solution towards the optimal solution (i.e. the value of the most matching node combination). It is the value that makes the current solution away from the worst solution (that is, the most mismatched node combination).

[0348] Preferably, based on the evaluation algorithm, the identified and marked target acquisition abnormal data is compared with the known abnormal data in the database, and the impact of the target acquisition abnormal data is evaluated, including the following steps:

[0349] S41, initializing the size of the candidate evaluation set of target collection abnormal data, setting the dynamic comparison probability and the maximum number of iterations, and selecting the initial best abnormal impact evaluation result, and calculating its impact score;

[0350] S42. Based on the dynamic comparison probability, compare the size of the dynamic comparison probability and the random number. If the dynamic comparison probability is greater than the random number, generate a new abnormal impact assessment candidate solution through a cross-category integration formula; if the dynamic comparison probability is less than the random number, generate a new abnormal impact assessment candidate solution through a same-category integration formula;

[0351] S43, applying the strategy optimization algorithm to the newly generated abnormal impact assessment candidate solution, and recalculating its impact score;

[0352] S44, comparing the impact scores of the newly generated abnormal impact assessment candidate solution and the current optimal abnormal impact assessment result. If the impact score of the current optimal abnormal impact assessment result is less than the impact score of the newly generated assessment result, then replacing the impact score of the current optimal abnormal impact assessment result with the new solution; otherwise, keeping the impact score of the current optimal abnormal impact assessment result unchanged, and proceeding to the next round of iteration;

[0353] S45. Check whether the maximum number of iterations has been reached. If so, output the impact score of the global current optimal abnormal impact assessment result as the impact degree of the abnormal data collected by the assessment target. If not, return to step S42 to continue optimization.

[0354] Specifically, the evaluation algorithm is a hybrid pollen algorithm, which combines the strategy optimization algorithm with the pollen algorithm, integrates the strategy optimization algorithm into the improved pollen algorithm, updates and selects pollen individuals through strategy optimization, dynamically switches the optimization mode between global and local, and introduces regularization strategies in the middle and late stages of algorithm iteration to strengthen the information exchange between general pollen individuals and optimal pollen individuals. The role of the strategy optimization algorithm is to explore a wider range of solution space in the global stage; to use existing information to accurately adjust pollen individuals in the local stage to ensure that they move closer to the optimal solution; and to improve the convergence and robustness of the algorithm.

[0355] It is necessary to explain that, for example, taking the potential impact assessment of speeding vehicles on traffic safety in intelligent transportation systems as an example:

[0356] 1) Initialize the size of the candidate evaluation set for target collection of abnormal data

[0357] enter:

[0358] Alternative evaluation set size: Determine the number of abnormal data records to be evaluated (such as data marked as abnormal).

[0359] Dynamic comparison probability: initialized to a certain value (such as P0), indicating the selection probability between cross-category and same-category evaluation.

[0360] Maximum number of iterations: Limits the number of cycles of the optimization process.

[0361] Initial best evaluation result: Select a set of records that are most similar to the current target anomaly data from the known anomaly data in the database and calculate their initial influence scores.

[0362] deal with:

[0363] Set the initial parameters of the candidate evaluation set for subsequent dynamic comparison and strategy optimization.

[0364] 2) Generate new alternative solutions for abnormal impact assessment based on dynamic comparison probability

[0365] enter:

[0366] Current dynamic comparison probability P t .

[0367] Random number r (value range [0, 1][0, 1]).

[0368] deal with:

[0369] Compare P t >r, select the cross-category integration formula to generate a new evaluation solution:

[0370] Select different categories of abnormal data (such as speeding, drunk driving, etc.) from the database, combine the characteristics of the target abnormal data, and generate a new evaluation solution.

[0371] If P t ≤r, select the same category integration formula to generate a new evaluation solution:

[0372] Abnormal data of the same category as the target abnormal data (such as speeding records) are selected from the database, and a new evaluation solution is generated by combining the characteristics of the target data.

[0373] The new evaluation solution contains multi-dimensional features, such as speed, frequency, time, location, etc.

[0374] Output: New anomaly impact assessment candidate solutions.

[0375] 3) Apply the strategy optimization algorithm and recalculate the influence score

[0376] enter:

[0377] Newly generated anomaly impact assessment candidate solutions.

[0378] The current evaluation policy optimization algorithm.

[0379] deal with:

[0380] Apply the policy optimization algorithm to process the new evaluation solution:

[0381] Adjust the feature weights to make them more consistent with the actual characteristics of the target anomaly data.

[0382] Recalculate the influence score of the newly evaluated solution:

[0383] The weight and matching degree of the features are comprehensively considered to obtain the influence score of the new solution.

[0384] Output: The influence score of the new evaluation solution.

[0385] 4) Compare the impact score of the new solution with the current optimal evaluation result

[0386] enter:

[0387] The impact score of the current optimal abnormal impact assessment result.

[0388] The influence score of the newly generated evaluation solution.

[0389] deal with:

[0390] If the impact score of the new solution is higher than the current optimal evaluation result:

[0391] Replace the current optimal evaluation result with the new solution.

[0392] If the impact score of the new solution is lower than or equal to the current best evaluation result:

[0393] Keep the current best evaluation result.

[0394] Output: The current optimal abnormal impact assessment result.

[0395] 5) Check whether the maximum number of iterations has been reached

[0396] Input: current number of iterations and maximum number of iterations.

[0397] deal with:

[0398] If the maximum number of iterations is reached: the impact score of the current global optimal anomaly impact assessment result is output as the final impact assessment result of the target collection anomaly data.

[0399] If the maximum number of iterations is not reached: adjust the dynamic comparison probability P t (For example, gradually reduce it to increase the possibility of the same category evaluation.) Return to step S42 to continue generating new evaluation solutions and optimizing them.

[0400] Output: Final impact score, which indicates the impact of abnormal data collected by the target.

[0401] Preferably, applying the strategy optimization algorithm to the newly generated abnormal impact assessment candidate solution and recalculating its impact score comprises the following steps:

[0402] S431, randomly generate an initial abnormal impact assessment candidate solution set, calculate the corresponding impact score of each candidate solution in the initial abnormal impact assessment candidate solution set, and perform fast non-dominated sorting on the candidate solutions, calculate the congestion degree of each solution, and form a sorted preferred candidate solution set;

[0403] S432, based on the roulette wheel selection method, selecting the candidate solution with the highest influence score from the preferred candidate solution set, and duplicating it to form a new candidate solution set;

[0404] S433, performing combination and perturbation operations (i.e., crossover and mutation operations) on the new candidate solution set, eliminating the candidate solutions that do not meet the constraints, retaining the candidate solutions that meet the constraints, and regenerating the candidate solution data that meets the constraints using the initialization function;

[0405] S434, merging the basic candidate solution set and the extended candidate solution set to form a new optimal candidate solution set, eliminating duplicate candidate solutions, and retaining only unique candidate solution data to form a non-duplicate candidate solution set;

[0406] S435, performing fast non-dominated sorting on the non-repetitive candidate solution set, and calculating the congestion degree of each solution to obtain a new non-repetitive candidate solution data set;

[0407] S436: Determine whether the current number of iterations reaches the threshold. If not, return to step S432 to continue iterating. If reached, output the final optimal candidate solution set as the optimization strategy result of abnormal impact assessment, and recalculate its impact score.

[0408] Specifically, the strategy optimization algorithm is an improved NSGA-II algorithm. The NSGA-II algorithm is a classic multi-objective optimization algorithm. When used to deal with multi-objective optimization problems, it can effectively find a solution set that meets the multi-objective requirements. The present invention improves the basic NSGA-II algorithm, designs an effective integer encoding method, and introduces a deduplication operation.

[0409] Specifically, Roulette Wheel Selection is a selection operator in genetic algorithms that simulates the roulette process, assigning selection probabilities to individuals in a population, thereby selecting individuals for the next generation. In the Roulette Wheel Selection method, each individual in a population has a selection probability proportional to its fitness value. This probability is similar to the probability corresponding to each number on a roulette wheel. The higher the fitness value of an individual, the larger its corresponding "share" on the roulette wheel.

[0410] It needs to be explained that, taking the impact assessment of abnormal vehicle behavior in intelligent transportation systems as an example:

[0411] 1) Generate an initial abnormal impact assessment candidate solution set

[0412] enter:

[0413] Abnormal data set: contains multiple abnormal behavior records.

[0414] Initial candidate solution set size: Determine the number of candidate solutions that need to be generated.

[0415] deal with:

[0416] An initial set of candidate solutions is randomly generated, each solution containing a different combination or feature of the abnormal data.

[0417] An impact score is calculated for each candidate solution, and the impact score may be based on factors such as the severity, frequency, and time of the abnormal behavior.

[0418] Perform fast non-dominated sorting on candidate solutions and sort the solutions according to influence scores and other constraints (such as cost, efficiency, etc.).

[0419] The crowding degree of each solution is calculated to determine its position in the sorted set of preferred solutions.

[0420] Output: sorted set of optimal alternative solutions.

[0421] 2) Select alternative solutions using the roulette wheel selection method

[0422] Input: Select the set of candidate solutions.

[0423] deal with:

[0424] According to the crowding degree of the candidate solutions, the roulette wheel selection method is used to select the candidate solution with the highest influence score. The selected candidate solution is copied to generate a new set of candidate solutions.

[0425] Output: New set of candidate solutions.

[0426] 3) Perform combination and perturbation operations

[0427] Input: New set of candidate solutions.

[0428] deal with:

[0429] Perform combination and perturbation operations on new sets of candidate solutions to explore new solution spaces.

[0430] Eliminate the alternative solutions that do not meet the constraints.

[0431] Use the initialization function to regenerate alternative solution data that meets the constraints.

[0432] Output: Expanded set of alternative solutions.

[0433] 4) Merge the candidate solution sets and eliminate duplicates

[0434] Input: basic alternative solution set and extended alternative solution set.

[0435] deal with:

[0436] Combine the basic candidate solution set with the expanded candidate solution set.

[0437] Eliminate duplicate candidate solutions generated during the merging process.

[0438] Keep only the candidate solution data.

[0439] Output: A set of alternative solutions without duplication.

[0440] 5) Fast non-dominated sorting and calculation of congestion

[0441] Input: A set of non-repeated alternative solutions.

[0442] deal with:

[0443] Fast non-dominated sorting of a set of non-repeating candidate solutions.

[0444] Compute the crowding degree of each solution.

[0445] Output: A new dataset of candidate solutions without duplication.

[0446] 6) Determine the number of iterations and output the results

[0447] enter:

[0448] The current iteration number.

[0449] The preset iteration threshold.

[0450] deal with:

[0451] Determine whether the current number of iterations reaches the preset iteration threshold.

[0452] If not reached, return to step S432 to continue iteration.

[0453] If it is reached, the final optimal candidate solution set is output.

[0454] Output: The final set of optimal candidate solutions, as the optimization strategy result for abnormal impact assessment.

[0455] Preferably, the formula for the roulette wheel selection method is:

[0456]

[0457] Where P i represents the cumulative probability of the ith alternative solution; m i represents the influence score of the ith alternative solution; m j represents the influence score of the jth alternative solution; n represents the total number of alternative solutions.

[0458] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data collection method based on big data, characterized in that: The data collection method comprises the following steps: S1. Obtain the data source to be collected, and pre-process the data source to be collected to obtain the data to be collected; S2. Based on the feature extraction algorithm, feature extraction is performed on the obtained data to be collected to obtain target collected data; S3. Based on the abnormality recognition algorithm, the obtained target collection data is identified and analyzed, and the abnormal target collection data is marked; S4. Based on the evaluation algorithm, the identified and marked target collection abnormal data is compared with the known abnormal data in the database to evaluate the impact of the target collection abnormal data; S5. Based on the result of evaluating the impact of the target collection abnormal data, determine whether the target collection abnormal data meets the preset data collection standard.

2. The data collection method based on big data according to claim 1, characterized in that: The feature extraction algorithm is used to extract features from the acquired data to be collected, and obtaining the target collected data comprises the following steps: S21, initializing the data set to be collected, determining the data set to be extracted, and adjusting the data to be collected; S22, clustering the adjusted data to be collected, and determining a combination of data to be collected that has the greatest impact on the target collected data; S23, using the feature extraction algorithm to adjust the data to be collected to perform a new classification, determine the category to which each data to be collected belongs, and find the optimal similarity measurement between different classifications; S24, comparing the data to be collected obtained through the adjustment and classification operations with the standard requirements of the target collection data, and selecting the combination of data to be collected with the smallest deviation from the target collection data as the optimal data set to be collected; S25, determine whether the adjustment and screening process of the current data to be collected meets the preset termination condition. If so, terminate the adjustment and screening, take the current optimal data set to be collected as the target collection data and output it; otherwise, go to step S21 to continue processing the data to be collected.

3. The data collection method based on big data according to claim 2 is characterized in that: The method of reclassifying the data to be collected after being adjusted by the feature extraction algorithm, determining the category to which each data to be collected belongs, and finding the optimal similarity measurement between different classifications includes the following steps: S231, randomly selecting a number of adjusted data to be collected as initial classification clustering points, calculating the similarity measure between each unclassified data to be collected and all classification clustering points, and assigning the data to be collected to the classification clustering point with the highest similarity measure, thereby forming a preliminary classification result; S232, randomly selecting data points that are not used as classification clustering points from the adjusted data to be collected as potential new classification clustering point candidates; S233, using the randomly selected classification cluster point candidates, evaluating the overall error change caused by replacing the current classification cluster point, and updating the classification cluster point set according to the evaluation result; S234. When the classification clustering point no longer changes, determine the final category to which each data to be collected belongs, and calculate the optimal similarity measure between different categories.

4. The data collection method based on big data according to claim 1, characterized in that: The method of identifying and analyzing the obtained target acquisition data based on the abnormality recognition algorithm and marking the target acquisition abnormal data includes the following steps: S31, obtaining the historical status record of the target collected data and counting the number of abnormalities, and calculating the abnormal distribution ratio of the target collected data; S32. Analyze the abnormal distribution of the target collected data based on data mining technology, and draw a distribution frequency diagram of the abnormal proportion of the target collected data; S33, based on the distribution frequency diagram of the abnormal proportion of the target collected data, construct an abnormal data recognition model, and deploy the constructed abnormal data recognition model to the area with high incidence of abnormal data; S34. If the data in the high-incidence area of ​​abnormal data detected in the target collected data exceeds the threshold, the abnormal data recognition algorithm is used to identify and analyze the current target collected data, and the abnormal data points existing in the current target collected data are marked.

5. The data collection method based on big data according to claim 4 is characterized in that: If the data in the high-incidence area of ​​abnormal data exceeds the threshold value, the abnormal data points existing in the current target collected data are marked by using an abnormal recognition algorithm, and the abnormal data points are included in the following steps: S341, discretizing the target collected data, and simplifying the attributes of the target collected data dimensions using an attribute importance reduction algorithm; S342, initializing a pool including a plurality of analysis nodes, and setting a recognition threshold range for each analysis node; S343, selecting a target collection data record from the current target collection data, and selecting a number of analysis nodes that best match the current target collection data through a node optimization algorithm, and performing sampling analysis on the target collection data; S344, generating an abnormal identification signal from the target acquisition data according to the simplified attributes, extracting relevant signal values, calculating signal concentrations, and updating signal accumulation values ​​in the extracted analysis nodes; S345, for each analysis node, determine whether its signal strength is less than a preset recognition threshold, if less than the recognition threshold, return to S343 to continue sampling; S346, comparing the semi-mature signal and the mature signal of each analysis node, if the semi-mature signal is larger than the mature signal, marking the analysis node as an abnormal state node; if the semi-mature signal is smaller than the mature signal, marking the analysis node as a normal state node; S347. Determine whether the analysis of several analysis nodes has been completed in the target collection data. If the analysis is completed, calculate the abnormal evaluation value of the target collection data record; if the abnormal evaluation value is greater than the preset abnormal threshold, mark the target collection data as an abnormal data point, otherwise mark it as a normal data point.

6. The data collection method based on big data according to claim 5 is characterized in that: The step of selecting a target acquisition data record from the current target acquisition data, selecting a number of analysis nodes that best match the current target acquisition data through a node optimization algorithm, and performing sampling analysis on the target acquisition data includes the following steps: S3431. Setting the initial node pool size, number of optimization variables and termination conditions of the optimization process for target data collection and analysis; S3432, performing fitness evaluation on the set of analysis nodes currently selected from the node pool, and identifying the node combination that best matches the target collection data and the node combination that does not match the target collection data the most; S3433, based on the feature difference between the most matching node combination and the least matching node combination, using the updating rule of the node optimization algorithm to generate an updated analysis node set; S3434, comparing the fitness of the updated analysis node set with the previous analysis node set, if the updated node set has the best fitness, then replacing the old analysis node set, otherwise retaining the old analysis node set; S3435. If the termination condition of the optimization process is reached, the best analysis node set is output as the analysis nodes that best match the current target acquisition data; otherwise, return to step S3432 to continue optimizing the analysis node set.

7. The data collection method based on big data according to claim 6 is characterized in that: The update rule formula is: In the formula, Indicates J a,b,c The updated value of the node; J a,b,c Indicates the value of the ath node in the bth node pool size in the cth iteration; Indicates the value of the a-th node in the corresponding best matching node combination in the c-th iteration; It represents the value of the a-th node in the c-th iteration corresponding to the least matching node combination; o1 and o2 both represent random numbers.

8. The data collection method based on big data according to claim 1, characterized in that: Based on the evaluation algorithm, the identified and marked target acquisition abnormal data is compared with the known abnormal data in the database to evaluate the impact of the target acquisition abnormal data, including the following steps: S41, initializing the size of the candidate evaluation set of target collection abnormal data, setting the dynamic comparison probability and the maximum number of iterations, and selecting the initial best abnormal impact evaluation result, and calculating its impact score; S42. Based on the dynamic comparison probability, compare the size of the dynamic comparison probability and the random number. If the dynamic comparison probability is greater than the random number, generate a new abnormal impact assessment candidate solution through a cross-category integration formula; if the dynamic comparison probability is less than the random number, generate a new abnormal impact assessment candidate solution through a same-category integration formula; S43, applying the strategy optimization algorithm to the newly generated abnormal impact assessment candidate solution, and recalculating its impact score; S44, comparing the impact scores of the newly generated abnormal impact assessment candidate solution and the current optimal abnormal impact assessment result. If the impact score of the current optimal abnormal impact assessment result is less than the impact score of the newly generated assessment result, then replacing the impact score of the current optimal abnormal impact assessment result with the new solution; otherwise, keeping the impact score of the current optimal abnormal impact assessment result unchanged, and proceeding to the next round of iteration; S45. Check whether the maximum number of iterations has been reached. If so, output the impact score of the global current optimal abnormal impact assessment result as the impact degree of the abnormal data collected by the assessment target. If not, return to step S42 to continue optimization.

9. The data collection method based on big data according to claim 8, characterized in that: The step of applying the strategy optimization algorithm to the newly generated abnormal impact assessment candidate solution and recalculating its impact score comprises the following steps: S431, randomly generate an initial abnormal impact assessment candidate solution set, calculate the corresponding impact score of each candidate solution in the initial abnormal impact assessment candidate solution set, and perform fast non-dominated sorting on the candidate solutions, calculate the congestion degree of each solution, and form a sorted preferred candidate solution set; S432, based on the roulette wheel selection method, selecting the candidate solution with the highest influence score from the preferred candidate solution set, and duplicating it to form a new candidate solution set; S433, performing combination and perturbation operations on the new candidate solution set, eliminating the candidate solutions that do not meet the constraints, retaining the candidate solutions that meet the constraints, and regenerating the candidate solution data that meets the constraints using the initialization function; S434, merging the basic candidate solution set and the extended candidate solution set to form a new optimal candidate solution set, eliminating duplicate candidate solutions, and retaining only unique candidate solution data to form a non-duplicate candidate solution set; S435, performing fast non-dominated sorting on the non-repetitive candidate solution set, and calculating the congestion degree of each solution to obtain a new non-repetitive candidate solution data set; S436: Determine whether the current number of iterations reaches the threshold. If not, return to step S432 to continue iterating. If reached, output the final optimal candidate solution set as the optimization strategy result of abnormal impact assessment, and recalculate its impact score.

10. The data collection method based on big data according to claim 9, characterized in that: The formula for the roulette wheel selection method is: Where P i represents the cumulative probability of the ith alternative solution; m i represents the influence score of the ith alternative solution; m j represents the influence score of the jth alternative solution; n represents the total number of alternative solutions.

Images