Data acquisition processing method and system based on acquisition model

Abstract

The invention relates to the technical field of data processing, and discloses a data acquisition processing method and system based on an acquisition model, and the method comprises the steps: determining a plurality of pieces of source data of a data flow source based on the acquisition model, carrying out the priority analysis of each piece of source data in a data set of the same type, on the basis of the data format and the data state of each piece of source data in the source data chain, abnormal data in the source data chain is determined and removed, and whether first to-be-processed data and adjacent to-be-processed data are associated or not is determined according to data features and adjacent data features; according to the association score and the non-association score, whether the remaining data to be processed on the target source data chain is associated with the first data to be processed or not is sequentially judged, and the storage position of the associated data chain and the storage position of the non-associated data chain are determined according to the association score and the non-association score respectively. And the reliability of data processing is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data acquisition and processing method and system based on an acquisition model. Background Art

[0002] With the development of science and technology, we are faced with certain limitations when utilizing and collecting massive amounts of data from multiple channels such as radio, television, and media.

[0003] Chinese patent publication number CN111143651B discloses a data collection and analysis system for integrated operation and management of new media, including: resource management module, data collection module, data storage module, data processing module, data analysis module, data management module, and application service module; the data analysis module is used for data extraction, conversion, cleaning and loading processing, providing data inspection rule definition, data quality report, plan task management, and exception management functions, timely defining data processing solutions, and realizing the loading of massive data. It can be seen that in the process of data collection and processing, no processing is performed on the association between data, resulting in data loss or difficulty in query when data is stored or retrieved.

[0004] Therefore, it is necessary to design a data acquisition processing method and system based on the acquisition model to solve the problems existing in the current technology. Summary of the invention

[0005] In view of this, the present invention proposes a data collection and processing method and system based on an acquisition model, aiming to solve the problem that during the data collection and processing process, the association between data is not processed, resulting in data loss or query difficulty when storing or retrieving data.

[0006] In one aspect, the present invention provides a data acquisition and processing method based on an acquisition model, comprising:

[0007] Determine the source of the data stream, determine multiple source data of the source of the data stream based on the acquisition model, classify all the source data into the same type, determine a data set of the same type, perform a priority analysis on each source data in the data set of the same type, and construct a source data chain based on the result of the priority analysis;

[0008] Based on the data format and data status of each source data in the source data chain, determine and remove abnormal data in the source data chain, obtain a target source data chain, and use each remaining source data in the target source data chain as data to be processed;

[0009] Acquire the first data to be processed on the target source data chain and the adjacent data to be processed adjacent to the first data to be processed, extract the data features of the first data to be processed and the adjacent data features of the adjacent data to be processed, determine whether the first data to be processed and the adjacent data to be processed are associated according to the data features and the adjacent data features, and determine in turn whether the remaining data to be processed on the target source data chain are associated with the first data to be processed, and determine the associated data chain and the non-associated data chain based on the determination result;

[0010] The associated score is determined according to the amount of data to be processed in the associated data chain, and the non-associated score is determined according to the amount of data to be processed in the non-associated data chain. The storage locations of the associated data chain and the non-associated data chain are determined respectively according to the associated score and the non-associated score.

[0011] Further, when determining the multiple source data of the data stream source based on the acquisition model, it includes:

[0012] Acquire historical captured data, construct a data set based on the historical captured data, and sample a training set and a test set according to the data ratio;

[0013] Preselecting a neural network model, iteratively training the neural network model according to the training set, using grid search to find hyperparameters of the neural network model during the iterative training, and performing F1 score verification on the iteratively trained neural network model according to the test set;

[0014] If the F1 score of the neural network model after the current iterative training is less than the F1 score of the neural network model after the previous iterative training, the hyperparameters of the neural network model are adjusted, and iterative training is continued until the preset number of iterations is reached;

[0015] If the F1 score of the neural network model after the current iterative training is greater than or equal to the F1 score of the neural network model after the previous iterative training, the iterative training is stopped and the neural network model after the current iterative training is used as the acquisition model;

[0016] The data stream source is connected to the acquisition model to determine multiple source data.

[0017] Furthermore, when each source data in the same type of data set is subjected to a priority analysis and a source data chain is constructed based on the result of the priority analysis, it includes:

[0018] Substituting all source data in the same type of data set into the entropy model to determine the data entropy value of each source data;

[0019] When all data entropy values ​​are unique, all data entropy values ​​are sorted in order from large to small to construct the source data chain;

[0020] When there are data with the same entropy values, an entropy value set is established with the same data entropy values, a data entropy value is randomly selected from the entropy value set, an entropy value series is established with different data entropy values, and the randomly selected data entropy value is substituted into the entropy value series, the data entropy values ​​are arranged in descending order, and the source data chain is constructed, wherein each source data in the source data chain is recorded as data to be verified.

[0021] Furthermore, based on the data format and data status of each source data in the source data chain, determining and removing abnormal data in the source data chain to obtain a target source data chain includes:

[0022] Determine whether the data format of the data to be verified conforms to the source format of the data stream source;

[0023] Perform data fitting on all the data to be verified to determine a fitting curve, and respectively determine the data value ratio of the data to be verified and its two adjacent data to be verified, which are recorded as the first data ratio and the second data ratio;

[0024] Determine a data odds ratio of the first data odds and the second data odds of each to-be-verified data, and record it as a fitting value;

[0025] When there are no two adjacent data to be verified, the first data ratio or the second data ratio of an adjacent data to be verified is used as the fitting value;

[0026] Determine the data state of each to-be-verified data according to the fitting curve and the fitting value, wherein the data state includes a discrete state and a continuous state;

[0027] Determine whether the data format of the data to be verified conforms to the determination result of the source format of the data stream source and the data status according to the data format of the data to be verified, determine whether the data to be verified is abnormal data and eliminate it.

[0028] Further, when determining whether the data format of the data to be verified conforms to the determination result of the source format of the data stream source and the data state, determining whether the data to be verified is abnormal data and eliminating it includes:

[0029] Obtaining the fitting value of each data to be verified, determining an average fitting value, and determining a preset fitting range based on the average fitting value;

[0030] Determine a verification distance between each to-be-verified data and the fitting curve, and determine a data state of each to-be-verified data according to the verification distance and the fitting value;

[0031] When the verification distance is less than or equal to the preset distance, and the fitting value is within the preset fitting range, the data state is determined to be a continuous state, otherwise, the data state is a discrete state;

[0032] When the data format of the data to be verified conforms to the source format of the data stream source and its data state is a continuous state, the data to be verified is determined to be valid data;

[0033] When the data format of the data to be verified conforms to the source format of the data stream source, and / or its data state is a discrete state, the data to be verified is determined to be abnormal data and is removed.

[0034] Further, when determining whether the first data to be processed and the adjacent data to be processed are associated according to the data feature and the adjacent data feature, it includes:

[0035] Determine a feature data value corresponding to each data feature, and determine adjacent feature data values ​​of each adjacent data feature, wherein each feature data value corresponds to each adjacent feature data value one-to-one;

[0036] The association similarity is determined based on all feature data values ​​and all adjacent feature data values, and the association similarity is obtained by the following formula:

[0037]

[0038] Wherein, Q represents the correlation similarity, n represents the number of feature data values, Fi represents the i-th feature data value, and Gi represents the i-th adjacent feature data value;

[0039] Presetting an association similarity threshold, and when the association similarity is greater than or equal to the association similarity threshold, determining that the first data to be processed and the adjacent data to be processed are associated;

[0040] When the association similarity is less than the association similarity threshold, it is determined that there is no association between the first data to be processed and the adjacent data to be processed.

[0041] Further, when determining in sequence whether the remaining data to be processed on the target source data chain is associated with the first data to be processed, and determining the associated data chain and the non-associated data chain based on the determination result, it includes:

[0042] When the remaining data to be processed on the target source data chain is associated with the first data to be processed, determining an associated data chain and an unassociated data chain according to whether the first data to be processed is associated with the adjacent data to be processed;

[0043] If the first data to be processed is associated with the adjacent data to be processed, all the data to be processed on the target source data chain are constructed as an associated data chain; if the first data to be processed is not associated with the adjacent data to be processed, all the associated data to be processed on the target source data chain are constructed as an associated data chain, and the remaining unassociated data to be processed are constructed as non-associated data chains;

[0044] When the remaining data to be processed on the target source data chain is not associated with the first data to be processed, and the first data to be processed is associated with the adjacent data to be processed, the first data to be processed and the adjacent data to be processed are constructed as an associated data chain, and the remaining data to be processed that is not associated is constructed as a non-associated data chain;

[0045] When the remaining data to be processed on the target source data chain is not associated with the first data to be processed, and the first data to be processed is not associated with the adjacent data to be processed, the target source data chain is treated as a non-associated data chain.

[0046] Further, when determining the association score according to the amount of data to be processed in the association data chain, and determining the non-association score according to the amount of data to be processed in the non-association data chain, it includes:

[0047] Obtain the characteristic data value corresponding to each data characteristic of the data to be processed in the associated data chain, and determine the association score according to the following formula:

[0048]

[0049] Among them, A represents the association score, B represents the number of unprocessed data in the association data chain, j represents the jth unprocessed data in the association data chain, represents the kth characteristic data value of the jth data to be processed in the associated data chain;

[0050] Obtain the characteristic data value corresponding to each data characteristic of the data to be processed in the non-correlated data chain, and determine the non-correlated score according to the following formula:

[0051]

[0052] Where F represents the non-correlated score, H represents the number of unprocessed data in the non-correlated data chain, and x represents the xth unprocessed data in the non-correlated data chain. Indicates the yth characteristic data value of the xth data to be processed in the non-associated data chain, Indicates the maximum characteristic data value of the data to be processed in the non-associated data chain, Indicates the minimum characteristic data value of the data to be processed in a non-associated data chain.

[0053] Further, when the storage locations of the associated data chain and the non-associated data chain are determined respectively according to the associated score and the non-associated score, it includes:

[0054] The storage location includes a primary sub-storage repository, a secondary sub-storage repository, and a tertiary sub-storage repository;

[0055] Presetting a first preset score and a second preset score, wherein the first preset score is greater than the second preset score;

[0056] When the association score or the non-association score is greater than or equal to the first preset score, determining that the storage location of the association data chain or the non-association data chain is the first-level sub-storage repository;

[0057] When the association score or the non-association score is less than the first preset score and greater than the second preset score, determining to store the association data chain or the non-association data chain in the secondary sub-storage repository;

[0058] When the association score or the non-association score is less than or equal to the second preset score, it is determined that the storage location of the association data chain or the non-association data chain is the third-level sub-storage bank.

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows: by classifying multiple source data into the same type, the consistency of data structure and data type is ensured, priority analysis is adopted to ensure the priority of processing each source data in the data set, and the efficiency of data processing is improved, and abnormal data is screened and eliminated based on data format and data status to ensure the reliability and accuracy of data, and whether the first data to be processed and the adjacent data to be processed are associated according to data characteristics and adjacent data characteristics, and finally the associated data chain and the non-associated data chain are determined, ensuring the accuracy of data classification processing, quantifying the associated score and the non-associated score, comprehensively weighing the data of the associated data chain and the non-associated data chain, and ensuring the accuracy of data storage and call.

[0060] On the other hand, the present application also provides a data acquisition and processing system based on an acquisition model, which is applied to the above-mentioned data acquisition and processing method based on an acquisition model, including:

[0061] The acquisition module is configured to determine the source of the data stream, determine multiple source data of the source of the data stream based on the acquisition model, classify all the source data into the same type, determine a data set of the same type, perform a priority analysis on each source data in the data set of the same type, and construct a source data chain based on the result of the priority analysis;

[0062] A verification module is configured to determine and remove abnormal data in the source data chain based on the data format and data status of each source data in the source data chain, obtain a target source data chain, and use each remaining source data in the target source data chain as data to be processed;

[0063] an association module, configured to obtain the first data to be processed on the target source data chain and the adjacent data to be processed adjacent to the first data to be processed, extract the data features of the first data to be processed and the adjacent data features of the adjacent data to be processed, determine whether the first data to be processed and the adjacent data to be processed are associated according to the data features and the adjacent data features, and determine in turn whether the remaining data to be processed on the target source data chain are associated with the first data to be processed, and determine an associated data chain and an unassociated data chain based on the determination result;

[0064] The processing module is configured to determine an associated score according to the amount of data to be processed in the associated data chain, determine a non-associated score according to the amount of data to be processed in the non-associated data chain, and determine storage locations of the associated data chain and the non-associated data chain according to the associated score and the non-associated score, respectively.

[0065] It can be understood that the above-mentioned data acquisition and processing method and system based on the acquisition model have the same beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0067] Figure 1 A flow chart of a data acquisition and processing method based on an acquisition model provided by an embodiment of the present invention;

[0068] Figure 2 A functional block diagram of a data acquisition and processing system based on an acquisition model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0070] In some embodiments of the present application, see Figure 1 As shown, a data acquisition processing method based on an acquisition model includes:

[0071] S100: Determine the source of the data stream, determine multiple source data of the data stream source based on the acquisition model, and classify all source data into the same type, determine the same type of data set, perform priority analysis on each source data in the same type of data set, and build a source data chain based on the result of the priority analysis.

[0072] S200: Based on the data format and data status of each source data in the source data chain, determine and remove abnormal data in the source data chain, obtain a target source data chain, and use each remaining source data in the target source data chain as data to be processed.

[0073] S300: Obtain the first data to be processed on the target source data chain and the adjacent data to be processed that is adjacent to the first data to be processed, extract the data features of the first data to be processed and the adjacent data features of the adjacent data to be processed, determine whether the first data to be processed and the adjacent data to be processed are associated based on the data features and the adjacent data features, and determine in turn whether the remaining data to be processed on the target source data chain is associated with the first data to be processed, and determine the associated data chain and the non-associated data chain based on the judgment result.

[0074] S400: determining an associated score according to the amount of data to be processed in the associated data chain, determining a non-associated score according to the amount of data to be processed in the non-associated data chain, and determining storage locations of the associated data chain and the non-associated data chain respectively according to the associated score and the non-associated score.

[0075] Specifically, the data stream sources include radio, television, media and other channels. The acquisition model is used to capture multiple source data of the data stream source. The multiple source data include text data, image data and video data, etc. All source data are divided into the same type, and the source data is divided according to the source format when dividing. For example, all text data are established as a data set of the same type, or all image data are established as a data set of the same type. Ensure that the data is processed according to one type of data during the data processing process to avoid data distortion or errors caused by inconsistent data formats during processing. Priority analysis is performed on each source data in the same type of data set to build a source data chain. Priority analysis can remove duplicate data when using the acquisition model to capture the source of the data stream. For example, when two identical text data are captured, priority analysis can only retain one text data to avoid duplication of work and ensure priority in the data processing process. Only judging duplicate data will ignore whether the source data of the source data chain established by the acquisition model after capturing the source data meets the data format and data status. The acquisition model may capture abnormal data due to factors such as noise in the data transmission process and network environment interference. Verification is performed based on the data format and data status of each source data in the source data chain to identify whether the data format meets the source format and whether the data state is discrete. Then, abnormal data (format errors, discrete states, and damaged data) are eliminated to obtain the target source data chain. The target source data chain is the result after eliminating abnormal data. At this time, the remaining source data in the target source data chain is the result of eliminating abnormal data and deleting duplicate data. Each remaining source data in the target source data chain is used as data to be processed, which improves the accuracy of data processing.

[0076] It can be understood that, according to the data features of the first data to be processed and the adjacent data features of the adjacent data to be processed, it is confirmed whether the first data to be processed and the adjacent data to be processed are associated. The data features include data size, encoding method, timestamp, data change rate, etc. By judging whether the first data to be processed and the adjacent data to be processed are associated, the foundation is laid for whether the data to be processed can be stored together. When judging whether the remaining data to be processed on the target source data chain is associated with the first data to be processed, it is also judged whether there is an association according to the data features of the first data to be processed and the data features of other remaining data to be processed. Based on the judgment result, the associated data chain and the non-associated data chain are determined. The data to be processed in the associated data chain are all associated data, while the data to be processed in the non-associated data chain are all non-associated data. The associated score and the non-associated score are calculated to determine the storage location of the associated data chain and the non-associated data chain, so that the storage between the data is relatively independent. When retrieving, a category of data can also be retrieved according to the associated data chain, which improves the efficiency of data storage. Through the steps of data classification, abnormal elimination, and association analysis, the efficiency of data collection and processing is effectively improved, and the data storage structure is optimized, which lays a reliable foundation for subsequent retrieval and use.

[0077] In some embodiments of the present application, when determining multiple source data of a data stream source based on an acquisition model, it includes: obtaining historical captured data, constructing a data set based on the historical captured data, sampling a training set and a test set according to the data ratio, pre-selecting a neural network model, iteratively training the neural network model according to the training set, and using grid search to find hyperparameters of the neural network model during the iterative training, and verifying the F1 score of the iteratively trained neural network model according to the test set. If the F1 score of the neural network model after the current iterative training is less than the F1 score of the neural network model after the previous iterative training, the hyperparameters of the neural network model are adjusted, and iterative training is continued until a preset number of iterations is reached. If the F1 score of the neural network model after the current iterative training is greater than or equal to the F1 score of the neural network model after the previous iterative training, the iterative training is stopped, the neural network model after the current iterative training is used as the acquisition model, the data stream source is connected to the acquisition model, and multiple source data are determined.

[0078] Specifically, a data set is constructed based on historical captured data. The data set includes the historical methods of successfully capturing source data, data capture time, various types of source data, etc. Usually 50% to 70% of the data set is used as a training set, and the rest is used as a test set. The training set is used to train the neural network model, and the test set is used to evaluate the performance of the trained model to improve the generalization ability of the model. The neural network model contains different levels and types of activation functions to capture complex relationships and patterns between data. Grid search is used to find the hyperparameters of the neural network model in iterative training. Grid search finds the optimal hyperparameters of the model by traversing the hyperparameter combinations in the parameter space to improve the prediction performance of the model and avoid judgment errors. During each iteration, the model will try to learn the patterns and relationships of the data in the training set to improve its prediction or classification capabilities. After each iterative training, the F1 score of the model is verified using the data in the test set. If the F1 score of the neural network model after the current iterative training is less than the F1 score of the neural network model after the previous iterative training, it means that the model has not yet achieved a stable performance. In this case, it is necessary to adjust the hyperparameters of the neural network model and continue iterative training. If the F1 score of the neural network model after the current iterative training is greater than or equal to the F1 score of the neural network model after the previous iterative training, it means that the model performance has improved or remained stable. At this time, the iterative training can be stopped, indicating that the model can stably approach the optimal solution and output the source data, thereby improving the efficiency and stability of capturing the source data.

[0079] In some embodiments of the present application, when each source data in a data set of the same type is prioritized for analysis and a source data chain is constructed based on the result of the prioritized analysis, it includes: substituting all source data in the data set of the same type into an entropy value model to determine the data entropy value of each source data; when all data entropy values ​​are unique, sorting all data entropy values ​​in order from large to small to construct a source data chain; when there are identical data entropy values, establishing an entropy value set for the identical data entropy values, randomly extracting a data entropy value from the entropy value set, establishing an entropy value series for different data entropy values, and substituting the randomly extracted data entropy value into the entropy value series, arranging the data entropy values ​​in descending order to construct a source data chain, wherein each source data in the source data chain is recorded as data to be verified.

[0080] Specifically, the entropy model is pre-trained, and the training process is consistent with the above-mentioned neural network model. The data set used for training is based on different source data and the data entropy values ​​corresponding to the source data. According to the entropy model, the data entropy value of each source data can be obtained, and the source data chain is constructed by sorting the source data through the data entropy value, which effectively removes duplicate data in the source data and responds to different degrees of data processing needs, thereby ensuring the priority of data processing and providing reliable support for subsequent verification.

[0081] In some embodiments of the present application, based on the data format and data status of each source data in the source data chain, abnormal data in the source data chain is determined and eliminated to obtain a target source data chain, including: determining whether the data format of the data to be verified conforms to the source format of the data stream source, performing data fitting on all the data to be verified to determine a fitting curve, respectively determining the data value ratios of the data to be verified and its two adjacent data to be verified, recorded as the first data ratio and the second data ratio, determining the data ratio ratio of the first data ratio and the second data ratio of each data to be verified, recorded as the fitting value, when there are no two adjacent data to be verified, the first data ratio or the second data ratio of an adjacent data to be verified is used as the fitting value, and the data state of each data to be verified is determined according to the fitting curve and the fitting value, the data state includes a discrete state and a continuous state, determining whether the judgment result and the data state of the data format of the data to be verified conform to the source format of the data stream source, and determining whether the data to be verified is abnormal data and eliminating it.

[0082] Specifically, according to the data format of the data to be verified, it is determined whether it conforms to the source format of the data stream source. After confirming the target source data chain or using the acquisition model to capture, the data format may deviate from the source format. By comparison, it can be checked whether the data format of the data to be verified conforms to the source format. When fitting all the data to be verified to determine the fitting curve, Origin or Matlab is used to fit the data to determine the fitting curve. The source format represents the format of a type of data, such as: the source format of image data is represented by JPG, PNG, etc. Each data to be verified has a corresponding source format. According to the data format of the data to be verified, it is determined whether it conforms to the source format of the data stream source, which can avoid data processing errors caused by format inconsistency and ensure the stability of data processing. The data value ratio of the data to be verified and its two adjacent data to be verified is determined respectively, recorded as the first data ratio and the second data ratio. The first data ratio is the ratio of the data value of the current data to be verified to the data value of the previous data to be verified, and the second data ratio is the ratio of the data value of the current data to be verified to the data value of the next data to be verified.

[0083] It is understandable that when the data to be verified is the first data or the last data, there are not two adjacent data to be verified. If it is the first data, the second data ratio is used as the fitting value, and if it is the last data, the first data ratio is used as the fitting value. When each source data in the source data chain is captured by the acquisition model, due to factors such as network environment, bandwidth limitation and data packet loss, the data state of the data to be verified will be discrete, thereby affecting the data quality of the data to be verified. According to the judgment result of the source format of the data stream source and the comprehensive evaluation of the data state, it can effectively determine whether the data to be verified is abnormal data and eliminate it, thereby improving the efficiency of data processing.

[0084] In some embodiments of the present application, when determining whether the data format of the data to be verified conforms to the judgment result and data status of the source format of the data stream source, determining whether the data to be verified is abnormal data and eliminating it, it includes: obtaining the fitting value of each data to be verified, determining the average fitting value, determining the preset fitting range based on the average fitting value, determining the verification distance between each data to be verified and the fitting curve, determining the data status of each data to be verified according to the verification distance and the fitting value, when the verification distance is less than or equal to the preset distance, and the fitting value is within the preset fitting range, then the data status is determined to be a continuous state, otherwise, the data status is a discrete state, when the data format of the data to be verified conforms to the source format of the data stream source, and its data status is a continuous state, then the data to be verified is determined to be valid data, when the data format of the data to be verified conforms to the source format of the data stream source, and / or its data status is a discrete state, then the data to be verified is determined to be abnormal data and it is eliminated.

[0085] Specifically, the verification distance between each data to be verified and the fitting curve is determined. When the fitting curve is determined by data fitting through Origin or Matlab, the horizontal axis of the fitting curve is represented by the timestamp of the data to be verified, and the vertical axis of the fitting curve is the data value of each data to be verified. The verification distance is the straight-line distance vertically connecting the data to be verified and the Y-axis. The preset distance is preferably the variance to 1.5 times the variance of the data values ​​of all the data to be verified that constitute the fitting curve. The larger the preset distance is set, the more erroneous elimination will occur (i.e., a surge in abnormal data). The preset fitting range is determined based on the average fitting value. The left boundary of the preset fitting range is the difference between the average fitting value and the fitting value, and the right boundary of the preset fitting range is the sum of the average fitting value and the fitting value. Through the dual verification of the fitting value and the verification distance, the continuous state and discrete state of the data to be verified can be accurately identified, and the interference of abnormal data on subsequent data analysis can be avoided, thereby improving the accuracy and stability of data processing. The preset fitting range is determined based on the average fitting value, and the preset distance is determined according to the variance of the data values ​​of all the data to be verified. The data status of each data to be verified is dynamically determined, which avoids the judgment error caused by a single setting, helps to screen out abnormal data from the data to be verified, and ensures the data quality of the target source data chain.

[0086] In some embodiments of the present application, when determining whether the first to-be-processed data and the adjacent to-be-processed data are associated according to the data features and the adjacent data features, it includes: determining the feature data value corresponding to each data feature, determining the adjacent feature data value of each adjacent data feature, wherein each feature data value corresponds to each adjacent feature data value one-to-one, and determining the association similarity according to all feature data values ​​and all adjacent feature data values, and the association similarity is obtained by the following formula:

[0087]

[0088] Among them, Q represents the association similarity, n represents the number of feature data values, Fi represents the i-th feature data value, Gi represents the i-th adjacent feature data value, and an association similarity threshold is set in advance. When the association similarity is greater than or equal to the association similarity threshold, it is determined that the first data to be processed and the adjacent data to be processed are associated. When the association similarity is less than the association similarity threshold, it is determined that the first data to be processed and the adjacent data to be processed are not associated.

[0089] In some embodiments of the present application, when determining in sequence whether the remaining data to be processed on the target source data chain is associated with the first data to be processed, and determining the associated data chain and the non-associated data chain based on the determination result, it includes: when the remaining data to be processed on the target source data chain is associated with the first data to be processed, determining the associated data chain and the non-associated data chain according to whether the first data to be processed and the adjacent data to be processed are associated; if the first data to be processed and the adjacent data to be processed are associated, constructing all the data to be processed on the target source data chain into an associated data chain; if the first data to be processed and the adjacent data to be processed are not associated, constructing the target source data chain into an associated data chain. All associated data to be processed on the data chain are constructed as an associated data chain, and the remaining unassociated data to be processed are constructed as a non-associated data chain. When the remaining data to be processed on the target source data chain is not associated with the first data to be processed, and the first data to be processed and the adjacent data to be processed are associated, the first data to be processed and the adjacent data to be processed are constructed as an associated data chain, and the remaining unassociated data to be processed are constructed as a non-associated data chain. When the remaining data to be processed on the target source data chain is not associated with the first data to be processed, and the first data to be processed and the adjacent data to be processed are not associated, the target source data chain is used as a non-associated data chain.

[0090] Specifically, after eliminating abnormal data, the target source data chain is obtained, and each remaining source data in the target source data chain is used as data to be processed. At this time, each remaining source data is represented as data to be verified remaining after eliminating abnormal data. Since data verification has been performed, it is used as data to be processed. When determining whether the first data to be processed and the adjacent data to be processed are associated based on the data features and the features of the adjacent data, the determination is still based on the data value of the data to be processed, except that the data value used at this time is the feature data value corresponding to the data feature of the data to be processed. The data features of the data to be processed include data size, encoding method, timestamp, data change rate, etc., and the corresponding feature data values ​​can comprehensively reflect the association between the data. The adjacent feature data values ​​corresponding to the adjacent data features of the adjacent data to be processed are similar. Each feature data value corresponds to each adjacent feature data value one by one. By calculating the association similarity, it is possible to accurately measure whether the first data to be processed and the adjacent data to be processed are associated. The association similarity threshold is preferably 0.8. When judging in turn whether the remaining data to be processed on the target source data chain are associated with the first data to be processed, the judgment is also made based on the feature data value and the feature data value of the remaining data to be processed. The judgment process is consistent with the confirmation of the association similarity between the first data to be processed and the adjacent data to be processed, and will not be repeated here. By determining the associated data chain and the non-associated data chain, the associated data to be processed can be summarized and stored together during storage, thereby improving the efficiency of data storage without affecting the stability of the data. When the data is retrieved, the deviation caused by the loss of data is avoided, thereby improving the reliability and accuracy of data processing.

[0091] In some embodiments of the present application, when determining the association score according to the number of data to be processed in the associated data chain, and determining the non-association score according to the number of data to be processed in the non-association data chain, it includes: obtaining a feature data value corresponding to each data feature of the data to be processed in the associated data chain, and determining the association score according to the following formula:

[0092]

[0093] Among them, A represents the association score, B represents the number of unprocessed data in the association data chain, j represents the jth unprocessed data in the association data chain, Represent the kth characteristic data value of the jth data to be processed in the associated data chain, obtain the characteristic data value corresponding to each data feature of the data to be processed in the non-associated data chain, and determine the non-associated score according to the following formula:

[0094]

[0095] Where F represents the non-correlated score, H represents the number of unprocessed data in the non-correlated data chain, and x represents the xth unprocessed data in the non-correlated data chain. Indicates the yth characteristic data value of the xth data to be processed in the non-associated data chain, Indicates the maximum characteristic data value of the data to be processed in the non-associated data chain, Indicates the minimum characteristic data value of the data to be processed in a non-associated data chain.

[0096] In some embodiments of the present application, when the storage locations of associated data chains and non-associated data chains are determined respectively according to the associated scores and non-associated scores, it includes: the storage location includes a first-level sub-repository, a second-level sub-repository, and a third-level sub-repository, a first preset score and a second preset score are preset, the first preset score is greater than the second preset score, when the associated score or the non-associated score is greater than or equal to the first preset score, it is determined that the storage location of the associated data chain or the non-associated data chain is the first-level sub-repository, when the associated score or the non-associated score is less than the first preset score and greater than the second preset score, it is determined that the storage location of the associated data chain or the non-associated data chain is the second-level sub-repository, and when the associated score or the non-associated score is less than or equal to the second preset score, it is determined that the storage location of the associated data chain or the non-associated data chain is the third-level sub-repository.

[0097] Specifically, the first preset score is preferably 9, and the second preset score is preferably 4. By calculating the associated score and the non-associated score, the associated data chain or the non-associated data chain is stored in the corresponding sub-repository, thereby achieving accurate processing and management of the data. The number of feature data values ​​corresponding to the data to be processed in the associated data chain is 3, 5, and 4, respectively. The feature data values ​​are 2, 2, 4 (the first one); 2, 10, 5, 4, 3 (the second one), and 5, 4, 3, 5 (the third one). The associated score A=3*ln(49), A is 11.67, and the associated data chain is stored in the first-level sub-repository. The calculation of the non-associated score is not repeated here. The first-level sub-repository, the second-level sub-repository, and the third-level sub-repository are divided according to the encryption level. The higher the score, the more and higher the number of data to be processed and the feature data values ​​involved, and the higher the encryption level, which helps to reasonably allocate the data according to the degree of association and importance, thereby improving the reliability and stability of data processing.

[0098] In summary, the beneficial effects of the present invention are as follows: by classifying multiple source data into the same type, the consistency of data structure and data type is ensured, priority analysis is adopted to ensure the priority of processing each source data in the data set, and the efficiency of data processing is improved, and abnormal data is screened and eliminated based on data format and data status to ensure the reliability and accuracy of the data, and whether the first data to be processed and the adjacent data to be processed are associated according to data characteristics and adjacent data characteristics, and finally the associated data chain and the non-associated data chain are determined, ensuring the accuracy of data classification processing, quantifying the associated score and the non-associated score, and comprehensively weighing the data of the associated data chain and the non-associated data chain, ensuring the accuracy of data storage and call.

[0099] In another preferred embodiment based on the above embodiment, refer to Figure 2 As shown, this embodiment provides a data acquisition and processing system based on an acquisition model, which is applied to the above-mentioned data acquisition and processing method based on an acquisition model, including:

[0100] The acquisition module is configured to determine the source of the data stream, determine multiple source data of the data stream source based on the acquisition model, classify all source data into the same type, determine the same type of data set, perform priority analysis on each source data in the same type of data set, and build a source data chain based on the result of the priority analysis.

[0101] The verification module is configured to determine and eliminate abnormal data in the source data chain based on the data format and data status of each source data in the source data chain, obtain the target source data chain, and use each remaining source data in the target source data chain as data to be processed.

[0102] The association module is configured to obtain the first data to be processed on the target source data chain and the adjacent data to be processed adjacent to the first data to be processed, extract the data features of the first data to be processed and the adjacent data features of the adjacent data to be processed, determine whether the first data to be processed and the adjacent data to be processed are associated according to the data features and the adjacent data features, and determine in turn whether the remaining data to be processed on the target source data chain is associated with the first data to be processed, and determine the associated data chain and the non-associated data chain based on the judgment result.

[0103] The processing module is configured to determine the association score according to the number of data to be processed in the associated data chain, determine the non-association score according to the number of data to be processed in the non-association data chain, and determine the storage locations of the associated data chain and the non-association data chain respectively according to the association score and the non-association score.

[0104] Specifically, by classifying multiple source data into the same type, the consistency of data structure and data type is ensured. Priority analysis is used to ensure the priority of processing each source data in the data set, thereby improving the efficiency of data processing. The data format and data status are screened and abnormal data is eliminated to ensure the reliability and accuracy of the data. According to the data characteristics and adjacent data characteristics, it is determined whether the first data to be processed and the adjacent data to be processed are associated, and finally the associated data chain and the non-associated data chain are determined to ensure the accuracy of data classification processing. The associated score and the non-associated score are quantified, and the data of the associated data chain and the non-associated data chain are comprehensively weighed to ensure the accuracy of data storage and call.

[0105] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0106] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A data acquisition and processing method based on an acquisition model, characterized in that: include: Determine the source of the data stream, determine multiple source data of the source of the data stream based on the acquisition model, and classify all the source data into the same type, determine a data set of the same type, perform a priority analysis on each source data in the data set of the same type, and construct a source data chain based on the result of the priority analysis; Based on the data format and data status of each source data in the source data chain, determine and remove abnormal data in the source data chain, obtain a target source data chain, and use each remaining source data in the target source data chain as data to be processed; Acquire the first data to be processed on the target source data chain and the adjacent data to be processed adjacent to the first data to be processed, extract the data features of the first data to be processed and the adjacent data features of the adjacent data to be processed, determine whether the first data to be processed and the adjacent data to be processed are associated according to the data features and the adjacent data features, and determine in turn whether the remaining data to be processed on the target source data chain are associated with the first data to be processed, and determine the associated data chain and the non-associated data chain based on the determination result; The associated score is determined according to the amount of data to be processed in the associated data chain, and the non-associated score is determined according to the amount of data to be processed in the non-associated data chain. The storage locations of the associated data chain and the non-associated data chain are determined respectively according to the associated score and the non-associated score.

2. The data acquisition and processing method based on the acquisition model according to claim 1 is characterized in that: When determining the multiple source data of the data stream source based on the acquisition model, it includes: Acquire historical captured data, construct a data set based on the historical captured data, and sample a training set and a test set according to the data ratio; Preselecting a neural network model, iteratively training the neural network model according to the training set, using grid search to find hyperparameters of the neural network model during the iterative training, and performing F1 score verification on the iteratively trained neural network model according to the test set; If the F1 score of the neural network model after the current iterative training is less than the F1 score of the neural network model after the previous iterative training, the hyperparameters of the neural network model are adjusted, and iterative training is continued until the preset number of iterations is reached; If the F1 score of the neural network model after the current iterative training is greater than or equal to the F1 score of the neural network model after the previous iterative training, the iterative training is stopped and the neural network model after the current iterative training is used as the acquisition model; The data stream source is connected to the acquisition model to determine multiple source data.

3. The data acquisition and processing method based on the acquisition model according to claim 2 is characterized in that: When each source data in the same type of data set is subjected to a priority analysis and a source data chain is constructed based on the result of the priority analysis, it includes: Substituting all source data in the same type of data set into the entropy model to determine the data entropy value of each source data; When all data entropy values ​​are unique, all data entropy values ​​are sorted in order from large to small to construct the source data chain; When there are data with the same entropy values, an entropy value set is established with the same data entropy values, a data entropy value is randomly selected from the entropy value set, an entropy value series is established with different data entropy values, and the randomly selected data entropy value is substituted into the entropy value series, the data entropy values ​​are arranged in descending order, and the source data chain is constructed, wherein each source data in the source data chain is recorded as data to be verified.

4. The data acquisition and processing method based on the acquisition model according to claim 3 is characterized in that: When determining and removing abnormal data in the source data chain based on the data format and data status of each source data in the source data chain to obtain a target source data chain, the method includes: Determine whether the data format of the data to be verified conforms to the source format of the data stream source; Perform data fitting on all the data to be verified to determine a fitting curve, and respectively determine the data value ratio of the data to be verified and its two adjacent data to be verified, which are recorded as the first data ratio and the second data ratio; Determine a data odds ratio of the first data odds and the second data odds of each to-be-verified data, and record it as a fitting value; When there are no two adjacent data to be verified, the first data ratio or the second data ratio of an adjacent data to be verified is used as the fitting value; Determine the data state of each to-be-verified data according to the fitting curve and the fitting value, wherein the data state includes a discrete state and a continuous state; Determine whether the data format of the data to be verified conforms to the determination result of the source format of the data stream source and the data status according to the data format of the data to be verified, determine whether the data to be verified is abnormal data and eliminate it.

5. The data acquisition and processing method based on the acquisition model according to claim 4 is characterized in that: When determining whether the data format of the data to be verified conforms to the source format of the data stream source and the data status, determining whether the data to be verified is abnormal data and eliminating it, it includes: Obtaining the fitting value of each data to be verified, determining an average fitting value, and determining a preset fitting range based on the average fitting value; Determine a verification distance between each to-be-verified data and the fitting curve, and determine a data state of each to-be-verified data according to the verification distance and the fitting value; When the verification distance is less than or equal to the preset distance, and the fitting value is within the preset fitting range, the data state is determined to be a continuous state, otherwise, the data state is a discrete state; When the data format of the data to be verified conforms to the source format of the data stream source and its data state is a continuous state, the data to be verified is determined to be valid data; When the data format of the data to be verified conforms to the source format of the data stream source, and / or its data state is a discrete state, the data to be verified is determined to be abnormal data and is removed.

6. The data acquisition and processing method based on the acquisition model according to claim 5 is characterized in that: When determining whether the first data to be processed and the adjacent data to be processed are associated according to the data feature and the adjacent data feature, it includes: Determine a feature data value corresponding to each data feature, and determine adjacent feature data values ​​of each adjacent data feature, wherein each feature data value corresponds to each adjacent feature data value one-to-one; The association similarity is determined based on all feature data values ​​and all adjacent feature data values, and the association similarity is obtained by the following formula: Among them, Q represents the association similarity, n represents the number of feature data values, Fi represents the i-th feature data value, represents the i-th adjacent feature data value; Presetting an association similarity threshold, and when the association similarity is greater than or equal to the association similarity threshold, determining that the first data to be processed and the adjacent data to be processed are associated; When the association similarity is less than the association similarity threshold, it is determined that there is no association between the first data to be processed and the adjacent data to be processed.

7. The data acquisition and processing method based on the acquisition model according to claim 6 is characterized in that: When determining in sequence whether the remaining data to be processed on the target source data chain is associated with the first data to be processed, and determining the associated data chain and the non-associated data chain based on the determination result, the method includes: When the remaining data to be processed on the target source data chain is associated with the first data to be processed, determining an associated data chain and an unassociated data chain according to whether the first data to be processed is associated with the adjacent data to be processed; If the first data to be processed is associated with the adjacent data to be processed, all the data to be processed on the target source data chain are constructed as an associated data chain; if the first data to be processed is not associated with the adjacent data to be processed, all the associated data to be processed on the target source data chain are constructed as an associated data chain, and the remaining unassociated data to be processed are constructed as non-associated data chains; When the remaining data to be processed on the target source data chain is not associated with the first data to be processed, and the first data to be processed is associated with the adjacent data to be processed, the first data to be processed and the adjacent data to be processed are constructed as an associated data chain, and the remaining data to be processed that is not associated is constructed as a non-associated data chain; When the remaining data to be processed on the target source data chain is not associated with the first data to be processed, and the first data to be processed is not associated with the adjacent data to be processed, the target source data chain is treated as a non-associated data chain.

8. The data acquisition and processing method based on the acquisition model according to claim 7 is characterized in that: When determining the association score according to the amount of data to be processed in the association data chain, and determining the non-association score according to the amount of data to be processed in the non-association data chain, it includes: Obtain the characteristic data value corresponding to each data characteristic of the data to be processed in the associated data chain, and determine the association score according to the following formula: Among them, A represents the association score, B represents the number of data to be processed in the association data chain, and j represents the first data to be processed, represents the kth characteristic data value of the jth data to be processed in the associated data chain; Obtain the characteristic data value corresponding to each data characteristic of the data to be processed in the non-correlated data chain, and determine the non-correlated score according to the following formula: Where F represents the non-correlated score, H represents the number of unprocessed data in the non-correlated data chain, and x represents the xth unprocessed data in the non-correlated data chain. Indicates the yth characteristic data value of the xth data to be processed in the non-associated data chain, Indicates the maximum characteristic data value of the data to be processed in the non-associated data chain, Indicates the minimum characteristic data value of the data to be processed in a non-associated data chain.

9. The data acquisition and processing method based on the acquisition model according to claim 8 is characterized in that: When the storage locations of the associated data chain and the non-associated data chain are determined respectively according to the associated score and the non-associated score, the method includes: The storage location includes a primary sub-storage repository, a secondary sub-storage repository, and a tertiary sub-storage repository; Presetting a first preset score and a second preset score, wherein the first preset score is greater than the second preset score; When the association score or the non-association score is greater than or equal to the first preset score, determining that the storage location of the association data chain or the non-association data chain is the first-level sub-storage repository; When the association score or the non-association score is less than the first preset score and greater than the second preset score, determining to store the association data chain or the non-association data chain in the secondary sub-storage repository; When the association score or the non-association score is less than or equal to the second preset score, it is determined that the storage location of the association data chain or the non-association data chain is the third-level sub-storage bank.

10. A data acquisition and processing system based on an acquisition model, applied to the data acquisition and processing method based on an acquisition model as claimed in any one of claims 1 to 9, characterized in that: include: The acquisition module is configured to determine the source of the data stream, determine multiple source data of the source of the data stream based on the acquisition model, classify all the source data into the same type, determine a data set of the same type, perform a priority analysis on each source data in the data set of the same type, and construct a source data chain based on the result of the priority analysis; A verification module is configured to determine and remove abnormal data in the source data chain based on the data format and data status of each source data in the source data chain, obtain a target source data chain, and use each remaining source data in the target source data chain as data to be processed; an association module, configured to obtain the first data to be processed on the target source data chain and the adjacent data to be processed adjacent to the first data to be processed, extract the data features of the first data to be processed and the adjacent data features of the adjacent data to be processed, determine whether the first data to be processed and the adjacent data to be processed are associated according to the data features and the adjacent data features, and determine in turn whether the remaining data to be processed on the target source data chain are associated with the first data to be processed, and determine an associated data chain and an unassociated data chain based on the determination result; The processing module is configured to determine an associated score according to the amount of data to be processed in the associated data chain, determine a non-associated score according to the amount of data to be processed in the non-associated data chain, and determine storage locations of the associated data chain and the non-associated data chain according to the associated score and the non-associated score, respectively.

Images