Data processing method and system based on big data

Abstract

The embodiment of the invention provides a data processing method and system based on big data, and belongs to the technical field of data processing. The method comprises the steps that first data corresponding to a target object under a first sensor and second data corresponding to the target object under a second sensor are collected, the first data are used for representing structured data corresponding to the target object, and the second data are used for representing semi-structured data or non-structured data corresponding to the target object; performing structured processing on the second data to obtain third data, and performing data preprocessing on the third data to obtain fourth data; performing data preprocessing on the first data to obtain fifth data, and performing information fusion on the fourth data and the fifth data to obtain target data corresponding to the target object; and performing data analysis on the target data to obtain a target analysis result corresponding to the target object. The method solves the problem that in the prior art, valuable information cannot be extracted from the multi-source data, so that relevance between different data sources cannot be recognized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method and system based on big data. Background Art

[0002] In the era of big data, the amount of data is growing at an explosive rate, and the diversity of data sources is also increasing. Currently, data not only comes from traditional business systems, but also from IoT devices, various applications, servers, smart devices, and social media platforms. These emerging data sources bring rich content and opportunities to data analysis, but also raise many challenges. Despite the increasing diversity of data sources, existing technologies are unable to extract valuable information from these multi-source data, and thus cannot identify the correlation between different data sources. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide a data processing method and system based on big data, aiming to solve the problem that the prior art cannot extract valuable information from these multi-source data, and thus cannot identify the correlation between different data sources.

[0004] In a first aspect, an embodiment of the present invention provides a data processing method based on big data, comprising:

[0005] Collecting first data corresponding to a target object under a first sensor and second data corresponding to a target object under a second sensor, wherein the first data is used to characterize structured data corresponding to the target object, and the second data is used to characterize semi-structured data or unstructured data corresponding to the target object;

[0006] Performing structured processing on the second data to obtain third data, and performing data preprocessing on the third data to obtain fourth data;

[0007] Performing data preprocessing on the first data to obtain fifth data, and performing information fusion on the fourth data and the fifth data to obtain target data corresponding to the target object;

[0008] Perform data analysis on the target data to obtain a target analysis result corresponding to the target object.

[0009] In a second aspect, an embodiment of the present invention provides a data processing system based on big data, including:

[0010] A data acquisition module, used to acquire first data corresponding to a target object under a first sensor and second data corresponding to a second sensor, wherein the first data is used to characterize structured data corresponding to the target object, and the second data is used to characterize semi-structured data or unstructured data corresponding to the target object;

[0011] a data processing module, configured to perform structured processing on the second data to obtain third data, and perform data preprocessing on the third data to obtain fourth data;

[0012] a data fusion module, configured to perform data preprocessing on the first data to obtain fifth data, and perform information fusion on the fourth data and the fifth data to obtain target data corresponding to the target object;

[0013] The data analysis module is used to perform data analysis on the target data to obtain a target analysis result corresponding to the target object.

[0014] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any one of the data processing methods based on big data provided in the specification of the present invention are implemented.

[0015] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any data processing method based on big data provided in the specification of the present invention.

[0016] The embodiment of the present invention provides a data processing method and system based on big data, the method comprising: through the first data collected under the first sensor and the second data collected under the second sensor, all aspects of the target object can be fully understood, thereby ensuring the diversity and integrity of the data sources. The second data is structured to obtain the third data, and the third data is further preprocessed to obtain the fourth data, so that the semi-structured or unstructured data can be converted into a format that is easier to analyze and process, thereby improving the operability and analysis efficiency of the data. At the same time, the first data is preprocessed to obtain the fifth data. Information fusion of the fourth data and the fifth data helps to integrate the information of multi-source data, thereby obtaining more comprehensive and accurate target data. Finally, the target data is analyzed to extract valuable information and achieve more accurate target identification and decision support. This method solves the problem that it is impossible to extract valuable information from multi-source data and identify the correlation between data sources in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a data processing method based on big data provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the module structure of a data processing system based on big data provided by an embodiment of the present invention;

[0020] Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0023] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0024] The embodiment of the present invention provides a data processing method and system based on big data. The data processing method based on big data can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The terminal device can be a server or a server cluster.

[0025] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0026] Please refer to Figure 1 , Figure 1 A flowchart of a data processing method based on big data provided by an embodiment of the present invention.

[0027] like Figure 1 As shown, the data processing method based on big data includes steps S101 to S104.

[0028] Step S101, collecting first data corresponding to a target object under a first sensor and second data corresponding to a target object under a second sensor, wherein the first data is used to characterize structured data corresponding to the target object, and the second data is used to characterize semi-structured data or unstructured data corresponding to the target object.

[0029] Exemplarily, the target object for which data needs to be collected and the type of sensor to be used are determined, and it is ensured that the selected sensor can effectively capture the structured data (first sensor) and semi-structured or unstructured data (second sensor) required by the target object.

[0030] Exemplarily, the first sensor and the second sensor are started to collect data from the target object. Ensure that the sensors work normally and can stably record data. Store the collected first data (structured data) and second data (semi-structured or unstructured data). Set up a suitable data storage mechanism for subsequent processing and analysis.

[0031] For example, real-time data is collected through IoT devices or other sensors; log information of applications, servers and devices is collected; structured or unstructured data is collected from social media platforms; data is collected from websites through web crawlers. Specifically, IoT devices include temperature sensors, humidity sensors, pressure sensors, etc. The real-time data collected by these sensors include parameters such as temperature, humidity, and pressure. Log information of applications, servers and devices includes operation records, error logs, performance monitoring data, etc. Data from social media platforms include messages, comments, likes, etc. posted by users. Web crawlers collect news, blogs, forum posts and other data from websites.

[0032] Step S102: perform structured processing on the second data to obtain third data, and perform data preprocessing on the third data to obtain fourth data.

[0033] Exemplarily, the second data is usually semi-structured or unstructured, such as text, images, or videos. Before starting processing, it is necessary to clarify the basic characteristics and content type of the data in order to select an appropriate processing method. Useful information is extracted from the second data. This may include identifying keywords in text, extracting features from images, or extracting frames from videos. The extracted information is converted into a structured format, such as a table, a database record, or a data model with clear fields. This may involve parsing text data into structured fields, converting image features into numerical values, etc. The extracted and converted data is integrated into a unified structured data set to obtain the third data.

[0034] Exemplarily, the third data is cleaned, including removing duplicates, processing missing values, correcting errors and abnormal data, etc. This is to ensure the quality and consistency of the data. The data is standardized, such as normalizing numerical data, converting categorical data into a unified format, etc., so that the data can be processed consistently in subsequent analysis. The data is converted, such as converting the date and time format into a unified format, encoding text data into digital representation, etc. This can improve the analysis efficiency and accuracy of the data. If necessary, the data can be enhanced, such as by generating additional features, combining other data sources, etc., to increase the expressiveness of the data, thereby obtaining the corresponding fourth data under data preprocessing.

[0035] For example, the third data is subjected to preprocessing processes such as cleaning, conversion, integration and dimensionality reduction, including removing or correcting noise, missing values ​​and outliers in the data; standardizing, normalizing and encoding the data; merging and integrating data from different sources; and using PCA technology to reduce the data dimension to obtain the fourth data. Specifically, during the data cleaning process, data cleaning tools are used to remove or correct noise, missing values ​​and outliers. During the data conversion process, standardization, normalization, encoding and other technologies are used to convert the data into a format suitable for analysis. During the data integration process, ETL (Extract, Transform, Load) tools are used to merge and integrate data from different sources. PCA technology projects the data into a new coordinate system through linear transformation, so that the data dimension in the new coordinate system is reduced, thereby obtaining the fourth data.

[0036] Step S103: preprocess the first data to obtain fifth data, and fuse the fourth data with the fifth data to obtain target data corresponding to the target object.

[0037] Exemplarily, the fifth data is obtained by performing data preprocessing such as noise identification, missing value supplementation, and normalization processing on the first data.

[0038] Exemplarily, if the fourth data is in a dictionary format, the fourth data consists of a key and a value, and each item is a key-value pair. If the fifth data is also in a dictionary format, the fifth data consists of a key and a value, and each item is a key-value pair, and then the keys of the fourth data and the fifth data are subjected to intersection processing, so that the values ​​corresponding to the same key in the fourth data and the fifth data are weighted, and then the fourth data and the fifth data have the same key. The first fusion result; when the fourth data and the fifth data have different keys, the data is directly merged to obtain the second fusion result. Finally, the first fusion result and the second fusion result are determined as the target data corresponding to the target object.

[0039] The data format of the fourth data is a dictionary, in which each item is a key-value pair, a key (key) and a corresponding value (value). The data format of the fifth data is also a dictionary, which is also composed of key-value pairs, and each item is also a key (key) and a corresponding value (value). All keys are extracted from the fourth data and the fifth data to form two sets respectively. Find the common keys in these two sets, that is, the intersection part. The common key is the key that exists in both the fourth data and the fifth data. For each common key, extract the corresponding value from the fourth data and the fifth data. According to a predefined weighting strategy (such as weighted average, weighted sum, etc.), these values ​​are weighted to generate weighted values. The processed weighted values ​​are combined with the corresponding common keys to form a new dictionary, which represents the first fusion result.

[0040] Exemplarily, unique keys in the fourth data and the fifth data are found, that is, keys that exist in one dictionary but not in the other dictionary. Unique keys and their corresponding values ​​are extracted from the fourth data and the fifth data. These unique keys and their values ​​are directly merged into the result to form a second fusion result.

[0041] Exemplarily, the first fusion result includes all the keys that are present in both the fourth data and the fifth data, after weighted processing. The second fusion result includes all the keys and their values ​​that are unique to the fourth data and the fifth data. The first fusion result and the second fusion result are merged to form the target data corresponding to the complete target object. This includes all processed key-value pairs, and takes into account the weighted processing of common keys and the direct merging of unique keys. In this way, the fourth data and the fifth data can be effectively fused to obtain the target data corresponding to the target object. This process ensures that data with the same key is weighted and data with different keys are directly merged, thereby providing a comprehensive and consistent fusion result.

[0042] In some embodiments, the data preprocessing of the first data to obtain the fifth data includes: identifying missing values ​​of the first data to obtain missing data; obtaining relevant data corresponding to the missing data from the first data, and then calculating the mean result corresponding to the relevant data; filling the first data according to the mean result to obtain intermediate data, and normalizing the intermediate data to obtain normalized data; using a principal component analysis algorithm to perform dimensionality reduction on the normalized data to obtain the fifth data corresponding to the first data.

[0043] Exemplarily, the first data is checked to identify which data items have missing values. Missing values ​​are usually expressed as blanks, NA, null, etc. The location of the missing values ​​and the related fields are marked for subsequent processing. According to the location of the missing values, data associated with the missing data are extracted from the first data, which data can be other field values ​​in the same record, or data from other records related to the missing fields. The related data are organized into a data set or list. These related data are statistically analyzed to calculate the mean. The mean can be calculated by a simple average value, or by using methods such as weighted average. Based on the calculated mean result, the missing values ​​are replaced with the mean result, thereby filling the missing parts in the first data. After the filling is completed, a new data set, i.e., intermediate data, is obtained, in which there are no missing values.

[0044] Exemplarily, a suitable normalization method (such as minimum-maximum normalization, standardization, etc.) is selected according to the characteristics and requirements of the data. All values ​​in the intermediate data are normalized to conform to a specified range or distribution, thereby obtaining normalized data.

[0045] Exemplarily, the covariance matrix of the normalized data is first calculated. By performing eigenvalue decomposition on the covariance matrix, eigenvalues ​​and eigenvectors are obtained. The main eigenvectors, i.e., principal components, are selected according to the size of the eigenvalues. The first few principal components are usually selected to explain the variance of most of the data. The normalized data is projected onto the selected principal components to complete the dimensionality reduction operation. The data processed by PCA is sorted into the final format to obtain the fifth data.

[0046] Specifically, the first data is completely processed from missing value processing, data filling, normalization to dimensionality reduction, and finally the fifth data corresponding to the first data is obtained. This process not only ensures the integrity and consistency of the data, but also extracts the main features of the data through dimensionality reduction operations.

[0047] In some embodiments, the information fusion of the fourth data and the fifth data to obtain the target data corresponding to the target object includes: using a feature characterization model to vectorize the fourth data to obtain a first characterization vector and using the feature characterization model to vectorize the fifth data to obtain a second characterization vector; determining a first weight corresponding to the first characterization vector and a second weight corresponding to the second characterization vector; determining a third characterization vector corresponding to the target object when the first data is collected and determining a fourth characterization vector corresponding to the target object when the second data is collected; determining a first error corresponding to the target object based on the first weight, the first characterization vector and the third characterization vector; determining a second error corresponding to the target object based on the second weight, the second characterization vector and the fourth characterization vector; correcting the first weight based on the first error to obtain a third weight, and correcting the second weight based on the second error to obtain a fourth weight; information fusion of the first characterization vector and the second characterization vector based on the third weight and the fourth weight to obtain the target data corresponding to the target object.

[0048] Exemplarily, the fourth data is processed using a feature representation model (e.g., a word embedding model, a deep learning model, etc.). The fourth data is converted into a vector representation, i.e., a first representation vector is obtained. These vectors capture the characteristics and information of the data. The fifth data is processed using the feature representation model in the same manner. The fifth data is converted into a vector representation, i.e., a second representation vector is obtained.

[0049] Exemplarily, a first weight corresponding to the first characterization vector and a second weight corresponding to the second characterization vector are determined based on expert experience according to a data source corresponding to the fourth data and a data source corresponding to the fifth data.

[0050] Exemplarily, a feature representation model (e.g., a word embedding model, a deep learning model, etc.) is used to determine a third representation vector corresponding to the target object when the first data is collected, and a feature representation model (e.g., a word embedding model, a deep learning model, etc.) is used to determine a fourth representation vector corresponding to the target object when the second data is collected.

[0051] Exemplarily, each element of the first characterization vector is subtracted from the corresponding element of the third characterization vector, and the result is a new vector representing the difference between the two, and the vector difference is obtained, and then the vector difference obtained by the above calculation is multiplied element by element with the first weight. This process can be understood as a weighted operation on the vector difference, and the result is a weighted difference vector. Finally, the mean or sum of these weighted difference vectors is calculated to obtain a scalar value representing the first error. This reflects the weighted difference between the first characterization vector and the third characterization vector.

[0052] Exemplarily, a vector difference between the second characterization vector and the fourth characterization vector is obtained. Again, this can be obtained by element-by-element subtraction to form a new difference vector. The vector difference is element-by-element multiplied by the second weight to weight the difference. The result is a weighted difference vector. Finally, the mean or sum of these weighted difference vectors is calculated to obtain a scalar value representing the second error. This reflects the weighted difference between the second characterization vector and the fourth characterization vector.

[0053] Exemplarily, based on the analysis result of the first error, it is determined how to adjust the first weight. Common correction methods include increasing or decreasing the weight value, or adjusting the weight in proportion to the specific value of the error. The determined correction strategy is applied to the first weight to generate a new weight value, i.e., the third weight. This can be achieved by calculating a correction factor and combining it with the original weight so that the first error is minimized under the third weight.

[0054] Exemplarily, based on the analysis result of the second error, determine how to adjust the second weight. Common correction methods include increasing or decreasing the weight value, or adjusting the weight in proportion to the specific value of the error. Apply the determined correction strategy to the second weight to generate a new weight value, i.e., the fourth weight. This can be achieved by calculating a correction factor and combining it with the original weight so that the second error is minimized under the fourth weight.

[0055] Exemplarily, the modified third weight and the fourth weight are used to perform weighted processing on the first characterization vector and the second characterization vector, respectively. The first characterization vector is multiplied by the third weight to obtain a weighted first characterization vector. This weighted vector reflects the influence of the modified weight on the first characterization vector. The second characterization vector is multiplied by the fourth weight to obtain a weighted second characterization vector. This weighted vector reflects the influence of the modified weight on the second characterization vector. The weighted first characterization vector and the second characterization vector are fused to obtain the target data corresponding to the target object.

[0056] Specifically, by determining the weights associated with the representation vectors, the influence of each feature on the target object can be reflected. These weights help the model to more accurately identify and utilize important features, thereby improving prediction accuracy. By correcting the weights according to the first error and the second error, the importance of the features can be dynamically adjusted. This correction mechanism can improve the robustness of the model, making it more stable when facing different data situations. By correcting the weights, the feature fusion process can be optimized. The corrected weights can more reasonably combine different representation vectors, thereby more accurately reflecting the comprehensive characteristics of the target object. Therefore, by fusing the representation vectors according to the corrected weights, information from different data sources can be fully utilized. This fusion can improve the comprehensiveness and depth of data analysis, make the representation of target data more comprehensive and accurate, and provide more reliable support for data-driven decision-making.

[0057] In some embodiments, the information fusion of the first characterization vector and the second characterization vector according to the third weight and the fourth weight to obtain the target data corresponding to the target object includes: obtaining a first data mean corresponding to the target object under the first sensor and obtaining a second data mean corresponding to the target object under the second sensor; obtaining a first sum result by summing the first characterization vector and the first data mean and obtaining a second sum result by summing the second characterization vector and the second data mean; and obtaining the target data corresponding to the target object by information fusion of the first sum result and the second sum result using the third weight and the fourth weight.

[0058] Exemplarily, a first data mean corresponding to historical data collected by the target object under the first sensor is obtained, and a second data mean corresponding to historical data collected by the target object under the second sensor is obtained, and then the first characterization vector and the first data mean are summed to obtain a first sum result, and the second characterization vector and the second data mean are summed to obtain a second sum result.

[0059] Exemplarily, each element of the first sum result is divided by the smallest first error. This step is to standardize the sum result so that it is not affected by the size of the original error, thereby more accurately reflecting the true characteristics of the data. Next, the result obtained in the previous step is multiplied by the third weight to obtain the adjusted first result. This result comprehensively considers the characteristics of the first data, the error of the model, and the weight factor.

[0060] Exemplarily, each element of the second sum result is divided by the smallest second error. This step is also to standardize the data and eliminate the influence of the error on the result. The standardized second sum result is multiplied by the fourth weight. This weight reflects the relative importance of the second data in the final target data, and an adjusted second result is obtained. This result combines the characteristics of the second data, model performance, and weight information. Finally, the adjusted first result and the second result are summed. This step is to combine the information in the two results to form a more comprehensive and accurate target data.

[0061] Specifically, by calculating the mean of the data under the first sensor and the second sensor, a comprehensive data view of the target object can be obtained. Mean calculation helps to eliminate accidental noise and outliers, making the data more stable and reliable. By processing the first sum result and the second sum result, the characteristics of the target object under different sensors can be accurately expressed. This processing method makes the data expression more accurate and reduces the deviation that may occur due to different data sources. Using the third weight and the fourth weight to fuse the first sum result and the second sum result helps to comprehensively consider the characteristics and importance of different sensor data. Weight adjustment can reasonably distribute the contribution of different features to the target data, thereby improving the expression ability of the final target data.

[0062] Step S104: performing data analysis on the target data to obtain a target analysis result corresponding to the target object.

[0063] Exemplarily, TensorFlow, PyTorch, or SparkMLlib is used to train machine learning models. These tools provide a rich library of algorithms and powerful computing power to help extract valuable information from data. Select and construct appropriate features based on the characteristics of the target data for input into the machine learning model. This may include extracting features from the original data, selecting important features, etc. Select an appropriate machine learning model, such as a regression model, a classification model, or a clustering model, based on the analysis goal. For example, decision trees, random forests, or support vector machines (SVMs) can be used for classification tasks, while K-means or hierarchical clustering can be used for clustering tasks.

[0064] For example, data can be grouped using clustering techniques (such as K-means, DBSCAN, or hierarchical clustering) to discover natural structures and patterns in the data. Cluster analysis helps identify similar data points or behavior patterns in the target data. Or use classification techniques (such as logistic regression, decision trees, or neural networks) to predict labels for the target data. This can help identify and classify the characteristics or behaviors of the target object. Or use association rule techniques (such as Apriori algorithm, FP-Growth) to discover the relationship or association pattern between items in the target data. This helps understand the implicit rules and relationships in the target data.

[0065] In some embodiments, the data analysis of the target data to obtain the target analysis result corresponding to the target object includes: determining a clustering algorithm corresponding to the target data, and performing data clustering on the target data according to the clustering algorithm to obtain an initial cluster; performing classification analysis on the initial cluster using a data classification model to obtain a target type corresponding to the initial cluster; performing data screening on the initial cluster according to the target type to obtain a target cluster; and performing data association analysis on the target cluster to obtain the target analysis result corresponding to the target object.

[0066] Exemplarily, a suitable clustering algorithm is selected according to the characteristics of the target data (such as the size, distribution and structure of the data). Factors to be considered include the dimension of the data, the shape of the clusters and the expected number of clusters. The target data is clustered according to the selected clustering algorithm. The algorithm divides the data into multiple initial clusters, each of which contains similar data points.

[0067] Exemplarily, an appropriate classification model is selected based on the characteristics of the initial clusters and the analysis objectives. Commonly used classification models include logistic regression, support vector machine (SVM), decision tree, random forest, and neural network. The classification model is trained using labeled data. The labeled data should include label information of the initial clusters so that the model can learn the relationship between the characteristics of the clusters and the target types. The trained classification model is applied to the initial clusters for classification analysis to predict the target type corresponding to each cluster. This step can help identify the role or significance of each cluster in the target analysis.

[0068] Exemplarily, according to the target type obtained by the classification model, the initial clusters that meet the specific target type are screened out. This process can help focus on clusters of specific categories for further analysis. Select the cluster corresponding to the target type from the initial cluster to form a target cluster. The target cluster contains data related to the target type, which will be used for subsequent association analysis. Select a suitable association analysis method, such as association rule mining (Apriori, FP-Growth, etc.) to discover the relationship between items in the data. Association analysis aims to reveal the potential relationship and pattern between data items in the target cluster. Apply the selected association analysis method to analyze the data of the target cluster and mine meaningful association rules or patterns. These rules can reveal common combinations or relationships between data items and help understand the characteristics and behaviors of the target object. Determine the target analysis results of the target object based on the results of the association analysis. The target analysis results can include the discovered laws, patterns, and relationships between data items, etc., providing valuable insights for subsequent decisions and strategies.

[0069] Specifically, by implementing the above steps, the accuracy, efficiency and effectiveness of data analysis can be significantly improved. By selecting a suitable clustering algorithm, accurately identifying the target type, and optimizing the data screening and analysis process, the associations and patterns in the data can be deeply explored, thereby providing stronger support for business decisions and strategies. These steps help to understand the inherent laws of the data, improve the interpretability and practicality of the analysis results, and thus achieve more targeted and efficient data-driven decisions.

[0070] In some embodiments, the method also includes: encrypting the first data to obtain a first encryption result and encrypting the second data to obtain a second encryption result; determining a database and storing the first encryption result and the second encryption result in the database to obtain a target storage result; and setting access rights for the target storage result.

[0071] For example, encryption technology is used to protect data during transmission and storage; role management and permission control are used to ensure that only authorized users can access data; and regulations such as GDPR are followed to ensure that data processing complies with relevant legal requirements. Specifically, encryption technologies include SSL / TLS protocols, AES encryption algorithms, etc., which are used to protect the security of data during transmission and storage. Role management and permission control are used to limit data access rights to ensure that only authorized users can access data. Regulations such as GDPR are used to regulate data processing behaviors and ensure that data processing complies with relevant legal requirements.

[0072] Exemplarily, an encryption algorithm is determined, and then the encryption algorithm is used to encrypt the first data to obtain a first encryption result, and the encryption algorithm is used to encrypt the second data to obtain a second encryption result; thereby determining a database, and storing the first encryption result and the second encryption result in the database to obtain a target storage result; and access permissions are set for the target storage result to ensure that only authorized users can access the data.

[0073] Exemplarily, Hadoop Distributed File System (HDFS) is used to store and manage large-scale unstructured data; Cassandra database is used to store and manage structured or semi-structured data; AWS S3 cloud storage service is used to store and manage data. Specifically, HDFS has a large storage capacity and can handle large-scale data, and is suitable for storing unstructured data such as pictures, videos, log files, etc. Cassandra database has high availability and high scalability, and is suitable for storing structured or semi-structured data such as user information, transaction records, etc. AWS S3 cloud storage service provides a secure and scalable storage solution suitable for storing various types of data.

[0074] Specifically, by encrypting the first data and the second data, storing the encrypted data in a database, and setting access permissions, the security protection level of the data can be effectively improved. These measures help prevent data leakage, tampering, and unauthorized access, while meeting compliance requirements and enhancing user trust. Ultimately, these measures improve the security of data management, reduce security risks, and increase the transparency and credibility of data processing.

[0075] In some embodiments, the method also includes: performing data analysis on the fourth data to obtain a first change trend corresponding to the target object; performing data analysis on the fifth data to obtain a second change trend corresponding to the target object; determining a visualization tool, displaying the first change trend and the second change trend in the visualization tool, and obtaining a visualization result corresponding to the target object.

[0076] Exemplarily, such as moving average, exponential smoothing, ARIMA model, etc., are used to identify trends, seasonality and periodicity in the fourth data. Or used to model the relationship between the fourth data and predict the change of trend. Statistical methods such as mean, standard deviation, variance analysis, etc. can also be used to summarize the basic characteristics of the fourth data. Calculate key statistical indicators or model parameters to describe and explain the trend of data changes. Analyze trends, periodic changes or other patterns in the data, for example, by drawing time series graphs, trend lines, etc. Summarize the obtained change trends, including descriptions such as growth, decline, stability, etc., and obtain the first change trend corresponding to the target object.

[0077] Similarly, use methods such as moving average, exponential smoothing, ARIMA model, etc. to identify trends, seasonality and periodicity in the fifth data. Or use them to model the relationship between the fifth data and predict changes in trends. Statistical methods such as mean, standard deviation, variance analysis, etc. can also be used to summarize the basic characteristics of the fifth data. Calculate key statistical indicators or model parameters to describe and explain the trend of data changes. Analyze trends, periodic changes or other patterns in the data, such as by drawing time series graphs, trend lines, etc. Summarize the obtained change trends, including descriptions such as growth, decline, stability, etc., and obtain the second change trend corresponding to the target object.

[0078] For example, according to the characteristics of the data and the requirements of the final display, select a suitable visualization tool, such as Tableau, Power BI, Excel, Python visualization library (such as Matplotlib, Seaborn), Google DataStudio, etc., so as to import the analysis results of the first change trend and the second change trend into the selected visualization tool. Ensure that the data format meets the requirements of the tool. Thus, obtain the visualization result corresponding to the target object.

[0079] Specifically, through the above steps, the fourth and fifth data can be analyzed in detail and the results can be displayed in the visualization tool. This process includes data preparation, selecting appropriate analysis methods, obtaining change trends, selecting visualization tools, and the final visualization display. The entire process helps to deeply understand the trend of data changes, and at the same time, the visualization effect makes the data more intuitive and easy to understand, which is convenient for decision-making and communication.

[0080] The embodiment of the present invention provides a data processing method and system based on big data, the method comprising: through the first data collected under the first sensor and the second data collected under the second sensor, all aspects of the target object can be fully understood, thereby ensuring the diversity and integrity of the data sources. The second data is structured to obtain the third data, and the third data is further preprocessed to obtain the fourth data, so that the semi-structured or unstructured data can be converted into a format that is easier to analyze and process, thereby improving the operability and analysis efficiency of the data. At the same time, the first data is preprocessed to obtain the fifth data. Information fusion of the fourth data and the fifth data helps to integrate the information of multi-source data, thereby obtaining more comprehensive and accurate target data. Finally, the target data is analyzed to extract valuable information and achieve more accurate target identification and decision support. This method solves the problem that it is impossible to extract valuable information from multi-source data and identify the correlation between data sources in the prior art.

[0081] See also Figure 2 , Figure 2 A data processing system 200 based on big data is provided in an embodiment of the present application. The data processing system 200 based on big data includes a data acquisition module 201, a data processing module 202, a data fusion module 203, and a data analysis module 204, wherein the data acquisition module 201 is used to collect first data corresponding to a target object under a first sensor and second data corresponding to a second sensor, wherein the first data is used to characterize structured data corresponding to the target object, and the second data is used to characterize semi-structured data or unstructured data corresponding to the target object; the data processing module 202 is used to perform structured processing on the second data to obtain third data, and perform data preprocessing on the third data to obtain fourth data; the data fusion module 203 is used to perform data preprocessing on the first data to obtain fifth data, and perform information fusion on the fourth data and the fifth data to obtain target data corresponding to the target object; the data analysis module 204 is used to perform data analysis on the target data to obtain a target analysis result corresponding to the target object.

[0082] In some implementations, the data fusion module 203 performs, during the process of preprocessing the first data to obtain the fifth data:

[0083] Performing missing value identification on the first data to obtain missing data;

[0084] Obtain relevant data corresponding to the missing data from the first data, and then calculate the mean result corresponding to the relevant data;

[0085] Filling the first data according to the mean value result to obtain intermediate data, and normalizing the intermediate data to obtain normalized data;

[0086] A principal component analysis algorithm is used to perform dimensionality reduction processing on the normalized data to obtain the fifth data corresponding to the first data.

[0087] In some implementations, the data fusion module 203 performs, during the process of fusing the fourth data and the fifth data to obtain the target data corresponding to the target object:

[0088] Performing vector representation on the fourth data using the feature representation model to obtain a first representation vector, and performing vector representation on the fifth data using the feature representation model to obtain a second representation vector;

[0089] Determine a first weight corresponding to the first characterization vector and a second weight corresponding to the second characterization vector;

[0090] Determine a third characterization vector corresponding to the target object when the first data is collected, and determine a fourth characterization vector corresponding to the target object when the second data is collected;

[0091] Determine a first error corresponding to the target object according to the first weight, the first characterization vector, and the third characterization vector;

[0092] Determine a second error corresponding to the target object according to the second weight, the second characterization vector, and the fourth characterization vector;

[0093] Correcting the first weight according to the first error to obtain a third weight, and correcting the second weight according to the second error to obtain a fourth weight;

[0094] Information fusion is performed on the first characterization vector and the second characterization vector according to the third weight and the fourth weight to obtain target data corresponding to the target object.

[0095] In some implementations, the data fusion module 203 performs, during the process of fusing the first characterization vector and the second characterization vector according to the third weight and the fourth weight to obtain the target data corresponding to the target object:

[0096] Obtaining a first data mean corresponding to the target object under the first sensor and obtaining a second data mean corresponding to the target object under the second sensor;

[0097] Summing the first characterization vector and the first data mean to obtain a first sum result, and summing the second characterization vector and the second data mean to obtain a second sum result;

[0098] The first sum result and the second sum result are subjected to information fusion using the third weight and the fourth weight to obtain the target data corresponding to the target object.

[0099] In some implementations, the data analysis module 204 performs, during the process of performing data analysis on the target data to obtain the target analysis result corresponding to the target object:

[0100] Determine a clustering algorithm corresponding to the target data, and perform data clustering on the target data according to the clustering algorithm to obtain initial clusters;

[0101] Using a data classification model to classify and analyze the initial clusters to obtain target types corresponding to the initial clusters;

[0102] Performing data screening on the initial cluster according to the target type to obtain a target cluster;

[0103] Performing data association analysis on the target cluster to obtain the target analysis result corresponding to the target object.

[0104] In some implementations, the big data-based data processing system 200 further performs:

[0105] Encrypting the first data to obtain a first encryption result and encrypting the second data to obtain a second encryption result;

[0106] Determine a database, and store the first encryption result and the second encryption result in the database to obtain a target storage result;

[0107] Set access permissions for the target storage result.

[0108] In some implementations, the big data-based data processing system 200 further performs:

[0109] Performing data analysis on the fourth data to obtain a first change trend corresponding to the target object;

[0110] Performing data analysis on the fifth data to obtain a second change trend corresponding to the target object;

[0111] A visualization tool is determined, and the first change trend and the second change trend are displayed in the visualization tool to obtain a visualization result corresponding to the target object.

[0112] In some implementations, the big data-based data processing system 200 may be applied to a terminal device.

[0113] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the data processing system 200 based on big data described above can refer to the corresponding process in the aforementioned data processing method embodiment based on big data, and will not be repeated here.

[0114] See also Figure 3 , Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0115] like Figure 3 As shown, the terminal device 300 includes a processor 301 and a memory 302, and the processor 301 and the memory 302 are connected via a bus 303, such as an I2C (Inter-integrated Circuit) bus.

[0116] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0117] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0118] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the terminal device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0119] The processor is used to run a computer program stored in the memory, and implement any one of the data processing methods based on big data provided by the embodiments of the present invention when executing the computer program.

[0120] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0121] Collecting first data corresponding to a target object under a first sensor and second data corresponding to a target object under a second sensor, wherein the first data is used to characterize structured data corresponding to the target object, and the second data is used to characterize semi-structured data or unstructured data corresponding to the target object;

[0122] Performing structured processing on the second data to obtain third data, and performing data preprocessing on the third data to obtain fourth data;

[0123] Performing data preprocessing on the first data to obtain fifth data, and performing information fusion on the fourth data and the fifth data to obtain target data corresponding to the target object;

[0124] Perform data analysis on the target data to obtain a target analysis result corresponding to the target object.

[0125] In some implementations, during the process of preprocessing the first data to obtain the fifth data, the processor 301 executes:

[0126] Performing missing value identification on the first data to obtain missing data;

[0127] Obtain relevant data corresponding to the missing data from the first data, and then calculate the mean result corresponding to the relevant data;

[0128] Filling the first data according to the mean value result to obtain intermediate data, and normalizing the intermediate data to obtain normalized data;

[0129] A principal component analysis algorithm is used to perform dimensionality reduction processing on the normalized data to obtain the fifth data corresponding to the first data.

[0130] In some implementations, the processor 301, in the process of fusing the fourth data and the fifth data to obtain the target data corresponding to the target object, executes:

[0131] Performing vector representation on the fourth data using the feature representation model to obtain a first representation vector, and performing vector representation on the fifth data using the feature representation model to obtain a second representation vector;

[0132] Determine a first weight corresponding to the first characterization vector and a second weight corresponding to the second characterization vector;

[0133] Determine a third characterization vector corresponding to the target object when the first data is collected, and determine a fourth characterization vector corresponding to the target object when the second data is collected;

[0134] Determine a first error corresponding to the target object according to the first weight, the first characterization vector, and the third characterization vector;

[0135] Determine a second error corresponding to the target object according to the second weight, the second characterization vector, and the fourth characterization vector;

[0136] Correcting the first weight according to the first error to obtain a third weight, and correcting the second weight according to the second error to obtain a fourth weight;

[0137] Information fusion is performed on the first characterization vector and the second characterization vector according to the third weight and the fourth weight to obtain target data corresponding to the target object.

[0138] In some implementations, the processor 301, in the process of fusing the first characterization vector and the second characterization vector according to the third weight and the fourth weight to obtain the target data corresponding to the target object, executes:

[0139] Obtaining a first data mean corresponding to the target object under the first sensor and obtaining a second data mean corresponding to the target object under the second sensor;

[0140] Summing the first characterization vector and the first data mean to obtain a first sum result, and summing the second characterization vector and the second data mean to obtain a second sum result;

[0141] The first sum result and the second sum result are subjected to information fusion using the third weight and the fourth weight to obtain the target data corresponding to the target object.

[0142] In some implementations, during the process of performing data analysis on the target data to obtain the target analysis result corresponding to the target object, the processor 301 executes:

[0143] Determine a clustering algorithm corresponding to the target data, and perform data clustering on the target data according to the clustering algorithm to obtain initial clusters;

[0144] Using a data classification model to classify and analyze the initial clusters to obtain target types corresponding to the initial clusters;

[0145] Performing data screening on the initial cluster according to the target type to obtain a target cluster;

[0146] Performing data association analysis on the target cluster to obtain the target analysis result corresponding to the target object.

[0147] In some implementations, the processor 301 further executes:

[0148] Encrypting the first data to obtain a first encryption result and encrypting the second data to obtain a second encryption result;

[0149] Determine a database, and store the first encryption result and the second encryption result in the database to obtain a target storage result;

[0150] Set access permissions for the target storage result.

[0151] In some implementations, the processor 301 further executes:

[0152] Performing data analysis on the fourth data to obtain a first change trend corresponding to the target object;

[0153] Performing data analysis on the fifth data to obtain a second change trend corresponding to the target object;

[0154] A visualization tool is determined, and the first change trend and the second change trend are displayed in the visualization tool to obtain a visualization result corresponding to the target object.

[0155] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned data processing method based on big data embodiment, and will not be repeated here.

[0156] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any data processing method based on big data provided in the description of the embodiment of the present invention.

[0157] The storage medium may be an internal storage unit of the terminal device in the aforementioned embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device.

[0158] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0159] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0160] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data processing method based on big data, characterized in that: The method comprises: Collecting first data corresponding to a target object under a first sensor and second data corresponding to a target object under a second sensor, wherein the first data is used to characterize structured data corresponding to the target object, and the second data is used to characterize semi-structured data or unstructured data corresponding to the target object; Performing structured processing on the second data to obtain third data, and performing data preprocessing on the third data to obtain fourth data; Performing data preprocessing on the first data to obtain fifth data, and performing information fusion on the fourth data and the fifth data to obtain target data corresponding to the target object; Perform data analysis on the target data to obtain a target analysis result corresponding to the target object.

2. The method according to claim 1, characterized in that The performing data preprocessing on the first data to obtain fifth data includes: Performing missing value identification on the first data to obtain missing data; Obtain relevant data corresponding to the missing data from the first data, and then calculate the mean result corresponding to the relevant data; Filling the first data according to the mean value result to obtain intermediate data, and normalizing the intermediate data to obtain normalized data; A principal component analysis algorithm is used to perform dimensionality reduction processing on the normalized data to obtain the fifth data corresponding to the first data.

3. The method according to claim 1, characterized in that The fusing the fourth data and the fifth data to obtain target data corresponding to the target object includes: Performing vector representation on the fourth data using the feature representation model to obtain a first representation vector, and performing vector representation on the fifth data using the feature representation model to obtain a second representation vector; Determine a first weight corresponding to the first characterization vector and a second weight corresponding to the second characterization vector; Determine a third characterization vector corresponding to the target object when the first data is collected, and determine a fourth characterization vector corresponding to the target object when the second data is collected; Determine a first error corresponding to the target object according to the first weight, the first characterization vector, and the third characterization vector; Determine a second error corresponding to the target object according to the second weight, the second characterization vector, and the fourth characterization vector; Correcting the first weight according to the first error to obtain a third weight, and correcting the second weight according to the second error to obtain a fourth weight; Information fusion is performed on the first characterization vector and the second characterization vector according to the third weight and the fourth weight to obtain target data corresponding to the target object.

4. The method according to claim 3, characterized in that The performing information fusion on the first characterization vector and the second characterization vector according to the third weight and the fourth weight to obtain target data corresponding to the target object includes: Obtaining a first data mean corresponding to the target object under the first sensor and obtaining a second data mean corresponding to the target object under the second sensor; Summing the first characterization vector and the first data mean to obtain a first sum result, and summing the second characterization vector and the second data mean to obtain a second sum result; The first sum result and the second sum result are subjected to information fusion using the third weight and the fourth weight to obtain the target data corresponding to the target object.

5. The method according to claim 1, characterized in that The performing data analysis on the target data to obtain a target analysis result corresponding to the target object includes: Determine a clustering algorithm corresponding to the target data, and perform data clustering on the target data according to the clustering algorithm to obtain initial clusters; Using a data classification model to classify and analyze the initial clusters to obtain target types corresponding to the initial clusters; Performing data screening on the initial cluster according to the target type to obtain a target cluster; Performing data association analysis on the target cluster to obtain the target analysis result corresponding to the target object.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Encrypting the first data to obtain a first encryption result and encrypting the second data to obtain a second encryption result; Determine a database, and store the first encryption result and the second encryption result in the database to obtain a target storage result; Set access permissions for the target storage result.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Performing data analysis on the fourth data to obtain a first change trend corresponding to the target object; Performing data analysis on the fifth data to obtain a second change trend corresponding to the target object; A visualization tool is determined, and the first change trend and the second change trend are displayed in the visualization tool to obtain a visualization result corresponding to the target object.

8. A data processing system based on big data, characterized in that: include: A data acquisition module, used to acquire first data corresponding to a target object under a first sensor and second data corresponding to a second sensor, wherein the first data is used to characterize structured data corresponding to the target object, and the second data is used to characterize semi-structured data or unstructured data corresponding to the target object; a data processing module, configured to perform structured processing on the second data to obtain third data, and perform data preprocessing on the third data to obtain fourth data; a data fusion module, configured to perform data preprocessing on the first data to obtain fifth data, and perform information fusion on the fourth data and the fifth data to obtain target data corresponding to the target object; The data analysis module is used to perform data analysis on the target data to obtain a target analysis result corresponding to the target object.

9. A terminal device, characterized in that: The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program and implement the data processing method based on big data as described in any one of claims 1 to 7 when executing the computer program.

10. A computer storage medium for computer storage, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the data processing method based on big data as described in any one of claims 1 to 7.

Images