A big data processing method and system for realizing hybrid data analysis

Abstract

The present invention provides a big data processing method and system for realizing hybrid data analysis. The present invention receives real-time data streams from different sources, and the real-time data streams include structured data and unstructured data; based on the data fusion logic in a pre-constructed dynamic data fusion framework, the structured data and the unstructured data are fused and processed to obtain fused data, and the data fusion logic includes a multi-level fusion logic and a dynamic weight adjustment mechanism; the fused data is abstracted to obtain a data view; based on the data view, an analysis task is defined, and the analysis task is executed to obtain an analysis result; the technical solution provided by the present invention realizes efficient, real-time and flexible hybrid data analysis, and improves the overall performance and support capability of big data processing.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data processing technology, and in particular, to a big data processing method and system for implementing hybrid data analysis. Background Art

[0002] With the advent of the big data era, enterprises and organizations are facing an increasing demand for data processing. These data not only come from traditional structured databases, but also include unstructured data generated from social media, IoT devices, text files and other channels. Traditional data processing methods often have difficulty effectively coping with the coexistence of structured and unstructured data, especially when large amounts of heterogeneous data need to be analyzed and processed in real time. Traditional data fusion technologies and methods seem to be unable to cope with this situation.

[0003] Existing data processing technologies use methods that directly merge structured data and unstructured data, which can easily lead to information loss or inaccuracy; when faced with high-speed data streams, existing technologies find it difficult to achieve real-time data fusion and analysis; when processing complex data, data analysis results are not accurate enough; when faced with different application scenarios, the data fusion rules used by existing technologies are fixed and cannot be dynamically adjusted according to actual analysis needs. Summary of the invention

[0004] The embodiments of the present invention provide a big data processing method and system for realizing hybrid data analysis, so as to solve the problems in the prior art of information loss or inaccuracy, difficulty in realizing real-time data fusion and analysis, inaccurate data analysis results, and inability to dynamically adjust fused data according to actual analysis needs.

[0005] In a first aspect, an embodiment of the present invention provides a big data processing method for implementing hybrid data analysis, including:

[0006] Receiving real-time data streams from different sources, the real-time data streams containing structured data and unstructured data;

[0007] Based on the data fusion logic in the pre-built dynamic data fusion framework, the structured data and the unstructured data are fused and processed to obtain fused data, wherein the data fusion logic includes a multi-level fusion logic and a dynamic weight adjustment mechanism;

[0008] Abstracting the fused data to obtain a data view;

[0009] Based on the data view, an analysis task is defined, and the analysis task is executed to obtain an analysis result.

[0010] Optionally, the data fusion logic based on the pre-built dynamic data fusion framework fuses the structured data and the unstructured data to obtain fused data, and the data fusion logic includes a multi-level fusion logic and a dynamic weight adjustment mechanism, including:

[0011] Applying a data matching algorithm to identify the correlation between the structured data and the unstructured data according to a preset data association rule to obtain aligned data elements, wherein the preset data association rule is used to guide the initial data fusion process;

[0012] The data elements are merged to obtain preliminary merged data, and complementary information results of the structured data and the unstructured data are determined based on the preliminary merged data, and fused data are obtained by applying data fusion logic, multi-level fusion logic and dynamic weight adjustment mechanism.

[0013] Optionally, merging the data elements to obtain preliminary merged data, determining complementary information results of the structured data and the unstructured data based on the preliminary merged data, and applying data fusion logic, multi-level fusion logic and dynamic weight adjustment mechanism to obtain fused data includes:

[0014] Using a rule-based merging method, the data elements are preliminarily merged to obtain preliminary merged data;

[0015] Based on the preliminary merged data, using a complementary information recognition algorithm, calculating an information gain ratio between the structured data and the unstructured data to obtain a complementary information result between the structured data and the unstructured data;

[0016] Based on the complementary information results, a deep fusion network is used to fuse different types of data to obtain preliminary fused data;

[0017] The preliminary fusion data is multi-level fused using a multi-level fusion logic and a dynamic weight adjustment mechanism to obtain fused data. The multi-level fusion logic includes a multi-scale feature extraction technology, a hierarchical feature aggregation technology, and a cross-layer consistency verification mechanism that are executed in sequence.

[0018] Optionally, the application of multi-level fusion logic and dynamic weight adjustment mechanism to perform multi-level fusion on the preliminary fusion data to obtain fusion data, the multi-level fusion logic includes multi-scale feature extraction technology, hierarchical feature aggregation technology and cross-layer consistency verification mechanism executed in sequence, including:

[0019] Using multi-scale feature extraction technology to extract different levels of features in the preliminary fusion data;

[0020] Using hierarchical feature aggregation technology, aggregate the features at different levels to obtain a multi-level feature representation;

[0021] A cross-layer consistency verification mechanism is introduced to determine the consistency of the multi-level feature representation through a consistency loss function to obtain target features at different levels;

[0022] A dynamic weight adjustment mechanism is introduced to adjust the importance of the target features at different levels to obtain a target weight vector, and the target weight vector is used to perform weighted summation on the target features at different levels to generate fused data.

[0023] Optionally, the introduction of a dynamic weight adjustment mechanism to adjust the importance of the target features at different levels to obtain a target weight vector, and using the target weight vector to perform weighted summation on the target features at different levels to generate fused data includes:

[0024] Define the initialized weight vector;

[0025] Based on the deep learning network structure, a dynamic weight adjustment model is constructed, and a weight learning mechanism is introduced into the dynamic weight adjustment model, wherein the weight learning mechanism is used to adjust the weight vector through a back propagation algorithm;

[0026] Define the weight adjustment loss function;

[0027] Based on the weight adjustment loss function, a gradient descent optimization algorithm is used to update the weight vector to obtain a target weight vector;

[0028] The target weight vector is used to perform weighted summation on the target features at different levels to generate fused data.

[0029] Optionally, the abstracting the fused data to obtain a data view includes:

[0030] Using a data abstraction module, the fused data is aggregated to obtain aggregated data, wherein the aggregation includes calculating an average value, a maximum value, a minimum value, and a total value;

[0031] Using data dimensionality reduction technology to reduce the dimension of the aggregated data to obtain reduced-dimensional data;

[0032] Based on the dimensionally reduced data, a visual representation is established to obtain a data view.

[0033] Optionally, defining an analysis task based on the data view, executing the analysis task, and obtaining an analysis result includes:

[0034] Based on the data view, defining analysis objectives, and defining analysis tasks based on the analysis objectives, wherein the analysis objectives include identifying patterns, predicting trends, and detecting anomalies;

[0035] According to the analysis task, a machine learning model is selected, and the data in the data view is input into the machine learning model, the analysis task is executed, and the analysis result is obtained.

[0036] In a second aspect, an embodiment of the present invention provides a big data processing system for implementing hybrid data analysis, including:

[0037] A receiving module, used for receiving real-time data streams from different sources, wherein the real-time data streams include structured data and unstructured data;

[0038] A construction module is used to fuse and process the structured data and the unstructured data based on the data fusion logic in the pre-built dynamic data fusion framework to obtain fused data, wherein the data fusion logic includes a multi-level fusion logic and a dynamic weight adjustment mechanism;

[0039] A processing module, used for performing abstract processing on the fused data to obtain a data view;

[0040] The definition module is used to define the analysis task based on the data view, and execute the analysis task to obtain the analysis result.

[0041] In a third aspect, an embodiment of the present invention provides a computing device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the big data processing methods for implementing hybrid data analysis as described in the first aspect.

[0042] In a fourth aspect, an embodiment of the present invention provides a computer storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the big data processing method for implementing hybrid data analysis as described in any one of the first aspects.

[0043] In an embodiment of the present invention, real-time data streams from different sources are received, and the real-time data streams include structured data and unstructured data; based on the data fusion logic in the pre-constructed dynamic data fusion framework, the structured data and the unstructured data are fused to obtain fused data, and the data fusion logic includes multi-level fusion logic and dynamic weight adjustment mechanism; the fused data is abstracted to obtain a data view; based on the data view, an analysis task is defined, and the analysis task is executed to obtain an analysis result. The technical solution provided by the present invention realizes efficient, real-time, and flexible hybrid data analysis, and improves the overall performance and support capabilities of big data processing; wherein, the dynamic weight adjustment mechanism can adapt to changes in data streams in real time to ensure the accuracy and timeliness of data fusion; the dynamic weight adjustment mechanism ensures the effective use of features at different levels and improves the quality of fused data; and the weight adjustment loss function ensures the consistency and integrity between features at different levels.

[0044] These and other aspects of the present invention will become more apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a big data processing method for implementing hybrid data analysis provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the structure of a big data processing system for implementing hybrid data analysis provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the structure of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0050] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0052] Figure 1 A flowchart of a big data processing method for implementing hybrid data analysis is provided for an embodiment of the present invention. Figure 1 As shown, the method includes:

[0053] In the era of big data, data streams from different sources, including structured data and unstructured data, need to be efficiently fused to provide reliable support for decision making. Existing data fusion methods are difficult to adapt to changes in data streams in real time, resulting in inaccurate analysis results. Based on this, the present invention provides a big data processing method for realizing hybrid data analysis, such as Figure 1 ,include:

[0054] Step 101: receiving real-time data streams from different sources, wherein the real-time data streams include structured data and unstructured data;

[0055] In this step, real-time data stream refers to data received continuously from different sources such as sensors, social media, databases, etc. in a short period of time; structured data refers to data with a fixed format and structure, such as records in a database; unstructured data refers to data without a fixed format, such as social media posts, emails, etc.

[0056] This step receives data from different sources in real time, such as sensor data, social media posts, log files, database records, etc.; and classifies the received data into structured data and unstructured data by type, for example, database records are considered structured data, while social media posts are considered unstructured data.

[0057] Step 102: Based on the data fusion logic in the pre-built dynamic data fusion framework, the structured data and the unstructured data are fused to obtain fused data, wherein the data fusion logic includes a multi-level fusion logic and a dynamic weight adjustment mechanism;

[0058] In this step, the dynamic data fusion framework refers to a data processing framework that can dynamically adjust the association rules between structured data and unstructured data according to real-time analysis needs; multi-level feature extraction refers to the technology of extracting features from different levels, including low-level features, mid-level features and high-level features; dynamic weight adjustment mechanism refers to a mechanism that automatically adjusts the importance of features at different levels through a learning algorithm.

[0059] This step applies multi-level fusion logic. First, the data is preprocessed, such as format conversion and noise filtering. According to the type and content of the data, a suitable fusion algorithm is selected, such as rule-based methods, machine learning models, etc. The dynamic weight adjustment mechanism is used to evaluate the reliability and relevance of each data source, and their contribution in the fusion process is adjusted accordingly. The processed structured data is combined with the unstructured data to form a comprehensive data set, namely the fused data.

[0060] Step 103: abstract the fused data to obtain a data view;

[0061] In this step, abstraction processing refers to converting specific data into higher-level conceptual representations for easier understanding or further analysis; data view refers to a way of observing a data set from a specific perspective, usually to meet specific query requirements or analysis purposes.

[0062] For example, through data analysis techniques such as clustering, classification, and association rule mining, the core information of the data can be extracted; data models can be designed to convert complex data into a format that is easy to understand and use; data views from multiple perspectives can be created to meet the needs of different users. For example, business analysts may need to focus on sales trends, while product teams may be concerned about user feedback.

[0063] Step 104: Based on the data view, define an analysis task, and execute the analysis task to obtain an analysis result;

[0064] In this step, the data view refers to the abstract representation of the fused data, which facilitates the definition and execution of subsequent analysis tasks.

[0065] In this step, specific analysis tasks are defined based on the content of the data view, such as predicting future trends, identifying abnormal patterns, etc.; selected machine learning models or data analysis algorithms, such as time series analysis, anomaly detection algorithms, etc., are applied to perform analysis tasks; by executing analysis tasks, detailed analysis reports are generated, which include main findings, key indicators, visualization charts, etc.

[0066] The embodiments of the present invention can achieve the following beneficial effects through the above steps:

[0067] Through multi-level feature extraction and dynamic weight adjustment mechanism, it can adapt to changes in data streams in real time to ensure the accuracy and timeliness of data fusion;

[0068] Multi-level feature extraction technology ensures the effective use of features at different levels, and the dynamic weight adjustment mechanism enables the importance of features to be reasonably distributed, thus improving the quality of fused data;

[0069] Through the generated data views, various analysis tasks can be flexibly defined and executed to meet the analysis needs in different scenarios;

[0070] With good scalability, it can easily integrate new data sources and analysis tools to support more diverse and complex data processing needs.

[0071] In the era of big data, real-time data streams from different sources are increasing day by day. These data include both easy-to-process structured data and complex and changeable unstructured data. In order to make full use of these valuable information resources, based on this, the present invention provides a specific embodiment, in which step 102, based on the data fusion logic in the pre-built dynamic data fusion framework, fuses the structured data and the unstructured data to obtain fused data. The data fusion logic includes a multi-level fusion logic and a dynamic weight adjustment mechanism, and specifically includes the following steps:

[0072] Step 210: applying a data matching algorithm to identify the correlation between the structured data and the unstructured data according to a preset data association rule to obtain aligned data elements, wherein the preset data association rule is used to guide the initial data fusion process;

[0073] In this step, the data matching algorithm refers to a technology used to detect whether there is a corresponding relationship between two or more data sources, by comparing the similarities of data items to determine whether they represent the same entity or concept; the preset data association rules refer to a series of pre-defined rules used to guide the system on how to identify and associate data from different sources to ensure the accuracy of data matching.

[0074] In this step, a suitable matching algorithm is selected, such as the Levenshtein algorithm based on string similarity and the TF-IDF algorithm based on content; rules are defined. For example, if two data items have similar names and close timestamps, they are considered to describe the same event; matching algorithms are applied to identify related items in structured and unstructured data according to preset rules to form aligned data elements.

[0075] Step 211: merging the data elements to obtain preliminary merged data, and determining complementary information results of the structured data and the unstructured data based on the preliminary merged data, and applying data fusion logic, multi-level fusion logic and dynamic weight adjustment mechanism to obtain fused data;

[0076] In this step, complementary information results refer to the complementary information between structured data and unstructured data found on the basis of preliminary merged data, which can provide a more comprehensive perspective; multi-level fusion logic refers to the use of multi-stage or multi-level processing methods in the data fusion process, and each layer may focus on different data features or attributes to achieve more refined data integration; dynamic weight adjustment mechanism refers to the dynamic adjustment of the weights of each data source according to the quality, reliability and relevance of the data in the data fusion process to optimize the fusion effect.

[0077] This step combines the aligned data elements to form preliminary merged data; Analyze the preliminary merged data to find complementary information between structured data and unstructured data, for example, structured data may provide precise time and location, while unstructured data provides detailed descriptions or sentiment analysis; Ensure that the identified complementary information is accurate and meaningful, which can be done through manual review or automated verification; Design multi-level fusion logic based on the characteristics and needs of the data, for example, the first layer handles basic matching of data, and the second layer performs deep integration of data; Dynamically adjust the weight of each data source based on the quality and reliability of the data. For example, if a data source has a good historical performance, it can be given a higher weight; Input the preliminary merged data and complementary information results into the multi-level fusion logic, and generate the final fused data after processing.

[0078] The embodiments of the present invention can achieve the following beneficial effects through the above steps:

[0079] Through data matching and multi-level fusion, the integrity and accuracy of the data are ensured, and data missing and errors are avoided;

[0080] Identify and utilize the complementary information between structured data and unstructured data to make the data richer and more comprehensive, and improve the usability of the data;

[0081] High-quality fusion data provides decision makers with more comprehensive and accurate information support, helping them make more scientific decisions;

[0082] The dynamic weight adjustment mechanism enables the system to dynamically adjust the processing strategy according to the actual quality of the data, improving the flexibility and adaptability of the system.

[0083] In order to further process the aligned data elements, based on this, the present invention provides a specific embodiment, wherein the step 211, merging the data elements to obtain preliminary merged data, and determining the complementary information result of the structured data and the unstructured data based on the preliminary merged data, applying the data fusion logic multi-level fusion logic and the dynamic weight adjustment mechanism to obtain the fused data, specifically includes the following steps:

[0084] Step 221: using a rule-based merging method to preliminarily merge the data elements to obtain preliminary merged data;

[0085] In this step, rule-based merging methods are used to combine multiple data elements together based on preset rules and conditions. These rules may include matching of data fields, comparison of timestamps, etc.

[0086] This step defines a series of merging rules based on the characteristics of the data and business requirements. For example, if the timestamps of two data elements differ by no more than 5 minutes and some key fields are the same, they are considered to belong to the same event; the above rules are applied to match and merge the aligned data elements; the matched data elements are combined to form preliminary merged data.

[0087] Step 222: Based on the preliminary merged data, using a complementary information recognition algorithm, calculate the information gain ratio between the structured data and the unstructured data to obtain a complementary information result between the structured data and the unstructured data;

[0088] In this step, the complementary information identification algorithm refers to an algorithm used to identify and quantify complementary information between structured data and unstructured data, usually by calculating the information gain ratio; the information gain ratio refers to an indicator for measuring the importance of a feature in classification, and is used to evaluate the correlation and complementarity between structured data and unstructured data.

[0089] In this step, a suitable algorithm is selected, such as an information gain ratio algorithm; the structured data and unstructured data in the preliminary merged data are analyzed, the information gain ratio between them is calculated, and their complementarity is evaluated; based on the calculation results, complementary information results between the structured data and the unstructured data are generated to show which data items have complementary relationships.

[0090] Step 223: Based on the complementary information result, use a deep fusion network to fuse different types of data to obtain preliminary fused data;

[0091] In this step, the deep fusion network refers to a complex neural network model that can process and fuse different types of data to generate high-level feature representations.

[0092] This step inputs the complementary information results and preliminary merged data into the deep fusion network; through the processing of the network, preliminary fused data is generated, which contains higher-level feature representations.

[0093] Step 224: Apply multi-level fusion logic and dynamic weight adjustment mechanism to perform multi-level fusion on the preliminary fusion data to obtain fusion data, wherein the multi-level fusion logic includes a multi-scale feature extraction technology, a hierarchical feature aggregation technology, and a cross-layer consistency verification mechanism that are executed in sequence;

[0094] This step uses multi-scale feature extraction technology to extract features of different scales from the preliminary fusion data; aggregate features of different scales to form a higher-level feature representation; perform consistency checks between features at different levels to ensure the accuracy and consistency of the fusion results; dynamically adjust the weights of each data source based on the quality and reliability of the data to optimize the fusion effect; and generate the final fused data after multi-level fusion logic processing.

[0095] The embodiments of the present invention can achieve the following beneficial effects through the above steps:

[0096] Through rule-based merging methods and complementary information identification algorithms, the integrity and accuracy of data are ensured, and data missing and redundancy are avoided;

[0097] Deep fusion networks and multi-level fusion logic can generate high-level feature representations, making data richer and more comprehensive, and improving data usability;

[0098] High-quality fusion data provides decision makers with more comprehensive and accurate information support, helping them make more scientific decisions;

[0099] The dynamic weight adjustment mechanism enables the system to dynamically adjust the processing strategy according to the actual quality of the data, improving the flexibility and adaptability of the system.

[0100] The present invention provides a specific embodiment, in which the step 224 applies a multi-level fusion logic and a dynamic weight adjustment mechanism to perform multi-level fusion on the preliminary fusion data to obtain fusion data, wherein the multi-level fusion logic includes a multi-scale feature extraction technology, a hierarchical feature aggregation technology, and a cross-layer consistency verification mechanism that are executed in sequence, and specifically includes the following steps:

[0101] Step 231: extracting features at different levels from the preliminary fusion data using a multi-scale feature extraction technique;

[0102] In this step, the multi-scale feature extraction technology refers to a technology for extracting different levels of features from data, including low-level features, mid-level features, and high-level features.

[0103] In this step, convolutional neural networks (CNN) and other technologies are used to extract low-level features from the preliminary fused data, such as image pixel values, color information, etc.; feature pyramid networks (FPN) and other technologies are used to extract mid-level features from the preliminary fused data, such as image edges and textures; and advanced semantic extraction technologies (such as attention mechanisms, long short-term memory networks (LSTM)) are used to extract high-level features from the preliminary fused data, such as semantic information, contextual relationships, etc.

[0104] Step 232: using a hierarchical feature aggregation technique to aggregate the features at different levels to obtain a multi-level feature representation;

[0105] In this step, the hierarchical feature aggregation technique refers to a technique for aggregating features at different levels together to obtain a multi-level feature representation.

[0106] Multi-level feature representation refers to a comprehensive feature representation formed by aggregating features at different levels.

[0107] This step uses hierarchical feature aggregation technology to aggregate low-level features, mid-level features, and high-level features together; through fusion methods such as weighted summation and splicing, features at different levels are integrated into a multi-level feature representation.

[0108] Step 233: introducing a cross-layer consistency verification mechanism, and determining the consistency of the multi-level feature representation through a consistency loss function to obtain target features at different levels;

[0109] In this step, the cross-layer consistency verification mechanism refers to a verification mechanism used to ensure the consistency between features at different levels; the consistency loss function refers to a loss function used to measure the consistency between features at different levels.

[0110] This step introduces a cross-layer consistency verification mechanism to ensure the consistency between features at different levels; uses a consistency loss function to calculate the consistency loss between features at different levels, such as mean square error (MSE), cross entropy loss, etc.; obtains target features at different levels through the consistency verification mechanism and consistency loss function.

[0111] Step 234: introducing a dynamic weight adjustment mechanism to adjust the importance of the target features at different levels to obtain a target weight vector, and using the target weight vector to perform weighted summation on the target features at different levels to generate fused data;

[0112] In this step, the dynamic weight adjustment mechanism refers to a mechanism that can automatically adjust the importance of features at different levels according to real-time analysis requirements; weighted summation refers to using the target weight vector to perform weighted summation of features at different levels to generate fused data.

[0113] This step introduces a dynamic weight adjustment mechanism to automatically adjust the importance of features at different levels according to real-time analysis requirements. The target weight vector is obtained through the dynamic weight adjustment mechanism to ensure that the importance of features at different levels can be reasonably distributed. The target weight vector is used to perform weighted summation of target features at different levels to generate the final fused data.

[0114] The embodiments of the present invention can achieve the following beneficial effects through the above steps:

[0115] Through multi-level feature extraction and dynamic weight adjustment mechanism, it can adapt to changes in data streams in real time to ensure the accuracy and timeliness of data fusion;

[0116] Multi-level feature extraction technology and dynamic weight adjustment mechanism ensure the effective use of features at different levels and improve the quality of fused data;

[0117] The consistency and integrity between features at different levels are ensured through the cross-layer consistency verification mechanism and consistency loss function.

[0118] The present invention provides a specific embodiment, wherein the step 234 introduces a dynamic weight adjustment mechanism to adjust the importance of the target features at different levels to obtain a target weight vector, and uses the target weight vector to perform weighted summation on the target features at different levels to generate fused data, specifically comprising the following steps:

[0119] Step 241: define an initialized weight vector;

[0120] In this step, the weight vector refers to the vector used to represent the importance of features at different levels, and each element represents the weight of the corresponding level feature.

[0121] This step defines a weight vector according to the number of different hierarchical features, and assigns an initial value to each weight. For example, all initial weights are set equal, or different initial values ​​are assigned based on empirical knowledge. Through the weight vector, the performance of different target feature representations in different tasks is calculated to determine the direction of weight adjustment, and the different tasks include calculating classification accuracy and calculating regression error.

[0122] Step 242: constructing a dynamic weight adjustment model based on the deep learning network structure, and introducing a weight learning mechanism into the dynamic weight adjustment model, wherein the weight learning mechanism is used to adjust the weight vector through a back propagation algorithm;

[0123] In this step, the dynamic weight adjustment model refers to a model used to dynamically adjust the importance of features at different levels; the weight learning mechanism refers to a mechanism used to adjust the weight vector through the back propagation algorithm; the back propagation algorithm refers to an algorithm used to train a neural network to adjust the weight by calculating the gradient.

[0124] In this step, a dynamic weight adjustment model is constructed by selecting a deep learning network structure suitable for the current application scenario, such as a convolutional neural network (CNN). A weight learning mechanism is introduced into the model, and the weight vector is adjusted through the back propagation algorithm to ensure that the weight can be dynamically adjusted according to the importance of multi-level features.

[0125] Step 243: define a weight adjustment loss function;

[0126] In this step, the weight adjustment loss function refers to the loss function used to measure the effect of weight vector adjustment.

[0127] In this step, a loss function 𝐿 is defined according to specific application requirements, such as mean square error (MSE), to measure the effect of weight adjustment; an appropriate form is selected to calculate the effect of weight adjustment.

[0128] More specifically, the embodiment of the present invention provides a weight adjustment loss function for evaluating the rationality of the weight vector. The specific calculation formula is as follows:

[0129] ;

[0130] ;

[0131] in, is the weight adjustment loss function, which is used to evaluate the weight vector the rationality of is the characteristic error term; is the inter-feature consistency term; is the weight smoothing term; Indicates The true value of the layer feature; Indicates The predicted value of the layer feature; is the weight coefficient of the feature error, which is used to control the influence of the feature error; It is the weight coefficient of consistency between features, which is used to control the consistency between features in different layers; is the weight coefficient of weight smoothing, which is used to control the smoothness of weight; It is The weights of layer features; It is The weights of layer features; is the number of feature layers;

[0132] The weight adjustment loss function introduces feature error terms, feature consistency terms, and weight smoothing terms, which can more effectively adjust the importance of features at different levels, thereby generating high-quality fusion data. The feature error term ensures that the error between the predicted value and the true value of each level is minimized; the feature consistency term ensures that the weight difference between features at different levels is as small as possible, improving the consistency of features; the weight smoothing term ensures the smoothness of the weight vector to avoid excessive weight fluctuations.

[0133] Step 244: Based on the weight adjustment loss function, the weight vector is updated using a gradient descent optimization algorithm to obtain a target weight vector;

[0134] In this step, the gradient descent optimization algorithm refers to an optimization algorithm that adjusts weights by calculating the gradient of the loss function with respect to the weights.

[0135] This step uses the gradient descent algorithm to calculate the gradient of the weight adjustment loss function with respect to the weight vector. According to the gradient and learning efficiency, the weight vector is updated to obtain the target weight vector.

[0136] Step 245: using the target weight vector to perform weighted summation on the target features at different levels to generate fused data;

[0137] This step uses the target weight vector obtained in step 244 to perform weighted summation on the target features at different levels to generate fused data.

[0138] The embodiments of the present invention can achieve the following beneficial effects through the above steps:

[0139] The dynamic weight adjustment mechanism can adapt to changes in data flow in real time to ensure the accuracy and timeliness of data fusion.

[0140] The dynamic weight adjustment mechanism ensures the effective utilization of features at different levels and improves the quality of fused data.

[0141] By adjusting the loss function through weights, the consistency and integrity between features at different levels are ensured.

[0142] In order to further improve the readability and analysis efficiency of the fused data, based on this, the present invention provides a specific embodiment, wherein the step 103, abstracting the fused data to obtain a data view, specifically includes the following steps:

[0143] Step 301: using a data abstraction module, aggregating the fused data to obtain aggregated data, wherein the aggregation includes calculating an average value, a maximum value, a minimum value, and a total value;

[0144] In this step, the data abstraction module refers to a tool or component used to convert raw data into a higher-level abstract representation, usually through aggregation, summary, and other methods.

[0145] In this step, select the appropriate aggregation operation according to business needs. For example, the average value can be used to reflect the overall level, the maximum and minimum values ​​can be used to identify extreme cases, and the total can be used to count the total amount; use the data abstraction module to aggregate the fused data. For example, for a fused dataset containing sales data, the average sales, highest sales, lowest sales, and total sales of each product can be calculated; save the aggregation results as a new dataset, that is, aggregated data.

[0146] Step 302: using data dimensionality reduction technology to reduce the dimension of the aggregated data to obtain dimensionality-reduced data;

[0147] In this step, data dimensionality reduction technology refers to a method of reducing the dimension of a data set. The purpose is to simplify the data structure while retaining important information. Common techniques include principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), etc.

[0148] In this step, select the appropriate data dimensionality reduction technology according to the characteristics of the data and the analysis requirements. For example, principal component analysis (PCA) is suitable for linear data, and t-distributed stochastic neighbor embedding (t-SNE) is suitable for nonlinear data; input the aggregated data into the selected dimensionality reduction technology for processing. For example, use PCA to reduce the high-dimensional sales data to two or three dimensions; save the result of dimensionality reduction as a new data set, that is, dimensionality-reduced data.

[0149] Step 303: Based on the dimension-reduced data, a visual representation is established to obtain a data view;

[0150] In this step, visual representation refers to presenting data in the form of graphs or charts to facilitate users to intuitively understand and analyze data; data view refers to data presented in a visual representation, usually to meet specific query requirements or analysis purposes.

[0151] In this step, select the appropriate visual representation form according to the characteristics of the reduced-dimensionality data and the purpose of analysis. For example, scatter plots are suitable for two-dimensional data, and heat maps are suitable for matrix data; use data visualization tools such as Matplotlib, Tableau, etc. to convert the reduced-dimensionality data into graphs or charts. For example, use a scatter plot to display the reduced-dimensionality sales data, where each point represents a product, and the color and size can represent different attributes, such as sales, profit, etc.; adjust the layout, color, label, etc. of the chart to make it clearer and more beautiful, so that users can understand and analyze it more easily.

[0152] The embodiments of the present invention can achieve the following beneficial effects through the above steps:

[0153] Through aggregation and dimensionality reduction techniques, complex data sets are simplified into a form that is easier to manage and analyze, reducing the complexity of data processing;

[0154] Visual representation makes data more intuitive and understandable, and users can quickly capture key information and trends in the data;

[0155] Aggregated data and dimensionality-reduced data provide a solid foundation for subsequent analysis, allowing users to explore and gain insights into data more quickly and improve analysis efficiency.

[0156] Through different aggregation operations and dimensionality reduction techniques, data can be analyzed from multiple angles and levels, providing comprehensive data support for decision makers;

[0157] The optimized data view is not only beautiful but also powerful, allowing users to easily conduct interactive analysis and improve the overall user experience.

[0158] The present invention provides a specific embodiment, wherein the step 104, based on the data view, defines an analysis task, executes the analysis task, and obtains an analysis result, specifically comprising the following steps:

[0159] Step 401: Based on the data view, define analysis objectives, and define analysis tasks based on the analysis objectives, wherein the analysis objectives include pattern recognition, trend prediction, and anomaly detection;

[0160] In this step, the specific analysis goal is determined based on the content of the data view. For example, if the data view displays time series data, the analysis goal can be to predict future trends; if it is sensor data, the analysis goal can be to identify patterns or detect anomalies; based on the analysis goal, the specific analysis task is determined. For example, if the analysis goal is to identify patterns, the analysis task can be to group the data using a clustering algorithm; if the analysis goal is to predict trends, the analysis task can be to predict future data points using time series analysis methods.

[0161] Step 402: Select a machine learning model according to the analysis task, input the data in the data view into the machine learning model, execute the analysis task, and obtain an analysis result;

[0162] In this step, you select a suitable machine learning model based on the requirements of the analysis task. For example, if the analysis task is to identify patterns, select a clustering algorithm (such as K-means); if the analysis task is to predict trends, select a time series analysis model (such as ARIMA); pre-process the data in the data view (such as data cleaning, standardization, etc.), and then input it into the selected machine learning model; use the selected machine learning model to perform specific analysis tasks, such as training models, making predictions or classifications, etc.; by performing analysis tasks, obtain specific analysis results, such as predicted future trends, identified patterns, or detected anomalies.

[0163] The embodiments of the present invention can achieve the following beneficial effects through the above steps:

[0164] By defining specific analysis goals based on data views, we can ensure that the analysis tasks have clear direction and specificity;

[0165] Select appropriate machine learning models based on the defined analysis tasks, ensuring that the selected models can effectively solve specific problems and thus obtain high-quality analysis results;

[0166] By defining a modular process for analysis goals, tasks, and model selection, the system can easily integrate new data sources and analysis tools in the future, while also making system maintenance and upgrades easier.

[0167] Figure 2 A schematic diagram of a big data processing system for implementing hybrid data analysis is provided for an embodiment of the present invention. Figure 2 As shown, the device comprises:

[0168] A receiving module 21, configured to receive real-time data streams from different sources, wherein the real-time data streams include structured data and unstructured data;

[0169] A construction module 22 is used to fuse the structured data and the unstructured data based on the data fusion logic in the pre-built dynamic data fusion framework to obtain fused data, wherein the data fusion logic includes a multi-level fusion logic and a dynamic weight adjustment mechanism;

[0170] A processing module 23 is used to perform abstract processing on the fused data to obtain a data view;

[0171] The definition module 24 is used to define an analysis task based on the data view, and execute the analysis task to obtain an analysis result.

[0172] Figure 2 The big data processing system for implementing hybrid data analysis can execute Figure 1The implementation principle and technical effect of the big data processing method for implementing hybrid data analysis described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the big data processing system for implementing hybrid data analysis in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0173] In one possible design, Figure 2 A big data processing system for implementing hybrid data analysis in the illustrated embodiment can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0174] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0175] The processing component 32 is used to: receive real-time data streams from different sources, the real-time data streams including structured data and unstructured data; based on the data fusion logic in the pre-built dynamic data fusion framework, fuse the structured data and the unstructured data to obtain fused data, the data fusion logic includes multi-level fusion logic and dynamic weight adjustment mechanism; abstract the fused data to obtain a data view; based on the data view, define analysis tasks, execute the analysis tasks, and obtain analysis results.

[0176] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0177] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0178] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0179] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0180] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0181] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0182] The embodiment of the present invention further provides a computer storage medium storing a computer program, which can achieve the above-mentioned Figure 1 The illustrated embodiment provides a big data processing method for implementing hybrid data analysis.

[0183] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0184] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A big data processing method for realizing hybrid data analysis, characterized in that: include: Receiving real-time data streams from different sources, the real-time data streams containing structured data and unstructured data; Based on the data fusion logic in the pre-built dynamic data fusion framework, the structured data and the unstructured data are fused and processed to obtain fused data, wherein the data fusion logic includes a multi-level fusion logic and a dynamic weight adjustment mechanism; Abstracting the fused data to obtain a data view; Based on the data view, define an analysis task, and execute the analysis task to obtain an analysis result; The data fusion logic based on the pre-built dynamic data fusion framework fuses the structured data and the unstructured data to obtain fused data. The data fusion logic includes a multi-level fusion logic and a dynamic weight adjustment mechanism, including: Applying a data matching algorithm to identify the correlation between the structured data and the unstructured data according to a preset data association rule to obtain aligned data elements, wherein the preset data association rule is used to guide the initial data fusion process; Merging the data elements to obtain preliminary merged data, and determining complementary information results of the structured data and the unstructured data based on the preliminary merged data, and applying data fusion logic, multi-level fusion logic and dynamic weight adjustment mechanism to obtain fused data; The merging of the data elements to obtain preliminary merged data, and determining the complementary information results of the structured data and the unstructured data based on the preliminary merged data, and applying data fusion logic, multi-level fusion logic and dynamic weight adjustment mechanism to obtain fused data include: Using a rule-based merging method, the data elements are preliminarily merged to obtain preliminary merged data; Based on the preliminary merged data, using a complementary information recognition algorithm, calculating an information gain ratio between the structured data and the unstructured data to obtain a complementary information result between the structured data and the unstructured data; Based on the complementary information results, a deep fusion network is used to fuse different types of data to obtain preliminary fused data; Applying multi-level fusion logic and dynamic weight adjustment mechanism to perform multi-level fusion on preliminary fusion data to obtain fusion data, wherein the multi-level fusion logic includes multi-scale feature extraction technology, hierarchical feature aggregation technology and cross-layer consistency verification mechanism executed in sequence; The abstract processing of the fused data to obtain a data view includes: Using a data abstraction module, the fused data is aggregated to obtain aggregated data, wherein the aggregation includes calculating an average value, a maximum value, a minimum value, and a total value; Using data dimensionality reduction technology to reduce the dimension of the aggregated data to obtain reduced-dimensional data; Based on the dimension-reduced data, a visual representation is established to obtain a data view; The step of defining an analysis task based on the data view, executing the analysis task, and obtaining an analysis result includes: Based on the data view, defining analysis objectives, and defining analysis tasks based on the analysis objectives, wherein the analysis objectives include identifying patterns, predicting trends, and detecting anomalies; According to the analysis task, a machine learning model is selected, and the data in the data view is input into the machine learning model, the analysis task is executed, and the analysis result is obtained.

2. The method according to claim 1, characterized in that The multi-level fusion logic and the dynamic weight adjustment mechanism are applied to perform multi-level fusion on the preliminary fusion data to obtain fusion data, wherein the multi-level fusion logic includes a multi-scale feature extraction technology, a hierarchical feature aggregation technology and a cross-layer consistency verification mechanism that are sequentially executed, including: Using multi-scale feature extraction technology to extract different levels of features in the preliminary fusion data; Using hierarchical feature aggregation technology, aggregate the features at different levels to obtain a multi-level feature representation; A cross-layer consistency verification mechanism is introduced to determine the consistency of the multi-level feature representation through a consistency loss function to obtain target features at different levels; A dynamic weight adjustment mechanism is introduced to adjust the importance of the target features at different levels to obtain a target weight vector, and the target weight vector is used to perform weighted summation on the target features at different levels to generate fused data.

3. The method according to claim 2, characterized in that The introduction of a dynamic weight adjustment mechanism adjusts the importance of the target features at different levels to obtain a target weight vector, and uses the target weight vector to perform weighted summation on the target features at different levels to generate fused data, including: Define the initialized weight vector; Based on the deep learning network structure, a dynamic weight adjustment model is constructed, and a weight learning mechanism is introduced into the dynamic weight adjustment model, wherein the weight learning mechanism is used to adjust the weight vector through a back propagation algorithm; Define the weight adjustment loss function; Based on the weight adjustment loss function, a gradient descent optimization algorithm is used to update the weight vector to obtain a target weight vector; The target weight vector is used to perform weighted summation on the target features at different levels to generate fused data.

4. A big data processing system for implementing hybrid data analysis, used to execute a big data processing method for implementing hybrid data analysis as claimed in any one of claims 1 to 3, characterized in that: include: A receiving module, used for receiving real-time data streams from different sources, wherein the real-time data streams include structured data and unstructured data; A construction module is used to fuse and process the structured data and the unstructured data based on the data fusion logic in the pre-built dynamic data fusion framework to obtain fused data, wherein the data fusion logic includes a multi-level fusion logic and a dynamic weight adjustment mechanism; A processing module, used for performing abstract processing on the fused data to obtain a data view; The definition module is used to define the analysis task based on the data view, and execute the analysis task to obtain the analysis result.

5. A computing device, characterized in that: It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a big data processing method for realizing hybrid data analysis as described in any one of claims 1 to 3.

6. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, a big data processing method for realizing hybrid data analysis as described in any one of claims 1 to 3 is implemented.

Images