Data analysis processing method and system based on big data

Abstract

The invention relates to the technical field of big data processing, in particular to a data analysis processing method and system based on big data, and the method comprises the following steps: S1, extracting original data from data sources of multiple fields; performing preliminary classification on the original data to generate a field classification list; s2, converting data fields with consistent or similar semantics to generate a standardized data set; s3, carrying out feature alignment on the data in different fields according to a field matching rule and the correlation between the data; s4, generating an interaction relation matrix by calculating the correlation between different domain features; and S5, taking the interaction relation matrix as input, and adopting a graph neural network to construct a cross-domain interaction relation graph. According to the cross-domain data fusion method, semantic unification of cross-domain data is realized through the semantic mapping rule, the complex interaction relationship between the data is accurately captured in combination with feature alignment and graph neural network modeling, and the efficiency and analysis precision of cross-domain data fusion are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a data analysis and processing method and system based on big data. Background Art

[0002] With the development of information technology, the application of big data in various industries has become more and more extensive, involving many fields such as medical care, finance, logistics, education, etc. However, the data in different fields are diverse, including structured data (such as tabular data in databases) and unstructured data (such as text, images and log data). These data usually come from heterogeneous systems, and the data semantics, format and type are different, making it difficult to directly compare, integrate and analyze the data. In this case, how to achieve standardized processing of cross-domain data, deep integration and effective modeling of multi-domain relationships has become one of the core issues in the field of big data analysis.

[0003] Existing technologies face the following major challenges in cross-domain data processing: First, the semantic standards of data in different fields are not unified, and the field and attribute names lack consistency, which makes data integration difficult; second, the matching and alignment methods of data features often lack flexibility and accuracy, and cannot fully utilize the potential associations between features in various fields; third, existing data modeling methods are difficult to capture the complex interactive relationships between cross-domain data, especially when the data scale is huge and the dimension is high, the existing technology has problems of insufficient efficiency and accuracy in building global association models. Therefore, it is of great technical value and practical significance to develop an analysis and processing method and system that can efficiently extract, fuse and model multi-domain data to solve the above problems. Summary of the invention

[0004] Based on the above objectives, the present invention provides a data analysis and processing method and system based on big data.

[0005] The data analysis and processing method based on big data includes the following steps:

[0006] S1: Extract raw data from data sources in multiple fields, including structured data and unstructured data; and preliminarily classify the raw data according to the attributes, types and field labels of the data sources to generate a field classification list;

[0007] S2: Based on the domain classification list generated by S1, the fields and attributes of data in different domains are semantically analyzed through pre-built semantic mapping rules, and the data fields with consistent or similar semantics are converted to generate a standardized data set;

[0008] S3: Take the standardized data set generated by S2 as input, perform feature alignment on data from different fields according to field matching rules and the correlation between data, and generate a cross-domain feature set;

[0009] S4: Based on the cross-domain feature set generated by S3, the interaction relationship matrix is ​​generated by calculating the correlation between features in different domains;

[0010] S5: Taking the interaction relationship matrix as input, a graph neural network is used to construct a cross-domain interaction relationship graph to describe the interaction patterns and their correlation strengths between different domains.

[0011] Optionally, the S1 specifically includes:

[0012] S11: For data sources in multiple fields, adapter interfaces matching the data types are configured respectively, wherein the data source for structured data accesses the data table through the database connection protocol; the data source for unstructured data obtains the original data through a file parser or API call;

[0013] S12: Extract raw data from the data source in batches or in real time according to the adapter interface configured in S11; for structured data, the extraction results include complete tables or data sets of specified fields; for unstructured data, the extraction results include text, pictures or logs; all extracted data are stored according to timestamps and data source identifiers;

[0014] S13: Perform field annotation on the original data extracted in S12 to generate an annotation information table; the annotation information table includes data source type, field name, field data type and corresponding field label;

[0015] S14: Based on the annotation information table generated in S13, classify the data according to the domain label, and classify the data belonging to the same domain into a unified data set; and assign a unique domain identifier to each classified data set to generate a domain classification list.

[0016] Optionally, the S2 specifically includes:

[0017] S21: Based on the domain classification list generated in S1, for each domain’s data, collect its commonly used field names, attribute values ​​and corresponding semantic definitions, and establish a domain semantic library; the semantic library includes the mapping relationship between field names, synonyms, hyponyms and hyponyms, and domain terms;

[0018] S22: Based on the domain semantic library constructed in S21, formulate semantic mapping rules, including synonym replacement rules, field name standardization rules, and data type conversion rules; each rule clearly specifies the mapping relationship between the source field and the target standard field and the conversion logic;

[0019] S23: Apply the semantic mapping rules defined in S22 to the data in each field classified by S1, and analyze the field names and attribute values ​​one by one; for fields with consistent or similar semantics, convert them according to the mapping rules, replace the source fields with standardized target field names, and adjust the data types and formats to meet the standardization requirements;

[0020] S24: Reorganize the data processed by S23 to form a standardized data set.

[0021] Optionally, the S3 specifically includes:

[0022] S31: Based on the standardized data set generated by S2, extract the data field name, data type and semantic label of each field; and establish a field matching rule library according to the field similarity and the field relationship defined in the field semantic library;

[0023] S32: Input the standardized data set of S2 into the field matching algorithm, and match the fields between different fields one by one according to the field matching rules of S31; during the matching process, by calculating the similarity of the field names, the semantic association weight, and the data type consistency score, select the field pair with the highest score as the matching result, and perform field alignment;

[0024] S33: For the fields matched by S32, the corresponding feature values ​​are extracted from each field data; and according to the mapping relationship in the field matching rule, the feature values ​​are converted into a unified unit, dimension or encoding method to ensure the consistency of the feature data;

[0025] S34: Reorganize the feature values ​​extracted and converted in S33 according to domain classification to generate a cross-domain feature set.

[0026] Optionally, the S31 specifically includes:

[0027] S311: Using the text similarity of the field names as the basis for field matching, calculate the similarity score S between the two field names name ;

[0028] S312: Calculate the semantic association weight S of the two fields according to the hyponym and synonym relationships of the field relationships in the domain semantic library. sem ;

[0029] S313: Compare the data types of the two fields and calculate the data type consistency score S type ;

[0030] S314: Based on the calculation results of S311, S312, and S313, the total score S of the field match is calculated. match , the formula is: S match =α·Sname +β·S sem +γ·S type , where α, β, and γ are corresponding weight parameters, which are used to balance the importance of field name similarity, semantic association weight, and data type consistency, and satisfy α+β+γ=1;

[0031] S315: S calculated based on S314 match , set the matching threshold T; when S match When ≥T, the two fields are judged as matching fields and recorded as matching rules.

[0032] Optionally, the S4 specifically includes:

[0033] S41: Based on the cross-domain feature set, the Pearson correlation coefficient is used to calculate the correlation between every two features;

[0034] S42: The feature correlation calculated in S41 is adjusted in combination with the domain weight matrix to calculate the weighted correlation;

[0035] S43: Organize the weighted correlations between all features into a matrix form to obtain an interaction relationship matrix.

[0036] Optionally, the S5 specifically includes:

[0037] S51: Take the interaction relationship matrix generated by S4 as input, determine the relationship between nodes and edges according to the element values ​​in the matrix; take each feature in the matrix as a node, and the correlation value of the matrix element as the weight of the edge between nodes to generate the initial graph structure;

[0038] S52: Assign a corresponding initial feature vector to each node. The initial feature vector comes from the feature value in the cross-domain feature set generated by S3; perform dimensionality reduction processing on the feature value to form a low-dimensional vector representation suitable for the input requirements of the graph neural network, and assign it to each node;

[0039] S53: Construct a graph neural network layer based on the message passing mechanism, and define the information transmission rules for each node to receive information from adjacent nodes; in each layer, the node features are updated by aggregating the feature information of neighboring nodes and its own features;

[0040] S54: Apply the graph neural network constructed in S53 to the initial graph structure, perform graph convolution operations layer by layer, and gradually update the feature representation of each node;

[0041] S55: After the multi-layer graph convolution operation is completed, the final embedded features of each node are output, and the final cross-domain interaction relationship graph is generated according to the weight of the edge; the cross-domain interaction relationship graph consists of a node set, an edge set and the embedded features of each node, which is used to describe the interaction pattern between different domains and their correlation strength.

[0042] Optionally, the S53 specifically includes:

[0043] S531: In the graph neural network layer, each node collects feature information from its neighboring nodes; for each node, first identify its neighboring node set, and then determine the node's neighboring node set based on the edges related to the node in the interaction relationship matrix;

[0044] S532: For each node, aggregate feature information of neighboring nodes by weight;

[0045] S533: After the aggregation is completed, the features of each node are updated through a nonlinear activation function;

[0046] S534: For each layer in the graph neural network, repeat the processes of S531 and S532 to gradually transfer information between nodes;

[0047] S535: The node features updated in S533 are used as input and processed in turn through a multi-layer graph convolutional network, where each layer includes the above-mentioned feature aggregation and update process until the set maximum number of layers is reached.

[0048] Optionally, the S54 specifically includes:

[0049] S541: The node initial feature embedding generated by S52 is used as input to the first layer of the graph neural network constructed by S53;

[0050] S542: In the first layer of the graph neural network, for each node, perform the following operations:

[0051] Operation 1: collect the feature vectors of all direct neighbor nodes of the node and perform weighted summation according to the weight of the edge to form an aggregate representation of the neighbor features;

[0052] Operation 2: The aggregated neighbor features are combined with the node’s own feature vector and input into a linear transformation function, which is then processed by a nonlinear activation function to generate an updated feature representation of the node in the first layer.

[0053] S543: For the lth layer of the graph neural network, repeat the following steps:

[0054] Step 1: Collect the feature representations of the neighbor nodes of the node at the l-1 layer, perform weighted summation according to the weight of the edge, and form an aggregate representation of the neighbor features;

[0055] Step 2: Combine the aggregated neighbor features with the feature representation of the node at layer l-1, input them into a linear transformation function, and then process them through a nonlinear activation function to generate an updated feature representation of the node at layer l;

[0056] S544: Repeat S543 until the preset number of graph neural network layers is reached;

[0057] S545: After completing the graph convolution operations of all layers, the final embedded feature representation of each node is output.

[0058] The data analysis and processing system based on big data is used to implement the above-mentioned data analysis and processing method based on big data, and includes the following modules:

[0059] Data collection module: used to extract raw data from data sources in multiple fields, including structured data and unstructured data; and to preliminarily classify the raw data according to the attributes, types and field labels of the data sources, and generate a field classification list;

[0060] Semantic standardization module: used to receive the domain classification list generated by the data acquisition module, perform semantic analysis on the fields and attributes of data in different fields according to the pre-built domain semantic library and semantic mapping rules; and convert the data fields with consistent or similar semantics to output a standardized data set;

[0061] Feature alignment module: used to receive the standardized data set output by the semantic standardization module, perform feature alignment on data from different fields according to the field matching rules and the correlation between data, and generate a cross-field feature set;

[0062] Interaction modeling module: used to receive the cross-domain feature set generated by the feature alignment module, calculate the correlation between features in different domains, and generate an interaction relationship matrix; and use the interaction relationship matrix as input to construct a cross-domain interaction relationship graph based on the graph neural network to describe the interaction pattern between domains and their correlation strength;

[0063] Result output module: used to receive the cross-domain interaction relationship diagram generated by the interaction modeling module, organize and store the node features and edge weight information of the diagram; and provide an interface to support external system calls.

[0064] Beneficial effects of the present invention:

[0065] The present invention, by introducing semantic mapping rules, performs semantically unified processing on data fields and attributes in different fields, eliminates the semantic differences between cross-domain data, and solves the problem of data fusion difficulty caused by semantic inconsistency in the prior art. At the same time, it utilizes field matching rules and feature alignment methods to achieve accurate matching and alignment of cross-domain features, so that data in different fields can be efficiently processed in a unified feature vector space.

[0066] The present invention, by constructing a cross-domain interaction relationship model based on graph neural network, can capture the complex interaction relationships between data, and perform information propagation and feature update layer by layer, so as to construct an accurate cross-domain interaction relationship graph. This method breaks through the limitations of existing technologies in cross-domain data analysis, not only significantly improves the efficiency and accuracy of data fusion, but also enhances the expression ability of cross-domain association patterns, providing a reliable technical means for multi-domain data analysis and decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 A schematic diagram of a data analysis and processing method according to an embodiment of the present invention;

[0069] Figure 2 Schematic diagram of a data analysis and processing system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0071] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0072] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0073] like Figure 1 As shown, the data analysis and processing method based on big data includes the following steps:

[0074] S1: Extract raw data from data sources in multiple fields, including structured data and unstructured data; and preliminarily classify the raw data according to the attributes, types and field labels of the data sources to generate a field classification list;

[0075] S2: Based on the domain classification list generated by S1, the fields and attributes of data in different domains are semantically analyzed through pre-built semantic mapping rules, and the data fields with consistent or similar semantics are converted to generate standardized data sets for unifying data representation between different domains.

[0076] S3: Take the standardized data set generated by S2 as input, perform feature alignment on data from different fields according to field matching rules and the correlation between data, generate a cross-domain feature set, extract features with high correlation and store them as a fused feature vector;

[0077] S4: Based on the cross-domain feature set generated by S3, the interaction relationship matrix is ​​generated by calculating the correlation between features in different domains;

[0078] S5: Taking the interaction relationship matrix as input, a graph neural network is used to construct a cross-domain interaction relationship graph to describe the interaction patterns and their correlation strengths between different domains.

[0079] S1 specifically includes:

[0080] S11: For data sources in multiple fields, adapter interfaces matching the data types are configured respectively. For data sources of structured data, data tables are accessed through database connection protocols (such as JDBC or ODBC); for data sources of unstructured data, raw data is obtained through file parsers or API calls; the interface configuration of each data source includes data source type, access method, field format, and security authentication parameters;

[0081] S12: Extract raw data from the data source in batches or in real time according to the adapter interface configured in S11; for structured data, the extraction results include complete tables or data sets of specified fields; for unstructured data, the extraction results include text, pictures or logs; all extracted data are stored according to timestamps and data source identifiers;

[0082] S13: Perform field annotation on the original data extracted in S12 to generate an annotation information table; the annotation information table includes data source type, field name, field data type and corresponding field label;

[0083] S14: Based on the annotation information table generated in S13, classify the data according to the domain label, and classify the data belonging to the same domain into a unified data set; and assign a unique domain identifier to each classified data set, and generate a domain classification list, which includes the classification name, data source path and classification feature description of each domain; through the clear processing of the above steps, it is possible to efficiently and accurately extract original data from data sources in multiple fields, and through unified attribute annotation and classification rules, quickly generate a domain classification list, which provides a structured and standardized data foundation for subsequent semantic mapping and data processing, thereby improving the efficiency and accuracy of data fusion processing.

[0084] S2 specifically includes:

[0085] S21: Based on the domain classification list generated in S1, for each domain’s data, collect its commonly used field names, attribute values ​​and corresponding semantic definitions, and establish a domain semantic library; the semantic library includes the mapping relationship between field names, synonyms, hyponyms and hyponyms, and domain terms;

[0086] S22: Based on the domain semantic library constructed in S21, formulate semantic mapping rules, including synonym replacement rules, field name standardization rules, and data type conversion rules; each rule clearly specifies the mapping relationship between the source field and the target standard field and the conversion logic;

[0087] S23: Apply the semantic mapping rules defined in S22 to the data in each field classified by S1, and analyze the field names and attribute values ​​one by one; for fields with consistent or similar semantics, convert them according to the mapping rules, replace the source fields with standardized target field names, and adjust the data types and formats to meet the standardization requirements;

[0088] S24: Reorganize the data processed by S23 to form a standardized data set. The field names, data types and formats in the data set are unified, eliminating the semantic differences between different fields, laying the foundation for subsequent data fusion and analysis. Through the above-mentioned step-by-step processing, it is possible to effectively perform semantic analysis and standardized conversion on data from different fields, unify field names and data formats, eliminate semantic differences between cross-domain data, generate standardized data sets, provide a consistent data basis for subsequent data fusion and analysis, and improve the efficiency and accuracy of data processing.

[0089] S3 specifically includes:

[0090] S31: Based on the standardized data set generated by S2, extract the data field name, data type and semantic label of each field; and establish a field matching rule library according to the field similarity and the field relationship defined in the field semantic library; the field matching rules include the field name similarity threshold, data type consistency requirements and weight parameters of the semantic association between fields;

[0091] S32: Input the standardized data set of S2 into the field matching algorithm, and match the fields between different fields one by one according to the field matching rules of S31; during the matching process, by calculating the similarity of the field names, the semantic association weight, and the data type consistency score, select the field pair with the highest score as the matching result, and perform field alignment;

[0092] S33: For the fields matched by S32, the corresponding feature values ​​are extracted from each field data; and according to the mapping relationship in the field matching rule, the feature values ​​are converted into a unified unit, dimension or encoding method to ensure the consistency of the feature data;

[0093] S34: The feature values ​​extracted and converted in S33 are reorganized according to the domain classification to generate a cross-domain feature set; this set contains the standardized feature values ​​of all matching fields and the identification information of their source domains, which serves as input for subsequent analysis and modeling; through the above-mentioned step-by-step processing, field matching and feature alignment between data in different domains can be achieved, the feature expression method of the field can be unified, and a cross-domain feature set can be generated. Under the guidance of the field matching rules, this feature set ensures the semantic consistency and relevance of the features, providing an accurate data basis for subsequent domain interaction relationship modeling.

[0094] The establishment of the field matching rule base in S31 specifically includes:

[0095] S311: Using the text similarity of the field names as the basis for field matching, calculate the similarity score S between the two field names name , the similarity formula is: Among them, N commonIndicates the number of common words in the two field names, N field1 and N field2 Respectively represent the total vocabulary of field names; the score range is [0, 1], the higher the score, the more similar the field names are;

[0096] S312: Calculate the semantic association weight S of the two fields according to the hyponym and synonym relationships of the field relationships in the domain semantic library. sem ; The specific formula is: Among them, W syn The weight indicating whether the fields are synonyms (the value is 1 if they are synonyms, otherwise 0), d indicates the hierarchical distance of the field in the hyponymy graph in the semantic library (if there is no hyponymy, d→∞, then the weight of this part is 0);

[0097] S313: Compare the data types of the two fields and calculate the data type consistency score S type ; The expression of consistency score is:

[0098] S314: Based on the calculation results of S311, S312, and S313, the total score S of the field match is calculated. match , the formula is: S match =α·S name +β·S sem +γ·S type , where α, β, and γ are corresponding weight parameters, which are used to balance the importance of field name similarity, semantic association weight, and data type consistency, and satisfy α+β+γ=1;

[0099] S315: S calculated based on S314 match , set the matching threshold T; when S match When ≥T, the two fields are judged as matching fields and recorded as matching rules.

[0100] The generated field matching rule base includes the following three forms:

[0101] Synonymous field matching rule: field pairs with highly similar field names and completely identical semantics, such as customer ID and user ID;

[0102] Superordinate and subordinate field matching rules: field pairs with high semantic association weight but low name similarity, such as product category and commodity type;

[0103] Type-compatible field matching rules: Field pairs with partially compatible data types but weak semantic and name associations, such as price (integer) and amount (floating point number); through the above steps and specific calculation formulas, a field matching rule library can be scientifically established to clarify the similarities, semantic associations, and type consistency between fields, and generate rules with clear matching logic. This rule library can provide a unified and accurate basis for field matching of cross-domain data, greatly improving the efficiency and accuracy of feature alignment.

[0104] S4 specifically includes:

[0105] S41: Based on the cross-domain feature set, the Pearson correlation coefficient is used to calculate the correlation between each two features. i,j , the correlation formula is: Among them, x i,k and x j,k Respectively represent the feature values ​​of feature i and j in field k, and are the means of features i and j respectively, and n is the number of fields involved in the calculation; the calculation result range is [-1, 1], a positive value indicates a positive correlation, and a negative value indicates a negative correlation;

[0106] S42: Feature correlation r calculated for S41 i,j , combined with the domain weight matrix to adjust and calculate the weighted correlation R i,j , the formula is: R i,j =r i,j ·W k , where W k is the weight factor of the field to which feature i and feature j belong, which is used to distinguish the importance of different fields;

[0107] S43: Weighted correlation R between all features i,j Organized into a matrix form, the interaction relationship matrix M is obtained; the expression is: Among them, the matrix element R i,j Represents the weighted correlation between features i and j. Through the above-mentioned steps and clear calculation formula, the correlation between features in different fields can be systematically calculated, and weighted adjustment can be performed in combination with the field weights to generate a highly expressive interaction relationship matrix. This matrix accurately quantifies the mutual relationship between cross-domain features and provides a rigorous and high-quality data foundation for the subsequent construction of graph models.

[0108] S5 specifically includes:

[0109] S51: Take the interaction relationship matrix generated by S4 as input, determine the relationship between nodes and edges according to the element values ​​in the matrix; take each feature in the matrix as a node, and the correlation value of the matrix element as the weight of the edge between nodes, and generate the initial graph structure, including a node set and an edge set;

[0110] S52: Assign a corresponding initial feature vector to each node. The initial feature vector comes from the feature value in the cross-domain feature set generated by S3; perform dimensionality reduction processing on the feature value to form a low-dimensional vector representation suitable for the input requirements of the graph neural network, and assign it to each node;

[0111] S53: Construct a graph neural network layer based on the message passing mechanism, and define the information transmission rules for each node to receive information from adjacent nodes; in each layer, the node features are updated by aggregating the feature information of neighboring nodes and its own features; the aggregation method includes weighted summation or average operation, and the weight is determined by the weight value of the edge;

[0112] S54: Apply the graph neural network constructed in S53 to the initial graph structure, perform graph convolution operations layer by layer, and gradually update the feature representation of each node; the output of each layer of convolution is used as the input of the next layer, so that the node features can gradually contain more contextual information of distant nodes;

[0113] S55: After the multi-layer graph convolution operation is completed, the final embedded features of each node are output, and the final cross-domain interaction relationship graph is generated according to the weight of the edge; the cross-domain interaction relationship graph is composed of a node set, an edge set and the embedded features of each node, which is used to describe the interaction pattern and its correlation strength between different fields; through the above steps, the interaction relationship matrix is ​​processed using a graph neural network, which can realize the full transmission and fusion of information between nodes based on the structure of the graph, and generate a cross-domain interaction relationship graph with high expressiveness. The graph intuitively and comprehensively reflects the complex relationship between cross-domain data, providing a scientific basis for subsequent analysis and decision-making.

[0114] S53 specifically includes:

[0115] S531: In the graph neural network layer, each node collects feature information from its neighboring nodes; for each node v i , first identify its neighbor node set N(v i ), and then according to the interaction relationship matrix with node v i Related edges, determine the node v i The neighbor node set; the feature value of the neighbor node is provided by the node embedding feature vector in S52;

[0116] S532: For each node v i, by weighted aggregation of neighbor node feature information; the weighting method is based on the edge weight w between nodes i,j , the specific formula is: in, For node v i The feature vector at level l, is the neighbor node v j The feature vector in the previous layer; this aggregation operation uses the edge weight w i,j To adjust the intensity of information transmission;

[0117] S533: After the aggregation is completed, the features of each node are updated through a nonlinear activation function; the update process formula is: Among them, σ is a nonlinear activation function, such as ReLU or Sigmoid, W (l) is the weight matrix of the lth layer, b (l) is the bias term, Represents a vector concatenation operation;

[0118] S534: For each layer in the graph neural network, repeat the process of S531 and S532 to gradually transfer information between nodes; information propagation is based on the graph structure and edge weights, allowing each node to obtain information about neighboring nodes, thereby learning richer feature representations; through the convolution operation of multi-layer graph neural networks, nodes can learn feature representations that contain broader context information;

[0119] S535: The updated node features in S533 are used as input and processed in turn through a multi-layer graph convolutional network, where each layer includes the above-mentioned feature aggregation and update process until the set maximum number of layers L is reached. The final feature representation of the node includes the information of the farther nodes in the graph. This multi-layer structure ensures the full propagation and fusion of the features of each node in the graph. The graph neural network layer based on the message passing mechanism constructed in the above steps can effectively carry out information propagation and feature update in the graph structure, ensuring that the node can receive and integrate the information of neighboring nodes layer by layer. This process enables the node features to gradually include global information, so that the complex relationship between cross-domain data can be accurately represented. The design of multi-layer graph convolution can further enhance the expressive power of node representation, laying the foundation for generating accurate cross-domain interaction relationship diagrams.

[0120] S54 specifically includes:

[0121] S541: The node initial feature embedding generated by S52 is used as input to the first layer of the graph neural network constructed by S53; the initial feature vector of each node represents its attribute in the cross-domain feature set;

[0122] S542: In the first layer of the graph neural network, for each node, perform the following operations:

[0123] Operation 1: collect the feature vectors of all direct neighbor nodes of the node and perform weighted summation according to the weight of the edge to form an aggregate representation of the neighbor features;

[0124] Operation 2: Combine the aggregated neighbor features with the node’s own feature vector and input them into a linear transformation function, which is then processed by a nonlinear activation function (such as ReLU) to generate an updated feature representation of the node in the first layer.

[0125] S543: For the lth layer (l>1) of the graph neural network, repeat the following steps:

[0126] Step 1: Collect the feature representations of the neighbor nodes of the node at the l-1 layer, perform weighted summation according to the weight of the edge, and form an aggregate representation of the neighbor features;

[0127] Step 2: Combine the aggregated neighbor features with the feature representation of the node at layer l-1, input them into a linear transformation function, and then process them through a nonlinear activation function to generate an updated feature representation of the node at layer l;

[0128] S544: Repeat S543 until the preset number of graph neural network layers L is reached; in each layer of convolution operation, the feature representation of the node gradually integrates more extensive neighbor information to capture deeper graph structure features;

[0129] S545: After completing the graph convolution operations of all layers, the final embedded feature representation of each node is output. This feature representation integrates the node's own attributes and multi-layer neighbor information, accurately reflecting the node's position and role in the cross-domain interaction relationship graph; through the above-mentioned step-by-step layer-by-layer graph convolution operations, the node feature representation is continuously updated in each layer, gradually integrating the information of neighboring nodes. The iteration of multi-layer convolution enables the node to capture deeper graph structure features. The generated final feature representation comprehensively reflects the complex associations of nodes in cross-domain interaction relationships, providing a solid data foundation for subsequent analysis and application.

[0130] like Figure 2 As shown, the data analysis and processing system based on big data is used to implement the above-mentioned data analysis and processing method based on big data, and includes the following modules:

[0131] Data collection module: used to extract raw data from data sources in multiple fields, including structured data and unstructured data; and to preliminarily classify the raw data according to the attributes, types and field labels of the data sources, and generate a field classification list;

[0132] Semantic standardization module: used to receive the domain classification list generated by the data acquisition module, and perform semantic analysis on the fields and attributes of data in different fields according to the pre-built domain semantic library and semantic mapping rules; and convert the data fields with consistent or similar semantics, output standardized data sets, and provide unified format input for subsequent data fusion;

[0133] Feature alignment module: It is used to receive the standardized data set output by the semantic standardization module, align the features of data from different fields according to the field matching rules and the correlation between data, and generate a cross-domain feature set as the input of interactive modeling;

[0134] Interaction modeling module: used to receive the cross-domain feature set generated by the feature alignment module, calculate the correlation between features in different domains, and generate an interaction relationship matrix; and use the interaction relationship matrix as input to construct a cross-domain interaction relationship graph based on the graph neural network to describe the interaction pattern between domains and their correlation strength;

[0135] Result output module: used to receive the cross-domain interaction relationship diagram generated by the interaction modeling module, organize and store the node features and edge weight information of the diagram; at the same time, it provides an interface to support external system calls for specific business scenario analysis or decision support.

[0136] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0137] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data analysis and processing method based on big data, characterized in that: The following steps are involved: S1: Extract raw data from data sources in multiple fields, including structured data and unstructured data; and preliminarily classify the raw data according to the attributes, types and field labels of the data sources to generate a field classification list; S2: Based on the domain classification list generated by S1, the fields and attributes of data in different domains are semantically analyzed through pre-built semantic mapping rules, and the data fields with consistent or similar semantics are converted to generate a standardized data set; S3: Take the standardized data set generated by S2 as input, perform feature alignment on data from different fields according to field matching rules and the correlation between data, and generate a cross-domain feature set; S4: Based on the cross-domain feature set generated by S3, the interaction relationship matrix is ​​generated by calculating the correlation between features in different domains; S5: Taking the interaction relationship matrix as input, a graph neural network is used to construct a cross-domain interaction relationship graph to describe the interaction patterns and their correlation strengths between different domains.

2. The data analysis and processing method based on big data according to claim 1 is characterized in that: The S1 specifically includes: S11: For data sources in multiple fields, adapter interfaces matching the data types are configured respectively, wherein the data source for structured data accesses the data table through the database connection protocol; the data source for unstructured data obtains the original data through a file parser or API call; S12: Extract raw data from the data source in batches or in real time according to the adapter interface configured in S11; for structured data, the extraction results include complete tables or data sets of specified fields; for unstructured data, the extraction results include text, pictures or logs; all extracted data are stored according to timestamps and data source identifiers; S13: Perform field annotation on the original data extracted in S12 to generate an annotation information table; the annotation information table includes data source type, field name, field data type and corresponding field label; S14: Based on the annotation information table generated in S13, classify the data according to the domain label, and classify the data belonging to the same domain into a unified data set; and assign a unique domain identifier to each classified data set to generate a domain classification list.

3. The data analysis and processing method based on big data according to claim 1 is characterized in that: The S2 specifically includes: S21: Based on the domain classification list generated in S1, for each domain’s data, collect its commonly used field names, attribute values ​​and corresponding semantic definitions, and establish a domain semantic library; the semantic library includes the mapping relationship between field names, synonyms, hyponyms and hyponyms, and domain terms; S22: Based on the domain semantic library constructed in S21, formulate semantic mapping rules, including synonym replacement rules, field name standardization rules, and data type conversion rules; each rule clearly specifies the mapping relationship between the source field and the target standard field and the conversion logic; S23: Apply the semantic mapping rules defined in S22 to the data in each field classified by S1, and analyze the field names and attribute values ​​one by one; for fields with consistent or similar semantics, convert them according to the mapping rules, replace the source fields with standardized target field names, and adjust the data types and formats to meet the standardization requirements; S24: Reorganize the data processed by S23 to form a standardized data set.

4. The data analysis and processing method based on big data according to claim 1 is characterized in that: The S3 specifically includes: S31: Based on the standardized data set generated by S2, extract the data field name, data type and semantic label of each field; and establish a field matching rule library according to the field similarity and the field relationship defined in the field semantic library; S32: Input the standardized data set of S2 into the field matching algorithm, and match the fields between different fields one by one according to the field matching rules of S31; during the matching process, by calculating the similarity of the field names, the semantic association weight, and the data type consistency score, select the field pair with the highest score as the matching result, and perform field alignment; S33: For the fields matched by S32, the corresponding feature values ​​are extracted from each field data; and according to the mapping relationship in the field matching rule, the feature values ​​are converted into a unified unit, dimension or encoding method to ensure the consistency of the feature data; S34: Reorganize the feature values ​​extracted and converted in S33 according to domain classification to generate a cross-domain feature set.

5. The data analysis and processing method based on big data according to claim 4 is characterized in that: The S31 specifically includes: S311: Using the text similarity of the field names as the basis for field matching, calculate the similarity score S between the two field names name ; S312: Calculate the semantic association weight S of the two fields based on the hyponym and synonym relationships of the field relationships in the domain semantic library. sem ; S313: Compare the data types of the two fields and calculate the data type consistency score S type ; S314: Based on the calculation results of S311, S312, and S313, the total score S of the field match is calculated. match , the formula is: S match =α·S name +β·S sem +γ·S type , where α, β, and γ are corresponding weight parameters, which are used to balance the importance of field name similarity, semantic association weight, and data type consistency, and satisfy α+β+γ=1; S315: S calculated based on S314 match , set the matching threshold T; when S match When ≥T, the two fields are judged as matching fields and recorded as matching rules.

6. The data analysis and processing method based on big data according to claim 1 is characterized in that: The S4 specifically includes: S41: Based on the cross-domain feature set, the Pearson correlation coefficient is used to calculate the correlation between every two features; S42: The feature correlation calculated in S41 is adjusted in combination with the domain weight matrix to calculate the weighted correlation; S43: Organize the weighted correlations between all features into a matrix form to obtain an interaction relationship matrix.

7. The data analysis and processing method based on big data according to claim 1 is characterized in that: The S5 specifically includes: S51: Take the interaction relationship matrix generated by S4 as input, determine the relationship between nodes and edges according to the element values ​​in the matrix; take each feature in the matrix as a node, and the correlation value of the matrix element as the weight of the edge between nodes to generate the initial graph structure; S52: Assign a corresponding initial feature vector to each node. The initial feature vector comes from the feature value in the cross-domain feature set generated by S3; perform dimensionality reduction processing on the feature value to form a low-dimensional vector representation suitable for the input requirements of the graph neural network, and assign it to each node; S53: Construct a graph neural network layer based on the message passing mechanism, and define the information transmission rules for each node to receive information from adjacent nodes; in each layer, the node features are updated by aggregating the feature information of neighboring nodes and its own features; S54: Apply the graph neural network constructed in S53 to the initial graph structure, perform graph convolution operations layer by layer, and gradually update the feature representation of each node; S55: After the multi-layer graph convolution operation is completed, the final embedded features of each node are output, and the final cross-domain interaction relationship graph is generated according to the weight of the edge; the cross-domain interaction relationship graph consists of a node set, an edge set and the embedded features of each node, which is used to describe the interaction pattern between different domains and their correlation strength.

8. The data analysis and processing method based on big data according to claim 7 is characterized in that: The S53 specifically includes: S531: In the graph neural network layer, each node collects feature information from its neighboring nodes; for each node, first identify its neighboring node set, and then determine the node's neighboring node set based on the edges related to the node in the interaction relationship matrix; S532: For each node, aggregate feature information of neighboring nodes by weight; S533: After the aggregation is completed, the features of each node are updated through a nonlinear activation function; S534: For each layer in the graph neural network, repeat the processes of S531 and S532 to gradually transfer information between nodes; S535: The node features updated in S533 are used as input and processed in turn through a multi-layer graph convolutional network, where each layer includes the above-mentioned feature aggregation and update process until the set maximum number of layers is reached.

9. The data analysis and processing method based on big data according to claim 8, characterized in that: The S54 specifically includes: S541: The node initial feature embedding generated by S52 is used as input to the first layer of the graph neural network constructed by S53; S542: In the first layer of the graph neural network, for each node, perform the following operations: Operation 1: collect the feature vectors of all direct neighbor nodes of the node and perform weighted summation according to the weight of the edge to form an aggregate representation of the neighbor features; Operation 2: The aggregated neighbor features are combined with the node’s own feature vector and input into a linear transformation function, which is then processed by a nonlinear activation function to generate an updated feature representation of the node in the first layer. S543: For the lth layer of the graph neural network, repeat the following steps: Step 1: Collect the feature representations of the neighbor nodes of the node at the l-1 layer, perform weighted summation according to the weight of the edge, and form an aggregate representation of the neighbor features; Step 2: Combine the aggregated neighbor features with the feature representation of the node at layer l-1, input them into a linear transformation function, and then process them through a nonlinear activation function to generate an updated feature representation of the node at layer l; S544: Repeat S543 until the preset number of graph neural network layers is reached; S545: After completing the graph convolution operations of all layers, the final embedded feature representation of each node is output.

10. A data analysis and processing system based on big data, used to implement a data analysis and processing method based on big data as described in any one of claims 1 to 9, characterized in that: Includes the following modules: Data collection module: used to extract raw data from data sources in multiple fields, including structured data and unstructured data; and to preliminarily classify the raw data according to the attributes, types and field labels of the data sources, and generate a field classification list; Semantic standardization module: used to receive the domain classification list generated by the data acquisition module, perform semantic analysis on the fields and attributes of data in different fields according to the pre-built domain semantic library and semantic mapping rules; and convert the data fields with consistent or similar semantics to output a standardized data set; Feature alignment module: used to receive the standardized data set output by the semantic standardization module, perform feature alignment on data from different fields according to the field matching rules and the correlation between data, and generate a cross-field feature set; Interaction modeling module: used to receive the cross-domain feature set generated by the feature alignment module, calculate the correlation between features in different domains, and generate an interaction relationship matrix; and use the interaction relationship matrix as input to construct a cross-domain interaction relationship graph based on the graph neural network to describe the interaction pattern between domains and their correlation strength; Result output module: used to receive the cross-domain interaction relationship diagram generated by the interaction modeling module, organize and store the node features and edge weight information of the diagram; and provide an interface to support external system calls.

Images