Digital twinning-oriented multi-dimensional data integration and analysis system

Abstract

The invention relates to the technical field of data integration and analysis, and discloses a digital twinning-oriented multi-dimensional data integration and analysis system, which comprises a data collection module, a data integration module, a data analysis module and a result output unit, in the digital twinning-oriented multi-dimensional data integration and analysis system, the same semantic elements in different data sources are accurately identified through a comprehensive algorithm in pattern matching, and the accuracy is improved; for data fusion, a weighted average method and a voting method are adopted for numeric data and classified data, reasonable fusion is achieved, and it is guaranteed that the result is reliable and effective; feature extraction can fully mine feature information from structured and unstructured data; according to attribute correlation analysis, the correlation relation between matching attributes is accurately grasped by means of a Pearson's correlation coefficient formula, and data understanding is deepened; results are output and displayed clearly, integration and analysis results can be presented visually, and subsequent decision-making and other operations are facilitated; the multi-dimensional data utilization efficiency and the analysis value in the digital twinning scene are effectively improved, and powerful data support is provided for development of related fields.

Description

Technical Field

[0001] The present invention relates to the technical field of data integration and analysis, and in particular to a multidimensional data integration and analysis system for digital twins. Background Art

[0002] Digital twin technology is widely used in many fields. It realizes various functions by building virtual models of physical entities. However, this is highly dependent on the effective integration and in-depth analysis of multidimensional data from multiple data sources. At present, data sources are diverse and scattered. The data generated by different systems and equipment differ in format, structure and semantics, forming data islands. For example, in industrial production, equipment operation data is mostly structured and stored in the database, while log files, operation records, etc. are unstructured text data. In the digital twin system, data integration faces many challenges. In terms of pattern matching, different data sources have different naming and structural definitions for entities and attributes with the same semantics. The traditional matching method that only considers a single factor is difficult to adapt to complex environments and is prone to inaccuracies or omissions.

[0003] Data fusion is also difficult. The data of the same entity in different data sources may differ due to various factors. For numerical and categorical data, appropriate fusion strategies are required, otherwise it will introduce erroneous information or reduce accuracy, affecting the reliability of the digital twin model.

[0004] Data analysis also has many demands and challenges. When extracting key features from structured data, traditional methods have difficulty in fully exploring potential patterns and relationships. Although unstructured text data contains value, it is difficult to directly analyze and utilize due to its characteristics. It requires effective feature extraction and combination with structured data. Moreover, in order to deeply understand the internal connections between data from different data sources, attribute association analysis is indispensable, which can reveal hidden laws and trends and provide support for decision-making and optimization. However, existing technologies have limitations when dealing with association analysis of multi-source heterogeneous data.

[0005] In short, digital twins urgently need to integrate and analyze multidimensional data. Existing data processing technologies are insufficient when dealing with multi-source heterogeneous data. Therefore, it is of great significance to develop an efficient and accurate multidimensional data integration and analysis system for digital twins. Summary of the invention

[0006] The purpose of the present invention is to provide a multidimensional data integration and analysis system for digital twins, which solves the technical problems raised in the background technology.

[0007] The purpose of the present invention can be achieved through the following technical solutions: Multidimensional data integration and analysis system for digital twins, including: A data collection module, used to collect relevant data from multiple data sources; The data integration module is used to collect relevant data from multiple data sources and integrate them through pattern matching and data fusion. In the pattern matching stage, the matching algorithms based on name similarity and structure similarity are used to calculate the edit distance and related similarity indicators of the attribute names, and the pattern matching similarity is comprehensively obtained, and compared with the threshold to determine whether it matches. In the data fusion stage, the weighted average method is used for numerical data, and the numerical attribute values ​​of the same entity are fused and calculated according to the weights of different data sources. The voting method is used for categorical data to count the number of occurrences of the category values ​​of the categorical attributes of the same entity in each data source, and the category value with the largest number of occurrences is selected as the fusion result. The data analysis module is used to collect relevant data from multiple data sources and conduct comprehensive analysis through feature extraction and attribute association analysis. In terms of feature extraction, the attribute parameters of key attributes are extracted for structured data and their mean, median and standard deviation are calculated as statistical features. For unstructured text data, the word segmentation is first performed, and then the word frequency and inverse document frequency are counted, and then the word frequency-inverse document frequency is obtained as the feature vector. In the attribute association analysis link, matching attributes are extracted based on the pattern matching results of the data integration module, and the correlation coefficient between the corresponding matching attributes is calculated using the Pearson correlation coefficient formula. The result output unit is used to output and display the integration results and analysis results obtained by the data integration module and the data analysis module.

[0008] As a further solution of the present invention: wherein, the collected relevant data formats include structured data and unstructured data.

[0009] As a further solution of the present invention: the integrated processing method is as follows: StepA1: Pattern matching Analyze multidimensional data patterns from different data sources and identify entities, attributes, and relationships with the same semantics; StepA1.1, name similarity calculation: First, extract the two attribute names and their lengths. Then create a two-dimensional array of the specified size and initialize its first row and first column, which respectively represent the number of operations of deleting part of the characters of one attribute name and inserting an empty string to generate part of the characters of another attribute name. Then, for the other elements of the array, calculate their values ​​according to the corresponding rules based on whether the characters at the corresponding positions of the two attribute names are the same. Then, the edit distance is obtained from the last element of the array. Finally, the name similarity is calculated based on the edit distance using the specified formula and recorded as S. name (A1, A2); StepA1.2, structural similarity calculation: Extract the number of all attributes in the table corresponding to the two attributes respectively, then mark the data types corresponding to the two attributes, and determine whether the two attributes are primary keys in the relational database and assign values ​​according to the conditions. Finally, calculate the structural similarity of the two attributes and record it as S struct (A1, A2); StepA1.3, pattern matching similarity calculation pass: ; Calculate the pattern matching similarity S0 (A1, A2); In the formula, γ is the preset weight coefficient; StepA1.4, pattern matching determination Compare the pattern matching similarity S0(A1, A2) with the preset pattern matching similarity threshold SY0: If S0(A1, A2)>SY0, then A1 and A2 are considered to match; If S0(A1, A2) ≤ SY0, it is determined that there is no match between A1 and A2; Step A2: Data Fusion Based on pattern matching, data of the same entity from different data sources are fused; StepA2.1, for numerical data, weighted average method is used for fusion; StepA2.2: For categorized data, voting method is used for fusion.

[0010] As a further solution of the present invention: the specific method of calculating the name similarity is as follows: StepA1.1.1, extract two attribute names and mark them as A1 and A2 respectively, and extract the lengths of the two attribute names and mark them as L1 and L2 respectively; StepA1.1.2. Create a two-dimensional array D of size (L1+1)×(L2+1); StepA1.1.3, Initialize the first row and first column: D[i][0]=i, where i=0, 1, 2,...L1; It represents the number of operations required to convert the first i characters of A1 into an empty string, that is, to delete i characters; D[0][j]=j, where j=0, 1, 2, ..., L2; It represents the number of operations required to convert an empty string into the first j characters of A2, that is, inserting j characters; StepA1.1.4. For i=0, 1, 2, ...L1 and j=0, 1, 2, ...L2, calculate D[i][j]: If the current characters are the same, that is, A1[i-1]=A2[j-1], then D[i][j]=D[i-1][j-1]; Otherwise, that is, the current characters are different, D[i][j]=min(D[i-1][j]+1, D[i][j-1]+1, D[i-1][j-1]+1); Among them, D[i-1][j]+1 means deleting the i-th character of A1; D[i][j-1]+1 means inserting the j-th character of A2 before the i-th character of A1; D[i-1][j-1]+1 means replacing the i-th character of A1 with the j-th character of A2; StepA1.1.5, by d(A1, A2) = D[L1][L2], we can get the edit distance d(A1, A2); StepA1.1.6, through: ; Calculate the name similarity S name (A1, A2).

[0011] As a further solution of the present invention: the specific method of calculating the structural similarity is as follows: StepA1.2.1, extract the number of all attributes in the table where A1 is located, and record it as N1, and extract the number of all attributes in the table where A2 is located, and record it as N2; StepA1.2.2, mark the data types corresponding to A1 and A2 as DT1 and DT2 respectively; StepA1.2.3, in the relational database, use KP1 and KP2 as marker variables to indicate whether A1 and A2 are primary keys; If A1 is the primary key, the value of KP1 is 1; if A1 is not the primary key, the value of KP1 is 0; If A2 is the primary key, the value of KP2 is 1; if A2 is not the primary key, the value of KP2 is 0; StepA1.2.4, through: ; Calculate the structural similarity S struct (A1, A2); In the formula, α1, α2, and α3 are the corresponding preset weight coefficients respectively, and “==” is a comparison operator; “==” is used to check whether the values ​​on the left and right sides are equal. If they are equal, the return value of this comparison expression is “1”, and if they are not equal, the return value is “0”.

[0012] As a further solution of the present invention: the fusion method in Step A2.1 is as follows: First, extract the values ​​of the specified numeric attributes of the same entity from m data sources and mark them as H e , e=1, 2, ... m; At the same time, the preset weight coefficients corresponding to each data source are extracted and recorded as E e ; Then through: ; Calculate the attribute value H0 after the fusion of the same entity data.

[0013] As a further solution of the present invention: the fusion method in Step A2.2 is as follows: Count the number of occurrences of each category value of the specified categorical attribute of the same entity from different data sources, and then use the category value with the largest number of occurrences as the fused category value.

[0014] As a further solution of the present invention: wherein, the edit distance is a measure of the difference between two strings; it calculates the minimum number of operations required to convert one string into another, and the operations include inserting, deleting and replacing characters.

[0015] As a further solution of the present invention: in StepA1.2.4, The size similarity of the tables where A1 and A2 are located is measured; Used to determine whether the data types are the same; Used to consider similarity of key constraints.

[0016] As a further solution of the present invention: the comprehensive analysis method is as follows: Step B1: Feature extraction Step B1.1: Structured data feature extraction In structured data, extract attribute parameters corresponding to key attributes; Then the mean, median and standard deviation of the attribute parameters corresponding to the key attributes are calculated and used as statistical features; Step B1.2: Unstructured data feature extraction Step B1.2.1, perform word segmentation on the text data; wherein, word segmentation is to divide the continuous text sequence into independent words according to certain rules; StepB1.2.2, count the number of times each word appears in the document; pass , calculate the frequency TF of the hth word in the fth document h，f; Among them, C h，f is the number of times the hth word appears in the fth document, h represents the word, h=1, 2, ...u, u represents the number of all words in the fth document, f represents the document, f=1, 2, ...v, v represents the number of documents; Step B1.2.3, select each word one by one, and count the number of documents containing each word, and record it as D h ; Then through , calculate the inverse document frequency IDF of each word h ; Step B1.2.4, through , calculate the word frequency-inverse document frequency TF-IDF of each word h，f ; And TF-IDF h，f As the feature vector corresponding to the text data; Step B2: Attribute association analysis Step B2.1, extracting attributes from different data sources but with matching relationships based on the results of pattern matching in the data integration module; StepB2.2, using the calculation formula of Pearson correlation coefficient: , calculate T1 g and T2 g The correlation coefficient Q 12 ; Where, T1 g 、T2 g are the number of observations of the matching attributes, g=1, 2, ...s, s represents the number of observations of the corresponding attribute, T1 p 、T2 p T1 g 、T2 g The average value of .

[0017] Beneficial effects of the present invention: The present invention can process related data in various formats including structured data and unstructured data, adapts to the complex and diverse data environment in reality, and can meet the needs of integrating and analyzing various types of data in different application scenarios.

[0018] The present invention performs pattern matching through a matching algorithm based on name similarity and structure similarity, comprehensively considers the edit distance of attribute names and multiple related similarity indicators to obtain pattern matching similarity, and compares it with a threshold to determine whether it matches. This method can more accurately identify entities, attributes and relationships with the same semantics from different data sources, thereby improving the accuracy of data integration.

[0019] The present invention adopts weighted average method for numerical data and voting method for categorized data to perform data fusion, and can perform reasonable fusion calculation according to the characteristics of different data types, so that the fused results can more accurately reflect the comprehensive situation of the same entity in different data sources, thereby ensuring the effectiveness and reliability of data integration.

[0020] In terms of feature extraction, the present invention can extract attribute parameters of key attributes for structured data and calculate their mean, median and standard deviation as statistical features. For unstructured text data, it can derive word frequency-inverse document frequency as a feature vector through operations such as word segmentation, statistical word frequency, and inverse document frequency. In this way, the feature information of the data can be fully mined from different angles, providing a rich data foundation for subsequent in-depth analysis.

[0021] The present invention, after extracting matching attributes according to the pattern matching results of the data integration module, uses the Pearson correlation coefficient formula to calculate the correlation coefficient between the corresponding matching attributes, which helps to accurately grasp the correlation relationship between the matching attributes in different data sources and further deepen the understanding and analysis of the data.

[0022] The present invention is provided with a result output unit, which can output and display the integration results and analysis results obtained by the data integration module and the data analysis module, so that users can intuitively and clearly obtain the final data situation after processing and analysis, which is convenient for subsequent decision-making, application and other operations.

[0023] To sum up, the multidimensional data integration and analysis system of the present invention can effectively improve the utilization efficiency and analysis value of multidimensional data in digital twin scenarios through its comprehensive data processing capabilities, precise data integration effects, in-depth data analysis functions and clear result output display, and provide strong data support for the development of related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below in conjunction with the accompanying drawings.

[0025] Figure 1 It is a system block diagram of the multidimensional data integration and analysis system for digital twins of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Embodiment 1 See also Figure 1 As shown, the present invention is a multidimensional data integration and analysis system for digital twins, comprising: A data collection module, used to collect relevant data from multiple data sources; In this embodiment, these data sources include databases, file systems, network interfaces, etc.; The collected data formats include structured data and unstructured data; Among them, structured data such as table data in relational databases, unstructured data such as text files, image data, etc.; Data integration module, used to collect relevant data from multiple data sources for integrated processing; The integration is handled as follows: StepA1: Pattern matching Analyze multidimensional data patterns from different data sources and identify entities, attributes, and relationships with the same semantics; It uses a matching algorithm based on name similarity and structural similarity; StepA1.1, name similarity calculation: StepA1.1.1, extract two attribute names and mark them as A1 and A2 respectively, and extract the lengths of the two attribute names and mark them as L1 and L2 respectively; StepA1.1.2. Create a two-dimensional array D of size (L1+1)×(L2+1); StepA1.1.3, Initialize the first row and first column: D[i][0]=i, where i=0, 1, 2,...L1; It represents the number of operations required to convert the first i characters of A1 into an empty string, that is, to delete i characters; D[0][j]=j, where j=0, 1, 2, ..., L2; It represents the number of operations required to convert an empty string into the first j characters of A2, that is, inserting j characters; StepA1.1.4. For i=0, 1, 2, ...L1 and j=0, 1, 2, ...L2, calculate D[i][j]: If the current characters are the same, that is, A1[i-1]=A2[j-1], then D[i][j]=D[i-1][j-1]; Otherwise, that is, the current characters are different, D[i][j]=min(D[i-1][j]+1, D[i][j-1]+1, D[i-1][j-1]+1); Among them, D[i-1][j]+1 means deleting the i-th character of A1; D[i][j-1]+1 means inserting the j-th character of A2 before the i-th character of A1; D[i-1][j-1]+1 means replacing the i-th character of A1 with the j-th character of A2; StepA1.1.5, by d(A1, A2) = D[L1][L2], we can get the edit distance d(A1, A2); Among them, the edit distance is a measure of the difference between two strings; it calculates the minimum number of operations required to transform one string into another, including inserting, deleting, and replacing characters; For example, for the strings "apple" and "aple", "apple" can be converted to "aple" by deleting the first "p", so the edit distance is 1; StepA1.1.6, through: ; Calculate the name similarity S name (A1, A2); For example, the value of d(A1, A2) is 2, the value of L1 is 11, the value of L2 is 9, and the value of max(L1, L2) is 11, then ; StepA1.2, structural similarity calculation: In this embodiment, the structural similarity calculation mainly considers factors such as the data table structure, data type, key constraints, etc. where the attribute is located; these factors combined can help determine whether two attributes are similar at the structural level; StepA1.2.1, extract the number of all attributes in the table where A1 is located, and record it as N1, and extract the number of all attributes in the table where A2 is located, and record it as N2; StepA1.2.2, mark the data types corresponding to A1 and A2 as DT1 and DT2 respectively; StepA1.2.3, in the relational database, use KP1 and KP2 as marker variables to indicate whether A1 and A2 are primary keys; If A1 is the primary key, the value of KP1 is 1; if A1 is not the primary key, the value of KP1 is 0; If A2 is the primary key, the value of KP2 is 1; if A2 is not the primary key, the value of KP2 is 0; For example, in a database containing a "student information table", the attribute A1 "student number" is usually the primary key, so for the attribute "student number", the corresponding primary key tag variable value is 1; while an attribute like "student name" is not a primary key, its corresponding primary key tag variable value is 0; StepA1.2.4, through: ; Calculate the structural similarity S struct (A1, A2); In the formula, α1, α2, and α3 are the corresponding preset weight coefficients respectively; In this example: The size similarity of the tables where A1 and A2 are located is measured; Used to determine whether the data types are the same; Similarity for considering key constraints; In the formula, "==" is a comparison operator; "==" is used to check whether the values ​​on the left and right sides are equal; if they are equal, the return value of this comparison expression is "1"; if they are not equal, the return value is "0"; If N1=5, N2=4, α1=0.3, then The value of is 2.7; If DT1 and DT2 are the same, then The value of is 1, otherwise it is 0; assuming α2=0.4, then The value of is 0.4; If A1 and A2 are both primary keys, that is, KP1=KP2=1; or if A1 and A2 are not primary keys, that is, KP1=KP2=0; then The value of is 1; If A1 and A2, one is a primary key and the other is not a primary key, then The value of is 0; Assume α3=0.3, and KP1=KP2, then The value of is 0.3; Under the above assumptions, ; StepA1.3, pattern matching similarity calculation pass: ; Calculate the pattern matching similarity S0 (A1, A2); In the formula, γ is the preset weight coefficient; StepA1.4, pattern matching determination Compare the pattern matching similarity S0(A1, A2) with the preset pattern matching similarity threshold SY0: If S0(A1, A2)>SY0, then A1 and A2 are considered to match; If S0(A1, A2) ≤ SY0, it is determined that there is no match between A1 and A2; Step A2: Data Fusion Based on pattern matching, data of the same entity from different data sources are fused; StepA2.1, for numerical data, weighted average method is used for fusion; The specific method is as follows: First, extract the values ​​of the specified numeric attributes of the same entity from m data sources and mark them as H e , e=1, 2, ... m; At the same time, the preset weight coefficients corresponding to each data source are extracted and recorded as E e ; Then through: ; Calculate the attribute value H0 after the fusion of the same entity data; StepA2.2, for the classified data, the voting method is used for fusion; The specific method is as follows: Count the number of occurrences of each category value of the specified categorical attribute of the same entity from different data sources, and then use the category value with the largest number of occurrences as the fused category value; The result output unit is used to output and display the integration results obtained by the data integration module in an intuitive, clear and easy-to-understand form; In this example: The attribute value H0 of the numerical data after fusion by weighted average method will be directly displayed in the form of numerical value, and the corresponding entity name and the attribute value from each data source before fusion will be marked; For the category values ​​of the categorized data after fusion through the voting method, the fused category values ​​will be directly displayed in text form, and the corresponding entity name and the category distribution of the categorized attributes of each data source before fusion will be marked.

[0028] This embodiment constructs the basic framework of a multidimensional data integration and analysis system for digital twins, which has many beneficial effects. Its data collection module can widely collect structured and unstructured data from a variety of data sources, providing rich materials for subsequent processing. The data integration module accurately identifies entities, attributes and relationships with the same semantics through a pattern matching algorithm, and then uses the weighted average method and voting method to reasonably integrate numerical and categorical data. Finally, the result output unit can display the integration results in an intuitive and clear form, which is convenient for users to understand and view the data integration situation, and effectively realizes the initial integration and presentation of data as a whole.

[0029] Embodiment 2 See also Figure 1As shown, as the second embodiment of the present invention, when the present application is implemented, compared with the first embodiment, the technical solution of this embodiment is different from that of the first embodiment only in that the second embodiment further includes: Data analysis module, used to collect relevant data from multiple data sources for comprehensive analysis; Step B1: Feature extraction Step B1.1: Structured data feature extraction In structured data, extract attribute parameters corresponding to key attributes; For example, for a database containing an "order information table", key attributes may include order number, order amount, order time, etc.; Calculate the mean, median and standard deviation of the attribute parameters corresponding to the key attributes and use them as statistical features; Step B1.2: Unstructured data feature extraction Step B1.2.1, perform word segmentation on the text data; wherein, word segmentation is to divide the continuous text sequence into independent words according to certain rules; StepB1.2.2, count the number of times each word appears in the document; pass , calculate the frequency TF of the hth word in the fth document h，f ; Among them, C h，f is the number of times the hth word appears in the fth document, h represents the word, h=1, 2, ...u, u represents the number of all words in the fth document, f represents the document, f=1, 2, ...v, v represents the number of documents; Step B1.2.3, select each word one by one, and count the number of documents containing each word, and record it as D h ; Then through , calculate the inverse document frequency IDF of each word h ; Step B1.2.4, through , calculate the word frequency-inverse document frequency TF-IDF of each word h，f ; And TF-IDF h，f As the feature vector corresponding to the text data; Step B2: Attribute association analysis Step B2.1, extracting attributes from different data sources but with matching relationships based on the results of pattern matching in the data integration module; For example, if the “user ID” attribute in database A is determined to match the “user ID” attribute in the file system in pattern matching; StepB2.2, using the calculation formula of Pearson correlation coefficient: , calculate T1 g and T2 g The correlation coefficient Q 12 ; Where, T1 g 、T2 g are the number of observations of the matching attributes, g=1, 2, ...s, s represents the number of observations of the corresponding attribute, T1 p 、T2 p T1 g 、T2 g The average value of In this embodiment, if the absolute value of the correlation coefficient is close to 1, it indicates that there is a strong linear association between the two attributes; if it is close to 0, it indicates that the association is weak; The result output unit is also used to output and display the analysis results obtained by the data analysis module in an intuitive, clear and easy-to-understand form; In this example: The statistical features obtained from the structured data analysis, such as the mean, median and standard deviation of the attribute parameters corresponding to the key attributes, will be presented in the form of a table; the column headings of the table are the key attribute name, mean, median and standard deviation, and each row corresponds to the relevant statistical information of a key attribute; The TF-IDF values ​​of each word calculated for unstructured text data will be displayed in the form of a word cloud. In the word cloud, the size of a word is proportional to its TF-IDF value, that is, the higher the TF-IDF value of a word, the larger the font size displayed in the word cloud, and the more eye-catching it is; The correlation coefficient Q12 between the matching attributes calculated by the Pearson correlation coefficient will be displayed in the form of a matrix diagram; the rows and columns of the matrix diagram correspond to different matching attributes, and the value of each cell in the matrix is ​​the correlation coefficient between the corresponding two attributes; This embodiment further expands the functions on the basis of the first embodiment, and the newly added data analysis module brings significant advantages. For the collected relevant data, it can extract key attribute parameters of structured data and calculate statistical features, obtain TF-IDF values ​​as feature vectors through series processing of unstructured text data, and can also perform attribute association analysis based on pattern matching results to obtain correlation coefficients. And the result output unit can display these analysis results in various intuitive forms such as tables, word cloud diagrams, matrix diagrams, etc., which helps to deeply explore the value of data, clearly present data characteristics and associations, so as to better assist decision-making and other applications.

[0030] Embodiment 3 See also Figure 1 As shown, as the third embodiment of the present invention, when the present application is specifically implemented, compared with the first and second embodiments, the technical solution of this embodiment is to combine the solutions of the first and second embodiments mentioned above for implementation.

[0031] This embodiment combines the solutions of embodiment one and embodiment two to fully integrate the advantages of both. It has the ability of comprehensive data collection, effective integration and clear display of integrated results of embodiment one, and the function of in-depth data analysis and diversified display of analysis results of embodiment two. This forms a complete data processing flow, which can fully meet the multi-dimensional data processing needs for digital twins, greatly improve the comprehensive data processing capabilities and utilization value, and provide more powerful support for related applications.

[0032] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0033] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multidimensional data integration and analysis system for digital twins, characterized by: include: A data collection module, used to collect relevant data from multiple data sources; Data integration module, used to collect relevant data from multiple data sources and perform integrated processing through pattern matching and data fusion; In the pattern matching stage, the matching algorithms based on name similarity and structure similarity are used to calculate the edit distance and related similarity indicators of the attribute names, and the pattern matching similarity is comprehensively obtained, and compared with the threshold to determine whether it matches; in the data fusion stage, the weighted average method is used for numerical data, and the numerical attribute values ​​of the same entity are fused and calculated according to the weights of different data sources; the voting method is used for categorical data, and the number of occurrences of the category values ​​of the categorical attributes of the same entity in each data source is counted, and the category value with the largest number of occurrences is selected as the fusion result; The data analysis module is used to collect relevant data from multiple data sources and conduct comprehensive analysis through feature extraction and attribute association analysis. In terms of feature extraction, the attribute parameters of key attributes are extracted for structured data and their mean, median and standard deviation are calculated as statistical features. For unstructured text data, the word segmentation is first performed, and then the word frequency and inverse document frequency are counted, and then the word frequency-inverse document frequency is obtained as the feature vector. In the attribute association analysis link, matching attributes are extracted based on the pattern matching results of the data integration module, and the correlation coefficient between the corresponding matching attributes is calculated using the Pearson correlation coefficient formula. The result output unit is used to output and display the integration results and analysis results obtained by the data integration module and the data analysis module.

2. The multidimensional data integration and analysis system for digital twins according to claim 1, characterized in that: The integration is handled as follows: StepA1: Pattern matching Analyze multidimensional data patterns from different data sources and identify entities, attributes, and relationships with the same semantics; StepA1.1, name similarity calculation: First, extract the two attribute names and their lengths. Then create a two-dimensional array of the specified size and initialize its first row and first column, which respectively represent the number of operations of deleting part of the characters of one attribute name and inserting an empty string to generate part of the characters of another attribute name. Then, for the other elements of the array, calculate their values ​​according to the corresponding rules based on whether the characters at the corresponding positions of the two attribute names are the same. Then, the edit distance is obtained from the last element of the array. Finally, the name similarity is calculated based on the edit distance using the specified formula and recorded as S. name (A1, A2); StepA1.2, structural similarity calculation: Extract the number of all attributes in the table corresponding to the two attributes respectively, then mark the data types corresponding to the two attributes, and determine whether the two attributes are primary keys in the relational database and assign values ​​according to the conditions. Finally, calculate the structural similarity of the two attributes and record it as S struct (A1, A2); StepA1.3, pattern matching similarity calculation pass: ; Calculate the pattern matching similarity S0 (A1, A2); In the formula, γ is the preset weight coefficient; StepA1.4, pattern matching determination Compare the pattern matching similarity S0(A1, A2) with the preset pattern matching similarity threshold SY0: If S0(A1, A2)>SY0, then A1 and A2 are considered to match; If S0(A1, A2) ≤ SY0, it is determined that there is no match between A1 and A2; Step A2: Data Fusion Based on pattern matching, data of the same entity from different data sources are fused; StepA2.1, for numerical data, weighted average method is used for fusion; StepA2.2: For categorized data, voting method is used for fusion.

3. The multidimensional data integration and analysis system for digital twins according to claim 2 is characterized in that: The specific method of calculating name similarity is as follows: StepA1.1.1, extract two attribute names and mark them as A1 and A2 respectively, and extract the lengths of the two attribute names and mark them as L1 and L2 respectively; StepA1.1.

2. Create a two-dimensional array D of size (L1+1)×(L2+1); StepA1.1.3, Initialize the first row and first column: D[i][0]=i, where i=0, 1, 2,...L1; It represents the number of operations required to convert the first i characters of A1 into an empty string, that is, to delete i characters; D[0][j]=j, where j=0, 1, 2, ..., L2; It represents the number of operations required to convert an empty string into the first j characters of A2, that is, inserting j characters; StepA1.1.

4. For i=0, 1, 2, ...L1 and j=0, 1, 2, ...L2, calculate D[i][j]: If the current characters are the same, that is, A1[i-1]=A2[j-1], then D[i][j]=D[i-1][j-1]; Otherwise, that is, the current characters are different, D[i][j]=min(D[i-1][j]+1, D[i][j-1]+1, D[i-1][j-1]+1); Among them, D[i-1][j]+1 means deleting the i-th character of A1; D[i][j-1]+1 means inserting the j-th character of A2 before the i-th character of A1; D[i-1][j-1]+1 means replacing the i-th character of A1 with the j-th character of A2; StepA1.1.5, by d(A1, A2) = D[L1][L2], we can get the edit distance d(A1, A2); StepA1.1.6, through: ; Calculate the name similarity S name (A1, A2).

4. The multidimensional data integration and analysis system for digital twins according to claim 2, characterized in that: The specific method of calculating structural similarity is as follows: StepA1.2.1, extract the number of all attributes in the table where A1 is located, and record it as N1, and extract the number of all attributes in the table where A2 is located, and record it as N2; StepA1.2.2, mark the data types corresponding to A1 and A2 as DT1 and DT2 respectively; StepA1.2.3, in the relational database, use KP1 and KP2 as marker variables to indicate whether A1 and A2 are primary keys; If A1 is the primary key, the value of KP1 is 1; if A1 is not the primary key, the value of KP1 is 0; If A2 is the primary key, the value of KP2 is 1; if A2 is not the primary key, the value of KP2 is 0; StepA1.2.4, through: ; Calculate the structural similarity S struct (A1, A2); In the formula, α1, α2, and α3 are the corresponding preset weight coefficients, and "==" is a comparison operator.

5. The multidimensional data integration and analysis system for digital twins according to claim 4 is characterized in that: The fusion method in StepA2.1 is as follows: First, extract the values ​​of the specified numeric attributes of the same entity from m data sources and mark them as H e , e=1, 2, ...m; At the same time, the preset weight coefficients corresponding to each data source are extracted and recorded as E e ; Then through: ; Calculate the attribute value H0 after the fusion of the same entity data.

6. The multidimensional data integration and analysis system for digital twins according to claim 4, characterized in that: The fusion method in StepA2.2 is as follows: Count the number of occurrences of each category value of the specified categorical attribute of the same entity from different data sources, and then use the category value with the largest number of occurrences as the fused category value.

7. The multidimensional data integration and analysis system for digital twins according to claim 3 is characterized in that: in, Levenshtein distance is a measure of the difference between two strings; it computes the minimum number of operations required to transform one string into the other, including insertions, deletions, and substitutions of characters.

8. The multidimensional data integration and analysis system for digital twins according to claim 4, characterized in that: In StepA1.2.4, Used to measure the size similarity of the tables where A1 and A2 are located; Used to determine whether the data types are the same; Used to analyze similarity of key constraints.

9. The multidimensional data integration and analysis system for digital twins according to claim 1, characterized in that: The comprehensive analysis method is as follows: Step B1: Feature extraction Step B1.1: Structured data feature extraction In structured data, extract attribute parameters corresponding to key attributes; Then the mean, median and standard deviation of the attribute parameters corresponding to the key attributes are calculated and used as statistical features; Step B1.2: Unstructured data feature extraction Step B1.2.1, perform word segmentation on the text data; wherein, word segmentation is to divide the continuous text sequence into independent words according to certain rules; StepB1.2.2, count the number of times each word appears in the document; pass , calculate the frequency TF of the hth word in the fth document h，f ; Among them, C h，f is the number of times the hth word appears in the fth document, h represents the word, h=1, 2, ...u, u represents the number of all words in the fth document, f represents the document, f=1, 2, ...v, v represents the number of documents; Step B1.2.3, select each word one by one, and count the number of documents containing each word, and record it as D h ; Then through , calculate the inverse document frequency IDF of each word h ; Step B1.2.4, pass , calculate the word frequency-inverse document frequency TF-IDF of each word h，f ; And TF-IDF h，f As the feature vector corresponding to the text data; Step B2: Attribute association analysis Step B2.1, extracting attributes from different data sources but with matching relationships based on the results of pattern matching in the data integration module; StepB2.2, using the calculation formula of Pearson correlation coefficient: , calculate T1 g and T2 g The correlation coefficient Q 12 ; Where, T1 g 、T2 g are the number of observations of the matching attributes, g=1, 2, ...s, s represents the number of observations of the corresponding attribute, T1 p 、T2 p T1 g 、T2 g The average value of .

10. The multidimensional data integration and analysis system for digital twins according to claim 1, characterized in that: in, The collected data formats include structured data and unstructured data.

Images