An intelligent data integration method and system based on data virtualization

Abstract

The present invention provides an intelligent data integration method and system based on data virtualization, belonging to the field of data integration technology, and the method includes: determining an association list of each heterogeneous data source that matches the demand type of business demand; analyzing the structural characteristics of each heterogeneous data source according to a predefined indicator set to obtain a parameter set, and constructing an analysis function corresponding to the heterogeneous data source; fusing all analysis functions to determine linear parameters and nonlinear parameters based on data integration, and configuring abstract symbols to the corresponding heterogeneous data sources respectively in combination with the association list of each heterogeneous data source; parsing business demand and optimizing the abstract symbol to obtain a unified symbol, so as to realize data integration of multiple heterogeneous data sources. Data integration is realized to effectively reduce the number of operations of calling and improve development efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of data integration, and in particular to an intelligent data integration method and system based on data virtualization. Background Art

[0002] Data virtualization is a data management approach that abstracts multiple heterogeneous data sources into a unified data layer and provides it to applications or users. Data virtualization is a bridge that enables the platform to access and display data from external source systems on demand. This innovative approach centralizes and simplifies data management without the platform itself actually storing the data. It establishes a virtual layer between data sources and users, enabling organizations to access and manage their data without copying or moving the data from its original location.

[0003] When using data virtualization, the relationship between external data sources is not considered during the data integration process. That is, the required data is retrieved from each external data source in turn. The number of source retrieval operations is not reduced due to the relationship between the sources, which indirectly leads to a decrease in the efficiency of application development.

[0004] Therefore, the present invention proposes an intelligent data integration method based on data virtualization. Summary of the invention

[0005] The present invention provides an intelligent data integration method and system based on data virtualization, which is used to accurately identify the associations and important situations of different sources based on the association list of heterogeneous data sources involved in the needs and the analysis function constructed by the predefined indicator set, and to set abstract symbols and symbol optimization through fusion processing to achieve data integration, so as to effectively reduce the number of call operations and improve development efficiency.

[0006] The present invention provides an intelligent data integration method based on data virtualization, comprising:

[0007] Step 1: Determine the associated list of each heterogeneous data source that matches the requirement type of the business need;

[0008] Step 2: Analyze the structural characteristics of each heterogeneous data source according to the predefined indicator set to obtain a parameter set, and construct an analysis function corresponding to the heterogeneous data source;

[0009] Step 3: All analysis functions are integrated to determine linear parameters and nonlinear parameters based on data integration, and abstract symbols are configured for the corresponding heterogeneous data sources in combination with the association list of each heterogeneous data source;

[0010] Step 4: Analyze the business requirements and optimize the abstract symbols to obtain unified symbols, so as to realize data integration of multiple heterogeneous data sources.

[0011] Preferably, step 1 comprises:

[0012] Acquire several heterogeneous data sources matching the requirement type of the business requirement from a type-source database;

[0013] An association analysis is performed on the data content of each heterogeneous data source to obtain an association list corresponding to the heterogeneous data source, wherein the association list includes an association content set with each remaining data source, a degree of association with each associated content, and a content importance of the corresponding associated content based on the corresponding heterogeneous data source.

[0014] Preferably, constructing an analysis function corresponding to heterogeneous data sources includes:

[0015] According to the standard conversion relationship between each predefined indicator in the predefined indicator set and the structural characteristics of the heterogeneous data source, the corresponding structural characteristics are converted into a first parameter value based on the corresponding predefined indicator, and a parameter set is formed;

[0016] According to the application scenario of each heterogeneous data source, a parameter description weight matrix corresponding to the heterogeneous data source is constructed. ,in, The parameter description weight vector representing the first application scenario; Represents the parameter description weight vector of the nth application scenario; the value of i is 1, 2, 3, ..., m, and m represents the parameter description of the mth predefined indicator;

[0017] Capturing the scenario log for issuing the business requirement, and extracting the scenario-specific representation in the scenario log, and then determining the representation weight consistent with the corresponding scenario-specific representation according to the representation coverage value and the semantic association value of the scenario-specific representation;

[0018] ;

[0019] in, The semantic association value representing the corresponding scene-specific representation; represents the length of the scene-specific representation associated with the position of the dth log paragraph in the scene log; Indicates the length of the dth log paragraph in the scenario log; Indicates that based on all The variance of represents the correlation coverage ratio of the corresponding scene-specific representation in the d-th log paragraph of the scene log; represents the coverage ratio of the number of times the corresponding scene-specific representation is represented in the d-th log paragraph of the scene log; represents the representation coverage value corresponding to the scene-specific representation; represents the number of occurrences of the corresponding scene-specific representation in the dth log paragraph in the scene log; represents the total number of scene feature representations in the dth log paragraph in the scene log; D represents the total number of log paragraphs in the scene log; Indicates that based on all The variance of Indicates that except for the corresponding scene feature representation The variance of the corresponding correlation coverage ratio under the remaining scene-specific representations other than ; Indicates that except for the corresponding scene feature representation The variance of the frequency coverage ratio corresponding to the remaining scene-specific representations other than ;

[0020] The representation weights are sorted by size, and the required scenes corresponding to the weights greater than the preset weights are selected;

[0021] Adjusting the representation weight of the required scene to obtain the dimension weight of the corresponding scene dimension;

[0022] ;

[0023] in, represents the dimension weight of the r1th required scene; represents the representation weight of the r1th required scene; M1 represents the number of required scenes; M2 represents the number of required scenes corresponding to the preset weight not greater than; ln represents the logarithmic function symbol; represents the sum of all representation weights less than the preset weight; represents the variance of the remaining weights after removing the representation weight corresponding to the r1th required scene from the representation weights that are greater than the preset weights;

[0024] Relying on the scene dimensions and dimension weights, and combining with A0, construct the current weight vector of the corresponding parameter set;

[0025] Based on the current weight vector and parameter set, an analysis function is constructed.

[0026] Preferably, the current weight vector of the corresponding parameter set is constructed based on the scene dimension and the dimension weight and combined with A0, including:

[0027] Based on the scenario dimensions and dimension weights, construct dimension horizontal quantities;

[0028] According to the order of the required scenes described in the dimensional horizontal quantity, vectors are extracted from the parameter description weight matrix A0 to obtain the required description matrix B0, and the dimensional horizontal quantity is respectively calculated with each column in the required description matrix B0 to obtain the weight of the corresponding parameter description, thereby obtaining the current weight vector.

[0029] Preferably, all analysis functions are fused to determine linear parameters and nonlinear parameters based on data integration, including:

[0030] Extract the parameter coefficients of the same parameter variable in all analysis functions and draw the coefficient curve to determine whether the corresponding coefficient curve tends to a horizontal linear relationship;

[0031] If it tends to, the corresponding parameter variable is first locked to obtain the linear parameter;

[0032] Otherwise, the corresponding parameter variables are locked for the second time to obtain nonlinear parameters.

[0033] Preferably, in combination with the association list of each heterogeneous data source, the abstract symbol is configured for the corresponding heterogeneous data source respectively, including:

[0034] The associated content set of each heterogeneous data source is divided according to the linear parameter and the nonlinear parameter to obtain the linear content for each linear parameter and the nonlinear content for each nonlinear parameter;

[0035] Obtaining a first total degree according to the degree of association of each linear content with each associated content, and obtaining a screening value according to the importance of the content of each linear content based on the corresponding heterogeneous data source;

[0036] Extracting the maximum value from all the screening values ​​corresponding to the same linear parameter, locking the matching content of the heterogeneous data source to which the maximum value belongs, and setting the first integration symbol;

[0037] Setting a call symbol to the matching contents of the remaining heterogeneous data sources corresponding to the same linear parameter, and combining the first integration symbol to set a first final symbol for the same linear parameter in different heterogeneous data sources;

[0038] Obtaining a second total degree according to the degree of association with each associated content involved in each nonlinear content, and sorting all second total degrees involved in the same nonlinear parameter to sequentially match the contents of the corresponding heterogeneous data sources, and setting second final symbols respectively;

[0039] The final symbols set for the parameter variables involved in each heterogeneous data source are counted to obtain abstract symbols.

[0040] Preferably, parsing the business requirements and optimizing the abstract symbols to obtain unified symbols includes:

[0041] Input the business requirements into the requirements analysis model to obtain the parameter importance of each requirement parameter;

[0042] Compare and analyze the demand parameters with the parameter variables, and obtain the non-important variables in combination with the parameter importance;

[0043] Add non-unique symbols to the first final symbols belonging to linear parameters in non-significant variables;

[0044] Add negligible symbols to the second final symbols of non-linear parameters in non-significant variables;

[0045] Based on the addition results, the abstract symbol is optimized to obtain a unified symbol.

[0046] The present invention provides an intelligent data integration system based on data virtualization, comprising:

[0047] A list determination module, for determining an associated list of each heterogeneous data source matching a requirement type of the business requirement;

[0048] A function construction module is used to analyze the structural characteristics of each heterogeneous data source according to a predefined indicator set to obtain a parameter set and construct an analysis function corresponding to the heterogeneous data source;

[0049] A fusion processing module is used to perform fusion processing on all analysis functions, determine linear parameters and nonlinear parameters based on data integration, and configure abstract symbols to the corresponding heterogeneous data sources respectively in combination with the association list of each heterogeneous data source;

[0050] The symbol optimization module is used to analyze the business requirements and optimize the abstract symbols to obtain unified symbols, so as to realize data integration of multiple heterogeneous data sources.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] Based on the association list of heterogeneous data sources involved in the needs and the analysis function constructed by the predefined indicator set, the association and important situations of different sources can be accurately determined. By fusion processing, linear and nonlinear parameters are determined to set abstract symbols and symbol optimization to achieve data integration, which can effectively reduce the number of call operations and improve development efficiency.

[0053] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0054] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0056] Figure 1 This is a flow chart of an intelligent data integration method based on data virtualization in an embodiment of the present invention;

[0057] Figure 2 This is a structural diagram of an intelligent data integration system based on data virtualization in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0059] The present invention provides an intelligent data integration method based on data virtualization, such as Figure 1 As shown, including:

[0060] Step 1: Determine the associated list of each heterogeneous data source that matches the requirement type of the business need;

[0061] Step 2: Analyze the structural characteristics of each heterogeneous data source according to the predefined indicator set to obtain a parameter set, and construct an analysis function corresponding to the heterogeneous data source;

[0062] Step 3: All analysis functions are integrated to determine linear parameters and nonlinear parameters based on data integration, and abstract symbols are configured for the corresponding heterogeneous data sources in combination with the association list of each heterogeneous data source;

[0063] Step 4: Analyze the business requirements and optimize the abstract symbols to obtain unified symbols, so as to realize data integration of multiple heterogeneous data sources.

[0064] In this embodiment, the requirement type is obtained by matching from a requirement-type comparison table, and the comparison table includes different business requirements and types that match the requirements. For example, the business requirement is to roughly retrieve resource information related to building application A.

[0065] In this embodiment, one heterogeneous data source corresponds to one association list, and the association list includes the associated content set of the heterogeneous data source and each remaining heterogeneous data source, the degree of association with each associated content, and the content importance of the corresponding associated content based on the corresponding heterogeneous data source.

[0066] The relevance is calculated based on a similarity function, and the importance of each related content is pre-set, that is, what kind of data exists in the heterogeneous data source is pre-set, and the importance of the data is also pre-set.

[0067] In this embodiment, the associated content set includes associated content that has a similar relationship between the heterogeneous data source and the corresponding remaining heterogeneous data sources, and the correlation degree ranges from 0 to 1, and each heterogeneous data source contains a number of set data contents, and the association is the association of data contents.

[0068] For example, heterogeneous data source 1 includes: data content 01, data content 02, and heterogeneous data source 2 includes: data content 01, data content 03. At this time, data content 01 in heterogeneous data source 1 is associated with data content 01 in heterogeneous data source 2, and the degree of association is 1.

[0069] In this embodiment, the structural features are related to the data structure, storage format, access method, and data content meaning of each data content in the heterogeneous data source, and the predefined indicator set is: structural indicator, storage indicator, access indicator, and meaning indicator.

[0070] In this embodiment, the parameter set is: the value of the data structure of each data content, the value of the storage format of each data content, the value of the access method of each data content, and the value of the content meaning of each data content.

[0071] In this embodiment, the analysis function = F (parameter set, current weight vector).

[0072] In this embodiment, the fusion process is to determine whether the same parameter is a linear parameter, so as to facilitate the configuration of symbols for the data source, and the abstract symbol is a symbol combination corresponding to each parameter in the heterogeneous data source.

[0073] In this embodiment, the unified symbols are used to further unify the processing of rough resources and reduce the number of call operations.

[0074] The beneficial effects of the above technical solution are: based on the association list of heterogeneous data sources involved in the needs and the analysis function constructed by the predefined indicator set, the association and important situations of different sources can be accurately determined, and the linear and nonlinear parameters are determined through fusion processing to set abstract symbols and symbol optimization to achieve data integration, so as to effectively reduce the number of call operations and improve development efficiency.

[0075] The present invention provides an intelligent data integration method based on data virtualization, step 1, comprising:

[0076] Acquire several heterogeneous data sources matching the requirement type of the business requirement from a type-source database;

[0077] An association analysis is performed on the data content of each heterogeneous data source to obtain an association list corresponding to the heterogeneous data source, wherein the association list includes an association content set with each remaining data source, a degree of association with each associated content, and a content importance of the corresponding associated content based on the corresponding heterogeneous data source.

[0078] In this embodiment, the type-source database includes heterogeneous data sources matching the required type, which are pre-set and convenient for direct retrieval.

[0079] In this embodiment, the association analysis is obtained through similarity function analysis.

[0080] The beneficial effects of the above technical solution are: heterogeneous data sources are obtained through type matching, and through association analysis, it is easy to build a content-based association list, providing a basis for subsequent symbol settings.

[0081] The present invention provides an intelligent data integration method based on data virtualization, which constructs an analysis function corresponding to heterogeneous data sources, including:

[0082] According to the standard conversion relationship between each predefined indicator in the predefined indicator set and the structural characteristics of the heterogeneous data source, the corresponding structural characteristics are converted into a first parameter value based on the corresponding predefined indicator, and a parameter set is formed;

[0083] According to the application scenario of each heterogeneous data source, a parameter description weight matrix corresponding to the heterogeneous data source is constructed. ,in, The parameter description weight vector representing the first application scenario; Represents the parameter description weight vector of the nth application scenario; the value of i is 1, 2, 3, ..., m, and m represents the parameter description of the mth predefined indicator;

[0084] Capturing the scenario log for issuing the business requirement, and extracting the scenario-specific representation in the scenario log, and then determining the representation weight consistent with the corresponding scenario-specific representation according to the representation coverage value and the semantic association value of the scenario-specific representation;

[0085] ;

[0086] in, The semantic association value representing the corresponding scene-specific representation; represents the length of the scene-specific representation associated with the position of the dth log paragraph in the scene log; Indicates the length of the dth log paragraph in the scenario log; Indicates that based on all The variance of represents the correlation coverage ratio of the corresponding scene-specific representation in the d-th log paragraph of the scene log; represents the coverage ratio of the number of times the corresponding scene-specific representation is represented in the d-th log paragraph of the scene log; represents the representation coverage value corresponding to the scene-specific representation; represents the number of occurrences of the corresponding scene-specific representation in the dth log paragraph in the scene log; represents the total number of scene feature representations in the dth log paragraph in the scene log; D represents the total number of log paragraphs in the scene log; Indicates that based on all The variance of Indicates that except for the corresponding scene feature representation The variance of the corresponding correlation coverage ratio under the remaining scene-specific representations other than ; Indicates that except for the corresponding scene feature representation The variance of the frequency coverage ratio corresponding to the remaining scene-specific representations other than ;

[0087] The representation weights are sorted by size, and the required scenes corresponding to the weights greater than the preset weights are selected;

[0088] Adjusting the representation weight of the required scene to obtain the dimension weight of the corresponding scene dimension;

[0089] ;

[0090] in, represents the dimension weight of the r1th required scene; represents the representation weight of the r1th required scene; M1 represents the number of required scenes; M2 represents the number of required scenes corresponding to the preset weight not greater than; ln represents the logarithmic function symbol; represents the sum of all representation weights less than the preset weight; represents the variance of the remaining weights after removing the representation weight corresponding to the r1th required scene from the representation weights that are greater than the preset weights;

[0091] Relying on the scene dimensions and dimension weights, and combining with A0, construct the current weight vector of the corresponding parameter set;

[0092] Based on the current weight vector and parameter set, an analysis function is constructed.

[0093] In this embodiment, the standard conversion relationship is to convert the structural features into numerical values ​​under the corresponding indicators, so as to obtain the values ​​of the corresponding indicators, and to standardize the indicators to facilitate the construction of parameter sets.

[0094] In this embodiment, the parameter set includes the first parameter value of each data content in the same heterogeneous data source under different predefined indicators.

[0095] In this embodiment, the application scenario refers to the development scenario applied to the corresponding heterogeneous data sources, such as takeaway applications, shopping applications, film and television applications, etc. In this process, the parameter description weights involved in different applications are pre-set, so the parameter description weight matrix A0 can be directly constructed.

[0096] In this embodiment, the scenario log is obtained by capturing the sending end when sending the service demand, and the scenario log is related to the service type of the sending end (client).

[0097] In this embodiment, the scene-specific representation is pre-set, for example, %% represents scene 1, Represents scenario 2, and so on, so the locks of the relevant representations can be made directly from the log.

[0098] In this embodiment, the weight is: ,in, Indicates the r2th representation weight.

[0099] In this embodiment, the preset weight value is 0.5.

[0100] In this embodiment, the number of required scenes is consistent with the number of dimensions.

[0101] In this embodiment, the required scenario is actually determined by capturing logs of the issued requirements, and then the representation weight is adjusted, thereby effectively ensuring the reliability of the scenario required in the business requirements and providing a basis for subsequent application development.

[0102] The beneficial effect of the above technical solution is: by determining the application scenarios of heterogeneous data sources to construct the matrix A0, and combining the scenario logs under business needs to screen the required scenarios, and then adjusting the weights to determine the reliability of the scenario existence, ensuring the accuracy of the current weight vector construction in the parameter set, and then facilitating the construction of analysis functions, providing a basis for subsequent function fusion processing.

[0103] The present invention provides an intelligent data integration method based on data virtualization, which relies on the scenario dimension and the dimension weight, and combines A0 to construct the current weight vector of the corresponding parameter set, including:

[0104] Based on the scenario dimensions and dimension weights, construct dimension horizontal quantities;

[0105] According to the order of the required scenes described in the dimensional horizontal quantity, vectors are extracted from the parameter description weight matrix A0 to obtain the required description matrix B0, and the dimensional horizontal quantity is respectively calculated with each column in the required description matrix B0 to obtain the weight of the corresponding parameter description, thereby obtaining the current weight vector.

[0106] In this embodiment, the dimensional lateral amount = {the dimensional weight of the required scene 1, ..., the dimensional weight of the required scene N};

[0107] Required Description Matrix ,in, The parameter description weight vector representing the first required scenario; The parameter description weight vector representing the Nth desired scenario.

[0108] In this embodiment, the calculation is: W1 = lateral dimension × ,...,Wm=dimensional lateral quantity× .

[0109] In this embodiment, the current weight vector is a vector of 1 row and m columns, that is, [W1, ..., Wm].

[0110] The beneficial effect of the above technical solution is: based on the constructed dimensional lateral quantity and each column in the required description matrix B0, calculation is performed to facilitate obtaining the current weight vector, thereby providing a basis for the construction of subsequent analysis functions.

[0111] The present invention provides an intelligent data integration method based on data virtualization, which performs fusion processing on all analysis functions and determines linear parameters and nonlinear parameters based on data integration, including:

[0112] Extract the parameter coefficients of the same parameter variable in all analysis functions and draw the coefficient curve to determine whether the corresponding coefficient curve tends to a horizontal linear relationship;

[0113] If it tends to, the corresponding parameter variable is first locked to obtain the linear parameter;

[0114] Otherwise, the corresponding parameter variables are locked for the second time to obtain nonlinear parameters.

[0115] In this embodiment, the current weight vector and parameter set involved in each analysis function are input into the parameter analysis model to obtain the true expression of the analysis function, that is: analysis function = a1x1+a2x2+a3x3+....+aRxR, wherein x1,...,xR are parameter variables involved in the corresponding analysis function in sequence, and a1,...,aR are parameter coefficients corresponding to each parameter in sequence, wherein the parameter analysis model is pre-trained, and the neural network model is trained using the vectors and parameter sets involved in the analysis function and the expressions of the functions as samples, therefore, the expression of the corresponding analysis function can be directly obtained.

[0116] In this embodiment, the parameter coefficients of the same parameter variable in each analysis function are extracted respectively to draw the curve. That is, when the curve tends to be horizontal (the linear coefficient tends to 0), the corresponding parameter variable is retained. It should be noted that the fluctuation of the coefficient curve should be controlled within the range of (the average value of all coefficients on the curve - b1, the average value of all coefficients on the curve + b1).

[0117] The beneficial effect of the above technical solution is: constructing the true expression of each analytical function based on the model, and then drawing the curve by extracting the parameter coefficients of the same parameter variable, effectively obtaining linear parameters and nonlinear parameters, and providing a basis for subsequent setting of symbols.

[0118] The present invention provides an intelligent data integration method based on data virtualization, which configures abstract symbols to the corresponding heterogeneous data sources respectively in combination with the association list of each heterogeneous data source, including:

[0119] The associated content set of each heterogeneous data source is divided according to the linear parameter and the nonlinear parameter to obtain the linear content for each linear parameter and the nonlinear content for each nonlinear parameter;

[0120] Obtaining a first total degree according to the degree of association of each linear content with each associated content, and obtaining a screening value according to the importance of the content of each linear content based on the corresponding heterogeneous data source;

[0121] Extracting the maximum value from all the screening values ​​corresponding to the same linear parameter, locking the matching content of the heterogeneous data source to which the maximum value belongs, and setting the first integration symbol;

[0122] Setting a call symbol to the matching contents of the remaining heterogeneous data sources corresponding to the same linear parameter, and combining the first integration symbol to set a first final symbol for the same linear parameter in different heterogeneous data sources;

[0123] Obtaining a second total degree according to the degree of association with each associated content involved in each nonlinear content, and sorting all second total degrees involved in the same nonlinear parameter to sequentially match the contents of the corresponding heterogeneous data sources, and setting second final symbols respectively;

[0124] The final symbols set for the parameter variables involved in each heterogeneous data source are counted to obtain abstract symbols.

[0125] In this embodiment, linear content+non-linear content=associated content set.

[0126] In this embodiment, there is a comparison relationship table between parameters and contents. Therefore, after the linear parameters and nonlinear parameters are determined, the corresponding contents for the linear parameters and nonlinear parameters can be obtained according to the comparison relationship table. For example, parameter 1 corresponds to content 01 and content 02, parameter 2 corresponds to content 03, and so on, so that linear content and sub-linear content can be obtained.

[0127] In this embodiment, the first total degree=the sum of the degrees of association of each associated content associated with the corresponding linear content. It should be noted that each linear parameter may have a first total degree in different heterogeneous data sources.

[0128] In this embodiment, the screening value=first total degree×ln(e+content importance), where e is a constant and its value is 2.7.

[0129] In this embodiment, the locked matching content is the data content corresponding to the linear parameter in the heterogeneous data source to which the maximum value belongs. The first integrated symbol is set to uniquely represent the locked matching content, which is convenient for symbol replacement and subsequent retrieval, and can effectively reduce space occupancy.

[0130] In this embodiment, the symbols are retrieved in order to reasonably display the matching contents of the remaining heterogeneous data sources, that is, to intuitively know that such contents are similar.

[0131] For example, the first integration symbol is: , the calling symbol for the matching content of the first remaining heterogeneous data source of the same linear parameter is d01. At this time, the first final symbol is: .

[0132] In this embodiment, the calculation method of the second total degree is similar to the calculation method of the first total degree, and will not be repeated here.

[0133] In this embodiment, the second final symbol is also unique and is obtained based on matching of a parameter-content-symbol comparison table, where the table contains representative symbols of data contents under different nonlinear parameters.

[0134] In this embodiment, abstract symbol: the final symbol to which each parameter variable relates.

[0135] The beneficial effects of the above technical solution are: linear and nonlinear content are obtained by dividing the associated content set according to linearity and nonlinearity, and symbols are set by classifying and discussing the linear content and the nonlinear content, so as to effectively achieve the unique representation of the content, and call symbols are set for linear parameters, so that similar content can be used directly, effectively reducing the call operation for similar content, and by setting symbols, the space occupancy is reduced, which effectively improves the development efficiency.

[0136] The present invention provides an intelligent data integration method based on data virtualization, which analyzes the business requirements and optimizes the abstract symbols to obtain unified symbols, including:

[0137] Input the business requirements into the requirements analysis model to obtain the parameter importance of each requirement parameter;

[0138] Compare and analyze the demand parameters with the parameter variables, and obtain the non-important variables in combination with the parameter importance;

[0139] Add non-unique symbols to the first final symbols belonging to linear parameters in non-significant variables;

[0140] Add negligible symbols to the second final symbols of non-linear parameters in non-significant variables;

[0141] Based on the addition results, the abstract symbol is optimized to obtain a unified symbol.

[0142] In this embodiment, the demand analysis model is obtained by training a neural network model based on different business requirements and the importance of parameters in the requirements as samples. Therefore, the parameter importance of different demand parameters can be directly obtained.

[0143] In this embodiment, the comparative analysis is implemented based on a demand-variable comparison table, and the table contains parameter variables that can be matched with different parameter parameters, and non-important variables refer to demand parameters whose parameter importance is less than the preset importance, and parameter variables that match demand parameters less than the preset importance.

[0144] Since the parameter variables include linear variables and nonlinear variables, they are divided again after the non-significant variables are determined.

[0145] In this embodiment, the purpose of the non-unique symbol is to indicate that the first final symbol of the parameter may be a corresponding first integrated symbol, and the data content corresponding to the first integrated symbol may be used during the application development process.

[0146] In this embodiment, the purpose of the ignorable symbol is to indicate that the second final symbol of the parameter can be slightly ignored, that is, the data content corresponding to the second final symbol can be ignored and may not be used in the application development process.

[0147] In this embodiment, the unified symbol is: all adjusted symbols.

[0148] The beneficial effect of the above technical solution is that through demand analysis and variable comparison analysis, new symbols can be set for related symbols of variables to further reduce data usage under different parameters, thereby improving development efficiency.

[0149] The present invention provides an intelligent data integration system based on data virtualization, such as Figure 2 As shown, including:

[0150] A list determination module, for determining an associated list of each heterogeneous data source matching a requirement type of the business requirement;

[0151] A function construction module is used to analyze the structural characteristics of each heterogeneous data source according to a predefined indicator set to obtain a parameter set and construct an analysis function corresponding to the heterogeneous data source;

[0152] A fusion processing module is used to perform fusion processing on all analysis functions, determine linear parameters and nonlinear parameters based on data integration, and configure abstract symbols to the corresponding heterogeneous data sources respectively in combination with the association list of each heterogeneous data source;

[0153] The symbol optimization module is used to analyze the business requirements and optimize the abstract symbols to obtain unified symbols, so as to realize data integration of multiple heterogeneous data sources.

[0154] The beneficial effects of the above technical solution are: based on the association list of heterogeneous data sources involved in the needs and the analysis function constructed by the predefined indicator set, the association and important situations of different sources can be accurately determined, and the linear and nonlinear parameters are determined through fusion processing to set abstract symbols and symbol optimization to achieve data integration, so as to effectively reduce the number of call operations and improve development efficiency.

[0155] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An intelligent data integration method based on data virtualization, characterized in that: include: Step 1: Determine the associated list of each heterogeneous data source that matches the requirement type of the business need; Step 2: Analyze the structural characteristics of each heterogeneous data source according to the predefined indicator set to obtain a parameter set, and construct an analysis function corresponding to the heterogeneous data source; Step 3: All analysis functions are integrated to determine linear parameters and nonlinear parameters based on data integration, and abstract symbols are configured for the corresponding heterogeneous data sources in combination with the association list of each heterogeneous data source; Step 4: Analyze the business requirements and optimize the abstract symbols to obtain unified symbols, so as to realize data integration of multiple heterogeneous data sources; Among them, building analysis functions corresponding to heterogeneous data sources includes: According to the standard conversion relationship between each predefined indicator in the predefined indicator set and the structural characteristics of the heterogeneous data source, the corresponding structural characteristics are converted into a first parameter value based on the corresponding predefined indicator, and a parameter set is formed; According to the application scenario of each heterogeneous data source, a parameter description weight matrix corresponding to the heterogeneous data source is constructed. ,in, The parameter description weight vector representing the first application scenario; Represents the parameter description weight vector of the nth application scenario; the value of i is 1, 2, 3, ..., m, and m represents the parameter description of the mth predefined indicator; Capturing the scenario log for issuing the business requirement, and extracting the scenario-specific representation in the scenario log, and then determining the representation weight consistent with the corresponding scenario-specific representation according to the representation coverage value and the semantic association value of the scenario-specific representation; ； in, The semantic association value representing the corresponding scene-specific representation; represents the length of the scene-specific representation associated with the position of the dth log paragraph in the scene log; Indicates the length of the dth log paragraph in the scenario log; Indicates that based on all The variance of represents the correlation coverage ratio of the corresponding scene-specific representation in the d-th log paragraph of the scene log; represents the coverage ratio of the number of times the corresponding scene-specific representation is represented in the d-th log paragraph of the scene log; represents the representation coverage value corresponding to the scene-specific representation; represents the number of occurrences of the corresponding scene-specific representation in the dth log paragraph in the scene log; represents the total number of scene feature representations in the dth log paragraph in the scene log; D represents the total number of log paragraphs in the scene log; Indicates that based on all The variance of Indicates that except for the corresponding scene feature representation The variance of the corresponding correlation coverage ratio under the remaining scene-specific representations other than ; Indicates that except for the corresponding scene feature representation The variance of the frequency coverage ratio corresponding to the remaining scene-specific representations other than ; The representation weights are sorted by size, and the required scenes corresponding to the weights greater than the preset weights are selected; Adjusting the representation weight of the required scene to obtain the dimension weight of the corresponding scene dimension; ； in, represents the dimension weight of the r1th required scene; represents the representation weight of the r1th required scene; M1 represents the number of required scenes; M2 represents the number of required scenes corresponding to the preset weight not greater than; ln represents the logarithmic function symbol; represents the sum of all representation weights less than the preset weight; represents the variance of the remaining weights after removing the representation weight corresponding to the r1th required scene from the representation weights that are greater than the preset weights; Relying on the scene dimensions and dimension weights, and combining with A0, construct the current weight vector of the corresponding parameter set; Based on the current weight vector and parameter set, an analysis function is constructed.

2. The intelligent data integration method based on data virtualization according to claim 1 is characterized in that: Step 1 includes: Acquire several heterogeneous data sources matching the requirement type of the business requirement from a type-source database; An association analysis is performed on the data content of each heterogeneous data source to obtain an association list corresponding to the heterogeneous data source, wherein the association list includes an association content set with each remaining data source, a degree of association with each associated content, and a content importance of the corresponding associated content based on the corresponding heterogeneous data source.

3. The intelligent data integration method based on data virtualization according to claim 1 is characterized in that: Relying on the scene dimensions and dimension weights, and in combination with A0, the current weight vector of the corresponding parameter set is constructed, including: Based on the scenario dimensions and dimension weights, construct dimension horizontal quantities; According to the order of the required scenes described in the dimensional horizontal quantity, vectors are extracted from the parameter description weight matrix A0 to obtain the required description matrix B0, and the dimensional horizontal quantity is respectively calculated with each column in the required description matrix B0 to obtain the weight of the corresponding parameter description, thereby obtaining the current weight vector.

4. The intelligent data integration method based on data virtualization according to claim 1 is characterized in that: All analytical functions are integrated to determine linear and nonlinear parameters based on data integration, including: Extract the parameter coefficients of the same parameter variable in all analysis functions and draw the coefficient curve to determine whether the corresponding coefficient curve tends to a horizontal linear relationship; If it tends to, the corresponding parameter variable is first locked to obtain the linear parameter; Otherwise, the corresponding parameter variables are locked for the second time to obtain nonlinear parameters.

5. The intelligent data integration method based on data virtualization according to claim 1 is characterized in that: Combine the association list of each heterogeneous data source to configure abstract symbols for the corresponding heterogeneous data source, including: The associated content set of each heterogeneous data source is divided according to the linear parameter and the nonlinear parameter to obtain the linear content for each linear parameter and the nonlinear content for each nonlinear parameter; Obtaining a first total degree according to the degree of association of each linear content with each associated content, and obtaining a screening value according to the importance of the content of each linear content based on the corresponding heterogeneous data source; Extracting the maximum value from all the screening values ​​corresponding to the same linear parameter, locking the matching content of the heterogeneous data source to which the maximum value belongs, and setting the first integration symbol; Setting a call symbol to the matching contents of the remaining heterogeneous data sources corresponding to the same linear parameter, and combining the first integration symbol to set a first final symbol for the same linear parameter in different heterogeneous data sources; Obtaining a second total degree according to the degree of association with each associated content involved in each nonlinear content, and sorting all second total degrees involved in the same nonlinear parameter to sequentially match the contents of the corresponding heterogeneous data sources, and setting second final symbols respectively; The final symbols set for the parameter variables involved in each heterogeneous data source are counted to obtain abstract symbols.

6. The intelligent data integration method based on data virtualization according to claim 1, characterized in that: Parsing the business requirements and optimizing the abstract symbols to obtain unified symbols include: Input the business requirements into the requirements analysis model to obtain the parameter importance of each requirement parameter; Compare and analyze the demand parameters with the parameter variables, and obtain the non-important variables in combination with the parameter importance; Add non-unique symbols to the first final symbols belonging to linear parameters in non-significant variables; Add negligible symbols to the second final symbols of non-linear parameters in non-significant variables; Based on the addition results, the abstract symbol is optimized to obtain a unified symbol.

7. An intelligent data integration system based on data virtualization, characterized in that: include: A list determination module, for determining an associated list of each heterogeneous data source matching a requirement type of the business requirement; The function construction module is used to analyze the structural characteristics of each heterogeneous data source according to the predefined indicator set to obtain a parameter set and construct an analysis function corresponding to the heterogeneous data source; A fusion processing module is used to perform fusion processing on all analysis functions, determine linear parameters and nonlinear parameters based on data integration, and configure abstract symbols to the corresponding heterogeneous data sources respectively in combination with the association list of each heterogeneous data source; A symbol optimization module, used to analyze the business requirements and optimize the abstract symbols to obtain unified symbols, so as to realize data integration of multiple heterogeneous data sources; Among them, the function building module is specifically used for: According to the standard conversion relationship between each predefined indicator in the predefined indicator set and the structural characteristics of the heterogeneous data source, the corresponding structural characteristics are converted into a first parameter value based on the corresponding predefined indicator, and a parameter set is formed; According to the application scenario of each heterogeneous data source, a parameter description weight matrix corresponding to the heterogeneous data source is constructed. ,in, The parameter description weight vector representing the first application scenario; Represents the parameter description weight vector of the nth application scenario; the value of i is 1, 2, 3, ..., m, and m represents the parameter description of the mth predefined indicator; Capturing the scenario log for issuing the business requirement, and extracting the scenario-specific representation in the scenario log, and then determining the representation weight consistent with the corresponding scenario-specific representation according to the representation coverage value and the semantic association value of the scenario-specific representation; ； in, The semantic association value representing the corresponding scene-specific representation; represents the length of the scene-specific representation associated with the position of the dth log paragraph in the scene log; Indicates the length of the dth log paragraph in the scenario log; Indicates that based on all The variance of represents the correlation coverage ratio of the corresponding scene-specific representation in the d-th log paragraph of the scene log; represents the coverage ratio of the number of times the corresponding scene-specific representation is represented in the d-th log paragraph of the scene log; represents the representation coverage value corresponding to the scene-specific representation; represents the number of occurrences of the corresponding scene-specific representation in the dth log paragraph in the scene log; represents the total number of scene feature representations in the dth log paragraph in the scene log; D represents the total number of log paragraphs in the scene log; Indicates that based on all The variance of Indicates that except for the corresponding scene feature representation The variance of the corresponding correlation coverage ratio under the remaining scene-specific representations other than ; Indicates that except for the corresponding scene feature representation The variance of the frequency coverage ratio corresponding to the remaining scene-specific representations other than ; The representation weights are sorted by size, and the required scenes corresponding to the weights greater than the preset weights are selected; Adjusting the representation weight of the required scene to obtain the dimension weight of the corresponding scene dimension; ; in, represents the dimension weight of the r1th required scene; represents the representation weight of the r1th required scene; M1 represents the number of required scenes; M2 represents the number of required scenes corresponding to the preset weight not greater than; ln represents the logarithmic function symbol; represents the sum of all representation weights less than the preset weight; represents the variance of the remaining weights after removing the representation weight corresponding to the r1th required scene from the representation weights that are greater than the preset weights; Relying on the scene dimensions and dimension weights, and combining with A0, construct the current weight vector of the corresponding parameter set; Based on the current weight vector and parameter set, an analysis function is constructed.

Images