Data model determination method and device, electronic equipment, storage medium and product

Abstract

The embodiment of the invention discloses a data model determination method and device, electronic equipment, a storage medium and a product, and the method comprises the steps: determining at least one piece of metadata corresponding to each to-be-fused data model according to operation scripts and input source association information of at least two to-be-fused data models; according to the labeling information corresponding to the at least one piece of metadata and the scheduling information corresponding to the metadata, determining a reuse degree attribute of the metadata; determining a model fusion attribute of the to-be-fused data model according to the multiplexing degree attribute of at least one piece of metadata belonging to the same to-be-fused data model; and performing fusion processing on the at least two to-be-fused data models according to the model fusion attributes associated with the to-be-fused data models and the annotation information of the at least one piece of metadata to obtain a target data model. According to the technical scheme, the redundancy of the model is reduced, and the utilization rate of computing power resources and storage resources is increased.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a data model determination method, device, electronic device, storage medium, and product. Background Art

[0002] In the era of big data, as the amount of data continues to grow, data warehouses, as a solution for data management and processing, have been widely used in efficient management and analysis of data. The architecture of data warehouses is mainly divided into source layer, general layer and application layer. Among them, the source layer is a data collection layer for collecting raw data; the general layer is a general model after unified and standardized processing; the application layer includes various data processing models that realize different business needs, and has the ability to build analysis and decision-making. At present, the models of the general layer often cannot meet the growing business data needs, and it is necessary to develop application layer data processing models that meet business data needs to fill the functional gaps of the general layer.

[0003] When implementing the technical solution based on the above method, the inventor found the following problems:

[0004] There are a large number of redundant data processing models in the application layer, which leads to a waste of storage resources. Furthermore, there are identical processing fields in the data processing models, which leads to a waste of computing resources when data processing is performed based on the redundant data processing models. Summary of the invention

[0005] The present invention provides a data model determination method, device, electronic device, storage medium and product to reduce the redundancy of the model and improve the utilization of computing resources and storage resources.

[0006] In a first aspect, an embodiment of the present invention provides a method for determining a data model, the method comprising:

[0007] Determine at least one metadata corresponding to each data model to be fused according to the running scripts of at least two data models to be fused and the input source association information;

[0008] Determining a reuse attribute of the metadata according to the annotation information corresponding to the at least one metadata and the scheduling information corresponding to the metadata;

[0009] Determining a model fusion attribute of the data model to be fused according to a reuse attribute of at least one metadata belonging to the same data model to be fused;

[0010] At least two data models to be fused are fused according to model fusion attributes associated with the data models to be fused and annotation information of at least one metadata to obtain a target data model.

[0011] Furthermore, the method further comprises:

[0012] Determine an abstract syntax tree corresponding to at least two data models to be merged according to the running scripts and keyword parsers of at least two data models to be merged; wherein the keyword parser is generated according to a pre-configured grammar rule file;

[0013] The abstract syntax tree is traversed based on the input source association information of each data model to be fused, and at least one metadata corresponding to each data model to be fused is obtained.

[0014] Furthermore, the method also includes: the input source associated information includes at least one parent table, a child table associated with the parent table, upstream information of the parent table and the child table, fields in the parent table and the child table, application fields, and at least one of data processing rules.

[0015] Furthermore, the method further comprises:

[0016] The metadata annotation information is determined in the following way:

[0017] For at least one metadata, performing primary labeling on the metadata according to a preset primary classification rule; and,

[0018] Performing secondary annotation on metadata according to the secondary classification rules associated with the primary classification rules;

[0019] The primary annotation and the secondary annotation corresponding to the metadata are used as the annotation information of the metadata.

[0020] Furthermore, the method further comprises:

[0021] Divide metadata with the same annotation information into a group to obtain a data set;

[0022] Determine the attributes to be fused of each metadata in the data set according to the scheduling information, operation information and weight information of each metadata in the data set;

[0023] According to all the attributes to be fused associated with the data set, the multiplexing attribute of the data set is determined, and the multiplexing attribute is used as the multiplexing attribute of each metadata in the data set.

[0024] Furthermore, the method further comprises:

[0025] The model fusion attribute of the data model to be fused is obtained by summing up the reuse attribute of at least one metadata associated with the same data model to be fused.

[0026] Furthermore, the method further comprises:

[0027] Determining a model fusion sequence according to the model fusion attributes of the data models to be fused; wherein the model fusion sequence includes the data models to be fused, and the data models to be fused are arranged in a fusion order;

[0028] Determine the model similarity coefficients of the first two data models to be fused in the model fusion sequence, and when the model similarity coefficients meet a preset coefficient threshold, fuse the first two data models to be fused to obtain a fused data model to be used;

[0029] The first two data models to be fused are updated to the data models to be used, and the data models to be used are used as the data models to be fused, so as to repeatedly perform the steps of determining the model coefficients of the first two data models to be fused in the model fusion sequence, and when the model coefficients meet the preset coefficient threshold, determining the data model to be used and updating it to the model fusion sequence, until the model fusion sequence includes a data model to be fused, and the data model to be fused is used as the target data model.

[0030] Furthermore, the method further comprises:

[0031] The model similarity coefficient is determined as follows:

[0032] Obtain the first data model to be fused and the second data model to be fused that are located at the first two positions in the model fusion sequence;

[0033] Determine, according to the annotation information of at least one metadata associated with the first data model to be fused and the annotation information of at least one metadata associated with the second data model to be fused, the number of first metadata with the same annotation information and the number of second metadata including all the annotation information;

[0034] A model similarity coefficient between the first data model to be fused and the second data model to be fused is determined according to the first metadata quantity and the second metadata quantity.

[0035] Furthermore, the method further comprises:

[0036] Determine a data candidate set having the same annotation information according to the annotation information of at least one metadata associated with the first data model to be fused and the annotation information of at least one metadata associated with the second data model to be fused;

[0037] According to the reuse attribute of each metadata in the data selection set, the data candidate set is updated;

[0038] According to the updated data candidate set, a data model to be used that is associated with the first data model to be fused and the second data model to be fused is determined.

[0039] Furthermore, the method further comprises:

[0040] When the model similarity coefficient does not meet the preset coefficient threshold, the data model to be fused that is located at the next position among the first two data models to be fused is removed from the model fusion sequence.

[0041] In a second aspect, an embodiment of the present invention further provides a data model determination device, the device comprising:

[0042] A metadata determination module, configured to determine at least one metadata corresponding to each of the data models to be fused according to the running scripts of at least two data models to be fused and the input source association information;

[0043] a multiplexing determination module, configured to determine a multiplexing attribute of the metadata according to the annotation information corresponding to the at least one metadata and the scheduling information corresponding to the metadata;

[0044] A fusion attribute determination module, used to determine the model fusion attribute of the data model to be fused according to the reuse attribute of at least one metadata belonging to the same data model to be fused;

[0045] The target model determination module is used to fuse the at least two data models to be fused according to the model fusion attributes associated with the data models to be fused and the annotation information of at least one metadata to obtain a target data model.

[0046] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0047] one or more processors;

[0048] a storage device for storing one or more programs,

[0049] When one or more programs are executed by one or more processors, the one or more processors implement a data model determination method as described in any one of the embodiments of the present invention.

[0050] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute a data model determination method as described in any one of the embodiments of the present invention.

[0051] The technical solution of the embodiment of the present invention, for the situation where there are multiple data processing models, these data processing models can be used as data models to be fused, and the key feature data of each data model to be fused are extracted according to the running scripts and input source association information of these data models to be fused, and at least one metadata corresponding to each data model to be fused is obtained. Subsequently, the reuse attribute of each metadata is determined according to the annotation information and scheduling information of each metadata, and then the model fusion attribute of the data model to be fused is determined according to the reuse attribute of the metadata belonging to the same data model to be fused, so that at least two data models to be fused are fused according to the model fusion attribute and the annotation information of the metadata to obtain the target data model. The solution of this embodiment obtains the target data model by fusing similar data models to be fused, which reduces the redundancy of the model and improves the utilization rate of storage resources. Furthermore, since the fused target data model merges the same processing fields of different data processing models, the repeated data processing process is reduced, thereby improving the utilization rate of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required to describe the embodiments. Obviously, the drawings introduced are only drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A flowchart of a method for determining a data model provided by an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a flow chart of another data model determination method provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a specific implementation process of determining a target data model involved in an embodiment of the present invention;

[0056] Figure 4 An example diagram of metadata involved in an embodiment of the present invention;

[0057] Figure 5 An example diagram of metadata and corresponding annotation information involved in an embodiment of the present invention;

[0058] Figure 6 A flowchart of another data model determination method provided by an embodiment of the present invention;

[0059] Figure 7 A schematic diagram of the structure of a data model determination device provided by an embodiment of the present invention;

[0060] Figure 8 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0062] Before introducing the present technical solution, the application scenario may be described first. The technical solution provided by the embodiment of the present invention may be applied in any scenario where it is necessary to reduce redundant models in a data processing framework, wherein the data processing framework includes a large number of data processing models. Exemplarily, for any data warehouse (i.e., a data processing framework), the application layer of the data warehouse may include a large number of data processing models to classify and store massive amounts of business data based on these data processing models. In this case, these data processing models may be fused to obtain a target data model, wherein the specific implementation method of the fusion processing may refer to the detailed description of this embodiment, so as to achieve the purpose of reducing redundant data processing models in the data warehouse.

[0063] Figure 1 A flow chart of a data model determination method provided in an embodiment of the present invention. This embodiment can be applied to any situation where it is necessary to reduce redundant models within a data processing framework. The method can be executed by a data model determination device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a PC or a server.

[0064] like Figure 1 As shown, the method includes:

[0065] S110: Determine at least one metadata corresponding to each data model to be fused according to the running scripts of at least two data models to be fused and the input source association information.

[0066] The data model to be fused is a data processing model to be subjected to model fusion processing. In this embodiment, the data model to be fused can be any data processing model in the data warehouse application layer. It should be noted that the data processing model can be understood as a data table, a form, etc. The running script is a computer program composed of a series of instructions and commands, which are used to implement the functions of the data model to be fused. The running script can exist in the form of a text file.

[0067] Among them, metadata is information about the organization of data, data domains and their relationships, that is, data used to describe data. In this embodiment, metadata can also be understood as feature data corresponding to the data model to be fused, and the representation form of metadata can be feature values, feature fields, etc. In particular, the metadata in this embodiment can include physical database table names, column names, field lengths, field types, constraint information, data dependencies, etc.; field-level lineage relationships, structured query language script information, interface programs, etc.; scheduling dependencies, progress and data update frequency, etc.

[0068] The input source association information is the feature dimension corresponding to the metadata required for subsequent model fusion. In this embodiment, the input source association information includes at least one parent table, a child table associated with the parent table, upstream information of the parent table and the child table, fields in the parent table and the child table, application fields, and at least one of data processing rules.

[0069] In this embodiment, the parent table is the data table that the data model to be fused depends on in the actual processing process; the child table associated with the parent table can be understood as a data table that needs to rely on the data content in the parent table for data processing. Exemplarily, for any of the data models A to be fused, if the processing operation of the data model A to be fused needs to rely on the data content in data table B, then data table B is the parent table of the data model A to be fused; the other data tables: data table C, data table D and data table E, the processing of these three data tables also depends on the data in data table B, then data table C, data table D and data table E are child tables associated with data table B;

[0070] In this embodiment, the upstream information of the parent table and the child table is the information that affects the content of the fields in the parent table and the child table; the fields in the parent table and the child table are the data content fields actually contained in the parent table and the child table; the application field is used to characterize the business scenarios in which the data model to be fused can be applied; the data processing rule is used to characterize the processing process that the data needs to go through, for example, the data processing rule can be a summation, counting and / or zone average calculation process. In this embodiment, the metadata that can reflect the characteristics of the data model to be fused can be determined through the multi-dimensional information of the input source association information.

[0071] Specifically, each data processing model in the data warehouse application layer can be used as a data model to be fused, and each data model to be fused has a corresponding running script. For any of the data models to be fused, the running script can be parsed and processed to obtain a feature data set corresponding to the data model to be fused. Since the obtained feature data set includes a large amount of data content, and only part of the data in the feature data set is required in the subsequent model fusion processing process, it is possible to select feature data corresponding to the data dimension pointed to by the input source association information from this feature data set, thereby using the screened feature data as at least one metadata corresponding to the data model to be fused. The metadata determination process of other data models to be fused is the same and will not be repeated here.

[0072] Exemplarily, for the data model A to be fused, the data model B to be fused, and the data model C to be fused, the corresponding running scripts A, running scripts B, and running scripts C are parsed and processed respectively, and feature data sets A, feature data sets B, and feature data sets C can be obtained. Further, feature screening is performed on the feature data sets A, feature data sets B, and feature data sets C respectively through input source association information to obtain N metadata corresponding to the data model A to be fused, N metadata corresponding to the data model B to be fused, and N metadata corresponding to the data model C to be fused.

[0073] S120: Determine a multiplexing attribute of the metadata according to annotation information corresponding to at least one metadata and scheduling information corresponding to the metadata.

[0074] Among them, the annotation information is a mark used to characterize the characteristics of metadata. In this embodiment, each metadata includes a characteristic field of at least one dimension. For example, at least one dimension may include: metadata identifier, metadata column name, metadata table name, source data set name, application field, data processing rules and other dimensions. One metadata may include multiple annotation information, and different annotation information corresponds to different characteristic field dimensions. Exemplarily, for any metadata S, the annotation information of the metadata S may include: first annotation information corresponding to the source data set name, second annotation information corresponding to the data processing rules, third annotation information corresponding to the application field, etc.

[0075] It should be noted that different annotation information may correspond to different annotation information levels, and annotation information of different levels may be applied with different priorities in subsequent processing. For example, based on the above exemplary embodiment, the first annotation information may be first-level annotation information, the second annotation information may be second-level annotation information, and the third annotation information may be third-level annotation information. When different metadata are classified based on the annotation information, all metadata are first divided into N large groups according to the first-level annotation information, and then the metadata in one of the large groups is divided into M small groups according to the second-level annotation information, and so on.

[0076] The scheduling information is used to represent the frequency of use of a certain metadata. For example, the scheduling information can be represented in the form of: the number of planned executions corresponding to a certain metadata per unit time.

[0077] The reusability attribute is used to characterize the reusable attribute characteristics of a metadata. For example, the reusability attribute can be represented in the form of a numerical value. The larger the reusability attribute value of a metadata is, the better the reusability of the metadata is.

[0078] Specifically, for each metadata of each data model to be fused, data annotation can be performed on each metadata according to a pre-set data annotation rule to obtain at least one annotation information corresponding to each metadata. The scheduling information of each metadata can be determined according to the actual business plan (the actual business plan includes the number of planned executions corresponding to a certain type of metadata per unit time); thus, all metadata can be divided into a plurality of different grouped metadata according to the annotation information of each metadata; for each grouped metadata, the multiplexing attribute value to be used corresponding to the current group can be determined according to the scheduling information of each metadata in the current group, and then the multiplexing attribute value to be used is used as the multiplexing attribute of each metadata in the current group.

[0079] Based on the above exemplary embodiment, after obtaining 3N metadata corresponding to the data model A to be fused, the data model B to be fused, and the data model C to be fused, the 3N metadata can be divided into H grouped metadata according to the annotation information corresponding to each of the 3N metadata. For any X group of the H grouped metadata, the X group includes 100 metadata. According to the scheduling information of the 100 metadata, the multiplexing attribute value Y to be used corresponding to the X group can be determined, so that the multiplexing attribute of the 100 metadata in the X group is Y. Based on the same method, the multiplexing attribute corresponding to each metadata can be determined.

[0080] S130: Determine a model fusion attribute of the data model to be fused according to a reuse attribute of at least one metadata belonging to the same data model to be fused.

[0081] Among them, the model fusion attribute is used to indicate the possibility that the data model to be fused can be fused. For example, the model fusion attribute can be represented in the form of a numerical value. The larger the numerical value of the model fusion attribute is, the higher the possibility that the data model to be fused can be fused.

[0082] Specifically, the model fusion attribute of the data model to be fused may be determined according to the sum of the reuse attributes of each metadata belonging to the same data model to be fused.

[0083] Based on the above example, the reuse attributes of the N metadata corresponding to the data model A to be fused are: A1, A2, A3, ..., A N , then according to A1, A2, A3, ... and A N The sum of is used to determine the model fusion attribute TR1 corresponding to the data model A to be fused. Based on the same method, the model fusion attribute TR2 corresponding to the data model B to be fused and the model fusion attribute TR3 corresponding to the data model C to be fused can be determined.

[0084] S140: Fusing at least two data models to be fused according to model fusion attributes associated with the data models to be fused and annotation information of at least one metadata to obtain a target data model.

[0085] The target data model is a data processing model finally obtained after at least two data models to be fused are fused.

[0086] Specifically, the data models to be fused can be sorted according to the numerical values ​​of the fusion attributes of each model to obtain a model sequence with a sequence of order. For the first two data models to be fused in the model sequence, the similarity of the two data models to be fused is determined based on the annotation information of the metadata corresponding to the two data models to be fused, so as to determine whether the two data models to be fused can be fused based on the similarity. If they can be fused, the two data models to be fused are fused according to the pre-fusion rules to obtain a fused data model. If they cannot be fused, it is determined whether the first data model to be fused and the third data model to be fused in the model sequence can be fused, until two data models to be fused that can be fused are determined. When there are no data models to be fused that can be fused in the model sequence, the fused data model obtained at this time is the target data model.

[0087] In this embodiment, the pre-fusion rule may be: determining the metadata intersection of the two data models to be fused to form a candidate set of all candidate metadata; then traversing the candidate set, filtering out metadata whose reuse attributes do not meet preset conditions, retaining only metadata that meet the preset conditions, and obtaining a target metadata set; establishing a corresponding data table through the characteristics of the metadata in the target metadata set, such as data source, association method, association relationship, field, and processing rules, which is a fused data model.

[0088] On the basis of the above example, the model fusion attribute TR1 corresponding to the data model A to be fused, the model fusion attribute TR2 corresponding to the data model B to be fused, and the model fusion attribute TR3 corresponding to the data model C to be fused, and TR1>TR2>TR3, then first, according to the annotation information of the metadata of the data model A to be fused and the data model B to be fused, determine whether the two can be fused. At this time, it can be divided into two situations. The first situation is: if they can be fused, then the data model A to be fused and the data model B to be fused are fused according to the preset fusion rules to obtain the fused data model M; then, further determine whether the fused data model M and the data model C to be fused can be fused. If so, the fused data model M and the data model C to be fused are fused to obtain the target data model. The second situation is: if they cannot be fused, then further determine whether the data model A to be fused and the data model C to be fused are fused to obtain the target data model.

[0089] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. For example, network requests are intercepted and processed with the authorization of the user.

[0090] The technical solution of the embodiment of the present invention, for the situation where there are multiple data processing models, these data processing models can be used as data models to be fused, and the key feature data of each data model to be fused are extracted according to the running scripts and input source association information of these data models to be fused, and at least one metadata corresponding to each data model to be fused is obtained. Subsequently, the reuse attribute of each metadata is determined according to the annotation information and scheduling information of each metadata, and then the model fusion attribute of the data model to be fused is determined according to the reuse attribute of the metadata belonging to the same data model to be fused, so that at least two data models to be fused are fused according to the model fusion attribute and the annotation information of the metadata to obtain the target data model. The solution of this embodiment obtains the target data model by fusing similar data models to be fused, which reduces the redundancy of the model and improves the utilization rate of storage resources. Furthermore, since the fused target data model merges the same processing fields of different data processing models, the repeated data processing process is reduced, thereby improving the utilization rate of computing resources.

[0091] Figure 2 This is another data model determination method schematic diagram provided by an embodiment of the present invention. Based on the above embodiment, steps S110 to S130 are described in detail. The specific implementation method can refer to the technical solution of this embodiment. The technical terms that are the same or corresponding to the above embodiment are not repeated here.

[0092] like Figure 2 As shown, the method specifically comprises the following steps:

[0093] S210: Determine an abstract syntax tree corresponding to at least two data models to be merged according to the running scripts and keyword parsers of at least two data models to be merged.

[0094] Among them, the keyword parser is generated according to the pre-configured grammar rule file. In this embodiment, the preset configuration grammar rule file may include: selecting and locating field information, capturing input source tables, and identifying screening conditions. It should be noted that in specific applications, according to the actual complexity of the script parsing situation running the data model, the grammar rule file can be expanded to cover various advanced structured query language grammars, such as: adding a sorting method (ORDER BY) grammar rule to capture the sorting rules of the results, and adding an insert (INSERT) grammar rule to support batch insert rules.

[0095] In the present embodiment, the keyword parser is used to convert the running script (for example, the running script of the structured query language statement) into an abstract syntax tree. Exemplarily, the keyword parser can decompose the structured query language statement into syntax units and use the data structure represented internally to represent these syntax units. The keyword parser uses a recursive algorithm to traverse the abstract syntax tree. At each recursive level, the keyword parser first converts the type of the current node into a node representing the type. Then, it connects the node representing the type and the child node to generate a new node. This new node may be a new node or a node inherited from the current node. Finally, the keyword parser recursively traverses the entire abstract syntax tree to generate the final abstract syntax tree.

[0096] The abstract syntax tree is an abstract data representation of the syntax structure of the running script. It represents the syntax structure of the programming language in a tree form, and each node on the tree represents a syntax structure and semantics in the running script. They are connected to each other through pointers (pointers can be values ​​or references). The type and content of these nodes are dynamically determined by the keyword parser.

[0097] Specifically, a keyword parser may be used to perform syntax extraction processing on all data models to be fused to obtain an abstract syntax tree.

[0098] S220 , traversing the abstract syntax tree based on the input source association information of each data model to be fused, and obtaining at least one metadata corresponding to each data model to be fused.

[0099] In this embodiment, for each data model to be fused, the abstract syntax tree can be traversed based on the input source association information of the current data model to be fused, so as to obtain at least one metadata corresponding to the current data model to be fused. Based on the same processing method, at least one metadata corresponding to each data model to be fused can be obtained. The advantage of this setting is that by parsing the running script of the data model to be fused, the data blood relationship is obtained, and the structured metadata features are extracted based on the data blood vessel relationship, providing a data basis for subsequent model fusion.

[0100] S230: for at least one metadata, perform primary labeling on the metadata according to a preset primary classification rule; and perform secondary labeling on the metadata according to a secondary classification rule associated with the primary classification rule.

[0101] Among them, the primary classification rules and the secondary classification rules are pre-set metadata annotation methods.

[0102] In this embodiment, each metadata includes a characteristic field of at least one dimension, for example, at least one dimension may include: metadata identifier, metadata column name, metadata table name, source data set name, application field, data processing rules, etc. For example, the first-level classification rule may be a labeling method corresponding to the "source data set name", and the second-level classification rule may be a labeling method corresponding to the "data processing rule". When the metadata is classified based on the labeling information, all metadata are first divided into several large groups according to the first-level labeling information, and then a large group is divided into several small groups according to the second-level labeling information.

[0103] Specifically, the annotation process of each metadata is consistent, and any one of them is used as the current metadata for exemplary description. The current metadata is annotated at the first level according to the first-level classification rule to obtain the first-level annotation information, and the current metadata is annotated at the second level according to the second-level classification rule to obtain the second-level annotation information. Based on this, each metadata corresponds to the first-level annotation information and the second-level annotation information.

[0104] Exemplarily, for metadata V, the current metadata is annotated at the first level according to the characteristic field of the "source data set name" dimension to obtain the first level annotation information "q"; the current metadata is annotated at the second level according to the characteristic field of the "data processing rule" dimension to obtain the second level annotation information "p". For metadata U, the current metadata is annotated at the first level according to the characteristic field of the "source data set name" dimension to obtain the first level annotation information "q" (which indicates that the characteristic field of the "source data set name" dimension of metadata V and metadata U is the same); the current metadata is annotated at the second level according to the characteristic field of the "data processing rule" dimension to obtain the second level annotation information "s".

[0105] S240. The primary annotation and the secondary annotation corresponding to the metadata are used as annotation information of the metadata.

[0106] In this embodiment, on the basis of obtaining the first-level annotation information and the second-level annotation information corresponding to each metadata, the first-level annotation information and the second-level annotation information are used as the annotation information of each metadata. The advantage of this setting is that by performing hierarchical annotation on metadata, a large number of metadata can be refined and classified, so that a unified reuse attribute can be determined according to the characteristic information of the same type of metadata, thereby improving the accuracy of the reuse attribute.

[0107] S250: Divide metadata with the same annotation information into a group to obtain a data set.

[0108] In this embodiment, metadata with consistent sub-annotation information in the annotation information are grouped into one group. For example, if the annotation information includes primary annotation information and secondary annotation information, for metadata V and metadata U, metadata V and metadata U can be grouped into one group only when the primary annotation information of metadata V is consistent with the primary annotation information of metadata U, and when the secondary annotation information of metadata V is consistent with the secondary annotation information of metadata U. Based on the above metadata grouping method, all metadata can be finally divided into multiple different data sets.

[0109] Next, the multiplexing attribute corresponding to each data set is determined respectively. S260 - S270 specifically explain how to determine the multiplexing attribute for any data set.

[0110] S260: Determine the attributes to be fused of each metadata in the data set according to the scheduling information, operation information, and weight information of each metadata in the data set.

[0111] The weight information is a preset weight value. In this embodiment, the weight information can be expressed as: i=Wai+Wci, where Wai refers to the application weight index and Wci refers to the general weight index. If the application weight is too high, the model can meet the high performance of the current business fine query, but the query performance for the newly added indicators is poor; if the general weight is too high, the model has a high degree of generalization and strong expansibility, but the query performance for the current business is not optimal. Therefore, the application weight index and the general weight index can be adaptively set according to the actual situation of the combination of the data warehouse model and the business.

[0112] The scheduling information is the number of planned executions corresponding to the metadata unit time. The running information is the number of times the metadata is calculated in a task instance. The attribute to be fused is the initial attribute value determined based on the scheduling information, running information and weight information.

[0113] Specifically, any metadata may be used as the current metadata, and the attributes to be fused of the current metadata may be obtained by performing a product operation on the scheduling information, the operation information, and the weight information corresponding to the current metadata.

[0114] In this embodiment, the attributes to be fused of the current metadata i can be expressed as:

[0115] R i =D i ×M i ×O i

[0116] In the formula, R i is the attribute to be fused of the current metadata i, D i The running information of the current metadata i, M i The scheduling information of the current metadata i, O i The weight information of the current metadata i.

[0117] S270: Determine the multiplexing attribute of the data set according to all attributes to be fused associated with the data set, and use the multiplexing attribute as the multiplexing attribute of each metadata in the data set.

[0118] In this embodiment, if the data set includes n metadata, the sum of the attributes to be fused corresponding to the n metadata can be used as the multiplexing attribute of the data set, which can be expressed as:

[0119]

[0120] Further, on the basis of obtaining the multiplexing attribute of the data set, the total multiplexing attribute is used as the multiplexing attribute of each metadata in the data set, that is, the multiplexing attribute of each metadata in the data set is R.

[0121] In this embodiment, the multiplexing attribute of each metadata is obtained by determining the multiplexing attribute of the metadata set with the same annotation information, which improves the efficiency of determining the multiplexing attribute and improves the accuracy of the multiplexing attribute.

[0122] S280 , obtaining a model fusion attribute of the data model to be fused by summing up the reuse attribute of at least one metadata associated with the same data model to be fused.

[0123] In this embodiment, at least one metadata associated with the same data model to be fused is determined, so it is easy to determine the multiplexing attributes corresponding to the same data model to be fused. These multiplexing attributes are summed up, and the sum of the multiplexing attributes obtained is the model fusion attribute of the data model to be fused. The model fusion attribute of the data model to be fused can be determined quickly and conveniently through the summing processing method.

[0124] S290: Fusing at least two data models to be fused according to model fusion attributes associated with the data models to be fused and annotation information of at least one metadata to obtain a target data model.

[0125] Next, a specific example is used to illustrate the target data model determination method of this embodiment. For a schematic diagram of the specific implementation process of determining the target data model, see Figure 3 .

[0126] First, inputting the operation scripts of at least two data models to be merged; then, parsing these operation scripts according to the keyword parser to obtain the abstract syntax trees corresponding to the at least two data models to be merged;

[0127] Second, traverse the abstract syntax tree according to the input source association information to obtain the metadata corresponding to each data model to be fused. For an example of metadata, see Figure 4 ,like Figure 4 As shown, metadata may include feature values ​​corresponding to each metadata feature name;

[0128] Third, each metadata is labeled to obtain the labeling information of each metadata, and all metadata are grouped according to the labeling information. Exemplarily, the metadata can be labeled at the first level according to the data source of the metadata, and the metadata can be labeled at the second level according to the constraint attribute characteristics. If the data sources of the two metadata v and u are the same, the first-level labeling information (label_level1) of v and u are equal, and the metadata v and metadata u are grouped into the same group according to the first-level labeling value. Furthermore, according to the constraint attribute characteristics of the metadata, the metadata in the same group of the first-level labeling information are grouped twice to obtain the second-level labeling information (label_level2) of each metadata in the same group.

[0129] In this step, according to the actual complexity of the metadata, the metadata can be structured in advance to facilitate the determination of whether the corresponding values ​​are equivalent when the metadata annotations are grouped. For example, when the field source depends on multiple tables and multiple fields, the data source is converted into a set of "table_name.col_name"; multiple constraint attribute features are converted into a set of a series of key-value pair-like data structures "attribute name: operator: attribute value". For an example diagram of metadata and corresponding annotation information, see Figure 5 ,like Figure 5 As shown, the source data sets of data sample 1 and data sample 2 are both "table_c.col_c", so the first-level labeling information of data sample 1 and data sample 2, that is, label-Ll is "0", and the source data set of data sample 3 is not "table_c.col_c", so the first-level labeling information of data sample 3, that is, label-L1 is "1".

[0130] Fourth, determine the reuse attribute of the metadata in each group, and determine the model fusion attribute of the data model to be fused according to the reuse attribute of the metadata belonging to the same data model to be fused.

[0131] Fifth, at least two data models to be fused are fused according to the model fusion attributes and the annotation information of the metadata to obtain a target data model.

[0132] The technical solution of the embodiment of the present invention determines the abstract syntax tree corresponding to at least two data models to be fused through the running scripts and keyword parsers of at least two data models to be fused; then, traverses the abstract syntax tree based on the input source association information of each data model to be fused to obtain at least one metadata corresponding to each data model to be fused, and obtains the data blood relationship by parsing the running script of the data model to be fused, and extracts structured metadata features based on the data blood vessel relationship to provide a data basis for subsequent model fusion; then, determines the annotation information of each metadata, and divides the metadata with the same annotation information into a group to obtain a data set, and then determines the metadata to be fused in the data set according to the scheduling information, running information and weight information of each metadata in the data set. Attribute, thus, according to all the attributes to be fused associated with the data set, the multiplexing attribute of the data set is determined, and the multiplexing attribute is used as the multiplexing attribute of each metadata in the data set. By determining the multiplexing attribute of the metadata set with the same annotation information, the multiplexing attribute of each metadata is obtained, which improves the efficiency of determining the multiplexing attribute and improves the accuracy of the multiplexing attribute; further, by summing the multiplexing attributes of at least one metadata associated with the same data model to be fused, the model fusion attribute of the data model to be fused is obtained, and according to the model fusion attribute associated with the data model to be fused and the annotation information of at least one metadata, at least two data models to be fused are fused to obtain the target data model, which reduces the redundancy of the model and improves the utilization rate of storage resources and computing resources.

[0133] Figure 6 This is another data model determination method schematic diagram provided by an embodiment of the present invention. Based on the above embodiment, the step S140 is further described. The specific implementation method can refer to the technical solution of this embodiment. The technical terms that are the same or corresponding to the above embodiment are not repeated here.

[0134] like Figure 6 As shown, the method specifically comprises the following steps:

[0135] S310: Determine at least one metadata corresponding to each data model to be fused according to the running scripts of at least two data models to be fused and the input source association information.

[0136] S320: Determine a multiplexing attribute of the metadata according to the annotation information corresponding to the at least one metadata and the scheduling information corresponding to the metadata.

[0137] S330: Determine a model fusion attribute of the data model to be fused according to a reuse attribute of at least one metadata belonging to the same data model to be fused.

[0138] S340: Determine a model fusion sequence according to the model fusion attributes of the data model to be fused.

[0139] The model fusion sequence is a model set of data models to be fused arranged in a fusion order. That is, the model fusion sequence includes data models to be fused, and the data models to be fused are arranged in a fusion order.

[0140] In this embodiment, the data models to be fused may be arranged in order according to the numerical values ​​of the fusion attributes of each model to obtain a model fusion sequence. Exemplarily, the data models to be fused may be arranged in descending order according to the numerical values ​​of the fusion attributes of the model to obtain a model fusion sequence.

[0141] S350, determining the model similarity coefficients of the first two data models to be fused in the model fusion sequence, and when the model similarity coefficients meet a preset coefficient threshold, fusing the first two data models to be fused to obtain a fused data model to be used.

[0142] Optionally, when the model similarity coefficient does not meet a preset coefficient threshold, the data model to be fused that is located next to the first two data models to be fused is removed from the model fusion sequence.

[0143] The model similarity coefficient is used to characterize the similarity between the two data models to be fused. The larger the value, the higher the similarity between the two. The preset coefficient threshold is a preset model similarity coefficient threshold. The data model to be used is an intermediate product when generating the target data model.

[0144] In this embodiment, the model similarity coefficient is determined in the following manner: the first data model to be fused and the second data model to be fused that are located at the first two positions in the model fusion sequence are obtained; based on the annotation information of at least one metadata associated with the first data model to be fused and the annotation information of at least one metadata associated with the second data model to be fused, the number of first metadata with the same annotation information and the number of second metadata including all annotation information are determined; based on the number of first metadata and the second number of metadata, the model similarity coefficient between the first data model to be fused and the second data model to be fused is determined.

[0145] In order to clearly explain how to determine the model similarity coefficient, a specific example is used here for illustration. The model fusion sequence is {model A, model B, model C}, then model A is obtained as the first data model to be fused, and model B is obtained as the second data model to be fused, model A is associated with 100 metadata, and model B is associated with 120 metadata; according to the annotation information of each metadata of model A and the annotation information of each metadata of model B, the number of first metadata with the same annotation information as model A and model B is 80; the number of second metadata including all annotation information of model A and model B is 220; thus, the model similarity coefficient of model A and model B can be determined according to the ratio of the first metadata number 80 to the second metadata number 220. In this embodiment, the model similarity coefficient of the two data models to be fused is determined by the intersection and union of the number of annotation information of the two data models to be fused. Due to the small amount of calculation, the model similarity coefficient can be determined quickly and accurately.

[0146] Specifically, on the basis of obtaining the model similarity coefficients of the first two data models to be fused, if the model similarity coefficient is greater than or equal to the preset coefficient threshold, the two are fused to obtain the fused data model to be used; if the model similarity coefficient is less than the preset coefficient threshold, the data model to be fused that is located in the latter position of the first two data models to be fused is removed from the model fusion sequence to update the model fusion sequence, so as to achieve the purpose of eliminating non-redundant models in the model fusion sequence. Further, the first two data models to be fused are determined from the updated model fusion sequence, and the model similarity coefficients of the two are determined.

[0147] Next, how to fuse the first two data models to be fused is described in detail.

[0148] In this embodiment, the first two data models to be fused are the first data model to be fused and the second data model to be fused. The first two data models to be fused are fused to obtain the fused data model to be used. The specific implementation method may include: determining a data candidate set with the same annotation information based on the annotation information of at least one metadata associated with the first data model to be fused and the annotation information of at least one metadata associated with the second data model to be fused; updating the data candidate set based on the reuse attribute of each metadata in the data selected set; and determining the data model to be used associated with the first data model to be fused and the second data model to be fused based on the updated data candidate set.

[0149] The data candidate set is a set of metadata with the same annotation information in the first data model to be fused and the second data model to be fused.

[0150] Specifically, according to the annotation information of at least one metadata corresponding to the first data model to be fused and the second data model to be fused, all metadata with the same annotation information are determined, and these metadata are used as the data candidate set. Furthermore, each metadata can be screened and processed according to the reuse attribute corresponding to each metadata in the data candidate set and the preset reuse attribute threshold. For example, metadata with a reuse attribute less than the preset reuse attribute threshold is eliminated to obtain an updated data candidate set. Finally, a corresponding data table is established based on the data source, association method, association relationship, specific field content and processing rules of each metadata in the updated data candidate set, which is the data model to be used associated with the first data model to be fused and the second data model to be fused. This embodiment constructs the fused data model to be used through the metadata with the same annotation information of the first data model to be fused and the second data model to be fused, and achieves the technical effect of reducing redundant models by merging the two models into one data model to be used.

[0151] S360, updating the first two data models to be fused to the data models to be used, and taking the data model to be used as the data model to be fused, to repeatedly execute the steps of determining the model coefficients of the first two data models to be fused in the model fusion sequence, and when the model coefficients meet the preset coefficient threshold, determining the data model to be used and updating it to the model fusion sequence, until the model fusion sequence includes a data model to be fused, and taking the data model to be fused as the target data model.

[0152] Based on the above exemplary embodiment, the model fusion sequence is {model A, model B, model C}. After the model A and model B are fused, the data model H to be used is obtained. At this time, the model fusion sequence is updated to {model H, model C}. Further, the S350 model processing method is used to fuse the model H and the model C to obtain the data model F to be used. At this time, the model fusion sequence is updated to {model F}, and model F is the target data model.

[0153] In this embodiment, the model similarity coefficient of two data models to be fused is determined, and the model similarity coefficient is used as a quantitative indicator to judge whether the two can be fused. Only when the two are highly similar, the two are fused, so that the redundant data models can be accurately fused.

[0154] In a specific experiment, for three specific data models: Model A, Model B and Model C, their respective corresponding resource consumption is as follows: Model A's computing consumption is 11.22 units and its storage consumption is 1.83 units; Model B's computing consumption is 3.45 units and its storage consumption is 0.8 units; Model C's computing consumption is 2.69 units and its storage consumption is 0.39 units. The total computing consumption of the three models is 17.36 units and the total storage consumption is 3.02 units. After the fusion of Model A, Model B and Model C provided by the embodiment, Model M is obtained. The computing consumption of Model M is 14.2 units and the storage consumption is 2.1 units. It can be seen that by fusing multiple data models to be fused, the waste of computing power resources and storage resources can be reduced, and resource utilization can be improved.

[0155] The technical solution of the embodiment of the present invention, when fusing at least two data models to be fused according to the model fusion attributes and the annotation information of the metadata to obtain the target data model, first, according to the model fusion attributes of the data models to be fused, determine the model fusion sequence, and then determine the model similarity coefficient of the first two data models to be fused in the model fusion sequence, and when the model similarity coefficient meets the preset coefficient threshold, fuse the first two data models to be fused to obtain the fused data model to be used, and fuse the first two data models to be fused in the model fusion sequence in turn based on the same processing method, until the model fusion sequence includes a data model to be fused, and use this data model to be fused as the target data model. The technical solution provided by this embodiment determines the model similarity coefficient of the two data models to be fused, and thus uses the model similarity coefficient as a quantitative indicator to judge whether the two can be fused. When the two are highly similar, the two are fused, so that the redundant data models can be accurately fused.

[0156] Figure 7 A schematic diagram of the structure of a data model determination device provided in an embodiment of the present invention, the device includes: a metadata determination module 410, a reuse determination module 420, a fusion attribute determination module 430 and a target model determination module 440.

[0157] The metadata determination module 410 is used to determine at least one metadata corresponding to each of the data models to be fused according to the running scripts and input source association information of at least two data models to be fused;

[0158] A multiplexing determination module 420, configured to determine a multiplexing attribute of the metadata according to the annotation information corresponding to the at least one metadata and the scheduling information corresponding to the metadata;

[0159] A fusion attribute determination module 430, configured to determine a model fusion attribute of the data model to be fused according to a reuse attribute of at least one metadata belonging to the same data model to be fused;

[0160] The target model determination module 440 is used to fuse the at least two data models to be fused according to the model fusion attributes associated with the data models to be fused and the annotation information of at least one metadata to obtain a target data model.

[0161] Based on the above device, optionally, the metadata determination module 410 includes:

[0162] An abstract syntax tree determination unit, configured to determine an abstract syntax tree corresponding to at least two data models to be merged according to the running scripts of at least two data models to be merged and a keyword parser; wherein the keyword parser is generated according to a pre-configured syntax rule file;

[0163] The metadata determination unit is used to traverse the abstract syntax tree based on the input source association information of each data model to be fused, and obtain at least one metadata corresponding to each data model to be fused.

[0164] Based on the above device, optionally, the input source associated information includes at least one parent table, a child table associated with the parent table, upstream information of the parent table and the child table, fields in the parent table and the child table, application fields, and at least one of data processing rules.

[0165] Based on the above device, optionally, the data model determination device further includes: a label information determination module;

[0166] The annotation information determination module is used to perform a primary annotation on at least one metadata according to a preset primary classification rule; and to perform a secondary annotation on the metadata according to a secondary classification rule associated with the primary classification rule; and to use the primary annotation and the secondary annotation corresponding to the metadata as the annotation information of the metadata.

[0167] Based on the above device, optionally, the multiplexing degree determination module 420 includes:

[0168] A data set determination unit, used to group metadata with the same annotation information into a group to obtain a data set;

[0169] The to-be-fused attribute determination unit is used to determine the to-be-fused attribute of each metadata in the data set according to the scheduling information, operation information and weight information of each metadata in the data set;

[0170] The multiplexing attribute determining unit is used to determine the multiplexing attribute of the data set according to all the attributes to be fused associated with the data set, and use the multiplexing attribute as the multiplexing attribute of each metadata in the data set.

[0171] On the basis of the above device, optionally, the fusion attribute determination module 430 is specifically used to obtain the model fusion attribute of the data model to be fused by summing up the reuse attribute of at least one metadata associated with the same data model to be fused.

[0172] Based on the above device, optionally, the target model determination module 440 includes:

[0173] A model fusion sequence determining unit, used to determine a model fusion sequence according to the model fusion attributes of the data models to be fused; wherein the model fusion sequence includes the data models to be fused, and the data models to be fused are arranged in a fusion order;

[0174] The to-be-used model determination unit is used to determine the model similarity coefficients of the first two to-be-fused data models in the model fusion sequence, and when the model similarity coefficients meet a preset coefficient threshold, fuse the first two to-be-fused data models to obtain a fused to-be-used data model;

[0175] The target model determination unit is used to update the first two data models to be fused to the data models to be used, and use the data model to be used as the data model to be fused, so as to repeatedly perform the steps of determining the model coefficients of the first two data models to be fused in the model fusion sequence, and when the model coefficients meet the preset coefficient threshold, determining the data model to be used and updating it to the model fusion sequence, until the model fusion sequence includes a data model to be fused, and the data model to be fused is used as the target data model.

[0176] Based on the above device, optionally, the target model determination module 440 further includes: a model similarity coefficient determination unit;

[0177] A model similarity coefficient determination unit is used to obtain the first data model to be fused and the second data model to be fused that are located at the first two positions in the model fusion sequence; determine the number of first metadata with the same annotation information and the number of second metadata including all annotation information based on the annotation information of at least one metadata associated with the first data model to be fused and the annotation information of at least one metadata associated with the second data model to be fused; and determine the model similarity coefficient between the first data model to be fused and the second data model to be fused based on the first metadata number and the second metadata number.

[0178] On the basis of the above device, optionally, the model determination unit to be used includes:

[0179] A candidate set determination subunit, configured to determine a data candidate set having the same annotation information according to the annotation information of at least one metadata associated with the first data model to be fused and the annotation information of at least one metadata associated with the second data model to be fused;

[0180] A candidate set updating subunit, used to update the data candidate set according to the reuse attribute of each metadata in the data selected set;

[0181] The model to be used determining subunit is used to determine the data model to be used associated with the first data model to be fused and the second data model to be fused according to the updated data candidate set.

[0182] Based on the above device, optionally, the target model determination unit is also used to remove the data model to be fused that is located at the latter position among the first two data models to be fused from the model fusion sequence when the model similarity coefficient does not meet the preset coefficient threshold.

[0183] The technical solution of the embodiment of the present invention, for the situation where there are multiple data processing models, these data processing models can be used as data models to be fused, and the key feature data of each data model to be fused are extracted according to the running scripts and input source association information of these data models to be fused, and at least one metadata corresponding to each data model to be fused is obtained. Subsequently, the reuse attribute of each metadata is determined according to the annotation information and scheduling information of each metadata, and then the model fusion attribute of the data model to be fused is determined according to the reuse attribute of the metadata belonging to the same data model to be fused, so that at least two data models to be fused are fused according to the model fusion attribute and the annotation information of the metadata to obtain the target data model. The solution of this embodiment obtains the target data model by fusing similar data models to be fused, which reduces the redundancy of the model and improves the utilization rate of storage resources. Furthermore, since the fused target data model merges the same processing fields of different data processing models, the repeated data processing process is reduced, thereby improving the utilization rate of computing resources.

[0184] The data model determination device provided in the embodiment of the present invention can execute the data model determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0185] It is worth noting that the various units and modules included in the above system are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0186] Figure 8The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 8 A block diagram of an exemplary electronic device 50 suitable for implementing exemplary embodiments of the present invention is shown. Figure 8 The electronic device 50 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0187] like Figure 8 As shown, the electronic device 50 is in the form of a general computing device. The components of the electronic device 50 may include but are not limited to: one or more processors or processing units 501, a system memory 502, and a bus 503 connecting different system components (including the system memory 502 and the processing unit 501).

[0188] Bus 503 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0189] The electronic device 50 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 50, including volatile and non-volatile media, removable and non-removable media.

[0190] System memory 502 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 504 and / or cache memory 505. Electronic device 50 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 506 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 not shown, usually called a "hard drive"). Although Figure 8 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 503 via one or more data medium interfaces. The memory 502 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0191] A program / utility 508 having a set (at least one) of program modules 507 may be stored, for example, in the memory 502, such program modules 507 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 507 generally perform the functions and / or methods of the embodiments described herein.

[0192] The electronic device 50 may also communicate with one or more external devices 509 (e.g., keyboards, pointing devices, displays 510, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 50, and / or communicate with any device that enables the electronic device 50 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 511. Furthermore, the electronic device 50 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 512. As shown, the network adapter 512 communicates with other modules of the electronic device 50 through the bus 503. It should be understood that although Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 50, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0193] The processing unit 501 executes various functional applications and page processing by running the programs stored in the system memory 502, for example, implementing the data model determination method provided in the embodiment of the present invention.

[0194] An embodiment of the present invention further provides a storage medium containing computer executable instructions, which are used to perform a data model determination method when executed by a computer processor. The method includes:

[0195] Determine at least one metadata corresponding to each data model to be fused according to the running scripts of at least two data models to be fused and the input source association information;

[0196] Determining a reuse attribute of the metadata according to the annotation information corresponding to the at least one metadata and the scheduling information corresponding to the metadata;

[0197] Determining a model fusion attribute of the data model to be fused according to a reuse attribute of at least one metadata belonging to the same data model to be fused;

[0198] At least two data models to be fused are fused according to model fusion attributes associated with the data models to be fused and annotation information of at least one metadata to obtain a target data model.

[0199] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0200] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0201] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0202] Computer program code for performing the operation of embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0203] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for determining a data model, characterized in that: include: Determine at least one metadata corresponding to each of the data models to be fused according to the running scripts of at least two data models to be fused and the input source association information; Determining a multiplexing attribute of the metadata according to the annotation information corresponding to the at least one metadata and the scheduling information corresponding to the metadata; Determining a model fusion attribute of the data model to be fused according to a reuse attribute of at least one metadata belonging to the same data model to be fused; The at least two data models to be fused are fused according to the model fusion attributes associated with the data models to be fused and the annotation information of at least one metadata to obtain a target data model.

2. The method according to claim 1, characterized in that The step of determining at least one metadata corresponding to each of the data models to be fused according to the running scripts of the at least two data models to be fused and the input source association information includes: Determine an abstract syntax tree corresponding to the at least two data models to be merged according to the running scripts and the keyword parser of the at least two data models to be merged; wherein the keyword parser is generated according to a pre-configured grammar rule file; The abstract syntax tree is traversed based on the input source association information of each of the data models to be fused, to obtain at least one metadata corresponding to each of the data models to be fused.

3. The method according to claim 1 or 2, characterized in that: The input source associated information includes at least one parent table, a child table associated with the parent table, upstream information of the parent table and the child table, fields in the parent table and the child table, an application field, and at least one of data processing rules.

4. The method according to claim 1, characterized in that: The annotation information of the metadata is determined by: For the at least one metadata, performing a primary labeling on the metadata according to a preset primary classification rule; and, Performing secondary annotation on the metadata according to the secondary classification rule associated with the primary classification rule; The first-level annotation and the second-level annotation corresponding to the metadata are used as annotation information of the metadata.

5. The method according to claim 1, characterized in that: The determining, according to the annotation information corresponding to the at least one metadata and the scheduling information corresponding to the metadata, the multiplexing attribute of the metadata includes: Divide metadata with the same annotation information into a group to obtain a data set; Determining the attributes to be fused of each metadata in the data set according to the scheduling information, operation information and weight information of each metadata in the data set; According to all the attributes to be fused associated with the data set, a multiplexing attribute of the data set is determined, and the multiplexing attribute is used as the multiplexing attribute of each metadata in the data set.

6. The method according to claim 1, characterized in that The step of determining the model fusion attribute of the data model to be fused according to the reuse attribute of at least one metadata belonging to the same data model to be fused includes: The model fusion attribute of the data model to be fused is obtained by summing up the reuse attribute of at least one metadata associated with the same data model to be fused.

7. The method according to claim 1, characterized in that The fusing the at least two data models to be fused according to the model fusion attributes associated with the data models to be fused and the annotation information of at least one metadata to obtain the target data model includes: Determining a model fusion sequence according to the model fusion attributes of the data models to be fused; wherein the model fusion sequence includes the data models to be fused, and the data models to be fused are arranged in a fusion order; Determine the model similarity coefficients of the first two data models to be fused in the model fusion sequence, and when the model similarity coefficients meet a preset coefficient threshold, fuse the first two data models to be fused to obtain a fused data model to be used; The first two data models to be fused are updated to the data models to be used, and the data models to be used are used as the data models to be fused, so as to repeatedly perform the steps of determining the model coefficients of the first two data models to be fused in the model fusion sequence, and when the model coefficients meet the preset coefficient threshold, determining the data model to be used and updating it to the model fusion sequence, until the model fusion sequence includes a data model to be fused, and the data model to be fused is used as the target data model.

8. The method according to claim 7, characterized in that The model similarity coefficient is determined in the following way: Obtain the first data model to be fused and the second data model to be fused that are located at the first two positions in the model fusion sequence; Determining, according to the labeling information of at least one metadata associated with the first data model to be fused and the labeling information of at least one metadata associated with the second data model to be fused, the number of first metadata having the same labeling information and the number of second metadata including all the labeling information; A model similarity coefficient between the first data model to be fused and the second data model to be fused is determined according to the first metadata quantity and the second metadata quantity.

9. The method according to claim 7, characterized in that: The first two data models to be fused are the first data model to be fused and the second data model to be fused. The fusion processing of the first two data models to be fused to obtain the fused data model to be used includes: Determining a data candidate set having the same annotation information according to the annotation information of at least one metadata associated with the first data model to be fused and the annotation information of at least one metadata associated with the second data model to be fused; updating the data candidate set according to the reuse attribute of each metadata in the data selected set; According to the updated data candidate set, a data model to be used associated with the first data model to be fused and the second data model to be fused is determined.

10. The method according to claim 7, characterized in that The method further comprises: When the model similarity coefficient does not meet the preset coefficient threshold, the data model to be fused that is located at the latter position among the first two data models to be fused is removed from the model fusion sequence.

11. A data model determination device, characterized in that: include: A metadata determination module, configured to determine at least one metadata corresponding to each of the data models to be fused according to the running scripts of at least two data models to be fused and the input source association information; a multiplexing determination module, configured to determine a multiplexing attribute of the metadata according to the annotation information corresponding to the at least one metadata and the scheduling information corresponding to the metadata; A fusion attribute determination module, used to determine the model fusion attribute of the data model to be fused according to the reuse attribute of at least one metadata belonging to the same data model to be fused; The target model determination module is used to fuse the at least two data models to be fused according to the model fusion attributes associated with the data models to be fused and the annotation information of at least one metadata to obtain a target data model.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data model determination method as described in any one of claims 1-10.

13. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the data model determination method as described in any one of claims 1 to 10 when executed by a computer processor.

14. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the data model determination method according to any one of claims 1-10.

Images