Model construction method, device, electronic device and storage medium

Abstract

The present invention relates to the technical field of data warehouses, and provides a model construction method, device, electronic device and storage medium. The method analyzes the lineage mapping relationship between source fields and target fields through database source table data and target table data, constructs an access count matrix of source field combinations using access records of target fields, divides source fields into updated database source tables according to the access count matrix using a clustering algorithm, and constructs a core model of an application system using updated database source tables containing source fields with high access popularity. This method realizes the automatic division of the data model structure of the application system, avoids the bloated data model caused by piling too many fields into the same data model, and can improve the application efficiency of the data model and the application system.

Description

Technical Field

[0001] The present invention relates to the technical field of data warehouses, and in particular to a model building method, device, electronic device and storage medium. Background Art

[0002] The data model is the core cornerstone and backbone of a data application system. To quickly meet business delivery needs, a large number of data fields are often piled into a single data model, resulting in an overly bloated model and delays in application system use. Summary of the Invention

[0003] In view of the above, it is necessary to propose a model construction method, device, electronic device and storage medium, which can use the source fields with higher access popularity to build the core model of the application system, automatically divide the structure of the data model of the application system, and improve the application efficiency of the data model and the application system.

[0004] A first aspect of the present invention provides a model building method, the method comprising: obtaining a database source table and a target table of the application system, and a correspondence between a source field in the database source table and a target field in the target table, based on a task log of the application system;

[0005] Obtaining an access record containing the target field from a user access log of the application system;

[0006] Constructing an access count matrix of source field combinations corresponding to the target field according to the access records and the corresponding relationship, wherein the source field combination includes a source field and / or a combination of multiple source fields, and the access count matrix includes the access count of each source field combination;

[0007] performing cluster analysis on the source field combinations according to the access times matrix, and dividing the source field combinations into multiple categories according to the results of the cluster analysis;

[0008] A plurality of updated database source tables are obtained according to the plurality of source field combinations of the plurality of categories, and a plurality of data models are generated based on the plurality of updated database source tables.

[0009] According to an optional embodiment of the present invention, obtaining a database source table and a target table of the application system, and a correspondence between a source field in the database source table and a target field in the target table based on a task log of the application system, includes:

[0010] Determine the database source table data and target table data in the task log, wherein the database source table data includes the database source table name and the source field name in the database source table, and the target table data includes the target table name and the target field name in the target table;

[0011] The database source table data and the target table data are parsed based on graph computing to obtain the corresponding relationship.

[0012] According to an optional embodiment of the present invention, the graph-based computing-based parsing of the database source table data and the target table data to obtain the corresponding relationship includes:

[0013] Build a graph computing model based on graph neural computing network;

[0014] Using the source field name and the target field name as nodes of the graph computing model;

[0015] The source field name and the target field name that have a mapping relationship with each other are connected, and the line connecting the source field name and the target field name is used as the edge of the graph computing model to obtain the corresponding relationship.

[0016] According to an optional embodiment of the present invention, constructing an access count matrix of the source field corresponding to the target field according to the access records and the corresponding relationship includes:

[0017] Obtaining the number of accesses to the source field combination corresponding to the target field according to the access record and the corresponding relationship;

[0018] A two-dimensional matrix is ​​constructed by taking the source field combination as a row dimension attribute and a column dimension attribute, and the access count of the source field combination is taken as an element of the two-dimensional matrix to obtain the access count matrix.

[0019] According to an optional embodiment of the present invention, obtaining the number of accesses of the source field combination corresponding to the target field according to the access record and the corresponding relationship includes:

[0020] Determine a target field combination in the access record, where the target field combination includes one target field and / or a combination of multiple target fields;

[0021] Determining the source field combination corresponding to the target field combination according to the corresponding relationship;

[0022] When the target field combination appears n times in the same access record, it is determined that the number of accesses to the source field combination corresponding to the target field combination is n times, where n represents an integer greater than or equal to 1.

[0023] According to an optional embodiment of the present invention, cluster analysis is performed on the source field combinations according to the access count matrix, and the source field combinations are divided into multiple categories according to the cluster analysis results, including:

[0024] Determine the value of the largest number of accesses in the access number matrix as the maximum value;

[0025] K numerical ranges are determined based on the maximum value, the numerical range to which the number of accesses of each source field combination belongs is determined, and source field combinations belonging to the same numerical range are determined to belong to the same category, where K represents an integer greater than 1.

[0026] According to an optional embodiment of the present invention, obtaining multiple updated database source tables according to the combination of source fields of the multiple categories, and generating multiple data models based on the multiple updated database source tables includes:

[0027] According to the source field combinations of the K categories, the source field combinations belonging to the same category are saved into an updated database source table, K updated database source tables are obtained, and an updated database source table name is defined for each updated database source table;

[0028] According to the K updated database source tables and the corresponding relationships, the target fields corresponding to the source fields in each updated database source table are merged into an updated target table, and an updated target table name is defined for each updated target table;

[0029] According to each updated database source table, each updated target table, and the task log, a data model for converting each updated database source table to each updated target table is constructed to generate the K data models.

[0030] A second aspect of the present invention provides a model building device, the device comprising an acquisition module, a construction module, a classification module, and a generation module:

[0031] The acquisition module is used to obtain the database source table and target table of the application system, and the corresponding relationship between the source field in the database source table and the target field in the target table based on the task log of the application system;

[0032] The acquisition module is further configured to acquire access records containing the target field from the user access log of the application system;

[0033] The construction module is configured to construct an access count matrix of a source field combination corresponding to the target field, wherein the source field combination includes a source field and / or a combination of a plurality of source fields, and the access count matrix includes the access count of each source field combination;

[0034] The classification module is configured to perform cluster analysis on the source field combinations according to the access times matrix, and divide the source field combinations into multiple categories according to the results of the cluster analysis;

[0035] The generating module is configured to obtain a plurality of updated database source tables according to the plurality of source field combinations of the plurality of categories, and generate a plurality of data models based on the plurality of updated database source tables.

[0036] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the processor is configured to implement the model building method when executing a computer program stored in the memory.

[0037] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the model building method when executed by a processor.

[0038] In summary, the model building method, device, electronic device and storage medium described in the present invention analyze the lineage mapping relationship between the source field and the target field through the database source table data and the target table data, use the access records of the target field to build the access count matrix of the source field combination, use the clustering algorithm to divide the source fields into updated database source tables according to the access count matrix, and use the updated database source table containing the source fields with higher access popularity to build the core model of the application system, thereby realizing the automatic division of the data model structure of the application system, avoiding the bloated data model caused by piling too many fields into the same data model, and being able to improve the application efficiency of the data model and the application system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flowchart of the model building method provided in Example 1 of the present invention.

[0040] Figure 2 This is an example diagram of the access times matrix provided by an embodiment of the present invention.

[0041] Figure 3 It is a structural diagram of the model building device provided in the second embodiment of the present invention.

[0042] Figure 4 It is a structural diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features therein can be combined with each other without conflict.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing the embodiments in an optional embodiment and are not intended to limit the present invention.

[0045] The model building method provided by the embodiment of the present invention is executed by an electronic device, and accordingly, the model building device runs in the electronic device.

[0046] Example 1

[0047] Figure 1 This is a flow chart of the model building method provided by the first embodiment of the present invention. The model building method specifically includes the following steps. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.

[0048] S11 , based on the task log of the application system, obtaining a database source table and a target table of the application system, and a correspondence between a source field in the database source table and a target field in the target table.

[0049] In an optional embodiment, the solution provided in the embodiment of the present application can be applied to re-dividing and constructing the data model of the application system. Based on the analysis of the access popularity of the combination of source fields in the database source table, the source fields are divided into updated database source tables according to the access popularity using a clustering algorithm. The core model of the application system is constructed using the updated database source table containing the source fields with higher access popularity, thereby realizing the automatic division of the structure of the data model of the application system, avoiding the bloated data model caused by piling too many fields into the same data model, and improving the application efficiency of the data model and the application system.

[0050] In an optional embodiment, obtaining a database source table and a target table of the application system, and a correspondence between a source field in the database source table and a target field in the target table based on a task log of the application system, includes:

[0051] Determine the database source table data and target table data in the task log, wherein the database source table data includes the database source table name and the source field name in the database source table, and the target table data includes the target table name and the target field name in the target table;

[0052] The database source table data and the target table data are parsed based on graph computing to obtain the corresponding relationship.

[0053] In an optional implementation, the task log (CronLog) includes a log of a scheduled task in the background of the application system, and the task may be an ETL (Extract-Transform-Load) task.

[0054] For example, a scheduled synchronization task of the application system is to synchronize a database to another database at 0:00 every day. The task log can record whether the synchronization task is successful, the time taken for success, etc.

[0055] For example, a scheduled conversion task of the application system is to complete the data integration conversion from a database source table in a certain database to a target table at 12:00 every day. The task log can record the database source table data, target table data, mapping logic and steps of the integration conversion, whether it is successful, time taken for success, etc. in the conversion task, wherein the mapping logic includes the mapping relationship between the source field and the target field.

[0056] In an optional embodiment, the database source table and the target table are corresponding concepts. The database source table of the source database is written or converted into the target database, and the obtained table is the target table corresponding to the database source table.

[0057] In an optional embodiment, the source fields include fields of multiple types and data types, and the database source table can store the source fields in a row or column format. For example, the source fields of database source table L1 include: user ID (integer) and time (date); the source fields of database source table L2 include: location (string) and amount (decimal); and the source fields of database source table L3 include: purchase quantity (integer) and product type (string). The data in brackets indicates the data type of the source field.

[0058] In one optional embodiment, during the conversion of the database source table to the target table, source fields from multiple database source tables may be mapped to the target table in parallel. This mapping may include multi-level hierarchical mapping, including multiple intermediate initial tables before ultimately obtaining the target table. Furthermore, based on the business rules of the application system, only some source fields from the database source tables may be mapped to the target table.

[0059] For example, the data source tables L1-L3 are converted to the target table L4 according to the user's purchase rules. The target table L4 can be a purchase record table of a certain user, including the target field of the purchase record of a certain user. For example, the target field in the target table L4 can include: user 567, a certain day, flower market, 10, mobile phone case, 96.8, which means that user 567 bought 10 mobile phone cases at the flower market on a certain day, spending a total of 96.8 yuan.

[0060] In an optional embodiment, the graph-based computing-based parsing of the database source table data and the target table data to obtain the corresponding relationship includes:

[0061] Build a graph computing model based on Graph Neural Networks (GNN);

[0062] Using the source field name and the target field name as nodes of the graph computing model;

[0063] The source field name and the target field name that have a mapping relationship with each other are connected, and the line connecting the source field name and the target field name is used as the edge of the graph computing model to obtain the corresponding relationship.

[0064] In an optional embodiment, graphs are highly abstract and flexible. Compared to linear lists, hierarchical trees, and other organizational methods, using graph computing to parse the mapping relationship between source fields and target fields has stronger representation capabilities. The graph neural networks (GNNs) may include graph convolutional networks.

[0065] The graph computing model consists of a finite non-empty set of nodes or vertices and a set of edges between the nodes, which can be expressed as G(V,E). Where G represents a graph, V is the set of nodes in the graph G, and E is the set of edges connecting the nodes in the graph G. When E has a direction, G is a directed graph.

[0066] In an optional implementation, a directed edge pointing from a source field name to a target field name is used to represent the mapping from the source field to the target field, thereby obtaining a more intuitive corresponding relationship.

[0067] S12: Obtain access records containing the target field from the user access log of the application system.

[0068] In an optional embodiment, the user access log (AccessLog) includes all user access logs to their own reports. The user access log generally records several key indicators, such as the request URL (Uniform Request Locator), request parameters, user ID, access time, interface return value, etc. For example, user 567 accessed his purchase record table (target table L4), generating a user access log.

[0069] When a user views a target field in a target table, an access record containing the target field viewed by the user will appear in the user access log. For example, the user views the purchase quantity 10 and the product type mobile phone case of the account with user ID 567. Therefore, the user access log records the access records of the target fields "567", "10" and "mobile phone case".

[0070] S13. Construct an access count matrix of the source field combination corresponding to the target field based on the access record and the corresponding relationship, wherein the source field combination includes a source field and / or a combination of multiple source fields, and the access count matrix includes the access count of each source field combination.

[0071] In an optional implementation, users can generally only access the target field through the target table. After obtaining the user's access record to the target field, the number of indirect accesses of the user to the source field corresponding to the target field can be reversed based on the corresponding relationship, thereby determining the access popularity of the source field.

[0072] In an optional implementation, constructing an access count matrix of the source field corresponding to the target field according to the access records and the corresponding relationship includes:

[0073] Obtaining the number of accesses to the source field combination corresponding to the target field according to the access record and the corresponding relationship;

[0074] A two-dimensional matrix is ​​constructed by taking the source field combination as a row dimension attribute and a column dimension attribute, and the access count of the source field combination is taken as an element of the two-dimensional matrix to obtain the access count matrix.

[0075] In an optional implementation, obtaining the access count of the source field combination corresponding to the target field according to the access record and the corresponding relationship includes:

[0076] Determine a target field combination in the access record, where the target field combination includes one target field and / or a combination of multiple target fields;

[0077] Determining the source field combination corresponding to the target field combination according to the corresponding relationship;

[0078] When the target field combination appears n times in the same access record, it is determined that the number of accesses to the source field combination corresponding to the target field combination is n times, where n represents an integer greater than or equal to 1.

[0079] For example, the target field combination in the access record may be [567 10 mobile phone case], and the corresponding source field combination is [user ID purchase quantity product type].

[0080] In an optional embodiment, row and column attributes of a two-dimensional matrix can be defined based on the source field combination, and the access counts of the source field combination can be used as elements of the two-dimensional matrix to obtain the access count matrix. It should be noted that the combination of row and column attributes also represents the source field combination.

[0081] For example Figure 2 As shown, it is an example diagram of the access count matrix provided in an embodiment of the present application. Among them, the source field combination [A BC] is composed of a single source field A (row dimension attribute, such as [user ID]) and a source field combination [BC] (column dimension attribute, such as [purchase quantity commodity type]), and the access count of the source field combination [A BC] is 5. It should be noted that when a user has two user IDs (for example, 567 and 789), and both user IDs are accessed, the situation where the access count of [AA] is 1 will also occur. In addition, the access count matrix can be an asymmetric matrix, for example, the access counts of the source field combination [A BC] and the source field combination [BC A] can be different; the dimension of the access count matrix is ​​determined by the source field combination, for example, the dimension of the access count matrix can be m×n, and m and n both represent positive integers greater than 1.

[0082] In an optional embodiment, the larger the value of the element in the access count matrix, the greater the access count and access popularity of the source field combination composed of the corresponding combination of row dimension attributes and column dimension attributes.

[0083] S14, performing cluster analysis on the source field combinations according to the access times matrix, and dividing the source field combinations into multiple categories according to the results of the cluster analysis.

[0084] In an optional embodiment, cluster analysis is performed on the source field combinations according to the access count matrix, and the source field combinations are divided into multiple categories according to the cluster analysis results, including:

[0085] Determine the value of the largest number of accesses in the access number matrix as the maximum value;

[0086] K numerical ranges are determined based on the maximum value, the numerical range to which the number of accesses of each source field combination belongs is determined, and source field combinations belonging to the same numerical range are determined to belong to the same category, where K represents an integer greater than 1.

[0087] For example, when the maximum access count value in the access count matrix is ​​100, it can be divided into three value ranges, namely 0-30, 31-60, and 61-100. The access count of the source field combination [A BC] is 5, so the source field combination [A BC] is divided into the 0-30 category.

[0088] In an optional embodiment, the cluster analysis method further includes a k-means clustering algorithm, etc. Based on the access count matrix, the source field combinations are classified according to access popularity, and the source fields in the source field combinations with greater access popularity can be grouped into one category.

[0089] S15 , obtaining a plurality of updated database source tables according to the plurality of source field combinations of the categories, and generating a plurality of data models based on the plurality of updated database source tables.

[0090] In an optional embodiment, obtaining multiple updated database source tables according to the combination of source fields of the multiple categories, and generating multiple data models based on the multiple updated database source tables includes:

[0091] According to the K categories of source fields, a combination of source fields belonging to the same category is saved into an updated database source table, K updated database source tables are obtained, and an updated database source table name is defined for each updated database source table, wherein the updated database source table includes the source fields in the source field combination;

[0092] According to the K updated database source tables and the corresponding relationships, the target fields corresponding to the source fields in each updated database source table are merged into an updated target table, and an updated target table name is defined for each updated target table;

[0093] According to each updated database source table, each updated target table, and the task log, a data model for converting each updated database source table to each updated target table is constructed to generate the K data models.

[0094] In an optional embodiment, the access popularity of source field combinations of different categories is different, so the application popularity (for example, the possibility of being applied) of the data model corresponding to each updated database source table is also different, thereby building a core model of the application system (for example, the data model that is most frequently accessed and applied), and extended models with lower popularity outside the core model, thereby splitting and grading the data model of the application system and improving the availability of the application system.

[0095] In an optional embodiment, the task log stores the mapping logic for converting each source field to the corresponding target field. A data model for converting each updated database source table to each updated target table is processed or constructed based on the mapping logic to obtain the K data models. Existing ETL tools (e.g., RestCloud, Kettle, etc.) can be used to construct the data models. ETL tools can automatically build documents, implement version control required for multi-person development environments, and automatically parallelize tasks, thereby enabling efficient and orderly model construction.

[0096] In an optional embodiment, the model building method provided in the embodiment of the present application analyzes the lineage mapping relationship between the source field and the target field through the database source table data and the target table data, constructs the access count matrix of the source field combination using the access records of the target field, and uses a clustering algorithm to divide the source fields into updated database source tables according to the access count matrix. The core model of the application system is constructed using the updated database source table containing the source fields with higher access popularity, thereby realizing the automatic division of the data model structure of the application system, avoiding the bloated data model caused by piling too many fields into the same data model, and improving the application efficiency of the data model and the application system.

[0097] Example 2

[0098] Figure 3 It is a structural diagram of the model building device provided in the second embodiment of the present invention.

[0099] In some embodiments, the model building device 20 may include a plurality of functional modules composed of computer program segments. The computer programs of the various program segments in the model building device 20 may be stored in a memory of an electronic device and executed by at least one processor to perform (see Figure 1 Description) Model building functionality.

[0100] In this embodiment, the model building device 20 can be divided into multiple functional modules based on the functions it performs. These functional modules may include an acquisition module 201, a construction module 202, a classification module 203, and a generation module 204. As used herein, a module refers to a series of computer program segments that can be executed by at least one processor and perform fixed functions, and is stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0101] The acquisition module 201 is configured to obtain a database source table and a target table of the application system, and a correspondence between a source field in the database source table and a target field in the target table, based on a task log of the application system.

[0102] In an optional embodiment, the solution provided in the embodiment of the present application can be applied to re-dividing and constructing the data model of the application system. Based on the analysis of the access popularity of the combination of source fields in the database source table, the source fields are divided into updated database source tables according to the access popularity using a clustering algorithm. The core model of the application system is constructed using the updated database source table containing the source fields with higher access popularity, thereby realizing the automatic division of the structure of the data model of the application system, avoiding the bloated data model caused by piling too many fields into the same data model, and improving the application efficiency of the data model and the application system.

[0103] In an optional embodiment, obtaining a database source table and a target table of the application system, and a correspondence between a source field in the database source table and a target field in the target table based on a task log of the application system, includes:

[0104] Determine the database source table data and target table data in the task log, wherein the database source table data includes the database source table name and the source field name in the database source table, and the target table data includes the target table name and the target field name in the target table;

[0105] The database source table data and the target table data are parsed based on graph computing to obtain the corresponding relationship.

[0106] In an optional implementation, the task log (CronLog) includes a log of a scheduled task in the background of the application system, and the task may be an ETL (Extract-Transform-Load) task.

[0107] For example, a scheduled synchronization task of the application system is to synchronize a database to another database at 0:00 every day. The task log can record whether the synchronization task is successful, the time taken for success, etc.

[0108] For example, a scheduled conversion task of the application system is to complete the data integration conversion from a database source table in a certain database to a target table at 12:00 every day. The task log can record the database source table data, target table data, mapping logic and steps of the integration conversion, whether it is successful, time taken for success, etc. in the conversion task, wherein the mapping logic includes the mapping relationship between the source field and the target field.

[0109] In an optional embodiment, the database source table and the target table are corresponding concepts. The database source table of the source database is written or converted into the target database, and the obtained table is the target table corresponding to the database source table.

[0110] In an optional embodiment, the source fields include fields of multiple types and data types, and the database source table can store the source fields in a row or column format. For example, the source fields of database source table L1 include: user ID (integer) and time (date); the source fields of database source table L2 include: location (string) and amount (decimal); and the source fields of database source table L3 include: purchase quantity (integer) and product type (string). The data in brackets indicates the data type of the source field.

[0111] In one optional embodiment, during the conversion of the database source table to the target table, source fields from multiple database source tables may be mapped to the target table in parallel. This mapping may include multi-level hierarchical mapping, including multiple intermediate initial tables before ultimately obtaining the target table. Furthermore, based on the business rules of the application system, only some source fields from the database source tables may be mapped to the target table.

[0112] For example, the data source tables L1-L3 are converted to the target table L4 according to the user's purchase rules. The target table L4 can be a purchase record table of a certain user, including the target field of the purchase record of a certain user. For example, the target field in the target table L4 can include: user 567, a certain day, flower market, 10, mobile phone case, 96.8, which means that user 567 bought 10 mobile phone cases at the flower market on a certain day, spending a total of 96.8 yuan.

[0113] In an optional embodiment, the graph-based computing-based parsing of the database source table data and the target table data to obtain the corresponding relationship includes:

[0114] Build a graph computing model based on Graph Neural Networks (GNN);

[0115] Using the source field name and the target field name as nodes of the graph computing model;

[0116] The source field name and the target field name that have a mapping relationship with each other are connected, and the line connecting the source field name and the target field name is used as the edge of the graph computing model to obtain the corresponding relationship.

[0117] In an optional embodiment, graphs are highly abstract and flexible. Compared to linear lists, hierarchical trees, and other organizational methods, using graph computing to parse the mapping relationship between source fields and target fields has stronger representation capabilities. The graph neural networks (GNNs) may include graph convolutional networks.

[0118] The graph computing model consists of a finite non-empty set of nodes or vertices and a set of edges between the nodes, which can be expressed as G(V,E). Where G represents a graph, V is the set of nodes in the graph G, and E is the set of edges connecting the nodes in the graph G. When E has a direction, G is a directed graph.

[0119] In an optional implementation, a directed edge pointing from a source field name to a target field name is used to represent the mapping from the source field to the target field, thereby obtaining a more intuitive corresponding relationship.

[0120] The acquisition module 201 is further configured to acquire access records containing the target field from the user access log of the application system.

[0121] In an optional embodiment, the user access log (AccessLog) includes all user access logs to their own reports. The user access log generally records several key indicators, such as the request URL (Uniform Request Locator), request parameters, user ID, access time, interface return value, etc. For example, user 567 accessed his purchase record table (target table L4), generating a user access log.

[0122] When a user views a target field in a target table, an access record containing the target field viewed by the user will appear in the user access log. For example, the user views the purchase quantity 10 and the product type mobile phone case of the account with user ID 567. Therefore, the user access log records the access records of the target fields "567", "10" and "mobile phone case".

[0123] The construction module 202 is used to construct an access count matrix of the source field combination corresponding to the target field based on the access record and the corresponding relationship, wherein the source field combination includes a source field and / or a combination of multiple source fields, and the access count matrix includes the access count of each source field combination.

[0124] In an optional implementation, users can generally only access the target field through the target table. After obtaining the user's access record to the target field, the number of indirect accesses of the user to the source field corresponding to the target field can be reversed based on the corresponding relationship, thereby determining the access popularity of the source field.

[0125] In an optional implementation, constructing an access count matrix of the source field corresponding to the target field according to the access records and the corresponding relationship includes:

[0126] Obtaining the number of accesses to the source field combination corresponding to the target field according to the access record and the corresponding relationship;

[0127] A two-dimensional matrix is ​​constructed by taking the source field combination as a row dimension attribute and a column dimension attribute, and the access count of the source field combination is taken as an element of the two-dimensional matrix to obtain the access count matrix.

[0128] In an optional implementation, obtaining the access count of the source field combination corresponding to the target field according to the access record and the corresponding relationship includes:

[0129] Determine a target field combination in the access record, where the target field combination includes one target field and / or a combination of multiple target fields;

[0130] Determining the source field combination corresponding to the target field combination according to the corresponding relationship;

[0131] When the target field combination appears n times in the same access record, it is determined that the number of accesses to the source field combination corresponding to the target field combination is n times, where n represents an integer greater than or equal to 1.

[0132] For example, the target field combination in the access record may be [567 10 mobile phone case], and the corresponding source field combination is [user ID purchase quantity product type].

[0133] In an optional embodiment, row and column attributes of a two-dimensional matrix can be defined based on the source field combination, and the access counts of the source field combination can be used as elements of the two-dimensional matrix to obtain the access count matrix. It should be noted that the combination of row and column attributes also represents the source field combination.

[0134] For example Figure 2 As shown, it is an example diagram of the access count matrix provided in an embodiment of the present application. Among them, the source field combination [A BC] is composed of a single source field A (row dimension attribute, such as [user ID]) and a source field combination [BC] (column dimension attribute, such as [purchase quantity commodity type]), and the access count of the source field combination [A BC] is 5. It should be noted that when a user has two user IDs (for example, 567 and 789), and both user IDs are accessed, the situation where the access count of [AA] is 1 will also occur. In addition, the access count matrix can be an asymmetric matrix, for example, the access counts of the source field combination [A BC] and the source field combination [BC A] can be different; the dimension of the access count matrix is ​​determined by the source field combination, for example, the dimension of the access count matrix can be m×n, and m and n both represent positive integers greater than 1.

[0135] In an optional embodiment, the larger the value of the element in the access count matrix, the greater the access count and access popularity of the source field combination composed of the corresponding combination of row dimension attributes and column dimension attributes.

[0136] The classification module 203 is configured to perform cluster analysis on the source field combinations according to the access times matrix, and divide the source field combinations into multiple categories according to the results of the cluster analysis.

[0137] In an optional embodiment, cluster analysis is performed on the source field combinations according to the access count matrix, and the source field combinations are divided into multiple categories according to the cluster analysis results, including:

[0138] Determine the value of the largest number of accesses in the access number matrix as the maximum value;

[0139] K numerical ranges are determined based on the maximum value, the numerical range to which the number of accesses of each source field combination belongs is determined, and source field combinations belonging to the same numerical range are determined to belong to the same category, where K represents an integer greater than 1.

[0140] For example, when the maximum access count value in the access count matrix is ​​100, it can be divided into three value ranges, namely 0-30, 31-60, and 61-100. The access count of the source field combination [A BC] is 5, so the source field combination [A BC] is divided into the 0-30 category.

[0141] In an optional embodiment, the cluster analysis method further includes a k-means clustering algorithm, etc. Based on the access count matrix, the source field combinations are classified according to access popularity, and the source fields in the source field combinations with greater access popularity can be grouped into one category.

[0142] The generating module 204 is configured to obtain multiple updated database source tables according to the multiple categories of source field combinations, and generate multiple data models based on the multiple updated database source tables.

[0143] In an optional embodiment, obtaining multiple updated database source tables according to the combination of source fields of the multiple categories, and generating multiple data models based on the multiple updated database source tables includes:

[0144] According to the K categories of source fields, a combination of source fields belonging to the same category is saved into an updated database source table, K updated database source tables are obtained, and an updated database source table name is defined for each updated database source table, wherein the updated database source table includes the source fields in the source field combination;

[0145] According to the K updated database source tables and the corresponding relationships, the target fields corresponding to the source fields in each updated database source table are merged into an updated target table, and an updated target table name is defined for each updated target table;

[0146] According to each updated database source table, each updated target table, and the task log, a data model for converting each updated database source table to each updated target table is constructed to generate the K data models.

[0147] In an optional embodiment, the access popularity of source field combinations of different categories is different, so the application popularity (for example, the possibility of being applied) of the data model corresponding to each updated database source table is also different, thereby building a core model of the application system (for example, the data model that is most frequently accessed and applied), and extended models with lower popularity outside the core model, thereby splitting and grading the data model of the application system and improving the availability of the application system.

[0148] In an optional embodiment, the task log stores the mapping logic for converting each source field to the corresponding target field. A data model for converting each updated database source table to each updated target table is processed or constructed based on the mapping logic to obtain the K data models. Existing ETL tools (e.g., RestCloud, Kettle, etc.) can be used to construct the data models. ETL tools can automatically build documents, implement version control required for multi-person development environments, and automatically parallelize tasks, thereby enabling efficient and orderly model construction.

[0149] In an optional embodiment, the model building method provided in the embodiment of the present application analyzes the lineage mapping relationship between the source field and the target field through the database source table data and the target table data, constructs the access count matrix of the source field combination using the access records of the target field, and uses a clustering algorithm to divide the source fields into updated database source tables according to the access count matrix. The core model of the application system is constructed using the updated database source table containing the source fields with higher access popularity, thereby realizing the automatic division of the data model structure of the application system, avoiding the bloated data model caused by piling too many fields into the same data model, and improving the application efficiency of the data model and the application system.

[0150] Example 3

[0151] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the above-mentioned model building method embodiment are implemented, for example: Figure 1 S11-S15 shown:

[0152] S11, based on the task log of the application system, obtaining a database source table and a target table of the application system, and a correspondence between a source field in the database source table and a target field in the target table;

[0153] S12, obtaining an access record containing the target field from the user access log of the application system;

[0154] S13, constructing an access count matrix of source field combinations corresponding to the target field based on the access records and the corresponding relationships, wherein the source field combination includes one source field and / or a combination of multiple source fields, and the access count matrix includes the access count of each source field combination;

[0155] S14, performing cluster analysis on the source field combinations according to the access count matrix, and dividing the source field combinations into multiple categories according to the results of the cluster analysis;

[0156] S15 , obtaining a plurality of updated database source tables according to the plurality of source field combinations of the categories, and generating a plurality of data models based on the plurality of updated database source tables.

[0157] Alternatively, when the computer program is executed by a processor, the functions of each module / unit in the above-mentioned device embodiment are realized, for example Figure 3 Modules 201-204:

[0158] The acquisition module 201 is used to obtain the database source table and target table of the application system, and the correspondence between the source field in the database source table and the target field in the target table based on the task log of the application system;

[0159] The acquisition module 201 is further configured to acquire access records containing the target field from the user access log of the application system;

[0160] The construction module 202 is configured to construct an access count matrix of source field combinations corresponding to the target field, wherein the source field combination includes a source field and / or a combination of multiple source fields, and the access count matrix includes the access count of each source field combination;

[0161] The classification module 203 is configured to perform cluster analysis on the source field combinations according to the access times matrix, and divide the source field combinations into multiple categories according to the results of the cluster analysis;

[0162] The generating module 204 is configured to obtain multiple updated database source tables according to the multiple categories of source field combinations, and generate multiple data models based on the multiple updated database source tables.

[0163] Example 4

[0164] See Figure 4 FIG. 3 is a schematic diagram of the structure of an electronic device according to Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the electronic device 3 includes a memory 31 , at least one processor 32 , at least one communication bus 33 and a transceiver 34 .

[0165] Those skilled in the art should understand that Figure 4 The structure of the electronic device shown does not constitute a limitation of the embodiments of the present invention, and can be either a bus structure or a star structure. The electronic device 3 can also include more or less other hardware or software than shown in the figure, or a different component arrangement.

[0166] In some embodiments, the electronic device 3 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The electronic device 3 may also include client devices, which include but are not limited to any electronic product capable of human-computer interaction with a client via a keyboard, mouse, remote control, touchpad, or voice-controlled device, such as a personal computer, tablet computer, smartphone, digital camera, etc.

[0167] It should be noted that the electronic device 3 is only an example. Other existing or future electronic products that are suitable for the present invention should also be included in the protection scope of the present invention and included here by reference.

[0168] In some embodiments, the memory 31 stores a computer program, and when the computer program is executed by the at least one processor 32, all or part of the steps in the model building method are implemented. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0169] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0170] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0171] In some embodiments, the at least one processor 32 is the control core (Control Unit) of the electronic device 3, which connects the various components of the entire electronic device 3 using various interfaces and lines, and executes various functions and processes data of the electronic device 3 by running or executing programs or modules stored in the memory 31, and calling data stored in the memory 31. For example, when the at least one processor 32 executes the computer program stored in the memory, it implements all or part of the steps of the model building method described in the embodiment of the present invention; or implements all or part of the functions of the model building device. The at least one processor 32 can be composed of an integrated circuit, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips.

[0172] In some embodiments, the at least one communication bus 33 is configured to implement connection and communication between the memory 31 and the at least one processor 32 .

[0173] Although not shown, the electronic device 3 may also include a power source (such as a battery) to power various components. Preferably, the power source may be logically connected to the at least one processor 32 via a power management device, thereby enabling the power management device to manage charging, discharging, and power consumption. The power source may also include one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components. The electronic device 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, cameras, etc., which will not be described in detail here.

[0174] The above-mentioned integrated unit implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module stored in a storage medium includes a number of instructions for causing a computer device (which can be a personal computer, electronic device, or network device, etc.) or a processor to execute portions of the methods described in various embodiments of the present invention.

[0175] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and actual implementation may employ other division methods.

[0176] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, and may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0177] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0178] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be regarded as exemplary and non-restrictive from any point of view, and the scope of the invention is defined by the appended claims rather than the above description, and it is intended that all variations that fall within the meaning and scope of the equivalents of the claims be encompassed within the present invention. Any figure marks in the claims should not be regarded as limiting the claims to which they relate. Furthermore, it is apparent that the word "comprising" does not exclude other units or, and the singular does not exclude the plural. Multiple units or devices stated in the specification may also be implemented by one unit or device through software or hardware. Words such as first and second are used to indicate names and do not indicate any particular order.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A model building method, characterized in that: The method comprises: Based on the task log of the application system, a database source table and a target table of the application system, as well as a correspondence between a source field in the database source table and a target field in the target table are obtained; Obtaining an access record containing the target field from a user access log of the application system; According to the access records and the corresponding relationship, an access count matrix of the source field combination corresponding to the target field is constructed, wherein the source field combination includes a source field and / or a combination of multiple source fields, and the access count matrix includes the access count of each source field combination; the construction of the access count matrix of the source field combination corresponding to the target field according to the access records and the corresponding relationship includes: obtaining the access count of the source field combination corresponding to the target field according to the access records and the corresponding relationship; constructing a two-dimensional matrix using the source field combination as row dimension attributes and column dimension attributes, and using the access count of the source field combination as an element of the two-dimensional matrix to obtain the access count matrix; Performing a cluster analysis on the source field combinations according to the access count matrix, and dividing the source field combinations into a plurality of categories according to the cluster analysis results, including: determining the value of the largest access count in the access count matrix as a maximum value; determining K value ranges according to the maximum value, determining the value range to which the access count of each source field combination belongs, and determining source field combinations belonging to the same value range as belonging to the same category, wherein K represents an integer greater than 1; A plurality of updated database source tables are obtained according to the plurality of source field combinations of the plurality of categories, and a plurality of data models are generated based on the plurality of updated database source tables.

2. The model building method according to claim 1, wherein: The step of obtaining a database source table and a target table of the application system, and a correspondence between a source field in the database source table and a target field in the target table based on the task log of the application system, includes: Determine the database source table data and target table data in the task log, wherein the database source table data includes the database source table name and the source field name in the database source table, and the target table data includes the target table name and the target field name in the target table; The database source table data and the target table data are parsed based on graph computing to obtain the corresponding relationship.

3. The model building method according to claim 2, wherein: The graph-based computing step of parsing the database source table data and the target table data to obtain the corresponding relationship includes: Build a graph computing model based on graph neural computing network; Using the source field name and the target field name as nodes of the graph computing model; The source field name and the target field name that have a mapping relationship with each other are connected, and the line connecting the source field name and the target field name is used as the edge of the graph computing model to obtain the corresponding relationship.

4. The model building method according to claim 1, wherein: The obtaining, according to the access record and the corresponding relationship, the number of accesses of the source field combination corresponding to the target field includes: Determine a target field combination in the access record, where the target field combination includes one target field and / or a combination of multiple target fields; Determining the source field combination corresponding to the target field combination according to the corresponding relationship; When the target field combination appears n times in the same access record, it is determined that the number of accesses to the source field combination corresponding to the target field combination is n times, where n represents an integer greater than or equal to 1.

5. The model building method according to claim 1, wherein: The step of obtaining a plurality of updated database source tables according to the plurality of source field combinations of the plurality of categories, and generating a plurality of data models based on the plurality of updated database source tables includes: According to the source field combinations of the K categories, the source field combinations belonging to the same category are saved into an updated database source table, K updated database source tables are obtained, and an updated database source table name is defined for each updated database source table; According to the K updated database source tables and the corresponding relationships, the target fields corresponding to the source fields in each updated database source table are merged into an updated target table, and an updated target table name is defined for each updated target table; According to each updated database source table, each updated target table, and the task log, a data model for converting each updated database source table to each updated target table is constructed to generate K data models.

6. A model building device for implementing the model building method according to claim 1, characterized in that: The device includes an acquisition module, a construction module, a classification module, and a generation module: The acquisition module is used to obtain the database source table and target table of the application system, and the corresponding relationship between the source field in the database source table and the target field in the target table based on the task log of the application system; The acquisition module is further configured to acquire access records containing the target field from the user access log of the application system; The construction module is configured to construct an access count matrix of the source field combination corresponding to the target field based on the access record and the corresponding relationship, wherein the source field combination includes a source field and / or a combination of multiple source fields, and the access count matrix includes the access count of each source field combination; The classification module is configured to perform cluster analysis on the source field combinations according to the access times matrix, and divide the source field combinations into multiple categories according to the results of the cluster analysis; The generating module is configured to obtain a plurality of updated database source tables according to the plurality of source field combinations of the plurality of categories, and generate a plurality of data models based on the plurality of updated database source tables.

7. An electronic device, characterized in that: The electronic device includes a processor and a memory, and the processor is configured to implement the model building method according to any one of claims 1 to 5 when executing a computer program stored in the memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model building method according to any one of claims 1 to 5 is implemented.

Images