Layered data verification method and its device, equipment and medium
Through the layered data verification method, the disassembled or cross-layer references in the layered data warehouse are automatically detected and optimized, which improves the efficiency of the data model and the utilization of computing resources, and solves the problem of inefficiency in the existing technology.
Patent Information
- Application Number
- CN202210399044.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-15
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-04-15
AI Technical Summary
There are strata references or cross-layer references in existing hierarchical data warehouses, resulting in waste of computing resources and low development efficiency, making it difficult for developers to effectively manage hierarchical data warehouses.
A hierarchical data verification method is proposed. By obtaining the calculation tasks in the data model and the standard table names of the model data tables, running the calculation tasks, generating resource consumption data, drawing a data table mapping relationship map, and performing referenced data tables Hierarchical verification, matches recommended data tables that are better than severed or cross-layer references, and pushes them to the development side for optimization.
Automatically manage hierarchical data warehouses, improve the efficiency of data models, reduce waste of computing resources, improve data acquisition efficiency, and optimize the rationality of data sources.
Smart Images

Figure CN114722126B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of database management, and in particular to a hierarchical data verification method, and further to corresponding devices, equipment and non-volatile storage media of the method. Background Art
[0002] Existing Internet platforms use hierarchical data warehouses to hierarchically store data and data tables in the platform. For example, Internet live broadcast platforms will usually aggregate the data of the live broadcast rooms, anchor users and audience users in the platform into data tables required for related live online services, and the data tables in the hierarchical data warehouse also reference the data tables stored in the warehouse. By developing corresponding data models, the data tables associated with them are obtained from the corresponding data sources, and then new data tables are created by obtaining aggregated data from other data tables for application to the newly deployed online services of the platform.
[0003] However, when data tables in existing hierarchical data warehouses reference other data sources in the warehouse, there are often staggered or cross-layer references. When the data model obtains data from its staggered or cross-layer referenced data tables, it will waste too many computing resources.
[0004] Moreover, when platform developers check whether the data models they develop have cross-layer or staggered references, they can often only determine it by checking the complicated codes in each computing task of the data model. This is too inefficient and cannot effectively manage the hierarchical data warehouse. When developing data models, developers can only manually query whether there are data sources suitable for reference in the hierarchical data warehouse, and the development efficiency is also very low.
[0005] In view of the problems existing in the existing hierarchical data warehouse, the applicant has made corresponding explorations in order to solve the problems. Summary of the invention
[0006] The purpose of the present application is to provide a hierarchical data verification method to meet user needs, and also relates to corresponding devices, equipment, non-volatile storage media and computer program products of the method.
[0007] In order to achieve the purpose of this application, the following technical solutions are adopted:
[0008] A hierarchical data verification method proposed for the purpose of this application includes the following steps:
[0009] In response to a data model adding instruction pushed by a development end and acting on a hierarchical data warehouse, a plurality of computing tasks included in the data model corresponding to the instruction are obtained, and a standard table name of a model data table corresponding to the data model is obtained, wherein the standard table name includes a data level, a data update frequency, and a data subject;
[0010] Performing a test run on each of the computing tasks to determine the computing resources consumed by each of the computing tasks, thereby generating resource consumption data representing the computing resources consumed by each of the computing tasks, and determining the reference data table corresponding to each of the computing tasks in the hierarchical data warehouse;
[0011] Based on the data hierarchy in the standard table name of the model data table, each of the reference data tables is traced upstream and downstream, the upstream and downstream data tables associated with each of the reference data tables in the hierarchical data warehouse are determined, and a data table mapping relationship diagram with the model data table as the central node is drawn;
[0012] According to the standard table names of the model data table and each of the referenced data tables, the referenced data tables are respectively checked at a hierarchical level. If the check result of a referenced data table indicates the existence of cross-layer references or staggered-layer references, a recommended data table that is better than the referenced data table is matched for the data model, and the recommended data table, the resource consumption data and the data table mapping relationship map are pushed to the development end.
[0013] In a further embodiment, before the step of responding to the data model adding instruction pushed by the development end and acting on the hierarchical data warehouse, the following steps are performed by the development end:
[0014] In response to a creation instruction acting on a model data table, obtain a data table name of the model data table corresponding to the creation instruction;
[0015] According to the preset naming rules, the data table name is verified to check whether the data table name contains the data level, data update frequency and data subject;
[0016] When the data table name passes the naming verification, the data table name will be packaged as a standard table name and the model data table as a data model adding instruction and pushed to the server.
[0017] In a further embodiment, the step of testing and running each of the computing tasks to determine the computing resources consumed by each of the computing tasks, and then generating resource consumption data representing the computing resources consumed by each computing task, includes the following steps performed by the server:
[0018] Run the computing task to determine the memory index and processor index occupied by the computing task when the computing task is completed;
[0019] Using the memory index and the processor index as computing resources of the computing task to generate mapping relationship data including the computing resources and the task name of the computing task;
[0020] By analogy, the mapping relationship data of all computing tasks are generated, and then the resource consumption data including the mapping relationship data is generated.
[0021] In a further embodiment, the step of tracing the upstream and downstream sources of each reference data table based on the data level of the model data table corresponding to the data model in the hierarchical data warehouse, determining the upstream and downstream data tables associated with each reference data table in the hierarchical data warehouse, and drawing a data table mapping relationship map with the model data table as the central node includes the following steps executed by the server:
[0022] Acquire the data level contained in the standard table name of the model data table as the first data level, and acquire the data level contained in the standard table name of the currently referenced data table as the second data level;
[0023] When the first data level exceeds the second data level, one or more data tables associated with the current referenced data table between the first and second data levels are determined as upstream data tables of the current referenced data table, and the current referenced data table is used as the terminal node of its upstream path. The terminal node is used as the drawing starting point, and the upstream data tables are drawn as nodes in the upstream path in order from low to high according to the data levels corresponding to the upstream data tables;
[0024] When the second data level exceeds the first data level, one or more data tables associated with the current referenced data table between the first and second data levels are determined as downstream data tables of the current referenced data table, and the current data table is used as the starting node of its downstream path. The starting node is used as the drawing starting point, and the downstream data tables are drawn as nodes in the downstream path in order from high to low according to the data levels corresponding to the downstream data tables;
[0025] Similarly, after drawing the upstream path or downstream path of each reference data table respectively, each upstream path and downstream path is connected with the model data table as the central node of the path to draw the data table mapping relationship map.
[0026] In a further embodiment, the step of performing hierarchical verification on the reference data tables according to the standard table names of the model data table and each of the reference data tables comprises the following steps performed by the server:
[0027] Acquire the data level contained in the standard table name of the model data table as the first data level, and acquire the data level contained in the standard table name of the currently referenced data table as the second data level;
[0028] When the first data level is greater than the second data level, and the level interval between the two exceeds one, it is characterized that the reference to the current referenced data table is a cross-layer reference, and a verification result representing the cross-layer reference will be generated for the current referenced data table;
[0029] When the first data level is smaller than the second data level, it indicates that the reference to the current referenced data table is a staggered reference, and a verification result indicating the staggered reference is generated for the current referenced data table.
[0030] In a further embodiment, the step of matching the data model with a recommended reference data table that is better than the reference data table includes the following steps performed by the server:
[0031] The data level contained in the standard table name of the model data table is used as the first data level and the data update frequency is used as the target data update frequency, and the data subject contained in the standard table name of the cross-layer or staggered reference data table is used as the target data subject and the data level is used as the second data level;
[0032] Determine one or more target data levels in the hierarchical data warehouse that are in a level interval that is equal to the first data level at most and higher than the second data level at least, so as to query one or more first data tables in the target data level whose standard table names have the target data subject;
[0033] Query whether there is a reference data table in the first data table whose standard table name has the target data update frequency and whose data level is closest to the first data level. If so, further query the reference data table to be used as the recommended reference data table. If not, use the reference data table in the first data table whose data level is closest to the first data level as the recommended reference data table.
[0034] In a further embodiment, after the step of pushing the recommended data table, the resource consumption data and the data table mapping relationship map to the development end, the following steps are performed by the server:
[0035] In response to a periodic layer scanning instruction acting on the layered data warehouse, performing a layer verification on the data tables of all data layers of the layered data warehouse;
[0036] Determine one or more target data models with hierarchical verification results that have cross-layer references or staggered-layer references, match the target data models with recommended reference data tables that are better than the reference data tables corresponding to the hierarchical verification results, and push them so that the development end corresponding to the target data models can perform model optimization.
[0037] A hierarchical data verification device proposed for the purpose of this application includes:
[0038] A computing task acquisition module is used to respond to a data model adding instruction pushed by a development end and acting on a hierarchical data warehouse, obtain a plurality of computing tasks contained in a data model corresponding to the instruction, and obtain a standard table name of a model data table corresponding to the data model, wherein the standard table name includes a data level, a data update frequency, and a data subject;
[0039] A computing task testing module, used to test and run each of the computing tasks, determine the computing resources consumed by each of the computing tasks, and then generate resource consumption data representing the computing resources consumed by each of the computing tasks, and determine the reference data table corresponding to each of the computing tasks in the hierarchical data warehouse;
[0040] A relationship map drawing module is used to trace the upstream and downstream sources of each reference data table based on the data hierarchy in the standard table name of the model data table, determine the upstream and downstream data tables associated with each reference data table in the hierarchical data warehouse, and draw a data table mapping relationship map with the model data table as the central node;
[0041] The hierarchical verification processing module is used to perform hierarchical verification on the reference data tables according to the standard table names of the model data table and each reference data table. If the verification result of a reference data table indicates the existence of cross-layer references or staggered layer references, a recommended data table that is better than the reference data table is matched for the data model, and the recommended data table, the resource consumption data and the data table mapping relationship map are pushed to the development end.
[0042] In a further embodiment, the computing task test module includes:
[0043] An occupation index determination submodule is used to run the computing task and determine the memory index and processor index occupied by the computing task after the task is completed.
[0044] A relational data generation submodule, used to use the memory index and the processor index as computing resources of the computing task to generate mapping relational data including the computing resources and the task name of the computing task;
[0045] The resource consumption data generation submodule is used to generate mapping relationship data of all computing tasks by analogy, and then generate the resource consumption data including the mapping relationship data.
[0046] In a further embodiment, the relationship map drawing module includes:
[0047] A data level acquisition submodule, used to acquire the data level contained in the standard table name of the model data table as the first data level, and acquire the data level contained in the standard table name of the current referenced data table as the second data level;
[0048] An upstream path drawing submodule, for determining, when the first data level exceeds the second data level, one or more data tables associated with the current referenced data table between the first and second data levels as upstream data tables of the current referenced data table, and then using the current referenced data table as the end node of its upstream path, taking the end node as the drawing starting point, and drawing the upstream data tables as nodes in the upstream path in order from low to high according to the data levels corresponding to the respective upstream data tables;
[0049] A downstream path drawing submodule, for determining, when the second data level exceeds the first data level, one or more data tables associated with the current referenced data table between the first and second data levels as downstream data tables of the current referenced data table, and then using the current data table as the starting node of its downstream path, taking the starting node as the drawing starting point, and drawing the downstream data tables as nodes in the downstream path in order from high to low according to the data levels corresponding to the respective downstream data tables;
[0050] The mapping relationship graph drawing submodule is used to, by analogy, draw the upstream path or downstream path of each reference data table respectively, connect each upstream path and downstream path with the model data table as the central node of the path, and draw the data table mapping relationship graph.
[0051] In a further embodiment, the layer verification processing module includes:
[0052] A data level acquisition submodule, used to acquire the data level contained in the standard table name of the model data table as the first data level, and acquire the data level contained in the standard table name of the current referenced data table as the second data level;
[0053] A cross-layer reference check submodule, used for characterizing that the reference to the current referenced data table is a cross-layer reference when the first data level is greater than the second data level and the level interval between the two sides exceeds one, and generating a check result characterizing the cross-layer reference for the current referenced data table;
[0054] The staggered-layer reference check submodule is used to characterize that the reference to the current referenced data table is a staggered-layer reference when the first data level is smaller than the second data level, and to generate a check result characterizing the staggered-layer reference for the current referenced data table.
[0055] In a preferred embodiment, the hierarchical verification processing module further includes:
[0056] A standard table name parsing submodule, used to use the data level contained in the standard table name of the model data table as the first data level and the data update frequency as the target data update frequency, and use the data subject contained in the standard table name of the cross-layer or staggered reference data table as the target data subject and the data level as the second data level;
[0057] A target data table query submodule is used to determine one or more target data levels in the hierarchical data warehouse that are in a level interval that is at most equal to the first data level and at least higher than the second data level, so as to query one or more first data tables in the target data level whose standard table names have the target data subject;
[0058] The recommended data table query submodule is used to query whether there is a reference data table in the first data table whose standard table name has the target data update frequency and whose data level is closest to the first data level. If so, further query the reference data table to be used as the recommended reference data table. If not, use the reference data table in the first data table whose data level is closest to the first data level as the recommended reference data table.
[0059] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned layered data verification method.
[0060] In order to solve the above technical problems, an embodiment of the present application further provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned layered data verification method.
[0061] In order to solve the above technical problem, an embodiment of the present application also provides a computer program product, including a computer program and computer instructions. When the computer program and computer instructions are executed by a processor, the processor executes the steps of the above-mentioned layered data verification method.
[0062] Compared with the prior art, the advantages of this application are as follows:
[0063] The present application automatically performs hierarchical verification on each data table in a hierarchical data warehouse, and matches a data model with cross-layer or staggered references to a recommended data table that is better than the corresponding one, so that developers can modify the staggered or cross-layer referenced data table to a recommended data table, so as to improve the data model to obtain data from the recommended data table to construct the data in the corresponding model data table. Specifically, the server performs hierarchical verification on the referenced data table through the standardized table name of the model data table and the referenced referenced data table, determines whether the referenced data table is a data table with cross-layer or staggered references, and then matches a recommended data table that is better than the data table with cross-layer or staggered references. Generally, a data table with a level close to the model data table and a consistent data update frequency and data subject is recommended for developers to optimize, so as to improve the efficiency of obtaining various types of data required for the model data table, and automatically manage the rationality of the data source for each data table in the hierarchical data warehouse.
[0064] Secondly, when developers develop new data models, this application will test the computing resources occupied by each computing task in the new data model during execution, and combine it with the recommended data tables that are automatically matched for developers to facilitate developers to understand the computing resources wasted by cross-layer or staggered data table references, so that developers can optimize the computing tasks in the new data model and improve the execution efficiency of the new data model.
[0065] In addition, the present application draws a data table mapping relationship map for the data model, which can represent the upstream and downstream relationships of the data tables referenced by the corresponding model data tables in the hierarchical database, so that developers can optimize the data tables referenced by the data model through the upstream and downstream relationships between each referenced data table and the model data table displayed in the data table mapping relationship map, or create new data tables to optimize the data source of the data model, thereby improving the execution efficiency of the data model and ensuring the data acquisition efficiency of the model data table.
[0066] Furthermore, the present application automatically and regularly scans the hierarchical data warehouse to verify the cross-layer or staggered referenced data tables for each data table in the hierarchical data warehouse, and matches the corresponding recommended data tables for the data models associated with these data tables, thereby improving the overall data update efficiency and data quality of the hierarchical data warehouse. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0068] Figure 1 A schematic diagram of a typical network deployment architecture related to implementing the technical solution of this application;
[0069] Figure 2A flowchart of a typical embodiment of the hierarchical data verification method of the present application;
[0070] Figure 3 A schematic diagram of each data level in a hierarchical data warehouse in this application;
[0071] Figure 4 A schematic diagram of an upstream path for referencing a data table in this application;
[0072] Figure 5 A schematic diagram of a downstream path for referencing a data table in this application;
[0073] Figure 6 A schematic diagram of a data table mapping relationship map in this application;
[0074] Figure 7 A flowchart diagram of a specific implementation method for performing naming verification on a data table name of a model data table in this application;
[0075] Figure 8 A flowchart of a specific implementation method for determining the computing resources occupied by a computing task during execution in this application;
[0076] Fig. 9 A schematic diagram of a process flow formed by a specific implementation method of drawing a data table mapping relationship map in this application;
[0077] Fig.10 A flowchart of a specific implementation method for layered verification of a reference data table of a model data table in this application;
[0078] Fig.11 A flowchart of a specific implementation method of matching a data model with a recommended data table that is superior to a data table referenced across layers or staggered layers in the present application;
[0079] Fig.12 A flowchart of a specific implementation method of periodic layer scanning processing performed by a server in this application;
[0080] Fig.13 It is a principle block diagram of a typical embodiment of the hierarchical data verification device of the present application;
[0081] Fig.14 This is a basic structural block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION
[0082] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.
[0083] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0084] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0085] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; traditional laptop and / or palmtop computers or other devices, which have and / or include radio frequency receivers. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.
[0086] The hardware referred to by the names such as "server", "client", and "work node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.
[0087] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.
[0088] See also Figure 1 , the hardware foundation required for the implementation of the relevant technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud. As an online server, it can be responsible for further connecting relevant data servers and other servers that provide relevant support, etc., so as to form a logically related service cluster to provide services for related terminal devices such as the smartphone 81 and personal computer 82 shown in the figure or a third-party server (not shown). The smartphone and personal computer can access the Internet through a well-known network access method and establish a data communication link with the server 80 in the cloud to run terminal applications related to the services provided by the server.
[0089] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.
[0090] The application described herein refers to an application running on a server or terminal device. This application implements the relevant technical solutions of the present application in a programming manner. Its program code can be stored in a non-volatile storage medium recognizable by a computer in the form of computer executable instructions, and can be loaded into the memory by the central processing unit for execution. The relevant device of the present application is constructed by the operation of the application on the computer.
[0091] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.
[0092] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.
[0093] See also Figure 2 , a hierarchical data verification method of the present application, in its typical embodiment, comprises the following steps:
[0094] Step S11, responding to the data model adding instruction pushed by the development end and acting on the hierarchical data warehouse, obtaining multiple computing tasks contained in the data model corresponding to the instruction, and obtaining the standard table name of the model data table corresponding to the data model, wherein the standard table name includes the data level, data update frequency and data subject:
[0095] The server responds to the data model adding instruction pushed by the development end for adding a new data model in the hierarchical data warehouse to obtain one or more computing tasks contained in the data model corresponding to the instruction, and obtains the standard table name of the model data table created corresponding to the data model.
[0096] The data model adding instruction is triggered by the development end to create a new data model. Specifically, after the developer at the development end completes the creation of the data model in the data model creation page, the development end will respond to the creation instruction acting on the data model, obtain the data table name configured by the developer for the data model, and perform a naming verification on the data table name according to the preset naming rules to verify whether the data table name contains the data hierarchy, data update frequency and data subject. When the data table name passes the naming verification, the data table name will be encapsulated as a standard table name and the model data table as a data model adding instruction and pushed to the server.
[0097] Please refer to Figure 3 , Figure 3It is a schematic diagram of each data layer in the hierarchical data warehouse. As shown in the figure, the hierarchical data warehouse includes multiple data layers, which are generally configured according to the unified standards in this field. For example, the hierarchical data warehouse generally includes three basic data layers: data application layer (APP), data warehouse layer (DW) and data operation layer (ODS), and the data layer of the data application layer (APP) is generally higher than that of the data warehouse layer (DW). Correspondingly, the data warehouse layer (DW) is generally higher than the data application layer (ODS). Among them, the data warehouse layer (DW) generally also includes a detailed data layer (DWV), a lightly summarized data layer (DWM), a wide table summary layer (DWS), a data mart layer (DM) and a common dimension layer (DIM). For the hierarchical relationship of each data layer in the data warehouse layer (DW), please refer to Figure 3 The relevant instructions in the description will not be repeated here.
[0098] The data model generally includes computing tasks for acquiring the data stored corresponding to the model data table to construct the model data table. These computing tasks respectively reference corresponding data tables from the hierarchical data warehouse to acquire the data required for the model data table. For example, when the model data table is a daily audience user information data table of all live broadcast rooms in the singing area, the data model corresponding to the model data table will include computing tasks for acquiring the live broadcast rooms in the singing area and computing tasks for acquiring audience user information entering the live broadcast rooms in the singing area. Of course, the computing task may also be a computing task for acquiring audience user information entering the live broadcast rooms in the singing area every day. These computing tasks will reference relevant data tables with the data they need to acquire to acquire the corresponding data. In specific actual business scenarios, the data tables referenced by each computing task are relatively flexible and changeable. Technical personnel in this field may refer to the data tables in the hierarchical data warehouse of their own platforms to configure the data tables referenced by each computing task. This step will not be repeated here.
[0099] Regarding the data level, data update frequency and data subject included in the standard table name corresponding to the model data table, the data level is used to specify the data level of the model data table in the hierarchical data warehouse, the data update frequency is used to specify the update frequency of the data stored in the model data table, and the data subject is used to specify the data type of the data stored in the model data table.
[0100] Step S12, test running each of the computing tasks to determine the computing resources consumed by each of the computing tasks, and then generate resource consumption data characterizing the computing resources consumed by each of the computing tasks, and determine the reference data table corresponding to each of the computing tasks in the hierarchical data warehouse:
[0101] After the server obtains the computing tasks included in the data model, it will test run these computing tasks to start the computing resources consumed when running these computing tasks, and then generate the resource consumption data representing the computing resources consumed by these computing tasks.
[0102] When the server is testing and running the computing task, if the code contained in the computing task has a format error or a compilation error and cannot be executed, the server will not continue to test and run the computing task, and the computing task will be determined as having an error in the code, and a notification of a code error in the computing task representing the code with a format error or a compilation error will be pushed to the development end corresponding to the computing task.
[0103] When testing the computing task, the server will determine the computing resources consumed by the computing task during operation to generate the resource consumption data representing the computing resources consumed by the computing tasks. Specifically, the server runs any of the computing tasks to determine the memory index (MBS) and processor index (CPUS) occupied when the computing task completes its operation, and uses the memory index and processor index as the computing resources of the computing task to generate mapping relationship data including the computing resources and the task name of the computing task. Similarly, mapping relationship data for all computing tasks are generated, and then the resource consumption data including the mapping relationship data are generated, so as to facilitate subsequent push to the development end that develops the computing tasks, so that developers at the development end can understand the computing resources occupied when each computing task is running for reference.
[0104] When the server tests and runs the computing task, it will determine the reference data table referenced by each computing task in the hierarchical data warehouse when obtaining data for the model data table by parsing the code of the computing task or using methods such as Atlas, so as to subsequently draw the data table mapping relationship map and perform hierarchical verification on the reference data table.
[0105] Step S13, based on the data hierarchy in the standard table name of the model data table, each of the reference data tables is traced upstream and downstream, the upstream and downstream data tables associated with each of the reference data tables in the hierarchical data warehouse are determined, and a data table mapping relationship diagram with the model data table as the central node is drawn:
[0106] After the server determines the reference data table referenced by the model data table, it will trace the upstream and downstream of each reference data table based on the data hierarchy in the standard table name of the model data table to draw the data table mapping relationship map that represents the upstream and downstream relationship between each data table and the model data table.
[0107] The data table mapping relationship map is used to characterize the upstream and downstream mapping relationship between the data table referenced by the model data table and the model data table. Specifically, after the server obtains all the referenced data tables referenced by the model data table, it obtains the data level contained in the standard table name of the model data table as the corresponding data level as the first data level, that is, the data level of the model data table in the hierarchical data warehouse is taken as the first data level, and the data level in the standard table name of the referenced data table whose path is currently to be drawn is obtained as the second data level, and then it is determined whether the first data level exceeds the second data level. If it exceeds, the model data table is the upstream of the referenced data table, and one or more data tables associated with the referenced data table between the first and second data levels are determined as the upstream data tables of the referenced data table, and then the referenced data table is used as the end node of its upstream path, and the end node is used as the drawing starting point. According to the data levels corresponding to each of the upstream data tables, these upstream data tables are drawn as nodes in the upstream path from low to high. For details, please refer to Figure 4 When the model data table is the daily audience user data table of the live broadcast room in the singing area, and the corresponding reference data table is the audience user data table, the model data table can generally be in the wide table summary layer (DWS), and the reference data table is generally in the detailed data layer (DWV). The data level of the model data table is greater than that of the reference data table. Correspondingly, when the reference data table has an associated upstream data table of the daily audience user data table in the light summary data layer (DWM), the upstream path drawn for the reference time table is as follows: Figure 4 As shown, the reference data table is located at Figure 4 The upstream data table is the node 403 in the upstream path 401.
[0108] Correspondingly, when the second data level of the reference data table exceeds the first data level of the model data table, one or more data tables associated with the reference data table between the first and second data levels are determined as downstream data tables of the reference data table, and then the data table is used as the starting node of its downstream path. With the starting node as the starting point for drawing, according to the data levels corresponding to each of the downstream data tables, the downstream data tables are drawn in order from high to low as nodes in the downstream path. For details, please refer to Figure 5When the model data table is the daily audience user data table of the live broadcast room in the singing area, and the corresponding reference data table is the live broadcast room conversion rate data table, the model data table can generally be in the wide table summary layer (DWS), and the reference data table is generally in the data application layer (APP). The data level of the reference data table is greater than that of the model data table. Correspondingly, when the reference data table has an associated downstream data table of the live broadcast room virtual gift giving data table in the light summary data layer (DM), the downstream path drawn for the reference time table is as follows: Figure 5 As shown, the reference data table is located at Figure 5 The starting node 502 in the downstream path 501 is the downstream data table, and the downstream data table is the node 503 in the downstream path.
[0109] After the server completes the drawing of the upstream and downstream paths of all referenced data tables according to the above-mentioned upstream path drawing method and downstream path drawing method, the server will connect the upstream paths and downstream paths with the model data table as the central node to draw the data table mapping relationship map. For details, please refer to Figure 6 , Figure 6 For the general Figure 4 The upstream path shown and Figure 5 The drawn downstream path is a data table mapping relationship diagram drawn with the model data table as the center, Figure 6 The central node 601 in is the model data table (daily audience user data table of the live broadcast room in the singing area).
[0110] Step S14: perform hierarchical verification on the reference data tables according to the standard table names of the model data table and each reference data table. If the verification result of a reference data table indicates the existence of cross-layer reference or staggered layer reference, match a recommended data table that is better than the reference data table for the data model, and push the recommended data table, the resource consumption data and the data table mapping relationship map to the development end:
[0111] After the server obtains all reference data tables associated with the model data table, it will perform hierarchical verification on the reference data tables respectively to determine whether there is a staggered reference or cross-layer reference between the model data table and the reference data tables.
[0112] After the server obtains the reference data table, it will parse the data level contained in the standard table name of the model data table, and parse the data level contained in the standard table name of the reference data table, and use the data level of the model data table as the first data level, and use the data level of the reference data table as the second level. When the first data level is greater than the second data level, and the level interval between the two sides exceeds one, it represents that the reference to the reference data table is a cross-level reference, and a verification result representing the cross-level reference will be generated for the reference data table. When the first data level is less than the second data level, it represents that the reference to the reference data table is a staggered-level reference, and a verification result representing the staggered-level reference will be generated for the reference data table. If the first data level is the same as the second data level, it represents that the reference to the reference data table does not have the staggered-level reference and the cross-level reference, and the verification result representing the same level of the reference data table will be used.
[0113] When the server determines through the hierarchical verification that the verification result of any of the referenced data tables is a staggered reference or a cross-layer reference, the server will match the model data table to a recommended data table that is superior to the referenced data table. Specifically, the server uses the data hierarchy of the model data table associated with the referenced data table whose verification result is a staggered reference or a cross-layer reference as the data hierarchy, and uses the data update frequency contained in the standard table name of the model data table as the target data update frequency, and uses the data subject contained in the standard table name of the referenced data table as the target data subject, and uses the data hierarchy contained in the standard table name as the second data hierarchy, so as to match the model data table with the referenced data table in the hierarchical data hierarchy. One or more target data levels are determined in the warehouse, with the maximum level being equal to the first data level and the minimum level being higher than the second data level. Then, one or more first data tables whose standard table names have the target data subject are re-derived from these target data levels, so as to query whether there is a reference data table whose standard table name has the target data update frequency and whose data level is closest to the first data level in these first data tables. If so, the reference data table is further queried to be used as the recommended reference data table. If not, the reference data table whose data level in the first data table is closest to the first data level is used as the recommended reference data table.
[0114] After the server completes the hierarchical verification and matching of the relevant recommended data tables for all the referenced data tables, the server pushes the verification results of each referenced data table and the corresponding recommended data table to the development end to which the model data table belongs, so that the development end can optimize the data table referenced by the corresponding computing task in the data model according to the verification results and the recommended data table. At the same time, the server pushes the resource consumption data and the data table mapping relationship map to the development end, so that the developers at the development end can refer to the resource consumption data and the data table mapping relationship map to optimize the computing tasks contained in the data model they developed.
[0115] It can be seen from the typical implementation of the method that the method automatically performs hierarchical verification on each data table in the hierarchical data warehouse, and matches the data model with cross-layer or staggered references to a recommended data table that is better than the corresponding one, so that the developer can modify the staggered or cross-layer referenced data table to a recommended data table, so as to improve the data model to obtain data from the recommended data table to construct the data in the corresponding model data table. Specifically, the server performs hierarchical verification on the referenced data table through the standard table name of the standardized naming of the model data table and the referenced referenced data table, determines whether the referenced data table is a data table with cross-layer or staggered references, and then matches the recommended data table that is better than the data table with cross-layer or staggered references. Generally, a data table with a level close to the model data table and a consistent data update frequency and data subject is recommended for the developer to optimize, so as to improve the efficiency of obtaining various types of data required for the model data table, and automatically manage the rationality of the data source for each data table in the hierarchical data warehouse.
[0116] Secondly, when developers develop new data models, this method will test the computing resources occupied by each computing task in the new data model during operation, and combine it with the recommended data tables automatically matched for developers, so that developers can understand the computing resources wasted by cross-layer or staggered references to data tables, so that developers can optimize the computing tasks in the new data model and improve the execution efficiency of the new data model.
[0117] In addition, this method draws a data table mapping relationship map for the data model, which can represent the upstream and downstream relationships of the data tables referenced by the corresponding model data tables in the hierarchical database, so that developers can optimize the data tables referenced by the data model through the upstream and downstream relationships between each referenced data table and the model data table displayed in the data table mapping relationship map, or create new data tables to optimize the data source of the data model, thereby improving the execution efficiency of the data model and ensuring the data acquisition efficiency of the model data table.
[0118] The above typical embodiments and their variations fully disclose the implementation scheme of the hierarchical data verification method of the present application. However, various variations of the method can still be derived by changing and amplifying some technical means. Other embodiments are briefly described as follows:
[0119] In one embodiment, please refer to Figure 7 , before the step of responding to the data model adding instruction pushed by the development end and acting on the hierarchical data warehouse, the step includes the following steps executed by the development end:
[0120] Step S08, in response to the creation instruction acting on the model data table, obtain the data table name of the model data table corresponding to the creation instruction:
[0121] After the developer at the development end completes the creation of the model data table he has developed, the creation instruction will be triggered so that the development end responds to the creation instruction and obtains the data table name edited by the developer for the model data table.
[0122] Step S09: perform a naming check on the data table name according to a preset naming rule to check whether the data table name contains the data level, data update frequency and data subject:
[0123] After the development end determines the data table name of the model data table, it will perform a naming verification on the data table name according to the mapped naming rule. The naming rule defines the naming format of the data table name of the model data table. The development end verifies whether the data table name contains the data subject used to define the data level of the model data table in the hierarchical data warehouse, the data update frequency for defining the model data table to update the data stored therein, and the data type stored in the model data table through the naming format defined in the naming rule.
[0124] Step S10, when the data table name passes the naming verification, the data table name will be encapsulated as a standard table name and the model data table as a data model adding instruction and pushed to the server:
[0125] When the development end determines that the data table name of the model data table passes the naming verification, the data table name will be determined by the development end as a standard table name. The development end will encapsulate the data table name of the model data table and its corresponding data model into the data model adding instruction, and push the data model adding instruction to the server to drive the server to respond to the data model adding instruction and deploy the data model and the model data table named with the standard table name to the hierarchical data warehouse.
[0126] In this embodiment, when the developer develops a new data model and its corresponding model data table on the development side, the development side will perform a name verification on the table name of the model data table so that when the server subsequently performs operation testing, graph drawing and hierarchical verification optimization on the model data table and data model, it can be implemented with the help of the standard naming of the model data table.
[0127] In one embodiment, please refer to Figure 8 The step of testing and running each of the computing tasks to determine the computing resources consumed by each of the computing tasks, and then generating resource consumption data representing the computing resources consumed by each computing task, includes the following steps performed by the server:
[0128] Step S121, running the computing task, and determining the memory index and processor index occupied by the computing task when the computing task is completed:
[0129] After the server obtains one or more computing tasks included in the data model, it will test and run these computing tasks respectively to determine the memory index (MBS) and processor index (CPUS) occupied by these computing tasks during operation.
[0130] Step S122: Using the memory index and the processor index as computing resources of the computing task to generate mapping relationship data including the computing resources and the task name of the computing task:
[0131] After the server determines the computing resources of the memory index and the processor index occupied by each computing task during execution, it will respectively generate mapping relationship data including the corresponding computing resources and task names for these computing tasks.
[0132] Step S123, and so on, generate mapping relationship data of all computing tasks, and then generate the resource consumption data including the mapping relationship data:
[0133] After the server obtains the mapping relationship data corresponding to all computing tasks, it will generate the resource consumption data including the mapping relationship data.
[0134] In this embodiment, the server tests and runs each computing task included in the data model to determine the computing resources occupied by these computing tasks during operation, so as to push them to the development end for reference by developers to perform code optimization and other processing on each computing task.
[0135] In one embodiment, please refer to Figures 4 to 6 and Fig. 9, the step of tracing the upstream and downstream sources of each reference data table based on the data level of the model data table corresponding to the data model in the hierarchical data warehouse, determining the upstream and downstream data tables associated with each reference data table in the hierarchical data warehouse, and drawing a data table mapping relationship map with the model data table as the central node includes the following steps executed by the server:
[0136] Step S131, obtaining the data level contained in the standard table name of the model data table as the first data level, and obtaining the data level contained in the standard table name of the current referenced data table as the second data level:
[0137] After the server obtains all reference data tables referenced by the data model, it draws the data table mapping relationship map representing the upstream and downstream relationships between the reference data tables and the model data tables corresponding to the data model in the hierarchical database.
[0138] The server obtains the data level contained in the standard table name of the model data table as the first data level, and obtains the data level contained in the standard table name of the current reference data table for which the path currently needs to be drawn as the second data level.
[0139] Step S132: when the first data level exceeds the second data level, one or more data tables associated with the current referenced data table between the first and second data levels are determined as upstream data tables of the current referenced data table, and the current referenced data table is used as the terminal node of its upstream path. The terminal node is used as the drawing starting point, and the upstream data tables are drawn as nodes in the upstream path in order from low to high according to the data levels corresponding to each of the upstream data tables:
[0140] Please refer to Figure 4 When the model data table is the daily audience user data table of the live broadcast room in the singing area, and the corresponding reference data table is the audience user data table, the model data table can generally be in the wide table summary layer (DWS), and the reference data table is generally in the detailed data layer (DWV). The first data level of the model data table is greater than the second data level of the reference data table. Correspondingly, when the reference data table has an associated upstream data table of the daily audience user data table in the light summary data layer (DWM), the upstream path drawn for the reference time table is as follows: Figure 4 As shown, the reference data table is located at Figure 4 The upstream data table is the node 403 in the upstream path 401.
[0141] Step S133: when the second data level exceeds the first data level, one or more data tables associated with the current referenced data table between the first and second data levels are determined as downstream data tables of the current referenced data table, and the current data table is used as the starting node of its downstream path. The starting node is used as the drawing starting point, and the downstream data tables are drawn as nodes in the downstream path in order from high to low according to the data levels corresponding to each of the downstream data tables:
[0142] Please refer to Figure 5 When the model data table is the daily audience user data table of the live broadcast room in the singing area, and the corresponding reference data table is the live broadcast room conversion rate data table, the model data table can generally be in the wide table summary layer (DWS), and the reference data table is generally in the data application layer (APP). The second data level where the reference data table is located is greater than the first data level where the model data table is located. Correspondingly, when the reference data table has an associated downstream data table of the live broadcast room virtual gift giving data table in the light summary data layer (DM), the downstream path drawn for the reference time table is as follows: Figure 5 As shown, the reference data table is located at Figure 5 The starting node 502 in the downstream path 501 is the downstream data table, and the downstream data table is the node 503 in the downstream path.
[0143] Step S134, and so on, after drawing the upstream path or downstream path of each reference data table respectively, connect each upstream path and downstream path with the model data table as the central node of the path, and draw the data table mapping relationship map:
[0144] Please refer to Figure 6 , Figure 6 For the general Figure 4 The upstream path shown and Figure 5 The drawn downstream path is a data table mapping relationship diagram drawn with the model data table as the center, Figure 6 The central node 601 in is the model data table (daily audience user data table of the live broadcast room in the singing area).
[0145] In this embodiment, a data table mapping relationship map is drawn to characterize the upstream and downstream relationships between the model data table and each referenced data table in the hierarchical data warehouse and is subsequently pushed to the development end, so that developers at the development end can determine the corresponding replacement data tables for the data tables in the data tables referenced by the data model they created that are wrong in the wrong layer or across layers based on the data table mapping relationship map.
[0146] In one embodiment, please refer to Fig.10The step of performing hierarchical verification on the reference data tables according to the standard table names of the model data table and each of the reference data tables comprises the following steps executed by the server:
[0147] Step S141, obtaining the data level contained in the standard table name of the model data table as the first data level, and obtaining the data level contained in the standard table name of the current referenced data table as the second data level:
[0148] When the server performs hierarchy verification on the reference data table associated with the model data table, the data hierarchy contained in the standard table name of the model data table is used as the first data hierarchy, and the data hierarchy contained in the standard table name of the current reference data table being verified is used as the second data hierarchy.
[0149] Step S142: When the first data level is greater than the second data level, and the level interval between the two exceeds one, the reference to the current referenced data table is characterized as a cross-layer reference, and a verification result representing the cross-layer reference is generated for the current referenced data table:
[0150] When the server determines that the first data level is greater than the second data level, and the level interval between the first data level and the second data level is greater than one, it represents that the model data table cross-layer references the current reference data table, and the server will be the verification result of the cross-layer reference represented by the current reference data table.
[0151] Step S143: When the first data level is smaller than the second data level, it indicates that the reference to the current referenced data table is a staggered reference, and a verification result indicating the staggered reference is generated for the current referenced data table:
[0152] When the server determines that the first data level is smaller than the second data level, it indicates that the model data table is staggered or reversely references the current reference data table, and the verification result generated by the server for the current reference data table will be a staggered reference.
[0153] In this embodiment, the server performs hierarchical verification on the reference data tables referenced by the model data table to determine whether there are reference data tables with cross-layer references or staggered-layer references in these reference data tables, so as to match the model data table with a recommended data table that is better than the reference data table with cross-layer or staggered-layer references for the developer to perform optimization processing.
[0154] In one embodiment, please refer to Fig.11 , the step of matching the data model to obtain a recommended reference data table that is better than the reference data table includes the following steps executed by the server:
[0155] Step S141', taking the data level contained in the standard table name of the model data table as the first data level and the data update frequency as the target data update frequency, and taking the data subject contained in the standard table name of the cross-layer or staggered reference data table as the target data subject and the data level as the second data level:
[0156] When the server determines through hierarchical verification that any referenced data table referenced by the model data table has a cross-layer reference or a staggered-layer reference, it will match the model data table with a recommended data table that is better than the referenced data table. First, the server takes the data layer in which the model data table is located as the first data layer, and the data update frequency of the model data table as the target data update frequency, and takes the data layer in which the referenced data table is located as the second data layer, and the data subject of the referenced data table as the target data subject.
[0157] Step S142', determining one or more target data levels in the hierarchical data warehouse that are in a level interval that is equal to the first data level at most and higher than the second data level at least, to query one or more first data tables in the target data level whose standard table names have the target data subject:
[0158] The server determines data levels in the hierarchical data warehouse that do not exceed the first data level, and determines target data levels in these data levels that are higher than the second data level, so as to query one or more first data tables having the target data subject in the standard table name in these target data levels.
[0159] If there is only one first data table, the first data table will be determined by the server as the recommended data table.
[0160] Step S143', query whether there is a reference data table in the first data table whose standard table name has the target data update frequency and whose data level is closest to the first data level. If so, further query the reference data table to be used as the recommended reference data table. If not, use the reference data table in the first data table whose data level is closest to the first data level as the recommended reference data table:
[0161] After the server queries out multiple first data tables, it queries out data tables with standard table names having the target data update frequency among these first data, and determines among these data tables the data table whose data level is closest to the first data level as the recommended data table, or first determines among each of the first data tables the data table whose data level is closest to the first data level, and determines among these data tables the data table whose standard table name has the target data update frequency as the recommended data table.
[0162] If there is no data table with the target data update frequency in the standard data table name in each of the first data tables, the server will determine a data table whose data level is closest to the first data level among the first data tables as the recommended data table.
[0163] In this embodiment, the server matches the data tables referenced across layers or staggered layers of the model data table with recommended data tables that are better than those data tables, and pushes these recommended data tables to the development end so that the developers can modify the data tables called in the corresponding computing tasks to the recommended data tables for optimization.
[0164] In one embodiment, please refer to Fig.12 After the step of pushing the recommended data table, the resource consumption data and the data table mapping relationship map to the development end, the following steps are performed by the server:
[0165] Step S15, in response to the periodic layer scanning instruction acting on the layered data warehouse, performing layer verification on the data tables of all data layers of the layered data warehouse:
[0166] By setting a periodic layer scanning task for the server, the server is driven to regularly perform layer verification on the data tables of each data level in the layered data warehouse. Specifically, when the time configured for the scheduled layer scanning task is reached, the server will respond to the periodic layer scanning instruction acting on the layered data warehouse, and perform layer verification on the data tables of each data level in the layered data warehouse.
[0167] For the said level verification, please refer to the relevant embodiments in the foregoing text, such as the relevant implementation methods in step S14 and steps S141 to S143, and this step will not be described in detail.
[0168] Step S16, determining one or more target data models with a hierarchical verification result of having cross-layer references or staggered layer references, matching the target data model with a recommended reference data table that is better than the reference data table corresponding to the hierarchical verification result and pushing it, so that the development end corresponding to the target data model performs model optimization:
[0169] When the server performs the periodic layer scanning task on the layered data warehouse, if there is a data table referenced across layers or staggered layers by any data model, the data model is determined by the server as the target data model, so that the server matches a recommended data table for the target data model that is better than the data table referenced across layers or staggered layers, and pushes the recommended referenced data table to the development end to which the target data model belongs, so that the developer at the development end can modify the data table referenced by the corresponding computing task to the recommended data table.
[0170] Regarding the specific implementation method of the server matching the recommended data table for the data model with cross-layer or staggered references, please refer to the relevant implementation methods in the previous text, such as the relevant embodiments of step S14 and steps S141' to step S143', and this step will not be repeated here.
[0171] In this embodiment, the server will periodically scan and verify the hierarchical data warehouse to verify the cross-layer or staggered referenced data tables for each data table in the hierarchical data warehouse, and match the corresponding recommended data tables for the data models associated with these data tables.
[0172] Furthermore, by functionalizing each step in the method disclosed in the above embodiments, a hierarchical data verification device of the present application can be constructed. Fig.13 In a typical embodiment, the device includes: a computing task acquisition module 11, which is used to respond to a data model addition instruction pushed by a development end and acting on a hierarchical data warehouse, obtain multiple computing tasks contained in the data model corresponding to the instruction, and obtain the standard table name of the model data table corresponding to the data model, wherein the standard table name includes the data level, data update frequency and data subject; a computing task test module 12, which is used to test and run each of the computing tasks, determine the computing resources consumed by each of the computing tasks, and then generate resource consumption data representing the computing resources consumed by each of the computing tasks, and determine the reference data table corresponding to each of the computing tasks in the hierarchical data warehouse; relationship map drawing Module 13 is used to perform upstream and downstream tracing of each of the referenced data tables based on the data hierarchy in the standard table name of the model data table, determine the upstream and downstream data tables associated with each of the referenced data tables in the hierarchical data warehouse, and draw a data table mapping relationship map with the model data table as the central node; the hierarchy verification processing module 14 is used to perform hierarchy verification on the referenced data tables according to the standard table names of the model data table and each of the referenced data tables. If the verification result of a referenced data table indicates the existence of cross-layer references or staggered layer references, a recommended data table that is better than the referenced data table is matched for the data model, and the recommended data table, the resource consumption data and the data table mapping relationship map are pushed to the development end.
[0173] In one embodiment, the computing task testing module 12 includes: an occupancy index determination submodule, used to run the computing task and determine the memory index and processor index occupied by the computing task when the computing task completes the operation; a relationship data generation submodule, used to use the memory index and processor index as the computing resources of the computing task to generate mapping relationship data containing the computing resources and the task name of the computing task; a resource consumption data generation submodule, used to generate mapping relationship data for all computing tasks by analogy, and then generate the resource consumption data containing the mapping relationship data.
[0174] In one embodiment, the relationship map drawing module 13 includes: a data level acquisition submodule, which is used to acquire the data level contained in the standard table name of the model data table as the first data level, and acquire the data level contained in the standard table name of the current reference data table as the second data level; an upstream path drawing submodule, which is used to determine, when the first data level exceeds the second data level, one or more data tables associated with the current reference data table between the first and second data levels as the upstream data tables of the current reference data table, and then use the current reference data table as the end node of its upstream path, take the end node as the drawing starting point, and draw the upstream data tables in the upstream path in order from low to high according to the data levels corresponding to each of the upstream data tables. a node; a downstream path drawing submodule, which is used to determine, when the second data level exceeds the first data level, one or more data tables associated with the current reference data table between the first and second data levels as downstream data tables of the current reference data table, and then use the current data table as the starting node of its downstream path, and use the starting node as the drawing starting point, and draw the downstream data tables as nodes in the downstream path from high to low according to the data levels corresponding to each of the downstream data tables; a mapping relationship graph drawing submodule, which is used to, by analogy, draw the upstream path or downstream path of each of the reference data tables respectively, and then connect each of the upstream paths and downstream paths with the model data table as the central node of the path to draw the data table mapping relationship graph.
[0175] In one embodiment, the hierarchical verification processing module 14 includes: a data hierarchy acquisition submodule, used to acquire the data hierarchy contained in the standard table name of the model data table as the first data hierarchy, and acquire the data hierarchy contained in the standard table name of the current referenced data table as the second data hierarchy; a cross-layer reference verification submodule, used to characterize the reference to the current referenced data table as a cross-layer reference when the first data hierarchy is greater than the second data hierarchy and the hierarchy interval between the two parties exceeds one, and to generate a verification result characterizing the cross-layer reference for the current referenced data table; a staggered-layer reference verification submodule, used to characterize the reference to the current referenced data table as a staggered-layer reference when the first data hierarchy is less than the second data hierarchy, and to generate a verification result characterizing the staggered-layer reference for the current referenced data table.
[0176] In another embodiment, the hierarchical verification processing module 14 also includes: a standard table name parsing submodule, which is used to use the data hierarchy contained in the standard table name of the model data table as the first data hierarchy and the data update frequency as the target data update frequency, and use the data subject contained in the standard table name of the cross-layer or staggered reference data table as the target data subject and the data hierarchy as the second data hierarchy; a target data table query submodule, which is used to determine one or more target data hierarchies in the hierarchical data warehouse that are in a hierarchy interval that is at most equal to the first data hierarchy and at least higher than the second data hierarchy, so as to query one or more first data tables in the target data hierarchy whose standard table name has the target data subject; a recommended data table query submodule, which is used to query whether there is a reference data table in the first data table whose standard table name has the target data update frequency and whose data hierarchy is closest to the first data hierarchy. If so, further query the reference data table to be used as the recommended reference data table. If not, use the reference data table in the first data table whose data hierarchy is closest to the first data hierarchy as the recommended reference data table.
[0177] In order to solve the above technical problems, the embodiment of the present application also provides a computer device for running a computer program implemented according to the layered data verification method. Fig.14 , Fig.14 This is a basic structural block diagram of the computer device in this embodiment.
[0178] like Fig.14As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a hierarchical data verification method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a hierarchical data verification method. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0179] In this embodiment, the processor is used to execute the specific functions of each module / submodule in the hierarchical data verification device of the present application, and the memory stores the program code and various data required to execute the above modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the hierarchical data verification device, and the server can call the server's program code and data to execute the functions of all submodules.
[0180] The present application also provides a non-volatile storage medium, wherein the hierarchical data verification method is written into a computer program and stored in the storage medium in the form of computer-readable instructions. When the computer-readable instructions are executed by one or more processors, it means that the program is run in the computer, thereby causing one or more processors to execute the steps of the hierarchical data verification method of any of the above-mentioned embodiments.
[0181] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0182] In summary, the present application effectively manages and upgrades the quality of data tables in a hierarchical data warehouse and improves the computational efficiency of data models in the warehouse.
[0183] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0184] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.
[0185] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A hierarchical data verification method, characterized in that: The steps include: In response to a data model adding instruction pushed by a development end and acting on a hierarchical data warehouse, a plurality of computing tasks included in the data model corresponding to the instruction are obtained, and a standard table name of a model data table corresponding to the data model is obtained, wherein the standard table name includes a data level, a data update frequency, and a data subject; Performing a test run on each of the computing tasks to determine the computing resources consumed by each of the computing tasks, thereby generating resource consumption data representing the computing resources consumed by each of the computing tasks, and determining the reference data table corresponding to each of the computing tasks in the hierarchical data warehouse; Based on the data hierarchy in the standard table name of the model data table, each of the reference data tables is traced upstream and downstream, the upstream and downstream data tables associated with each of the reference data tables in the hierarchical data warehouse are determined, and a data table mapping relationship diagram with the model data table as the central node is drawn; According to the standard table names of the model data table and each of the referenced data tables, the referenced data tables are respectively checked at a hierarchical level. If the check result of a referenced data table indicates the existence of cross-layer references or staggered-layer references, a recommended data table that is better than the referenced data table is matched for the data model, and the recommended data table, the resource consumption data and the data table mapping relationship map are pushed to the development end.
2. The method according to claim 1, characterized in that Before the step of responding to the data model adding instruction pushed by the development end and acting on the hierarchical data warehouse, the following steps are performed by the development end: In response to a creation instruction acting on a model data table, obtain a data table name of the model data table corresponding to the creation instruction; According to the preset naming rules, the data table name is verified to check whether the data table name contains the data level, data update frequency and data subject; When the data table name passes the naming verification, the data table name will be packaged as a standard table name and the model data table as a data model adding instruction and pushed to the server.
3. The method according to claim 1, characterized in that The step of testing and running each of the computing tasks to determine the computing resources consumed by each of the computing tasks, and then generating resource consumption data representing the computing resources consumed by each of the computing tasks, includes the following steps performed by the server: Run the computing task to determine the memory index and processor index occupied by the computing task when the computing task is completed; Using the memory index and the processor index as computing resources of the computing task to generate mapping relationship data including the computing resources and the task name of the computing task; By analogy, the mapping relationship data of all computing tasks are generated, and then the resource consumption data including the mapping relationship data is generated.
4. The method according to claim 1, characterized in that: The step of tracing the upstream and downstream sources of each reference data table based on the data level of the model data table corresponding to the data model in the hierarchical data warehouse, determining the upstream and downstream data tables associated with each reference data table in the hierarchical data warehouse, and drawing a data table mapping relationship map with the model data table as the central node includes the following steps executed by the server: Acquire the data level contained in the standard table name of the model data table as the first data level, and acquire the data level contained in the standard table name of the currently referenced data table as the second data level; When the first data level exceeds the second data level, one or more data tables associated with the current referenced data table between the first and second data levels are determined as upstream data tables of the current referenced data table, and the current referenced data table is used as the terminal node of its upstream path. The terminal node is used as the drawing starting point, and the upstream data tables are drawn as nodes in the upstream path in order from low to high according to the data levels corresponding to the upstream data tables; When the second data level exceeds the first data level, one or more data tables associated with the current referenced data table between the first and second data levels are determined as downstream data tables of the current referenced data table, and the current data table is used as the starting node of its downstream path. The starting node is used as the drawing starting point, and the downstream data tables are drawn as nodes in the downstream path in order from high to low according to the data levels corresponding to the downstream data tables; Similarly, after drawing the upstream path or downstream path of each reference data table respectively, each upstream path and downstream path is connected with the model data table as the central node of the path to draw the data table mapping relationship map.
5. The method according to claim 1, characterized in that The step of performing hierarchical verification on the reference data tables according to the standard table names of the model data table and each of the reference data tables includes the following steps performed by the server: Acquire the data level contained in the standard table name of the model data table as the first data level, and acquire the data level contained in the standard table name of the currently referenced data table as the second data level; When the first data level is greater than the second data level, and the level interval between the two exceeds one, it is characterized that the reference to the current referenced data table is a cross-layer reference, and a verification result representing the cross-layer reference will be generated for the current referenced data table; When the first data level is smaller than the second data level, it indicates that the reference to the current referenced data table is a staggered reference, and a verification result indicating the staggered reference is generated for the current referenced data table.
6. The method according to claim 1, characterized in that The step of matching the data model to find a recommended reference data table that is better than the reference data table includes the following steps performed by the server: The data level contained in the standard table name of the model data table is used as the first data level and the data update frequency is used as the target data update frequency, and the data subject contained in the standard table name of the cross-layer or staggered reference data table is used as the target data subject and the data level is used as the second data level; Determine one or more target data levels in the hierarchical data warehouse that are in a level interval that is equal to the first data level at most and higher than the second data level at least, so as to query one or more first data tables in the target data level whose standard table names have the target data subject; Query whether there is a reference data table in the first data table whose standard table name has the target data update frequency and whose data level is closest to the first data level. If so, further query the reference data table to be used as the recommended reference data table. If not, use the reference data table in the first data table whose data level is closest to the first data level as the recommended reference data table.
7. The method according to claim 1, characterized in that After the step of pushing the recommended data table, the resource consumption data and the data table mapping relationship map to the development end, the following steps are performed by the server: In response to a periodic layer scanning instruction acting on the layered data warehouse, performing a layer verification on the data tables of all data layers of the layered data warehouse; Determine one or more target data models with hierarchical verification results that have cross-layer references or staggered-layer references, match the target data models with recommended reference data tables that are better than the reference data tables corresponding to the hierarchical verification results, and push them so that the development end corresponding to the target data models can perform model optimization.
8. A hierarchical data verification device, characterized in that: include: A computing task acquisition module is used to respond to a data model adding instruction pushed by a development end and acting on a hierarchical data warehouse, obtain a plurality of computing tasks contained in a data model corresponding to the instruction, and obtain a standard table name of a model data table corresponding to the data model, wherein the standard table name includes a data level, a data update frequency, and a data subject; A computing task testing module, used to test and run each of the computing tasks, determine the computing resources consumed by each of the computing tasks, and then generate resource consumption data representing the computing resources consumed by each of the computing tasks, and determine the reference data table corresponding to each of the computing tasks in the hierarchical data warehouse; A relationship map drawing module is used to trace the upstream and downstream sources of each reference data table based on the data hierarchy in the standard table name of the model data table, determine the upstream and downstream data tables associated with each reference data table in the hierarchical data warehouse, and draw a data table mapping relationship map with the model data table as the central node; The hierarchical verification processing module is used to perform hierarchical verification on the reference data tables according to the standard table names of the model data table and each reference data table. If the verification result of a reference data table indicates the existence of cross-layer references or staggered layer references, a recommended data table that is better than the reference data table is matched for the data model, and the recommended data table, the resource consumption data and the data table mapping relationship map are pushed to the development end.
9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.
10. A non-volatile storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.
