Index management and scheduling method and system based on data lake

Abstract

The invention relates to an index management and scheduling method and system based on a data lake, and the method comprises the steps: configuring index information, classifying indexes, constructing an index tree structure, carrying out the authority management, configuring index conditions in a visual dragging manner, selecting a Hive table from a Hive database of the data lake, and carrying out the authority management of the selected Hive table. The method comprises the following steps: configuring a general index result table, mapping a field in a Hive table and a general index result table field on the basis of taking the field in the Hive table as an access condition and an access date, generating and verifying an SQL (Structured Query Language) statement, performing scheduling on the basis of configuration index information after verification is passed, performing index dependency ready judgment and continuity judgment through a scheduling engine, and submitting an index processing task to a Spark SQL for processing. And displaying in a view form. Therefore, by adopting the technical scheme, correlation analysis can be carried out on the data of each system, the data processing efficiency is improved, a larger data volume is supported, the configuration complexity is reduced, the flexibility is improved, index classification and isolation can be carried out, and the efficiency and quality of data processing and analysis are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and analysis, and in particular to an indicator management and scheduling method and system based on a data lake. Background Art

[0002] In the current data processing and analysis field, although data lakes, as an enterprise-level data architecture method, have demonstrated their advantages in centralized storage, support for diverse data types, and rapid data processing and analysis, especially in scenarios with more reads and fewer writes and large data volumes, compared to relational databases, existing technologies still face many problems in indicator management and scheduling:

[0003] 1. The existing technology lacks an intuitive and easy-to-use interface to guide business personnel to configure the source and conditions of indicator data, which makes the configuration process complicated and error-prone. At the same time, there is a lack of sufficient flexibility and convenience in the configuration of key processing information such as processing frequency and data cycle, as well as the setting of display information such as the number of retained digits and units.

[0004] 2. In terms of the organization and management of indicators, existing technologies often make it difficult to form a clear indicator tree structure, resulting in unclear correlation and hierarchy between indicators, making it difficult to effectively classify and isolate indicators, increasing the risk of duplicate indicator construction, and may also lead to inconsistencies in processing caliber, affecting the accuracy of data analysis.

[0005] 3. The existing technology cannot customize the scheduling engine, resulting in inconsistent dependency ready times in job processing, configuration indicators failing to take effect in a timely manner, and lack of periodic processing capabilities, all of which seriously restrict the efficiency and quality of data processing and analysis.

[0006] In summary, there are many problems and challenges in the existing technology in indicator management and scheduling. There is an urgent need for a more efficient, flexible and scalable indicator management and scheduling method to meet the increasingly complex and changeable data processing and analysis needs. Summary of the invention

[0007] In order to solve the deficiencies of the prior art, the present invention proposes an indicator management and scheduling method based on a data lake.

[0008] To achieve the above objectives, the technical solutions adopted by the present invention include:

[0009] A method for indicator management and scheduling based on a data lake, characterized by comprising:

[0010] Configuration indicator information, the indicator information includes basic information, execution information and result information;

[0011] Classify the indicators based on the configuration indicator information, build an indicator tree structure and perform authority management;

[0012] Configure indicator conditions, select a Hive table from the Hive database of the data lake, and use the fields in the Hive table as the data extraction conditions and data extraction date;

[0013] Map the fields in the Hive table with the fields in the general indicator result table, generate and verify the SQL statement, and after the verification passes, perform scheduling based on the configuration indicator information;

[0014] After the scheduling engine performs indicator dependency readiness judgment and continuity judgment, the scheduling engine submits the indicator processing task to Spark SQL for processing, writes the processing result into the general indicator result table, and displays it in the form of a view.

[0015] Furthermore, the indicator information includes basic information, execution information and result information, specifically including:

[0016] Configure basic information such as indicator name, description, and dimension;

[0017] Configure execution information such as storage granularity, calculation step size, calculation frequency, and first execution time;

[0018] Configure result information such as reserved bits, default values, and units.

[0019] Furthermore, the data retrieval conditions and data retrieval date based on the fields in the Hive table specifically include:

[0020] Combine the data retrieval condition fields in the Hive table by dragging and dropping to serve as the data retrieval condition;

[0021] Select the data date range based on the data retrieval date field in the Hive table.

[0022] Furthermore, we build an indicator tree structure and perform authority management, including:

[0023] Define the type of the root node and the associated indicators, select the category and set the indicator based on the indicator configuration, and manage the indicator authority according to the affiliated department.

[0024] Furthermore, the scheduling engine is used to determine the readiness of indicator dependencies and execution continuity, including:

[0025] The Hive table, the intermediate table, and the general indicator result table that the indicator depends on are judged to be ready through the scheduling engine, and the execution continuity is judged by querying the execution record.

[0026] Furthermore, after the scheduling engine submits the index processing task to Spark SQL for processing, it also includes:

[0027] Execution monitoring is performed through a status listener, and after the indicator processing task is completed, the data increment unloading task and the threshold screening task are triggered.

[0028] In addition, the present invention also relates to an indicator management and scheduling system based on a data lake, which is characterized by comprising:

[0029] An indicator information configuration module, used to configure indicator information, the indicator information includes basic information, execution information and result information;

[0030] An indicator tree construction module, used to classify indicators based on the configuration indicator information, construct an indicator tree structure and perform authority management;

[0031] An indicator condition configuration module is used to select a Hive table from the Hive database of the data lake and use the fields in the Hive table as the data retrieval conditions and data retrieval date;

[0032] A verification module is used to map the fields in the Hive table with the fields in the general indicator result table, generate and verify the SQL statement, and perform scheduling based on the configuration indicator information after the verification passes;

[0033] The processing display module is used to perform indicator dependency readiness judgment and continuity judgment through the scheduling engine, and the scheduling engine submits the indicator processing task to Spark SQL for processing, writes the processing result into the general indicator result table, and displays it in the form of a view.

[0034] In addition, the present invention also relates to an electronic device, characterized in that it comprises a processor and a memory;

[0035] The memory is used to store operation instructions;

[0036] The processor is used to execute the above method by calling the operation instruction.

[0037] In addition, the present invention also relates to a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and the computer program implements the above method when executed by a processor.

[0038] In addition, the present invention also relates to a computer program product, characterized in that the computer program product comprises a computer program, and the computer program implements the above method when executed by a processor.

[0039] Adopt the scheme provided by this application, by configuring indicator information (including basic information, execution information and result information), classify indicators based on the configuration information, build an indicator tree structure and perform permission management, and configure indicator conditions by visual drag and drop, select Hive table from the Hive database of the data lake, and map the fields in the Hive table with the fields in the general indicator result table based on the fields in the Hive table as the data acquisition conditions and the data acquisition date, generate and verify SQL statements, and after verification, schedule based on the configured indicator information, and the scheduling engine will perform indicator dependency readiness judgment and continuity judgment, and then submit the indicator processing task to Spark SQL for processing, write the processing results into the general indicator result table, and display them in the form of views. Therefore, adopting the above technical scheme, the data lake can unify the data source, and the correlation analysis of the data of each system can be performed, which improves the data processing efficiency and supports a larger amount of data; configuring indicator conditions by visual drag and drop can reduce the configuration complexity and improve flexibility; the indicator tree structure can be used to classify and isolate indicators, and duplicate construction and consistency of processing caliber can be achieved; judgment by the scheduling engine improves the efficiency and quality of data processing and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of an indicator management and scheduling method based on a data lake provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of the indicator configuration-basic information provided in the embodiment of the present application;

[0042] Figure 3 A schematic diagram of the indicator configuration-result mapping provided in the embodiment of the present application;

[0043] Figure 4 A schematic diagram of an indicator classification tree provided for an embodiment of the present application;

[0044] Figure 5 A schematic diagram of indicator configuration-indicator conditions provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the indicator display configuration provided in the embodiment of the present application;

[0046] Figure 7 A schematic diagram of the structure of an indicator management and scheduling system based on a data lake provided in an embodiment of the present application;

[0047] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present invention.

[0049] Those skilled in the art can understand that, unless otherwise stated, the singular forms "one", "an", "said" and "the" used herein may also include plural forms, and the "first", "second", etc., are only used to distinguish and define objects for the purpose of introducing the scheme clearly, and do not limit the objects themselves. Of course, the objects defined by "first" and "second" may be the same terminal, device, user, etc., or the same type of terminal, device and user. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0050] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0051] Figure 1 The flowchart of the indicator management and scheduling method based on the data lake provided by the embodiment of the present application is shown, which specifically includes:

[0052] S1. Configure indicator information, which includes basic information, execution information and result information;

[0053] In an optional embodiment, the configuration indicator information specifically includes:

[0054] Configure basic information such as indicator name, description, and dimension;

[0055] Configure execution information such as storage granularity, calculation step size, calculation frequency, and first execution time;

[0056] Configure result information such as reserved bits, default values, and units.

[0057] S2. Classify the indicators based on the configuration indicator information, build an indicator tree structure and perform authority management;

[0058] In an optional embodiment, constructing an indicator tree structure and performing authority management specifically includes:

[0059] Define the type of the root node and the associated indicators, select the category and set the indicator based on the indicator configuration, and manage the indicator authority according to the affiliated department; the indicator tree structure provides users with a form of indicator classification management, and the tree structure more intuitively shows the relationship between indicators.

[0060] S3. Configure indicator conditions, select a Hive table from the Hive database of the data lake, and use the fields in the Hive table as the data extraction conditions and data extraction date;

[0061] In an optional embodiment, the data retrieval conditions based on the fields in the Hive table and the data retrieval date specifically include:

[0062] Combine the data retrieval condition fields in the Hive table by dragging and dropping to serve as the data retrieval condition;

[0063] Select the data date range based on the data retrieval date field in the Hive table.

[0064] S4. Map the fields in the Hive table with the fields in the general indicator result table, generate and verify the SQL statement, and after verification, perform scheduling based on the configuration indicator information;

[0065] In an optional embodiment, by checking the formed sql statement, verifying the sql structure and returning the verification result, scheduling is performed based on the configuration indicator information after checking that the data list is correct.

[0066] S5. After the scheduling engine performs indicator dependency readiness judgment and continuity judgment, the scheduling engine submits the indicator processing task to Spark SQL for processing, and performs execution monitoring through the status listener. After the indicator processing task is completed, the data incremental unloading ElasticSearch and HBase jobs are triggered, and at the same time, it is determined whether the indicator has a threshold configuration. If so, the threshold screening job processing is performed, and the processing results are written into the general indicator result table, and displayed in the form of a view.

[0067] In an optional embodiment, the indicator dependency readiness judgment and execution continuity judgment are performed by the scheduling engine, specifically including:

[0068] The scheduling engine determines whether the current job has indicator dependencies on the Hive table, intermediate table, and general indicator result table that serve as indicator dependencies. If so, all dependent indicators are obtained, and then it is determined whether all dependencies are ready as of the current indicator. If so, the execution continuity is determined by querying the execution record. If the execution record is empty, the first execution time is taken, otherwise the latest successful execution date is taken.

[0069] Adopt the scheme provided by this application, by configuring indicator information (including basic information, execution information and result information), classify indicators based on the configuration information, build an indicator tree structure and perform permission management, and configure indicator conditions by visual drag and drop, select Hive table from the Hive database of the data lake, and map the fields in the Hive table with the fields in the general indicator result table based on the fields in the Hive table as the data acquisition conditions and the data acquisition date, generate and verify SQL statements, and after verification, schedule based on the configured indicator information, and the scheduling engine will perform indicator dependency readiness judgment and continuity judgment, and then submit the indicator processing task to Spark SQL for processing, write the processing results into the general indicator result table, and display them in the form of views. Therefore, adopting the above technical scheme, the data lake can unify the data source, and the correlation analysis of the data of each system can be performed, which improves the data processing efficiency and supports a larger amount of data; configuring indicator conditions by visual drag and drop can reduce the configuration complexity and improve flexibility; the indicator tree structure can be used to classify and isolate indicators, and duplicate construction and consistency of processing caliber can be achieved; judgment by the scheduling engine improves the efficiency and quality of data processing and analysis.

[0070] In order to more clearly introduce the technical solution of the present application, another specific implementation method is introduced below to help understand the solution disclosed in the present application:

[0071] Apply this solution to the operation management system, such as Figure 2 As shown in the figure, users configure the basic information of indicators through the visual configuration page, and then Figure 4 As shown in the figure, the operation indicator tree structure is formed based on the basic information, such as Figure 5 As shown, configure the indicator conditions through the visual configuration page, such as Figure 3 As shown in the figure, the number fields and counting results are mapped to the indicator result table, and the scheduling engine performs periodic data processing. The indicator results are queried through the unified query interface to provide the following Figure 6 Various display formats are shown.

[0072] For example: the number of transfers by a customer in the past 30 days (hereinafter referred to as "Indicator A") is displayed in a list;

[0073] Basic information: The dimension is customer number, the calculation step is 30 days, the calculation is performed every day, and a reminder is issued when the number of transactions exceeds 1,000;

[0074] Indicator configuration: Get data from the customer transaction flow table for the 30 days before the current date, and use the customer number and transfer transaction count as the numerical fields;

[0075] Mapping elements: Map the number fields and count results to the indicator result table, with the customer number as the dimension and the transfer transaction count as the result;

[0076] Indicator verification: After the above steps, the indicator elements are configured. During verification, real data is used to process the indicator, and the results are returned asynchronously and displayed in a list. After the user confirms that the data is correct, the indicator can be published and calculated according to the established frequency;

[0077] Processing scheduling: After the transaction flow record table is ready, the scheduling engine determines that all dependencies of indicator A are ready, and the previous day's execution is successful, then the processing of the day begins, the data is written to the hive table, and HBase and elasticsearch are uninstalled simultaneously; if there is an abnormal range "greater than 1000" configuration, the results greater than this value are written to the abnormal table, and a message reminder is sent to the indicator user.

[0078] Indicator display: The number of customer transfers is displayed in a list, and abnormal results are highlighted.

[0079] based on Figure 1 The present invention relates to an indicator management and scheduling method based on a data lake, and also relates to an indicator management and scheduling system based on a data lake, the structure of which is as follows: Figure 7 As shown, including:

[0080] The indicator information configuration module 701 is used to configure the indicator information, which includes basic information, execution information and result information;

[0081] An indicator tree construction module 702 is used to classify indicators based on the configuration indicator information, construct an indicator tree structure and perform authority management;

[0082] The indicator condition configuration module 703 is used to select a Hive table from the Hive database of the data lake, and use the fields in the Hive table as the data extraction conditions and data extraction date;

[0083] Verification module 704, used to map the fields in the Hive table with the fields in the general indicator result table, generate and verify the SQL statement, and perform scheduling based on the configuration indicator information after the verification passes;

[0084] The processing display module 705 is used to perform indicator dependency readiness judgment and continuity judgment through the scheduling engine, and the scheduling engine submits the indicator processing task to Spark SQL for processing, writes the processing result into the general indicator result table, and displays it in the form of a view.

[0085] Furthermore, the indicator information configuration module 701 is also used for:

[0086] Configure basic information such as indicator name, description, and dimension;

[0087] Configure execution information such as storage granularity, calculation step size, calculation frequency, and first execution time;

[0088] Configure result information such as reserved bits, default values, and units.

[0089] Furthermore, the indicator condition configuration module 703 is also used for:

[0090] Combine the data retrieval condition fields in the Hive table by dragging and dropping to serve as the data retrieval condition;

[0091] Select the data date range based on the data retrieval date field in the Hive table.

[0092] Furthermore, the indicator tree construction module 702 is also used for:

[0093] Define the type of the root node and the associated indicators, select the category and set the indicator based on the indicator configuration, and manage the indicator authority according to the affiliated department.

[0094] Furthermore, the processing display module 705 is also used for:

[0095] The Hive table, intermediate table and general indicator result table that the indicator depends on are judged to be ready by the scheduling engine, and the execution continuity is judged by querying the execution record;

[0096] Execution monitoring is performed through a status listener, and after the indicator processing task is completed, the data increment unloading task and the threshold screening task are triggered.

[0097] An embodiment of the present invention further provides an electronic device for executing the above method, as an implementation device of the method, comprising a processor and a memory;

[0098] A memory, used for storing operation instructions;

[0099] The processor is used to execute the data lake-based indicator management and scheduling method provided in any implementation of the present application by calling an operation instruction.

[0100] As an example, Figure 8A schematic diagram of the structure of an electronic device applicable to an embodiment of the present application is shown, and the electronic device 800 includes: a processor 801 and a memory 803. The processor 801 and the memory 803 are connected, such as being connected through a bus 802. Optionally, the electronic device 800 may also include a transceiver 804. It should be noted that in actual applications, the transceiver 804 is not limited to one. It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the specific structure of the electronic device 800. In other embodiments of the present application, the electronic device 800 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. Optionally, the electronic device may also include a display screen 805 for displaying images, or receiving user operation instructions when necessary.

[0101] The processor 801 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 804 may include a receiver and a transmitter. The transceiver 804 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0102] The processor 801 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 801 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0103] The processor 801 may also include one or more processing units, for example, the processor 801 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors. Among them, the controller may be the nerve center and command center of the electronic device 800. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. A memory may also be provided in the processor 801 for storing instructions and data. In some embodiments, the memory in the processor 801 is a high-speed cache memory. The memory may store instructions or data that the processor 801 has just used or circulated. If the processor 801 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor 801 is reduced, and the efficiency of the system is improved.

[0104] The processor 801 can run the indicator management and scheduling method based on the data lake provided in the embodiment of the present application. The processor 801 may include different devices. For example, when the CPU and GPU are integrated, the CPU and the GPU can cooperate to execute the indicator management and scheduling method based on the data lake provided in the embodiment of the present application. Part of the algorithm is executed by the CPU, and the other part of the algorithm is executed by the GPU to obtain faster processing efficiency.

[0105] The bus 802 may include a path to transmit information between the above components. The bus 802 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 802 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0106] The memory 803 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or may include a high-speed random access memory, or may include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0107] Optionally, the memory 803 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 801. The processor 801 is used to execute the application code stored in the memory 803 to implement the data lake-based indicator management and scheduling method provided in any embodiment of the present application.

[0108] The memory 803 can be used to store computer executable program codes, which include instructions. The processor 801 executes various functional applications and data processing of the electronic device 800 by running the instructions stored in the memory 803. The memory 803 may include a program storage area and a data storage area. Among them, the program storage area may store operating systems, application codes, etc. The data storage area may store data created during the use of the electronic device 800 (such as images, videos, etc. collected by camera applications), etc.

[0109] The memory 803 may also store one or more computer programs corresponding to the data lake-based indicator management and scheduling method provided in the embodiment of the present application. The one or more computer programs are stored in the above-mentioned memory 803 and are configured to be executed by the one or more processors 801. The one or more computer programs include instructions, and the above-mentioned instructions can be used to execute each step in the above-mentioned corresponding embodiment.

[0110] Of course, the code of the indicator management and scheduling method based on the data lake provided in the embodiment of the present application can also be stored in an external memory.

[0111] The display screen 805 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 800 may include 1 or N display screens 805, where N is a positive integer greater than 1. The display screen 805 can be used to display information input by a user or information provided to a user and various graphical user interfaces (GUIs). For example, the display screen 805 can display photos, videos, web pages, or files.

[0112] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, which will not be repeated here.

[0113] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the method in the above embodiment, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps in the method in the above embodiment are implemented.

[0114] An embodiment of the present invention also provides a computer program product capable of implementing all the steps of the method in the above embodiment. The computer program product stores a computer program, and when the computer program is executed by a processor, it implements all the steps of the method in the above embodiment.

[0115] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0117] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a box or multiple boxes. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0119] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for index management and scheduling based on a data lake, characterized in that: include: Configuration indicator information, the indicator information includes basic information, execution information and result information; Classify the indicators based on the configuration indicator information, build an indicator tree structure and perform authority management; Configure indicator conditions, select a Hive table from the Hive database of the data lake, and use the fields in the Hive table as the data extraction conditions and data extraction date; Map the fields in the Hive table with the fields in the general indicator result table, generate and verify the SQL statement, and after the verification passes, perform scheduling based on the configuration indicator information; After the scheduling engine performs indicator dependency readiness judgment and continuity judgment, the scheduling engine submits the indicator processing task to Spark SQL for processing, writes the processing result into the general indicator result table, and displays it in the form of a view.

2. The method according to claim 1, characterized in that Configuration indicator information, which includes basic information, execution information and result information, specifically including: Configure basic information such as indicator name, description, and dimension; Configure execution information such as storage granularity, calculation step size, calculation frequency, and first execution time; Configure result information such as reserved bits, default values, and units.

3. The method according to claim 1, characterized in that The data retrieval conditions based on the fields in the Hive table and the data retrieval date specifically include: Combine the data retrieval condition fields in the Hive table by dragging and dropping to serve as the data retrieval condition; Select the data date range based on the data retrieval date field in the Hive table.

4. The method according to claim 1, characterized in that Construct an indicator tree structure and perform permission management, including: Define the type of the root node and the associated indicators, select the category and set the indicator based on the indicator configuration, and manage the indicator authority according to the affiliated department.

5. The method according to claim 1, characterized in that The scheduling engine is used to determine the readiness of indicator dependencies and execution continuity, including: The Hive table, the intermediate table, and the general indicator result table that the indicator depends on are judged to be ready through the scheduling engine, and the execution continuity is judged by querying the execution record.

6. The method according to claim 1, characterized in that After the scheduling engine submits the index processing task to SparkSQL for processing, it also includes: Execution monitoring is performed through a status listener, and after the indicator processing task is completed, the data increment unloading task and the threshold screening task are triggered.

7. An indicator management and scheduling system based on a data lake, characterized in that: include: An indicator information configuration module, used to configure indicator information, the indicator information includes basic information, execution information and result information; An indicator tree construction module, used to classify indicators based on the configuration indicator information, construct an indicator tree structure and perform authority management; An indicator condition configuration module is used to select a Hive table from the Hive database of the data lake and use the fields in the Hive table as the data retrieval conditions and data retrieval date; A verification module is used to map the fields in the Hive table with the fields in the general indicator result table, generate and verify the SQL statement, and perform scheduling based on the configuration indicator information after the verification passes; The processing display module is used to perform indicator dependency readiness judgment and continuity judgment through the scheduling engine, and the scheduling engine submits the indicator processing task to Spark SQL for processing, writes the processing result into the general indicator result table, and displays it in the form of a view.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store operation instructions; The processor is used to execute the method according to any one of claims 1 to 6 by calling the operation instruction.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images