Whole-process data management method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a full-process data management method and device, equipment and a storage medium. The method comprises the following steps: collecting concurrent data of each data source through a preset interface, and respectively storing the concurrent data in a message flow form; performing data processing on the stored concurrent data based on a preset data processing task to obtain target data; performing hierarchical processing on the target data, and performing real-time storage on the target data after the hierarchical processing; and performing visual configuration on a preset data model table, and generating and executing a structured query language code based on the data model table after the visual configuration so as to call and display the target data which are stored in real time after layering. Visibly, according to the method, low coding is achieved, the structured query language code can be efficiently generated through the data model table subjected to visualized configuration, and therefore the structured query language code can be adopted to effectively and conveniently call the layered target data.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular to a method, apparatus, device and storage medium for full-process data management. Background Technology

[0002] Current data processing and analysis systems have made significant progress in the field of big data. Typical data management includes multiple stages such as data acquisition, data storage, data processing, and data output. Traditional data processing systems typically employ batch processing, which, while capable of handling large-scale data, has limitations in terms of real-time performance and flexibility. As business needs continue to evolve, enterprises require more efficient and real-time data processing and analysis capabilities to support rapid business decision-making.

[0003] In the current data management approach, the generation of structured query language instructions has not achieved sufficient low-code implementation. Most data calls still require writing code, and the data layering and related tabular data are very complex. This results in long code development cycles and low report generation efficiency, making it difficult to manage data throughout the entire process and meet the needs of rapidly changing business requirements. Summary of the Invention

[0004] This application provides a full-process data management method, apparatus, device, and storage medium, which solves the problems in related technologies where insufficient low-code implementation leads to low report generation rates and difficulty in data management due to manual coding. In scenarios with large amounts of concurrent data, based on the processed target data, the layered target data is made easy to call. In particular, after visually configuring the data model table, not only is low-code implementation achieved through configuring the model data table, but structured query language code can also be efficiently generated through the visually configured data model table. Thus, the layered target data can be effectively and conveniently called using structured query language code.

[0005] In a first aspect, embodiments of this application provide a full-process data management method, the method comprising: Concurrent data from various data sources is collected through a preset interface and stored in the form of message streams. Based on preset data processing tasks, the stored concurrent data is processed to obtain the target data; The target data is processed into layers, and the layered target data is stored in real time. The system performs visual configuration of the preset data model table, generates and executes structured query language code based on the visually configured data model table, and then displays the target data stored in real time after hierarchical processing.

[0006] Data processing tasks include data cleaning tasks and data transformation tasks. Data processing includes data cleaning operations and data transformation operations. Furthermore, based on preset data processing tasks, the stored concurrent data is processed to obtain target data, including: Configure corresponding data cleaning and data transformation tasks for concurrent data; The concurrent data is cleaned using a data cleaning task, and the cleaned concurrent data is then transformed using a data transformation task to obtain the target data.

[0007] Furthermore, after obtaining the target data, the method also includes: The target data is persistently stored in the pre-defined first data warehouse; The target data in the first data warehouse is synchronized to the preset second data warehouse in real time.

[0008] Furthermore, the target data is processed in layers, including: The system retrieves the log files from the second data warehouse and uses these files to determine the target data to be synchronized to the second data warehouse. Based on the preset business themes and stream processing framework, the identified target data is processed in layers.

[0009] Furthermore, based on preset business themes and stream processing frameworks, the identified target data is processed in layers, including: The determined target data is transformed into a preset data detail layer using a stream processing framework; Based on the business theme, the target data in the data detail layer is transformed to the preset data aggregation layer using a stream processing framework.

[0010] The data model table includes dimension tables, indicator tables, and / or fact tables, which are used to represent different categories of target data. Furthermore, the preset data model table is visually configured, and structured query language code is generated based on the visually configured data model table, including: Visual configuration is available for the table structure, field relationships, and calculation rules of the dimension table, indicator table, and / or fact table, respectively. Structured Query Language (SCL) code is generated based on the dimension tables, indicator tables, and / or fact tables configured visually.

[0011] Furthermore, the target data, stored in real time after being layered, is invoked for display, including: Based on any one or more of the dimension tables, indicator tables, and / or fact tables pointed to during the execution of the structured query language code, the corresponding hierarchical target data stored in real time is called for display.

[0012] Secondly, embodiments of this application also provide a full-process data management device, which includes: a data access module, a data processing module, a data layering module, a table configuration module, and a code processing module; The data access module is used to collect concurrent data from various data sources through preset interfaces and store them in the form of message streams. The data processing module is used to process the stored concurrent data based on preset data processing tasks to obtain the target data; The data layering module is used to process target data into layers. The data layering module is also used for real-time storage of the target data after layering processing; The table configuration module is used for visual configuration of preset data model tables; The code processing module is used to generate and execute structured query language code based on the data model table after visual configuration, so as to call the target data stored in real time after layering for display.

[0013] Thirdly, embodiments of this application also provide an electronic device, the device comprising: One or more processors; Storage device, configured to store one or more programs, When one or more programs are executed by one or more processors, the one or more processors implement the full-process data management method of the embodiments of this application.

[0014] Fourthly, embodiments of this application also provide a non-volatile storage medium for storing computer-executable instructions, which are configured to execute the full-process data management method of embodiments of this application when executed by a computer processor.

[0015] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the full-process data management method of embodiments of this application.

[0016] As can be seen from the above, the embodiments of this application provide a full-process data management method, apparatus, device and storage medium. Based on a large amount of concurrent data collected from multiple different data sources, the concurrent data is stored in the form of message streams, so that the stored concurrent data can form a message queue, which can more effectively and reliably realize the storage and transmission of concurrent data, and thus make it easier to perform data processing tasks on concurrent data.

[0017] After processing concurrent data, the target data is layered, transforming it from logically chaotic raw data into data that conforms to the relevant business logic layers, making it easier to call.

[0018] On the other hand, after configuring the data model table according to the business logic, it is possible to generate SQL code based on the data model table. This not only achieves low coding and avoids generating SQL code through a lot of coding work, but also effectively reduces the code development cycle and saves a lot of human resources.

[0019] Furthermore, when configuring the data model tables, the visual configuration not only achieves low coding but also reduces the difficulty of generating SQL code, thus enabling the use of structured query language code to effectively and conveniently call the hierarchical target data. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a full-process data management system provided in this application embodiment; Figure 2 This application provides a schematic diagram of the data flow of a data management system according to an embodiment of the present application. Figure 3 A flowchart illustrating a data processing procedure provided in this application embodiment; Figure 4 A flowchart illustrating the storage of target data provided in this application embodiment; Figure 5 A flowchart illustrating target data layering is provided for an embodiment of this application; Figure 6 A flowchart for generating and executing SQL code is provided as an embodiment of this application; Figure 7 A structural block diagram of a full-process data management device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the scope of the embodiments. Furthermore, it should be noted that, for ease of description, only the parts relevant to the embodiments of this application are shown in the accompanying drawings, not the entire structure.

[0023] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. Words such as "comprising" or "including" mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect. "Above," "below," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0024] As described in the background section, existing end-to-end data management methods are still insufficient to meet the needs of actual data processing.

[0025] In implementing this application, the applicant discovered that the main problem with the relevant end-to-end data management method is that it does not achieve sufficient low-code complexity when generating instructions for the structured query language.

[0026] In the process of managing data throughout the entire process, some data access requirements, such as when an application needs to access stored data or perform other operations on stored data, still need to be implemented by writing code. This requires professional personnel to write the relevant code.

[0027] However, the related tabular data is very complex, which leads to long code development cycles and low report generation efficiency, making it difficult to manage data throughout the entire process and meet the needs of rapidly changing business.

[0028] Based on this, one or more embodiments of this application provide a full-process data management method. Based on the constructed full-process data management device, in high-concurrency data scenarios, the target data after data processing is layered, making the layered target data easy to call. In particular, after visually configuring the data model table, not only is low-code achieved by configuring the model data table, but also structured query language code can be efficiently generated through the visually configured data model table, so that the layered target data can be effectively and conveniently called using structured query language code.

[0029] The full-process data management method provided in this application embodiment can be executed by a pre-built data management system. This system has one or more computer devices and executes the method through these computer devices. The computer device refers to any electronic device with data computing, processing and storage capabilities, such as mobile phones, PCs (Personal Computers), tablet computers and other terminal devices. This application embodiment does not limit this.

[0030] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0031] Figure 1 This is a flowchart illustrating a full-process data management method provided in an embodiment of this application. Figure 1 As shown, it includes the following steps: Step S101: Collect concurrent data from each data source through a preset interface and store them in the form of message streams.

[0032] In one embodiment, data can be accessed from multiple different data sources based on a preset interface. This preset interface is a high-performance interface for accessing high-concurrency data. When accessing concurrent data from various data sources, this preset interface can access different data sources in the form of data streams; for example, treating the data from each data source as a single data stream. Based on this, the preset interface can store the accessed data streams as message streams, thereby forming one or more message queues to complete the writing of high-concurrency data and facilitate the consumption of that data.

[0033] Step S102: Based on a preset data processing task, process the stored concurrent data to obtain the target data.

[0034] In one embodiment, the data stored in the aforementioned step S101 may contain invalid data such as null values, duplicate records, test data, and garbled characters generated by web crawlers; erroneous data such as data with disordered format and unreasonable logic; and data with inconsistent standardization such as data with different values ​​and measurement units. Therefore, the data needs to be processed to obtain reliable, consistent, and usable data.

[0035] Based on this, one or more data processing tasks can be pre-set, and each data processing task can be scheduled and monitored. Furthermore, by scheduling and monitoring each data processing task, incoming data can be retrieved from the message queue and processed accordingly.

[0036] In one embodiment, the data obtained after data processing can be used as the target data. This target data refers to raw data that has not been processed according to the pre-defined business logic.

[0037] Step S103: Perform layered processing on the target data and store the layered target data in real time.

[0038] In one embodiment, based on the target data obtained in the aforementioned step S102, since the target data is raw data containing original messy information, the target data cannot be directly used for reports related to preset business.

[0039] Based on this, the target data can be processed in layers, dividing it into different business logic layers. Specifically, during the layered processing, each newly obtained target data is processed in real-time. Furthermore, the layered target data can be stored in real-time, thus enabling real-time retrieval of the target data.

[0040] Step S104: Visually configure the preset data model table, generate and execute structured query language code based on the visualized data model table, and display the target data stored in real time after hierarchical processing.

[0041] In one embodiment, before making a real-time call to the layered target data, SQL (Structured Query Language) code can be generated, and the call to the target data can be implemented through the SQL code.

[0042] Before generating SQL code, you can visually configure the preset data model tables according to the relevant business logic, and then generate SQL code based on the configured data model tables. The data model tables contain one or more preset table types; during configuration, you can configure corresponding definitions for the data model tables according to the business logic.

[0043] Furthermore, during the configuration of the data model tables, various business logics and table types can be displayed in a pre-defined visual interface, allowing for a visual approach to configuring the data model tables. Based on this, the required SQL code can be generated using the table types defined in the data type table; that is, the generated SQL code can incorporate the definitions of each table type from the data type table.

[0044] Furthermore, after generating the SQL code, based on the hierarchical structure of the target data in the aforementioned embodiments, and assuming that the target data can be stored in the relevant business logic hierarchy, the generated SQL code can call the target data corresponding to the definition of each table type from the hierarchical target data and display the called data to the user.

[0045] As can be seen, the full-process data management method of the above embodiments of this application, based on a large amount of concurrent data collected from multiple different data sources, stores the concurrent data in the form of message streams, so that the stored concurrent data can form a message queue, which can more effectively and reliably realize the storage and transmission of concurrent data, and thus make it easier to perform data processing tasks on concurrent data.

[0046] After processing concurrent data, the target data is layered, transforming it from logically chaotic raw data into data that conforms to the relevant business logic layers, making it easier to call.

[0047] On the other hand, after configuring the data model tables according to business logic, SQL code can be generated based on these tables. This not only achieves low-code implementation and avoids extensive coding work to generate SQL code, but also effectively reduces the code development cycle and saves significant human resources. Furthermore, the visual configuration of the data model tables reduces the difficulty of generating SQL code while achieving low-code implementation, allowing for efficient and convenient retrieval of layered target data using Structured Query Language (SQL).

[0048] In some embodiments, Figure 2 This is a data flow diagram of a data management system provided in an embodiment of this application. The following is a combination of... Figure 2The specific examples of the data management system in this application will be used to illustrate the full-process data management method of this application.

[0049] like Figure 2 As shown, the data management system includes a data acquisition layer, a data processing layer, a bottom data warehouse layer, and a data modeling layer. The data acquisition layer communicates with the data source layer outside the data management system, and the data modeling layer communicates with the data application layer outside the data management system.

[0050] In one embodiment, the concurrent data accessed by the data management system from the data source layer can come from multiple different data sources, such as... Figure 2 As shown, concurrent data in the data source layer can specifically include, for example, R&D data, platform data, and media data.

[0051] Furthermore, when the data management system collects concurrent data from various data sources, a stream processing platform in the data acquisition layer can be used to collect the concurrent data and store it in the form of a message stream. In some examples, such as... Figure 2 As shown, the stream processing platform can be, for example, the Kafka platform (Kafka Distributed Stream Processing Platform); the preset interface can be, for example, a high-performance interface pre-written in the Kafka platform. When the high-performance interface collects concurrent data, it can write the data to disk and then write it to the Kafka platform, thereby effectively solving the problem of writing high-concurrency data.

[0052] Furthermore, a task process is set up at the data processing layer to consume concurrent data from the Kafka platform. This task process is used to configure data processing tasks for the concurrent data. In some specific examples, this task process may be, for example, a resident Kafka platform consumption framework based on a web interface for managing and monitoring data processing tasks.

[0053] Based on this, after configuring data processing tasks for concurrent data, the data processing of concurrent data can be achieved by managing and monitoring these tasks, thereby obtaining the target data. Furthermore, based on the data processing of concurrent data, the obtained target data can be stored in real-time in the underlying data warehouse layer. Further, based on the target data stored in the underlying data warehouse layer in real-time, the target data stored in the underlying data warehouse layer can be monitored in real-time, and the real-time stored target data can be layered, thus allowing it to be stored according to the layering of target data, and ensuring that the layered target data conforms to the business logic hierarchy.

[0054] Based on this, visualization configuration can be performed for the data model table in the data modeling layer. Then, based on the configured data model table, the corresponding SQL code can be generated using the SQL generation engine in the data modeling layer. The generated SQL code can then be used to call and display the layered target data in the underlying data warehouse layer.

[0055] In some embodiments, Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application is shown, such as... Figure 3 As shown, it includes the following steps: Step S301: Collect concurrent data from each data source through a preset interface and store them in the form of message streams.

[0056] Step S302: Configure corresponding data cleaning and data transformation tasks for concurrent data.

[0057] In one embodiment, data processing tasks may include data cleaning tasks and data transformation tasks.

[0058] Therefore, when configuring data processing tasks for target data, you can configure data cleaning tasks and data transformation tasks corresponding to the target data respectively.

[0059] Step S303: Perform data cleaning on concurrent data based on the data cleaning task, and perform data transformation on the cleaned concurrent data based on the data transformation task to obtain the target data.

[0060] In one embodiment, based on the data cleaning task and data transformation task described above, the data processing may include the data cleaning operation corresponding to the data cleaning task and the data transformation operation corresponding to the data transformation task.

[0061] Based on the data cleaning and data transformation tasks configured for the target data in the aforementioned steps, the data cleaning task can specifically perform data cleaning operations on the target data; the data transformation task can specifically perform data transformation operations on the target data.

[0062] Step S304: Perform layered processing on the target data and store the layered target data in real time.

[0063] Step S305: Visually configure the preset data model table, generate and execute structured query language code based on the visualized data model table, and display the target data stored in real time after hierarchical processing.

[0064] exist Figure 2 In specific examples, data processing tasks may include, for example: Figure 2The ETL (Extract, Transform, Load) task shown is a data cleaning task, data transformation task, and / or data loading task. This ETL task is a real-time task, which means that when new concurrent data is written to the Kafka platform, the ETL task can be configured in real time for the concurrent data.

[0065] Therefore, after concurrent data is written to the Kafka platform, data cleaning, transformation, and / or loading tasks can be configured for the Kafka platform through this task process during data processing. Furthermore, by managing and monitoring these tasks in real time, corresponding data cleaning, transformation, and / or loading operations can be performed on the concurrent data in Kafka in real time, thus simplifying the management and monitoring of data processing tasks.

[0066] like Figure 2 As shown, in the task configuration operation, in addition to configuring ETL tasks in real time, scheduled tasks, i.e. non-real-time tasks, can also be configured, so that corresponding scheduled tasks can be uniformly configured for concurrent data accumulated within a preset time period.

[0067] In some embodiments, Figure 4 A flowchart illustrating a method for storing target data according to an embodiment of this application is shown, such as... Figure 4 As shown, it includes the following steps: Step S401: Collect concurrent data from each data source through a preset interface and store them in the form of message streams. Step S402: Based on the preset data processing task, process the stored concurrent data to obtain the target data.

[0068] Step S403: Persistently store the target data in the preset first data warehouse.

[0069] In one embodiment, based on the target data obtained after data processing, the target data can be first stored in a first data warehouse. The first data warehouse can be used for persistent storage of the target data to obtain comprehensive target data and ensure reliable and stable storage of the target data.

[0070] Step S404: Synchronize the target data in the first data warehouse to the preset second data warehouse in real time.

[0071] In one embodiment, in addition to setting up a first data warehouse, a second data warehouse can also be set up. The second data warehouse is used for real-time storage of the target data. Specifically, after the target data is stored in the first data warehouse, before performing tiered processing on the target data, the target data stored in the first data warehouse can be synchronized to the second data warehouse to facilitate real-time tiering of the target data synchronized to the second data warehouse each time.

[0072] Step S405: Perform layered processing on the target data and store the layered target data in real time.

[0073] Step S406: Visually configure the preset data model table, generate and execute structured query language code based on the visualized data model table, and display the target data stored in real time after hierarchical processing.

[0074] exist Figure 2 In specific examples, the first data warehouse could be, for example, MySQL (My Structured Query Language, a relational database); the second data warehouse could be, for example, a Hologres (real-time interactive analytics service) data warehouse. In some examples, the target data in the underlying data warehouse layer can be divided into an ODS layer (Operational Data Store), a DWD layer (Data Warehouse Detail), and a DWS layer (Data Warehouse Summary). Among these, the data can be further divided into... Figure 2 The data in MySQL is used as the ODS layer, and the data in Hologres is used as the DWD layer and DWS layer.

[0075] like Figure 2 As shown, after storing the target data in MySQL, which serves as the first data warehouse, the target data stored in MySQL can be synchronized in real time to the Hologres data warehouse, which serves as the second data warehouse. In some examples of this application, during the process of synchronizing the target data from MySQL to the Hologres data warehouse, the MySQL log file can be queried in real time through a preset data platform to determine the first storage action when storing the target data, and then the target data to be stored in MySQL can be determined in real time based on the first storage action. Specifically, the log file can be, for example, MySQL's BinLog (Binary Log).

[0076] In some embodiments, the data platform used to query the first storage action can also integrate the target data stored in MySQL for a period of time according to a preset duration, and perform offline processing on the integrated target data according to a preset data processing method, such as executing a preset offline task on the integrated target data, or synchronizing the integrated target data to the Hologres data warehouse.

[0077] In other examples of this embodiment, the target data after data processing can also be stored in an offline data warehouse or a data warehouse, so that other offline operations can be performed on the target data in the offline data warehouse or data warehouse as needed.

[0078] As can be seen, in this embodiment, based on the constructed data management system, a large amount of high-concurrency data can be accessed through the high-performance interface of the Kafka platform, and stored in the form of message streams through Kafka, thereby ensuring the reliability of concurrent data storage. After configuring the concurrent data and completing the data processing tasks, persistent storage of the target data can be achieved by storing the obtained target data in MySQL, and real-time storage of the target data can be achieved by storing the target data in MySQL to Hologres in real time. In some embodiments, Figure 5 A flowchart illustrating a target data layering method provided in an embodiment of this application is shown, such as... Figure 5 As shown, it includes the following steps: Step S501: Call the log file of the second data warehouse and determine the target data to be synchronized to the second data warehouse based on the log file.

[0079] Step S502: Use a stream processing framework to convert the determined target data to a preset data detail layer.

[0080] In one embodiment, based on the target data synchronized to the second data warehouse in real time, the target data stored in the second data warehouse in real time can be layered using a stream processing framework, and the data can be layered according to a preset business theme during the layering process, so that the layered target data can conform to the business logic hierarchy.

[0081] When layering target data, the target data can first be converted into a coarser DWD layer.

[0082] Specifically, during the layering of target data, a stream processing framework can be used to call the BinLog of the second data warehouse in real time, i.e., the log file of the second data warehouse. This allows for real-time querying of the log file of the second data warehouse to determine the second storage action of the target data in the second data warehouse in real time. Furthermore, based on this second action, the target data to be stored in the second data warehouse can be determined in real time. Further, the stream processing framework can be used to transform the target data determined in real time to a preset DWD layer.

[0083] Step S503: Based on the business theme, use a stream processing framework to transform the target data in the data detail layer to the preset data summary layer.

[0084] In one embodiment, after conversion to the DWD layer, the target data in the DWD layer can be converted to the DWS layer according to a pre-set business theme, thereby obtaining target data suitable for business logic within the DWS layer. The business theme can be different categories of data, such as data about user online time and / or data about user recharge amounts.

[0085] Based on this, after layering the target data, since the target data in the DWS layer corresponds to different business themes, the target data in the DWS layer can be applied to the current business logic. Here, business logic refers to the logical rules such as conditions, operations, and / or outputs when executing corresponding business using target data from different business themes. Furthermore, when business execution involves target data from different layers, the corresponding business logic layers are also different. That is, when executing business using target data in the DWD layer, the business logic of the corresponding DWD layer should be followed; similarly, when executing business using target data in the DWS layer, the business logic of the corresponding DWS layer should be followed.

[0086] As can be seen, in one embodiment, the target data in Hologres is consumed using a stream processing framework, thereby achieving the layering of the target data. This enables the conversion of target data with disordered categories and business logic in the ODS layer into target data in the DWD layer or DWS layer that is applicable to the current business logic and business theme.

[0087] In some embodiments, Figure 6 This application provides a flowchart illustrating the generation and execution of SQL code according to an embodiment. Figure 6 As shown, it includes the following steps: Step S601: Visually configure the table structure, field relationships, and operation rules of the dimension table, indicator table, and / or fact table respectively.

[0088] In one embodiment, based on the hierarchical structure of the target data, SQL code can be generated in the data modeling layer according to the configured data model tables, and the hierarchical target data can be called through the SQL code.

[0089] The data model table includes one or more different table types.

[0090] In some specific examples, data model tables may include, for example, dimension tables, indicator tables, and / or fact tables, which are used to define different categories of target data. Thus, target data that conforms to the definition can be retrieved from multiple different categories of target data through the definition in each table type.

[0091] In one embodiment, the data model table can be configured visually during the process of generating SQL code for configuring the data model table.

[0092] Specifically, the table structure, field relationships, and calculation rules of the dimension table, indicator table, and / or fact table can be displayed separately in the preset visualization interface. Furthermore, the visualization configuration of the dimension table, indicator table, and / or fact table can be performed separately according to the displayed table structure, field relationships, and calculation rules.

[0093] Step S602: Generate structured query language code based on the dimension table, indicator table, and / or fact table after visualization configuration.

[0094] In one embodiment, during the generation of SQL code for calling target data, methods such as... Figure 2 The SQL generation engine in the data modeling layer shown generates SQL code.

[0095] Specifically, during the execution of the SQL generation engine, corresponding SQL code can be generated based on the definitions in the configured dimension tables, metric tables, and / or fact tables.

[0096] Step S603: Based on any one or more of the dimension table, indicator table, and / or fact table pointed to during the execution of the structured query language code, call the corresponding hierarchical and real-time stored target data for display.

[0097] In one embodiment, based on the SQL code generated in the preceding steps, when the SQL code is executed, the target data corresponding to the defined dimension table, indicator table, and / or fact table can be retrieved from the DWS layer. Then, the target data can be displayed to the user in a predetermined display interface.

[0098] As can be seen, in one embodiment, when the dimension table, indicator table, and / or fact table define different categories of target numbers according to the corresponding business themes, the hierarchical target data can be effectively applied to the configured dimension table, indicator table, and / or fact table. At the same time, by comprehensively considering the visual configuration process of the dimension table, indicator table, and / or fact table, the process of generating SQL code according to the dimension table, indicator table, and / or fact table effectively achieves low coding and can easily and quickly generate SQL without relying on professional programmers, thus providing rapid support for the business.

[0099] In some embodiments of this application, the data management system can establish a communication connection with one or more preset business application coefficients in the data application layer, thereby enabling the retrieval of target data based on instructions from the business application system. Specifically, as... Figure 2 As shown, the data management system can establish communication connections with application systems A, B, C, and D.

[0100] Based on this, when the data management system receives a call instruction for target data from any one or more application systems, the data management system can generate corresponding SQL code and call the corresponding target data through the SQL code. Alternatively, it can use pre-generated SQL to call the corresponding target data.

[0101] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the embodiments of this application also provide a full-process data management device.

[0102] Figure 7 This is a structural block diagram of a full-process data management device provided in an embodiment of this application. The device is configured to execute the full-process data management method provided in the above embodiment, and has corresponding functional modules and beneficial effects for executing the method. For example... Figure 7 As shown, the device includes: a data access module 701, a data processing module 702, a data layering module 703, a table configuration module 704, and a code processing module 705; The data access module 701 is used to collect concurrent data from various data sources through a preset interface and store them in the form of message streams. Data processing module 702 is used to process the stored concurrent data based on a preset data processing task to obtain target data; The data layering module 703 is used to perform layered processing on the target data; The data layering module 703 is also used for real-time storage of the target data after layering processing; Table configuration module 704 is used for visual configuration of preset data model tables; The code processing module 705 is used to generate and execute structured query language code based on the data model table after visualization configuration, so as to call the target data stored in real time after layering for display.

[0103] The aforementioned device stores a large amount of concurrent data collected from multiple different data sources in the form of message streams. This allows the stored concurrent data to form a message queue, enabling more efficient and reliable storage and transmission of concurrent data, and thus facilitating data processing tasks.

[0104] After processing concurrent data, the target data is layered, transforming it from logically chaotic raw data into data that conforms to the relevant business logic layers, making it easier to call.

[0105] On the other hand, after configuring the data model table according to the business logic, it is possible to generate SQL code based on the data model table. This not only achieves low coding and avoids generating SQL code through a lot of coding work, but also effectively reduces the code development cycle and saves a lot of human resources.

[0106] Furthermore, when configuring the data model tables, the visual configuration not only achieves low coding but also reduces the difficulty of generating SQL code, thus enabling the use of structured query language code to effectively and conveniently call the hierarchical target data.

[0107] In one possible embodiment, the data processing task includes a data cleaning task and a data transformation task, and the data processing includes data cleaning operations and data transformation operations.

[0108] Correspondingly, the data processing module 702 is specifically configured as follows: Configure corresponding data cleaning and data transformation tasks for concurrent data; The concurrent data is cleaned using a data cleaning task, and the cleaned concurrent data is then transformed using a data transformation task to obtain the target data.

[0109] In one possible embodiment, the end-to-end data management device further includes a data storage module 706, configured as follows: After obtaining the target data, the target data is persistently stored in the preset first data warehouse; The target data in the first data warehouse is synchronized to the preset second data warehouse in real time.

[0110] Correspondingly, the data layering module 703 is specifically configured as follows: The system retrieves the log files from the second data warehouse and uses these files to determine the target data to be synchronized to the second data warehouse. Based on the preset business themes and stream processing framework, the identified target data is processed in layers.

[0111] This includes performing layered processing on the identified target data based on preset business themes and stream processing frameworks, including: The determined target data is transformed into a preset data detail layer using a stream processing framework; Based on the business theme, the target data in the data detail layer is transformed to the preset data aggregation layer using a stream processing framework.

[0112] In one possible embodiment, the data model table includes dimension tables, indicator tables, and / or fact tables, which are used to represent different categories of target data.

[0113] Accordingly, the table configuration module 705 is further configured as follows: The table structure, field relationships, and calculation rules of the dimension table, indicator table, and / or fact table can be configured visually.

[0114] Accordingly, the code processing module 706 is specifically configured as follows: Structured Query Language (SCL) code is generated based on the dimension tables, indicator tables, and / or fact tables configured visually.

[0115] Furthermore, based on any one or more of the dimension tables, indicator tables, and fact tables pointed to during the execution of the structured query language code, the corresponding hierarchical target data stored in real time is called for display.

[0116] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0117] The apparatus described above is used to implement the corresponding full-process data management method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0118] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the embodiments of this application also provide a full-process data management device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the full-process data management method as described in any of the above embodiments.

[0119] Figure 8 A schematic diagram of the structure of a full-process data management device provided in this application embodiment is shown below. Figure 8 As shown, the device includes a processor 201, a memory 202, an input device 203, and an output device 204; the number of processors 201 in the device can be one or more. Figure 8 Taking a processor 201 as an example; the processor 201, memory 202, input device 203, and output device 204 in the device can be connected via a bus or other means. Figure 8 Taking a bus connection as an example, the memory 202, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as program instructions / modules for implementing the full-process data management method in this embodiment. The processor 201 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 202, thereby implementing the aforementioned full-process data management method. The input device 203 can be configured to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. The output device 204 may include a display screen or other display device.

[0120] The apparatus described above is used to implement the corresponding full-process data management method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0121] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-volatile storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are configured to perform a full-process data management method described in the above embodiments, comprising: collecting concurrent data from various data sources through a preset interface and storing them respectively in the form of message streams; processing the stored concurrent data based on a preset data processing task to obtain target data; performing layered processing on the target data and storing the layered target data in real time; visually configuring a preset data model table, generating and executing structured query language code based on the visually configured data model table to call and display the layered, real-time stored target data.

[0122] It is worth noting that in the embodiments of the above-mentioned full-process data management device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not configured to limit the protection scope of the embodiments of this application.

[0123] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product, which includes program code. When the program product is run on a computer device, the program code is configured to cause the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device can execute the full-process data management method described in the embodiments of this application. The program product can be implemented using any combination of one or more readable media, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated further here.

Claims

1. A full-process data management method, characterized in that, include: Concurrent data from various data sources is collected through a preset interface and stored in the form of message streams. The stored concurrent data is processed based on a preset data processing task to obtain the target data; The target data is processed into layers, and the layered target data is stored in real time. The preset data model table is configured visually, and structured query language code is generated and executed based on the visualized data model table to display the target data stored in real time after hierarchical processing.

2. The end-to-end data management method according to claim 1, characterized in that, The data processing task includes data cleaning task and data transformation task, and the data processing includes data cleaning operation and data transformation operation. The process of processing the stored concurrent data based on a preset data processing task to obtain target data includes: Configure corresponding data cleaning and data transformation tasks for concurrent data; The concurrent data is cleaned based on the data cleaning task, and the cleaned concurrent data is transformed based on the data transformation task to obtain the target data.

3. The end-to-end data management method according to claim 2, characterized in that, After obtaining the target data, the method further includes: The target data is persistently stored in a preset first data warehouse; The target data in the first data warehouse is synchronized to the preset second data warehouse in real time.

4. The end-to-end data management method according to claim 3, characterized in that, The hierarchical processing of the target data includes: The log file of the second data warehouse is invoked, and the target data to be synchronized to the second data warehouse is determined based on the log file; Based on the preset business themes and stream processing framework, the identified target data is processed in layers.

5. The end-to-end data management method according to claim 4, characterized in that, The hierarchical processing of the identified target data based on a preset business theme and stream processing framework includes: The identified target data is transformed to a preset data detail layer using the stream processing framework. Based on the business theme, the target data in the data detail layer is transformed to a preset data aggregation layer using the stream processing framework.

6. The end-to-end data management method according to any one of claims 1-5, characterized in that, The data model table includes a dimension table, an indicator table, and / or a fact table, wherein the dimension table, the indicator table, and the fact table are used to represent different categories of target data. The step of visually configuring the preset data model table and generating structured query language code based on the visually configured data model table includes: Visual configurations are made for the table structure, field relationships, and operation rules of the dimension table, the indicator table, and / or the fact table, respectively. Structured Query Language (SCL) code is generated based on the dimension table, indicator table, and / or fact table after visualization configuration.

7. The end-to-end data management method according to claim 6, characterized in that, The display of the target data stored in real time after the call layer includes: Based on any one or more of the dimension table, indicator table, and / or fact table pointed to during the execution of the structured query language code, the corresponding hierarchical target data stored in real time is called for display.

8. A full-process data management device, characterized in that, include: Data access module, data processing module, data layering module, table configuration module, and code processing module; The data access module is used to collect concurrent data from various data sources through preset interfaces and store them in the form of message streams. The data processing module is used to process the stored concurrent data based on a preset data processing task to obtain target data; A data layering module is used to perform layered processing on the target data; The data layering module is also used for real-time storage of the target data after layering processing; The table configuration module is used for visual configuration of preset data model tables; The code processing module is used to generate and execute structured query language code based on the data model table after visualization configuration, so as to call the target data stored in real time after layering for display.

9. An electronic device, characterized in that, The device includes: one or more processors; and a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the full-process data management method according to any one of claims 1-7.

10. A non-volatile storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are configured to perform the full-process data management method as described in any one of claims 1-7.

Images