Method, device and equipment for generating data product and storage medium

Abstract

The application relates to the technical field of data governance, and provides a data product generation method and device, equipment and a storage medium. The method comprises the following steps: when a first business rule model does not exist in a business rule model set, generating a business rule feature according to a business rule search engine, initiating an evaluation according to the business rule feature, and updating the business rule model set according to an evaluation result and the business rule feature; when a first technical rule model does not exist in a technical rule model set, generating a technical rule feature according to a technical rule search engine, initiating an evaluation according to the technical rule feature, and updating the technical rule model set according to an evaluation result and the technical rule feature; and data information and business content are processed according to the first business rule model and the first technical rule model to obtain a data product. The application can improve the efficiency of data product generation.

Description

Technical Field

[0001] This application relates to the field of data governance technology, and in particular to a method, apparatus, device and storage medium for generating data products. Background Technology

[0002] Traditional data governance methods typically involve manually layering, modeling, and configuring data transfer to process source data. The processed data is then used to create data products stored in a data warehouse to support business decisions across various scenarios. However, with the increasing segmentation and diversification of business scenarios, the demands on data products are growing, making the data processing of source data more complex. The increasing reliance on manual data processing leads to low efficiency in data product generation, failing to effectively support the timeliness requirements of business scenarios. Therefore, there is an urgent need for a data product generation method that can improve efficiency. Summary of the Invention

[0003] The main objective of this application is to provide a method, apparatus, device, and storage medium for generating data products, aiming to improve the efficiency of data product generation.

[0004] Firstly, according to an embodiment of this application, a method for generating a data product is provided, the method comprising:

[0005] Acquire data to be processed; wherein, the data to be processed includes data information and business content corresponding to the data information;

[0006] Determine whether a first business rule model exists in the business rule model set, wherein the first business rule model is used for data processing of the business content;

[0007] When the first business rule model does not exist, the business rule search engine generates business rule features corresponding to the business content, initiates a review based on the business rule features, and updates the business rule model set based on the review results and the business rule features.

[0008] Determine whether a first technical rule model exists in the set of technical rule models, wherein the first technical rule model is used for data processing of the data information;

[0009] When the first technical rule model does not exist, the technical rule search engine generates the technical rule features corresponding to the data information, initiates a review based on the technical rule features, and updates the technical rule model set based on the review results and the technical rule features.

[0010] When the first business rule model and the first technical rule model exist, the data information and the business content are processed according to the first business rule model and the first technical rule model to obtain data products.

[0011] Secondly, embodiments of this application provide a data product generation apparatus, comprising:

[0012] An acquisition module is used to acquire data to be processed; wherein, the data to be processed includes data information and business content corresponding to the data information;

[0013] The first judgment module is used to determine whether a first business rule model exists in the business rule model set for data processing of the business content;

[0014] The first review module is used to generate business rule features corresponding to the business content according to the business rule search engine when the first business rule model does not exist, and to initiate a review based on the business rule features, and to update the business rule model set according to the review results and the business rule features.

[0015] The second judgment module is used to determine whether a first technical rule model exists in the technical rule model set for data processing of the data information;

[0016] The second review module is used to generate technical rule features corresponding to the data information according to the technical rule search engine when the first technical rule model does not exist, and to initiate a review based on the technical rule features, and to update the technical rule model set according to the review results and the technical rule features.

[0017] The generation module is used to process the data information and the business content according to the first business rule model and the first technical rule model when the first business rule model and the first technical rule model exist, so as to obtain a data product.

[0018] Thirdly, embodiments of this application provide an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method for generating a data product as described in any of the first aspects.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for generating the data product according to any one of the first aspects.

[0020] This application proposes a method, apparatus, device, and storage medium for generating data products. By setting up a business rule search engine and a technical rule search engine, in the absence of a first business rule model and a first technical rule model, it automatically extracts data to be processed based on sets of business rule models and technical rule models, respectively. This allows for the review of business rule features and technical rule features, thereby reducing the difficulty of setting rules for data processing during data product generation. Simultaneously, the data product generation process is handled from both technical and business dimensions, separating business content from data information and simplifying the processing. As the amount of data to be processed increases, the business rule search engine and technical rule search engine can more accurately extract business rule features and technical rule features, thereby reducing the number of manual reviews. Therefore, compared with related technologies, the embodiments of this application can improve the efficiency of data product generation. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the method for generating data products provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the execution flow of a specific embodiment of the method for generating execution data products provided in this application.

[0023] Figure 3 This is a schematic diagram of the modules of the data product generation device provided in the embodiments of this application;

[0024] Figure 4 This is a schematic diagram of the hardware structure corresponding to the data product generation method of this application embodiment. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0027] The following is an explanation of the terms used in this application:

[0028] A data warehouse, abbreviated as DW or DWH, is a strategic collection of data that supports all types of data in decision-making processes at all levels of an enterprise. It is a single data store created for analytical reporting and decision support purposes.

[0029] Metadata is data about data, or structured data used to provide information about a resource. Metadata describes information resources or data objects, and its purpose is to: identify resources; evaluate resources; track changes in resources during use; achieve simple and efficient management of large amounts of networked data; and enable effective discovery, retrieval, integrated organization, and effective management of information resources.

[0030] Source data: In essence, it refers to the "data" itself, emphasizing that the data state is the "original state" after "creation", that is, the data that has not been processed. In the process of data management, source data generally refers to data that comes directly from source files (business system databases, offline files, IoT, etc.) or "copy data" that is directly copied from source files.

[0031] A search engine is a retrieval technology that uses specific strategies and algorithms to retrieve specific information from the internet and return it to the user based on their needs. The working mechanism of a search engine involves using efficient spider programs to traverse the entire Internet, starting from a specified URL and following hyperlinks, employing depth-first or breadth-first search algorithms to crawl the webpages and store the information in a local database. Then, an indexer is used to index important information units in the database, such as titles, keywords, and summaries, or the full text, for query navigation. Finally, the search engine matches the user's query submitted through the browser with the information in the indexed database using a certain retrieval technique, and returns the search results to the user according to a certain sorting method.

[0032] Traditional data governance methods typically involve manual layering, modeling, and configuration of data transfer to process source data. The processed data is then used to create data products stored in a data warehouse for use in various business scenarios, supporting business decision-making. However, with the increasing segmentation and diversification of business scenarios, the demands on data products are growing, making the data processing of source data more complex. The increasing reliance on manual data processing leads to low efficiency in data product generation, failing to effectively support the timeliness requirements of business scenarios. Therefore, there is an urgent need for a data product generation method that can improve efficiency. Based on this, this application provides a data product generation method, apparatus, device, and storage medium that can improve the efficiency of data product generation.

[0033] Reference Figure 1 As shown, a method for generating a data product according to an embodiment of this application includes:

[0034] Step S100: Obtain the data to be processed; wherein, the data to be processed includes data information and the business content corresponding to the data information;

[0035] Step S200: Determine whether a first business rule model exists in the business rule model set. The first business rule model is used for data processing of business content.

[0036] Step S300: When there is no first business rule model, generate business rule features corresponding to the business content according to the business rule search engine, initiate a review based on the business rule features, and update the business rule model set according to the review results and business rule features.

[0037] Step S400: Determine whether a first technical rule model exists in the set of technical rule models. The first technical rule model is used for data processing of data information.

[0038] Step S500: When there is no first technical rule model, generate technical rule features corresponding to the data information according to the technical rule search engine, initiate a review based on the technical rule features, and update the technical rule model set according to the review results and technical rule features.

[0039] Step S600: When a first business rule model and a first technical rule model exist, data information and business content are processed according to the first business rule model and the first technical rule model to obtain data products.

[0040] Therefore, by setting up a business rule search engine and a technical rule search engine, in the absence of a first business rule model and a first technical rule model, the data to be processed can be automatically extracted based on the business rule model set and the technical rule model set, respectively. This allows for the review of business rule features and technical rule features, thereby reducing the difficulty of setting rules for data processing during data product generation. Simultaneously, the data product generation process is handled separately from the dimensions of data information and business content, thus separating business content from data information and simplifying the processing. As the amount of data to be processed increases, the business rule search engine and the technical rule search engine can extract business rule features and technical rule features more accurately, thereby reducing the number of manual reviews. Therefore, compared with related technologies, the embodiments of this application can improve the efficiency of data product generation.

[0041] The data to be processed in step S100 can be obtained through data migration technology, which includes, but is not limited to, the acquisition triggered periodically or by events through tools such as the Flink CDC technology framework, Kafka, or Sqoop.

[0042] The data information recorded in step S100 is the field information (including field type and field name, etc.) that displays the business content. The business content records data generated due to business scenarios, such as personnel data and production data. For example, taking data to be processed obtained from the personnel database as an example, the data information corresponding to the business content of the personnel data is shown in Table 1 below:

[0043] ID Varchar(5) yes Employee Number StuffName Varchar(20) no Employee Name Folk Varchar(20) no nationality Kultur Varchar(14) no Education level

[0044] Table 1

[0045] The business content is a collection of records corresponding to Table 1 above. For example, one record is "123, xxx, Han, Bachelor's Degree". Here, "123" corresponds to "Employee ID" in Table 1, "xxx" corresponds to "Employee Name" in Table 1, "Han" corresponds to "Ethnicity" in Table 1, and "Bachelor's Degree" corresponds to "Education Level" in Table 1.

[0046] It should be noted that in some embodiments, the data to be processed also includes a data source. In this case, the data to be processed consists of a data source, data information, and business content. The data source represents the purpose of the business content. For example, if the business content records personnel data, the data source represents a data table that stores the business content or a database that stores personnel data separately. In some embodiments, due to different business scenarios, the personnel data will be further refined. In this case, the data source will also be specified to a specific application scenario, such as a personnel data table for the production department.

[0047] It should be noted that the judgments in steps S200 and S400 can be based on the data source to determine whether the first business rule model and the first technical rule model exist. Specifically, for example, each technical rule model in the technical rule model set and each business rule model in the business rule model set can be associated with its corresponding data source. This is achieved by comparing the data sources of each technical rule model in the technical rule model set and each business rule model in the business rule model set with the data source of the data to be processed, thereby determining whether the first business rule model and the first technical rule model exist. Alternatively, a flag can be set for the processed data source, and the existence of the first business rule model and the first technical rule model can be determined based on the value of the flag. In other embodiments, the judgments in steps S200 and S400 can be based on data information, such as whether the data information contains the field name specified by the data source, or whether the length of the data field defined in the data information meets the length value. In other embodiments, a business rule search engine and a technical rule search engine can be used directly for retrieval. If no results are found, business rule features and technical rule features are automatically generated.

[0048] It should be noted that the technical rule model set is the index database corresponding to the technical rule search engine. The business rule model set is the index database corresponding to the business rule search engine. The business rule features in step S300 are automatically generated by the business rule search engine. After approval, an index of the data to be processed will be established in the business rule model set, so that the matching first business rule model can be retrieved based on the index. The technical rule features in step S500 are automatically generated by the technical rule search engine. After approval, an index of the data to be processed will be established in the technical rule model set, so that the matching first technical rule model can be retrieved based on the index. Both the business rule search engine and the technical rule search engine adopt the principles of search engines, extracting keywords from the data to be processed and retrieving them from a pre-set index database to determine the required rule model. The implementation of the business rule search engine and the technical rule search engine in this embodiment will not be described in detail. As the amount of data to be processed increases, the business rule features and technical rule features extracted by the business rule search engine and the technical rule search engine respectively better meet the needs of the business scenario, thereby further reducing the probability of manual review and improving the efficiency of data product generation.

[0049] It should be noted that the initial review in steps S300 and S500 is manual. Automatically generated business rule features and technical rule features assist in the data governance process of the data to be processed, thereby reducing the requirements for auditors and improving processing efficiency. As the amount of data to be processed increases, the business rule features and technical rule features generated by the business rule search engine and technical rule search engine become more accurate. In some embodiments, when the accuracy rate of the business rule features and technical rule features exceeds an accuracy threshold, business content with importance below a preset importance threshold does not require manual review, thus achieving semi-automatic review. In other embodiments, when the accuracy rate of the business rule features and technical rule features exceeds an accuracy threshold, fully automatic review is achieved. Therefore, this application does not impose excessive limitations on the review mechanism and can selectively set it based on the business scenario.

[0050] The review in step S500 is used to verify whether the formulation, application, and implementation of the data governance framework, as well as the data governance strategies and rules, comply with data governance requirements. The data governance framework consists of data fields within the data information. Data governance strategies and rules indicate whether the field lengths and types of the selected data fields within the data governance framework meet governance requirements. The application of the data governance framework indicates whether the selected data governance strategies and rules meet business scenario requirements. In other words, the review in step S500 primarily verifies whether the field lengths and types of the data fields in the data information meet the needs of data warehouse modeling (e.g., taking a table recording personnel information as an example, determining whether the field type and length of the "employee number" field meet the governance requirements for building an employee model in the data warehouse), whether they meet the governance requirements of data governance theme processing, and whether they meet the various processes required for indicator extraction based on business scenario needs.

[0051] The review in step S300 mainly examines business content processes related to the business, as well as review services related to data dimensionality reduction, slicing, dicing, rotation, drill-down processing, etc.

[0052] It should be noted that by processing technology-related data and business-related content separately, the business and technology-related aspects of data processing can be made independent of each other, which can adapt to more business scenarios and improve compatibility.

[0053] It should be noted that in some embodiments, the processing of business content can be divided into multiple dimensions, such as themes, business metrics, etc. Correspondingly, there are also multiple business rule search engines, so there are multiple first business rule models.

[0054] It should be noted that steps S200 and S400 can be processed in parallel or in sequence. In this regard, the embodiments of this application do not impose too many restrictions on the order of the steps.

[0055] Understandably, the business rule model set includes the topic rule model set, and the first business rule model includes the first topic rule model. When the first topic rule model is not available in the topic rule model set for data processing of business content, step S300 generates business rule features corresponding to the business content based on the business rule search engine and initiates a review based on the business rule features, including:

[0056] Based on the topic extraction rule engine, semantic recognition is performed on the business content to obtain topic business rule features;

[0057] When a second topic rule model exists in the topic rule model set with a similarity greater than the first threshold to the characteristics of the topic business rule, the second topic rule model will be used as the approval content for review.

[0058] If a second topic rule model does not exist in the topic rule model set, a third topic rule model is created based on the characteristics of the topic business rules, and the third topic rule model is used as the approval content to initiate the review.

[0059] It should be noted that the topic extraction rule engine is one of the business rule search engines.

[0060] It should be noted that the topic extraction rule engine is used to perform semantic recognition of business information based on NLP (Natural Language Processing), and determine a second topic rule model based on the similarity between the identified topic business rule features and existing topic rule models. For example, taking a first threshold of 50% as an example, when the similarity of the second topic rule model is greater than the 50% threshold, the system points to the second topic rule model and sets its status to unapproved. Once the second topic rule model is approved, data processing of the business content is performed based on it. When the similarity between each topic rule model in the topic rule model set and the topic business rule features is less than the 50% threshold, a new data topic is created, forming a third topic rule model, and its status is set to unapproved. At this point, the created third topic rule model is reviewed to see if it conforms to the business scenario. Once approved, an index is created for the data to be processed, allowing retrieval of the third topic rule model to process the business content using the third topic rule model.

[0061] Understandably, the business rule model set includes the technical indicator model set, and the first business rule model includes the first technical indicator model. When the first technical indicator model is not available in the technical indicator model set for data processing of business content, step S300 generates business rule features corresponding to the business content based on the business rule search engine and initiates a review based on the business rule features, including:

[0062] Based on the technical indicator extraction rule engine, at least one data field related to the business content is extracted from the data information, and the technical indicator business rule characteristics are determined based on the data field.

[0063] Calculate the correlation between each technical indicator model in the technical indicator model set and the characteristics of the technical indicator business rules;

[0064] The second technical indicator model is obtained by selecting the N technical indicator models with the highest correlation from the set of technical indicator models; where N is an integer greater than 1.

[0065] The second technical indicator model will be used as the content of the approval process for review.

[0066] It should be noted that the technical indicator extraction rule engine is one of the business rule search engines.

[0067] It should be noted that the technical indicator extraction rule engine is used to perform correlation analysis on existing technical indicator models when new technical indicators are detected in the metadata. This correlation analysis primarily involves semantic analysis. Specifically, a matrix model is set based on the data type and dimensionality of the data information. The correlation analysis results are used to rank the relevance levels, and the N technical indicator models with the closest relevance are used as reference models and set to "unreviewed." During the review process, the reviewer can determine whether the second technical indicator model in the review process can be directly adopted or recreated with reference to it. Therefore, when the review is passed, a third technical indicator model is parsed from the review results and created in the technical indicator model set as an index for the model used to process the data for technical indicator processing.

[0068] It should be noted that for the data to be processed, its technical indicator can be a single indicator composed of multiple data fields. For example, referring to Table 1 above, the data fields "ethnicity" and "educational level" are both related to cultural background and can constitute metadata corresponding to that cultural background. Therefore, multiple data fields can be formatted into metadata. When the metadata extracted by the technical indicator extraction rule engine does not exist in the technical indicator model set, it indicates that the metadata is newly added, and the newly added metadata is used as a technical indicator business rule feature. In some embodiments, there are multiple metadata elements.

[0069] It should be noted that data fields related to business content indicate that the data field can be used to process data information for technical indicators. For example, in Table 1, the "primary key" column only identifies whether it is an index value and does not have a real business meaning. Therefore, it is not used as a field in the metadata of technical indicators.

[0070] It should be noted that during the approval process, if it is determined that the second technical indicator model can be directly applied to the data processing of the technical indicators of the data to be processed, then after approval, the second technical indicator model will be directly used for data processing. Correspondingly, for subsequent processing of the same type of data, the corresponding second technical indicator model can be found for data processing. In some embodiments, during the approval process, both the technical indicator business rule features and the second technical indicator model will be included in the review. When it is manually determined that the second technical indicator model is not suitable for processing the data to be processed, the reviewer can create a third technical model based on the technical indicator business rule features to establish an index between the third technical model and the data to be processed in the technical indicator model set. This facilitates the rapid retrieval of the third technical model when extracting rules from the data to be processed.

[0071] Understandably, the business rule model set includes the business indicator rule model set, and the first business rule model includes the first business indicator rule model. When the first business indicator rule model is not available in the business indicator rule model set for data processing of business content, step S300 generates business rule features corresponding to the business content based on the business rule search engine and initiates a review based on the business rule features, including:

[0072] Based on the business metric extraction rule engine, the first business content is extracted from the business content, and the business metric rule characteristics are determined based on the first business content.

[0073] Calculate the correlation between each business indicator rule model and the features of the business indicator rules in the business indicator rule model set;

[0074] The second business indicator rule model is obtained by selecting the N business indicator rule models with the highest relevance from the set of business indicator rule models; where N is an integer greater than 1.

[0075] The second business indicator rule model will be used as the approval content for the review process.

[0076] It should be noted that the business indicator extraction rule engine is one of the business rule search engines. When new business indicators are detected, the engine performs correlation analysis on the existing business indicator model. This correlation analysis primarily involves semantic analysis, segmenting and setting up a matrix model based on data type and dimensions, ranking the relevance, and using the most closely related business indicators as the reference. The status is then set to unreviewed, and a manual review process is initiated. For example, taking educational level as a business indicator classification, historically, all educational levels were below Master's (divided into three dimensions: Bachelor's and Master's, Associate's and Bachelor's, and Associate's and below). However, the current data to be processed includes records of PhD and Postdoctoral degrees. In this case, PhD and Postdoctoral degrees become new business indicators requiring review. The closest related business indicator model is the one corresponding to Bachelor's and Master's degrees.

[0077] It should be noted that in some embodiments, before the invocation, the source data extraction rule engine is also invoked to extract business content, and the extracted business content is further extracted using the business metric extraction rule engine. Source data extraction depends on the type of business data. Extraction rules are set on a pre-defined extraction model. If the extraction model restricts the type or theme of business data, relevant data is extracted according to these restrictions; otherwise, all data is extracted.

[0078] Understandably, the technical rule model set includes the data standard rule model set, and the first technical rule model includes the first data standard rule model. When the data standard rule model set does not have a first data standard rule model for data information processing, step S500 generates technical rule features corresponding to the data information based on the technical rule search engine and initiates a review based on the technical rule features, including:

[0079] Based on the standard extraction rule engine, the data fields to be processed by the data standard are determined from the data information, and the data standard technical rule features are obtained.

[0080] When a second data standard rule model exists in the data standard rule model set with a similarity greater than the third threshold to the characteristics of the data standard technical rules, the second data standard rule model will be used as the content to initiate the review.

[0081] When there is no second data standard rule model, the data standard technical rule features will be used as the approval content for the review.

[0082] It should be noted that the data standard extraction rule engine compares business data types with data standards. If they conform to the data standard, no additional data standard is set; otherwise, based on NLP (Natural Language Processing), relevant standard matching is performed, and data standards that meet the third threshold of similarity are provided to auditors for data standard review. For example, with a third threshold of 50%, when the matching degree between the second data standard rule model and the data standard technical rule features is greater than 50%, the first data rule model is used as the approval content, and its status is set to unreviewed. When the matching degree between the second data standard rule model and the data standard technical rule features is less than 50%, the data standard technical rule features are used as the new data standard, and its status is set to unreviewed.

[0083] For example, if the standard extraction rule engine is set to standardize the data field representing time, then the data field A representing time is extracted from the data information, and the data field A is compared with each data standard rule model in the data standard rule model set. If the data field A is an alias of a certain field B, then the data field A is replaced by the field B, thus completing the standardization process.

[0084] Understandably, the technical rule model set includes the data modeling rule model set, and the first technical rule model includes the first modeling rule model. When the data modeling rule model set does not have a first modeling rule model for data processing, step S500 generates technical rule features corresponding to the data information based on the technical rule search engine and initiates a review based on the technical rule features, including:

[0085] Based on the data modeling extraction rule engine, standard metadata for data modeling is determined from the data information, and modeling technical rule features are determined based on the standard metadata and modeling syntax template.

[0086] The modeling technology rules and features will be used as the content of the approval process to initiate the review.

[0087] It should be noted that the data modeling extraction rule engine is used to set a modeling syntax template based on the configured data warehouse type, and to establish technical rule features, i.e., a data model, based on the modeling syntax template and the metadata extracted from the first technical rule model. After review and approval, a second modeling rule model will be generated, and an index will be established between the second modeling rule model and the data to be processed in the data modeling rule model set, so that the second modeling rule model can be quickly retrieved before the data to be processed is processed.

[0088] Understandably, there are multiple first business rule models, including a first theme rule model, a first technical indicator model, and a first business indicator rule model; there are also multiple first technical rule models, including a first data standard rule model and a first modeling rule model. In step S600, based on the first business rule models and the first technical rule models, data information and business content are processed to obtain data products, including:

[0089] The theme data is determined based on the first theme rule model and business content;

[0090] The technical indicator data is determined based on the first technical indicator model and data information;

[0091] The business indicator data is determined based on the first business indicator rule model and business content;

[0092] Based on technical and business performance data, determine the business and technical performance data;

[0093] Based on the data information and the first data standard rule model, determine the data standard data;

[0094] Based on the first modeling rule model, data standard data, and data information, a business data model is obtained;

[0095] Data products are derived from business data models and business metrics data.

[0096] The data topic management service calls the topic extraction rule engine to determine the first topic rule model, and establishes a business topic based on the first topic rule model to obtain topic data.

[0097] The data standard management service invokes the standard extraction rule engine to determine the first data standard rule model. Based on this model, the data information is transformed to generate common data fields, resulting in standard data. The data standard management service extracts shared metadata attributes, reducing redundant modeling services.

[0098] The data modeling service calls the data modeling extraction rule engine to determine the first modeling rule model, and discovers metadata information based on the first modeling rule model. Then, a metadata model is built based on the metadata template to obtain the business data model.

[0099] The technical indicator management service calls the technical indicator extraction rule engine to determine the first technical indicator model. Based on this model, data fields are extracted from the data information to establish metadata, thus defining the technical indicator data. Similarly, the business indicator extraction rule engine is called to determine the first business indicator rule model. Based on this model, business content corresponding to the technical indicator data is extracted from the business content, resulting in the business indicator data. Therefore, the technical indicator management service establishes technical indicators for the data based on its technical attributes, monitoring the timeliness of data processing and the adequacy of data storage, thereby obtaining business technical indicator data.

[0100] It should be noted that the technical indicator management service, data modeling service, data theme management service, and data standard management service are all independent tasks created during program execution, such as microservices.

[0101] For example, see below. Figure 2 This application describes an execution process of an embodiment.

[0102] Data migration services are used to migrate data from the backup database to obtain the data to be processed. At this point, business process services are used to review the data to be processed according to business rules and standards. ， To determine whether a first business rule model exists in the business rule model set, wherein the first business rule model is used for data processing of business content, and to determine whether a first technical rule model exists in the technical rule model set by reviewing the data to be processed through technical process services, wherein the first technical rule model is used for data processing of data information.

[0103] Among them, reference Figure 2 As shown, the business rule model set includes the source data rule model set, the business indicator rule model set, the technical indicator model set, and the theme rule model set; the first business rule model includes at least one of the first source data rule model, the first business indicator rule model, the first theme rule model, and the first technical indicator model. When the business rule standard review fails, the following processing flow (1) to (4) is executed:

[0104] (1) If there is no first source data rule model in the source data rule model set, the source data extraction rule engine is called to generate source data business rule features and a review is initiated based on the source data business rule features.

[0105] (2) If there is no first business indicator rule model in the business indicator rule model set, the business indicator extraction rule engine is called to generate business indicator rule features and a review is initiated based on the business indicator rule features.

[0106] (3) If the first topic rule model does not exist in the topic rule model set, the topic extraction rule engine is called to generate topic business rule features and a review is initiated based on the topic business rule features;

[0107] (4) If the first technical indicator model does not exist in the technical indicator model set, the technical indicator extraction rule engine is called to generate technical indicator business rule features and a review is initiated based on the technical indicator business rule features.

[0108] When the review initiated in (1) to (4) above is passed, the corresponding business rule model set is updated. That is, the index library corresponding to the extraction rule engine is updated. At this time, the first business rule model exists in the business rule model set.

[0109] Among them, reference Figure 2 As shown, the business rule model set includes the data standard rule model set and the data modeling rule model set; the first business rule model includes at least one of the first data standard rule model and the first modeling rule model. When the technical rule standard review fails, the following processing flow (5) to (6) is executed:

[0110] (5) If the first data standard rule model is not found in the data standard rule model set, the data standard extraction rule engine is called to generate data standard technical rule features, and a review is initiated based on the data standard technical rule features.

[0111] (6) If the first modeling rule model is not found in the data modeling rule model set, the data modeling extraction rule engine is called to generate modeling technical rule features, and a review is initiated based on the modeling technical rule features.

[0112] When the review initiated by the above processing steps (5) and (6) is passed, the corresponding technical rule model set is updated, that is, the index library corresponding to the extraction rule engine is updated. At this time, the first technical rule model exists in the technical rule model set.

[0113] At this time, refer to Figure 2 As shown, the business content is processed by calling the first business rule model through the data theme service and technical indicator management service to obtain business technical indicator data; specifically, theme data is determined based on the first theme rule model and business content; technical indicator data is determined based on the first technical indicator model and data information; business indicator data is determined based on the first business indicator rule model and business content; and business technical indicator data is determined based on the technical indicator data and business indicator data; refer to... Figure 2As shown, the data information is processed by calling the first technical rule model through the data standard service and data modeling service to obtain the business data model. Specifically, based on the data information and the first data standard rule model, data standard data is determined; based on the first modeling rule model, the data standard data, and the data information, the business data model is obtained. Thus, data products are generated based on the business technical indicator data and the business data model and stored in the data warehouse.

[0114] Reference Figure 3 As shown, according to an embodiment of this application, a data product generation apparatus is provided, comprising:

[0115] The acquisition module 100 is used to acquire data to be processed; wherein, the data to be processed includes data information and business content corresponding to the data information;

[0116] The first judgment module 200 is used to determine whether the first business rule model exists in the business rule model set for data processing of business content;

[0117] The first review module 300 is used to generate business rule features corresponding to the business content based on the business rule search engine when the first business rule model does not exist, initiate a review based on the business rule features, and update the business rule model set based on the review results and business rule features.

[0118] The second judgment module 400 is used to determine whether the first technical rule model exists in the technical rule model set for data processing of data information;

[0119] The second review module 500 is used to generate technical rule features corresponding to data information based on the technical rule search engine when the first technical rule model does not exist, initiate a review based on the technical rule features, and update the technical rule model set based on the review results and technical rule features.

[0120] The generation module 600 is used to process data information and business content according to the first business rule model and the first technical rule model when a first business rule model and a first technical rule model exist, so as to obtain data products.

[0121] It is understood that the electronic device provided according to the embodiments of this application includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for generating data products.

[0122] This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0123] Please see Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0124] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0125] The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called by the processor 701 to execute the data product generation method of the embodiments of this application.

[0126] The input / output interface 703 is used to implement information input and output;

[0127] Communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth); and...

[0128] Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704);

[0129] The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0130] It is understood that the computer-readable storage medium provided according to the embodiments of this application stores a computer program, which, when executed by a processor, implements the above-described method for generating data products.

[0131] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0132] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0133] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0135] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0136] The terms “comprising” and “having”, and any variations thereof, in the specification and accompanying drawings of this application are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or apparatus.

[0137] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0139] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating a data product, characterized in that, The method includes: Acquire data to be processed; wherein, the data to be processed includes data information and business content corresponding to the data information; the business content records data generated due to business scenarios; the data information records field information for displaying business content; Determine whether a first business rule model exists in the business rule model set. The first business rule model is used to process the business content according to a preset processing dimension. The business rule model set includes a business indicator rule model set, a technical indicator model set, and a theme rule model set. The first business rule model includes at least one of a first theme rule model, a first technical indicator model, and a first business indicator rule model. When the first business rule model does not exist, a business rule feature corresponding to the business content is generated according to the business rule search engine, and a review is initiated based on the business rule feature. After the review is passed, the business rule model set is updated based on the business rule feature. The review initiated based on the business rule feature is used to review the business rule models in the business rule model set that are associated with the business rule feature. Determine whether a first technical rule model exists in the technical rule model set. The first technical rule model is used to process the data information to obtain a business data model. The technical rule model set includes a data standard rule model set and a data modeling rule model set. The first technical rule model includes at least one of a first data standard rule model and a first modeling rule model. When the first technical rule model does not exist, the technical rule search engine generates technical rule features corresponding to the data information and initiates a review based on the technical rule features. After the review is passed, the technical rule model set is updated based on the technical rule features. The review initiated based on the technical rule features is used to verify whether the field length and field type of the data fields of the data information meet the requirements of data governance. When the first business rule model and the first technical rule model exist, the data information and the business content are processed according to the first business rule model and the first technical rule model to obtain data products.

2. The method for generating data products according to claim 1, characterized in that, When the first business rule model includes a first topic rule model, and when the topic rule model set does not contain a first topic rule model for data processing of the business content, the step of generating business rule features corresponding to the business content based on the business rule search engine and initiating a review based on the business rule features includes: Based on the topic extraction rule engine, semantic recognition is performed on the business content to obtain topic business rule features; When there is a second topic rule model in the topic rule model set whose similarity to the features of the topic business rule is greater than the first threshold, the second topic rule model is used as the approval content for review. If a second topic rule model does not exist in the topic rule model set, a third topic rule model is created based on the topic business rule characteristics, and the third topic rule model is used as the approval content to initiate a review.

3. The method for generating data products according to claim 1, characterized in that, When the first business rule model includes a first technical indicator model, and when the set of technical indicator models does not contain a first technical indicator model for data processing of business content, the step of generating business rule features corresponding to the business content based on the business rule search engine and initiating a review based on the business rule features includes: Based on the technical indicator extraction rule engine, at least one data field associated with the business content is extracted from the data information, and the technical indicator business rule features are determined based on the data field. Calculate the correlation between each technical indicator model in the technical indicator model set and the business rule features of the technical indicators; The second technical indicator model is obtained by selecting the top N technical indicator models with the highest correlation from the set of technical indicator models; where N is an integer greater than 1. The second technical indicator model will be used as the content of the approval process to initiate a review.

4. The method for generating data products according to claim 1, characterized in that, When the first business rule model includes a first business indicator rule model, and when the set of business indicator rule models does not contain a first business indicator rule model for data processing of the business content, the step of generating business rule features corresponding to the business content based on the business rule search engine and initiating a review based on the business rule features includes: Based on the business metric extraction rule engine, the first business content is extracted from the business content, and the business metric rule features are determined based on the first business content. Calculate the correlation degree between each business indicator rule model in the business indicator rule model set and the business indicator rule features; The second business indicator rule model is obtained by selecting the top N business indicator rule models with the highest relevance from the set of business indicator rule models; where N is an integer greater than 1. The second business indicator rule model will be used as the approval content for the review.

5. The method for generating data products according to claim 1, characterized in that, When the first technical rule model includes a first data standard rule model, and when the data standard rule model set does not contain a first data standard rule model for data processing of the data information, the step of generating technical rule features corresponding to the data information according to the technical rule search engine and initiating a review based on the technical rule features includes: Based on the standard extraction rule engine, the data fields to be processed by the data standard are determined from the data information to obtain the data standard technical rule features; When a second data standard rule model exists in the data standard rule model set with a similarity greater than the third threshold to the characteristics of data standard technical rules, the second data standard rule model will be used as the approval content for review. When there is no second data standard rule model, the data standard technical rule features will be used as the approval content for review.

6. The method for generating data products according to claim 1, characterized in that, When the first technical rule model includes a first modeling rule model, and when the data modeling rule model set does not have a first modeling rule model for data processing of the data information, the step of generating technical rule features corresponding to the data information according to the technical rule search engine and initiating a review based on the technical rule features includes: Based on the data modeling extraction rule engine, standard metadata for data modeling is determined from the data information, and modeling technique rule features are determined based on the standard metadata and the modeling syntax template. The modeling technology rules and features will be used as the approval criteria for initiating a review.

7. The method for generating data products according to claim 1, characterized in that, There are multiple first business rule models, including a first theme rule model, a first technical indicator model, and a first business indicator rule model; there are multiple first technical rule models, including a first data standard rule model and a first modeling rule model. The step of processing the data information and the business content according to the first business rule models and the first technical rule models to obtain data products includes: The theme data is determined based on the first theme rule model and the business content; The technical indicator data is determined based on the first technical indicator model and the data information; The business indicator data is determined based on the first business indicator rule model and the business content; Based on the technical indicator data and the business indicator data, determine the business technical indicator data; Based on the data information and the first data standard rule model, determine the data standard data; Based on the first modeling rule model, the data standard data, and the data information, a business data model is obtained; Based on the business data model and the business indicator data, a data product is obtained.

8. A data product generation apparatus, characterized in that, include: The acquisition module is used to acquire data to be processed; wherein, the data to be processed includes data information and business content corresponding to the data information; the business content records data generated due to business scenarios; the data information records field information for displaying business content; The first judgment module is used to determine whether there exists a first business rule model in the business rule model set for processing the business content according to a preset processing dimension; the business rule model set includes a business indicator rule model set, a technical indicator model set, and a theme rule model set; the first business rule model includes at least one of a first theme rule model, a first technical indicator model, and a first business indicator rule model. The first review module is used to generate business rule features corresponding to the business content based on the business rule search engine when the first business rule model does not exist, and to initiate a review based on the business rule features, and to update the business rule model set based on the business rule features after the review is passed; the review initiated based on the business rule features is used to review the second business indicator rule model associated with the business rule features in the business rule model set. The second judgment module is used to determine whether there is a first technical rule model in the technical rule model set that is used to process the data information to obtain a business data model; the technical rule model set includes a data standard rule model set and a data modeling rule model set, and the first technical rule model includes at least one of a first data standard rule model and a first modeling rule model. The second review module is used to generate technical rule features corresponding to the data information according to the technical rule search engine when the first technical rule model does not exist, and to initiate a review based on the technical rule features. After the review is passed, the module updates the technical rule model set according to the technical rule features. The step of initiating a review based on the technical rule features is used to check whether the field length and field type of the data fields of the data information meet the requirements of data governance. The generation module is used to process the data information and the business content according to the first business rule model and the first technical rule model when the first business rule model and the first technical rule model exist, so as to obtain a data product.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method for generating the data product according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for generating the data product according to any one of claims 1 to 7.

Images