Offline feature management method, system, computer device and storage medium

Abstract

The application relates to an offline feature management method, system, computer device and storage medium. The system comprises a data management platform and a distributed relational database; the data management platform comprises a data acquisition module, a data processing module and a data transmission module; the distributed relational database is used for managing multiple offline feature tables created in advance; the data acquisition module is used for acquiring business data from a specified business system; the data processing module is used for regularly executing a pre-written data processing script to clean and process the specified business data and store feature data obtained through the cleaning and processing into a corresponding feature data table; and the data transmission module is used for regularly transmitting data in the feature data table to a corresponding offline feature table in the distributed relational database. The application can realize more efficient data processing and processing operation, has lower delay when processing large-scale data, is more convenient to operate, and thus business personnel can independently operate and process offline features.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an offline feature management method, system, computer device, and storage medium. Background Technology

[0002] With the rapid development of large model technology, the demand for offline features in modern large models is showing an increasing trend. Compared with the past, the amount of offline feature input and the amount of underlying business data in modern large models are much larger and more complex.

[0003] Currently, the main approach for offline feature generation in large-scale models is to synchronize relevant business data from the business database to Hive, perform data cleaning and processing in Hive, and finally store the processed data in HBase as an offline feature repository. Hive is a data warehouse tool that maps structured data files to a database table and provides SQL (Structured Query Language) query functionality, transforming SQL statements into MapReduce (a programming model) tasks for execution.

[0004] As the volume and complexity of offline feature inputs and underlying business data increase, the aforementioned solutions become increasingly difficult to meet practical needs. For example, components such as Hive and HBase (a distributed, column-oriented open-source database) require cluster deployment, resulting in high costs. The solutions involve the integration and configuration of multiple components and technologies, requiring a high level of technical expertise and maintenance costs. This makes related operations heavily reliant on technical personnel, hindering their use and operation by business users. Furthermore, components like Hive and HBase typically exhibit significant latency when processing large-scale data, impacting business decision-making and response speed, thus making them unsuitable for scenarios with high real-time data requirements. Additionally, the involvement of multiple components and data transmission processes presents challenges in data consistency and synchronization. Summary of the Invention

[0005] To address the aforementioned shortcomings or disadvantages, this application provides an offline feature management method, system, computer device, and storage medium. The embodiments of this application can achieve more efficient data processing and manipulation operations, have lower latency when processing large-scale data, and are more convenient to operate, thereby enabling business personnel to independently process offline features.

[0006] According to a first aspect, this application provides an offline feature management system. In some embodiments, the system includes a data management platform and a distributed relational database; the data management platform includes a data acquisition module, a data processing module, and a data transmission module.

[0007] A distributed relational database used to manage multiple pre-created offline feature tables;

[0008] The data acquisition module is used to collect business data from the specified business system;

[0009] The data processing module is used to periodically execute pre-written data processing scripts to clean and process the specified business data collected by the data acquisition module and store the cleaned and processed feature data into the corresponding pre-created feature data table.

[0010] The data transmission module is used to periodically transmit data from each feature data table to the corresponding offline feature table in a distributed relational database.

[0011] In some embodiments, the data management platform is a system built on Dataphin; the distributed relational database is a database built on TiDB.

[0012] In some embodiments, the data processing module is further configured to respond to a first configuration instruction, configure the data processing scripts pre-submitted by relevant business personnel as data processing tasks, and configure the scheduler so that the scheduler executes the data processing tasks periodically.

[0013] In some embodiments, the data processing scripts described above are SQL statements pre-written by business personnel based on business needs; the number of such data processing scripts is multiple.

[0014] In some embodiments, each feature data table is a row-based data table; each offline feature table is a column-based data table.

[0015] In some embodiments, when the data transmission module transmits data from the feature data table to the corresponding offline feature table in the distributed relational database, it converts the data in the feature data table into column mode and outputs the converted data to the corresponding offline feature table.

[0016] In some embodiments, the distributed relational database has an interface for batch querying offline features.

[0017] This application provides an offline feature management method according to a second aspect. In some embodiments, the method is applied to an offline feature management system, and the method includes:

[0018] Collect business data from the designated business system;

[0019] Regularly execute pre-written data processing scripts to clean and process specified business data and store the cleaned and processed feature data into the corresponding pre-created feature data table;

[0020] Periodically transfer data from the feature data table to the corresponding pre-created offline feature table.

[0021] According to a third aspect, this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the offline feature management method provided in any of the above embodiments.

[0022] According to a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the offline feature management method provided in any of the above embodiments.

[0023] The offline feature management system provided in the above embodiments includes a data management platform and a distributed relational database. The data management platform can automatically collect business data from designated business systems and periodically execute data processing scripts pre-configured by business personnel to clean and process the business data. The cleaned and processed data is stored in corresponding feature data tables, and subsequently, the data from each feature data table is periodically output to the corresponding offline feature tables in the distributed relational database. This application integrates data cleaning and data processing into the data management platform. Business personnel can pre-write corresponding data processing scripts on the platform according to business needs, and the platform can then implement an automated offline feature processing process based on the data processing scripts. This process is simpler than existing technologies, improves data processing efficiency, and lowers the technical threshold for manual operation. Furthermore, the platform is built on Dataphin, and can utilize Dataphin's scheduling function to set data processing scripts as computation tasks, and execute computation tasks periodically through the scheduler, thereby ensuring timely updates of offline feature data. The distributed relational database is built on TiDB, and corresponding output pipelines can be configured for each feature data table in the Dataphin platform to output feature data to TiDB. TiDB has good data storage and management capabilities, supports distributed storage and querying, and improves the efficiency of data storage and management. Furthermore, the feature data table is configured in row mode, and the offline feature table is configured in column mode. When outputting data from the feature data table, the feature data is processed by converting rows to columns, thereby improving the performance and response speed when querying feature data in TiDB. Attached Figure Description

[0024] Figure 1This is a schematic diagram illustrating the internal structure and application environment of an offline feature management system provided in one or more embodiments of this application;

[0025] Figure 2 This is a flowchart illustrating an offline feature management method provided in one or more embodiments of this application;

[0026] Figure 3 This is a schematic diagram of an offline feature processing flow based on Dataphin and TiDB provided in one or more embodiments of this application;

[0027] Figure 4 This is an internal structural diagram of a computer device provided in one or more embodiments of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0029] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0030] In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0031] The current solution for offline features in large-scale production models (hereinafter referred to as the traditional solution) mainly involves synchronizing relevant business data from the business database to Hive, then performing data cleaning and processing in Hive, and finally storing the processed data in HBase as an offline feature repository. As the amount of offline feature input and the amount of underlying business data become larger and more complex, the above traditional solution has at least the following shortcomings in actual use: (1) Components such as Hive and HBase in the traditional solution require cluster deployment, which results in high costs; (2) The traditional solution involves the integration and configuration of multiple components and technologies, which requires a high level of technical expertise and maintenance costs. This makes the implementation of related operations more dependent on technical personnel and inconvenient for business personnel to use and operate. For example, after synchronizing relevant business data to Hive in the traditional solution, data cleaning and processing are required in Hive. The data cleaning and processing mainly have the following functions: (1) removing sensitive information (such as mobile phone numbers and ID card numbers) from the business data. (1) Desensitize sensitive information such as user address, and use encryption algorithms such as MD5 to encrypt sensitive information; (2) Calculate new field information based on existing field information in business data, such as calculating the user's age based on the user's birthday field information; (3) Generate new data tables based on multiple data tables, such as calculating a new table user_order table based on the user table (user table) and the order table (order table). The fields in the user_order table can include some or all fields of the user table, some or all fields of the order table, and even new fields that need to be calculated based on the existing field information in the user table and / or the order table. The implementation of data cleaning and processing operations in traditional solutions involves writing code and using components in the Hive framework. Therefore, business personnel cannot use it directly and need to rely on technical personnel to write the corresponding code for implementing data cleaning and processing; (3) Components such as Hive and HBase have long delays when processing large-scale data. The delay in the data processing process will affect business decision-making and response speed, so it is not suitable for scenarios with high data real-time requirements; (4) Traditional solutions involve multiple components and data transmission processes, so there are certain problems in terms of data consistency and data synchronization.

[0032] To address the shortcomings of existing technologies, this application provides an offline feature management system. In some embodiments, such as... Figure 1 As shown, the offline feature management system includes a data management platform and a distributed relational database. In this embodiment, the distributed relational database serves as the offline feature library, in which multiple offline feature tables are pre-created. These offline feature tables are used to store feature data from the data management platform. The data management platform includes a data acquisition module, a data processing module, and a data transmission module. The following describes each module.

[0033] The data acquisition module is used to interface with business systems. These systems can be configured by business personnel according to their needs. There are typically multiple business systems, and the business data to be synchronized across each system is also configured by the business personnel based on their requirements. The data acquisition module automatically synchronizes the specified business data from each business system to the data management platform based on the information configured by the business personnel. Specifically, the specified business data from each business system can be synchronized to the data management platform in real time via Binlog. Binlog, or Binary Log, is a binary log file in a MySQL database used to record database change operations.

[0034] The data processing module is used for data cleaning and processing of business data. Business personnel can write data processing scripts according to actual business needs and submit them to the data management platform. The data processing module will then execute the scripts periodically. When executing the scripts, the data processing module cleans and processes the specified business data and stores the cleaned and processed feature data in a specified feature data table in the data management platform. Specifically, the data processing scripts written by business personnel need to specify the business data to be cleaned and processed, the cleaning and processing logic for that business data, and the feature data table used to store the cleaned and processed feature data.

[0035] Business personnel also need to pre-configure corresponding offline feature tables for each feature data table, so that the data transmission module can periodically transmit the data from each feature data table to the corresponding offline feature table in the distributed relational database.

[0036] In this embodiment, data cleaning and data processing are integrated into the data management platform. Business personnel configure the data source of feature data according to actual business needs, such as various business systems, specify the business data that needs to be synchronized, write corresponding data processing scripts, and configure the offline feature tables corresponding to each feature data table. The data management platform can then realize an automated offline feature processing process based on the data processing scripts. This process is simpler than the existing technology, improves data processing efficiency, and lowers the technical threshold for manual operation.

[0037] In some embodiments, Dataphin can be used to build the data management platform, and TiDB can be used to build the distributed relational database. Dataphin is a data platform product developed by Alibaba Group, providing end-to-end intelligent data construction and management services, including data synchronization, data transformation, data permissions, specification definition, model development, asset management, and data services. TiDB is an open-source distributed relational database developed by PingCAP, combining the advantages of traditional relational and non-relational databases. It supports Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP), and supports online horizontal scaling with excellent high availability. Accordingly, the data acquisition module of the data management platform in the above embodiments can be implemented using Dataphin's storage service, the data processing module can be implemented using Dataphin's data processing function, and the data transmission module can be implemented using Dataphin's transmission function.

[0038] Hive and HBase are both open-source middleware that only provide basic functionality. Implementing offline feature management using Hive+HBase requires long-term operation and maintenance by users (e.g., enterprises), resulting in high maintenance costs. Furthermore, in practice, users typically need to customize the entire process of offline feature data processing, as well as some functions within that process, such as data synchronization, data transformation, data recycling, and data permissions. Therefore, the technical cost of implementing traditional solutions is high. To address this, this embodiment selects Dataphin and TiDB to build the offline feature management system. The combination of Dataphin and TiDB enables a new offline feature management process that is simpler, more efficient, and more reliable than the offline feature management process implemented using the Hive+HBase combination. Moreover, since Dataphin and TiDB offer richer functionality, users do not need to develop corresponding functions during implementation, thus the technical cost of this embodiment is relatively low.

[0039] It should be noted that there are various data management platforms and distributed relational databases available, and the appropriate product can be selected based on the specific needs. For example, several data management platforms and distributed relational databases can be analyzed and tested based on actual requirements and the overall technology stack, and the final product can be selected based on the analysis and test results.

[0040] In some embodiments, the data processing scripts described above are SQL statements written in advance by business personnel in the Dataphin platform based on business needs; the number of the data processing scripts is usually multiple, but in some cases it may be just one.

[0041] In this embodiment, business personnel only need to write SQL to enable the data processing module to clean and process the specified business data and store the feature data in the corresponding feature data table, thus reducing the reliance on technical personnel.

[0042] For example, suppose a business requirement necessitates generating a new table based on the `user` and `order` tables. The `user` table contains fields such as `userId` (user ID), `name` (name), `phone` (phone number), and `registerTime` (registration time). The `order` table contains fields such as `userId`, `orderNo` (loan order code), `status` (status), `amount` (amount), `stage` (stage), and `orderTime` (order time). The SQL written by the business personnel could be as follows:

[0043] (1)

[0044] create table ifnot exists pdm_risk.user_order_statistic(

[0045] sample_row_num string comment'sequence row number',

[0046] userId string comment'user ID',

[0047] orderNumint comment'Total number of loan orders',

[0048] orderSuccessNumint comment'Number of successful loan orders'

[0049] orderFailNumint comment'Number of failed loan orders',

[0050] firstOrderTime date comment'First loan date'

[0051] `registerTime date comment'user registration time'`

[0052] )comment'User Order Statistics Table'partitioned by(ds string comment'Business date partition, format yyyymmdd')

[0053] ”;

[0054] (2)

[0055]

[0056]

[0057] Executing the above SQL(1) can create a table for statistical user order information, and partition it by date for easy data management and querying. Specifically, SQL(1) can create a new feature data table named pdm_risk.user_order_statistic (user order statistics table, hereinafter referred to as user_order_statistic table) (if the table already exists, no new table needs to be created). The table structure is: sample_row_num, userId, orderNum, orderSuccessNum, orderFailNum, firstOrderTime, registerTime; where sample_row_num, userId, orderNum, orderSuccessNum, orderFailNum, firstOrderTime, registerTime and other fields are all offline feature keys. And the table uses partitioned storage, the partition key is the ds field, the type is string, and the partition format is yyyymmdd (i.e., year, month, day). ds is the business date, which can be understood as the value of the ds field when executing this SQL every day is the date of that day.

[0058] The SQL(2) above can count the order information of each user in the partition on April 10, 2024 from the user table and the order table, including the total number of loan orders, the number of successful loan orders, the number of failed loan orders, the time of first loan and the user registration time, and insert these statistical results into the corresponding partition of the user_order_statistic table.

[0059] Furthermore, in some embodiments, the data processing module is also configured to respond to the first configuration instruction, configure the data processing scripts pre-submitted by relevant business personnel as data processing tasks, and configure the scheduler so that the scheduler executes the data processing tasks periodically.

[0060] In this embodiment, the data management platform is built on Dataphin, which can then utilize Dataphin's scheduling function to set data processing scripts as computation tasks and execute these tasks periodically through the scheduler, thereby ensuring timely updates of offline feature data.

[0061] Specifically, business users can use the scheduling function in the Dataphin platform to set the custom calculation SQL submitted by business users as a task (called a data processing task), and configure the scheduler according to actual needs, so that the scheduler can execute the corresponding data processing task regularly. For example, the scheduler can execute the corresponding data processing task at a specified time every day, week or month to ensure the timely update of offline feature data.

[0062] In addition, business personnel can divide the overall resource pool in the Dataphin platform, such as isolating resources between different businesses and departments; and set the upper limit of the number of CPUs that can be used for a single scheduled task, so as to optimize the scheduling of tasks, thereby improving the efficiency and stability of task execution and reducing operation and maintenance costs.

[0063] In some embodiments, the distributed relational database is built on TiDB. Traditional solutions store feature data in HBase, requiring dedicated management and maintenance, and are not conducive to data querying and management. TiDB, however, has excellent data storage and management capabilities, supporting distributed storage and querying. Therefore, business users can configure corresponding output pipelines for each feature data table in the Dataphin platform, and then the data transmission module can transfer feature data from each feature data table to the corresponding offline feature table in TiDB. This improves the efficiency of feature data storage and management.

[0064] Business users can create an offline feature table for each subject domain in TiDB. The table structure can include fields such as subject domain field, offline feature key, offline feature value, and business date to support efficient management and querying of feature data.

[0065] Furthermore, in some embodiments, the distributed relational database has an interface (hereinafter referred to as the query interface) for batch querying offline features, which is used to query data in the offline feature table in batches.

[0066] For example, the query SQL for the query interface can be as follows:

[0067]

[0068] The above query SQL is used to select data from the online_feature table for a specific user and offline feature key, and generate corresponding row numbers for the selected data.

[0069] Furthermore, each feature data table is a row-based data table, while each offline feature table is a column-based data table. That is, the feature data tables are configured in row mode, while the offline feature tables are configured in column mode. Accordingly, when the data transmission module transmits data from the feature data tables to the corresponding offline feature tables in the distributed relational database, it converts the data in the feature data tables into column mode. Specifically, it converts each row of data in the feature data tables into column-mode data, and then outputs the converted data to the corresponding offline feature tables. The table structure of the offline feature tables can be set according to actual needs; for example, its table structure typically includes some or all of the fields from the corresponding feature data tables.

[0070] The following example illustrates the row-to-column transformation operation.

[0071] For example, feature data from the `user_order_statistic` table needs to be transferred to an offline feature table in TiDB. As mentioned above, the `user_order_statistic` table has the following structure: `sample_row_num`, `userId`, `orderNum`, `orderSuccessNum`, `orderFailNum`, `firstOrderTime`, and `registerTime`. The offline feature table can have the following structure: subject field, offline feature key, offline feature value, and business date. The feature data for each row in the `user_order_statistic` table needs to be converted to columnar format before being stored in the offline feature table.

[0072] For example, after the transmission is completed, the data in the offline feature table can be as shown in Table 1:

[0073] Table 1:

[0074] Subject area Offline feature key Offline feature values Business Dates userId orderNum 10 20240410 userId orderSuccessNum 11 20240410 userId orderFailNum 12 20240410 userId firstOrderTime 13 20240410 userId registerTime 14 20240410

[0075] In traditional solutions, the performance of the query interface may be limited by the data storage and retrieval method. However, this embodiment uses TiDB to store offline feature data. Furthermore, by configuring the feature data table in row mode and the offline feature table in column mode, more efficient data retrieval and processing can be achieved based on the advantages of TiDB, thereby improving the performance and response speed of the query interface.

[0076] Based on the same inventive concept, this application also provides an offline feature management method. In some embodiments, this method can be applied to the offline feature management system provided in any of the above embodiments, which includes, for example: Figure 2 The steps shown are steps S110 to S130.

[0077] Step S110: Collect business data from the specified business system.

[0078] Step S120: Periodically execute pre-written data processing scripts to clean and process specified business data and store the cleaned and processed feature data into the corresponding pre-created feature data table.

[0079] Step S130: Periodically transfer the data from the feature data table to the corresponding pre-created offline feature table.

[0080] In some embodiments, this offline feature management system is built based on Dataphin and TiDB. In this case, the offline feature processing flow can be found in [reference needed]. Figure 3 As shown, the Dataphin platform's storage service interfaces with multiple business systems, such as the user center and order center, and synchronizes specified business data from these systems to the Dataphin platform in real time via Binlog. Business personnel submit pre-written SQL statements to the Dataphin platform based on actual business needs, and utilize the platform's scheduling function to configure each SQL statement as a data processing task, setting corresponding scheduling information (indicating task execution time) for each task. The Dataphin platform's data processing function periodically executes each data processing task according to its scheduling information. Each data processing task creates a feature data table and stores the feature data obtained from cleaning and processing the specified business data into this table. When the feature data table already exists, executing the data processing task ensures timely updates of the feature data. Business personnel also need to pre-configure corresponding output pipelines for each feature data table, so that the Dataphin platform's transmission function automatically and periodically transmits the feature data from each feature data table to the corresponding offline feature table in TiDB.

[0081] For specific limitations on the offline feature management method, please refer to the limitations on the offline feature management system above, which will not be repeated here. It should also be noted that, unless explicitly stated herein, the execution order of the steps included in the offline feature management method provided in any of the above embodiments is not strictly limited; these steps can be executed in other orders. Furthermore, at least some of these steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0082] This application provides a computer device in some embodiments, the internal structure of which can be shown as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores configuration data configured by business personnel, submitted data processing scripts (such as custom SQL calculations), and other data; the specific stored data may also be as defined in the above method embodiments. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an offline feature management method.

[0083] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0084] This embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the steps of the offline feature management method provided in any of the above embodiments.

[0085] This application also provides a computer-readable storage medium, in some embodiments of which a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of the offline feature management method provided in any of the above embodiments.

[0086] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0087] Those skilled in the art will understand that implementing all or part of the processes in the above method embodiments can be accomplished by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink, DRAM (SLDRAM), RAMbus, direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0088] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An offline feature management system, characterized in that, The system includes a data management platform built on Dataphin and a distributed relational database built on TiDB. The data management platform includes a data acquisition module, a data processing module, and a data transmission module. The data acquisition module is implemented through Dataphin's storage service, the data processing module is implemented through Dataphin's data processing function, and the data transmission module is implemented through Dataphin's transmission function. The distributed relational database is used to manage multiple pre-created offline feature tables; The data acquisition module is used to collect business data from designated business systems; there are multiple business systems, and the designated business data from each business system is synchronized to the data management platform in real time via Binlog; The data processing module is used to periodically execute pre-written data processing scripts to clean and process the specified business data collected by the data acquisition module and store the cleaned and processed feature data into corresponding pre-created feature data tables. The data processing scripts are SQL statements pre-written by business personnel based on business needs. There are multiple data processing scripts. The business personnel of the data management platform pre-configure corresponding output pipelines for each feature data table. The data transmission module is used to periodically transmit the data from each of the feature data tables to the corresponding offline feature tables in the distributed relational database; The data processing module is also used to respond to the first configuration instruction, configure the data processing scripts submitted in advance by relevant business personnel as data processing tasks, and configure the scheduler so that the scheduler executes the data processing tasks periodically. Each of the aforementioned feature data tables is a row-based data table; each of the aforementioned offline feature tables is a column-based data table. When the data transmission module transmits data from the feature data table to the corresponding offline feature table in the distributed relational database, it converts the data in the feature data table into column mode and outputs the converted data to the corresponding offline feature table.

2. The system as described in claim 1, characterized in that, The distributed relational database has an interface for batch querying offline features.

3. An offline feature management method, characterized in that, Applied to the offline feature management system as described in any one of claims 1-2, the method comprises: Collect business data from the designated business system; Regularly execute pre-written data processing scripts to clean and process specified business data and store the cleaned and processed feature data into the corresponding pre-created feature data table; Periodically transfer the data from each of the aforementioned feature data tables to the corresponding pre-created offline feature table.

4. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of claim 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of claim 3.

Images