Real-time feature data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a real-time feature data processing method and device, computer equipment and a storage medium. The method comprises the following steps: synchronizing business data of a plurality of business libraries to a business data table, calculating a pre-calculation result of a real-time feature according to the business data in the business data table, and writing the pre-calculation result into a pre-calculation feature result table; the method comprises the steps of synchronously updating a service data table when data change occurs in a service library, capturing corresponding change data when the service data table is updated, converting the change data into a Kafka message, sending the Kafka message to a specified theme of a Kafka message system, pulling a new Kafka message from the specified theme by using a stream processing engine Flink, processing the new Kafka message to obtain real-time feature update information, and sending the real-time feature update information to the service library. A pre-calculation characteristic result table is updated based on the information; and in response to a feature query request of the application program, obtaining feature data matched with the request from the pre-calculation feature result table and returning the feature data to the application program. The real-time performance of the feature data can be improved, the complexity and the maintenance cost of the system are reduced, and the feature access efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of feature data processing, and in particular to a real-time feature data processing method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development of big model technology, the real-time demand for feature data of big models is showing an increasing trend. Compared with ordinary models, the real-time feature input and underlying business data of big models are larger and more complex. In the face of these challenges, efficient data processing and efficient data acquisition methods are urgently needed to meet the needs of big models for larger and more complex data.

[0003] At present, the solutions for producing real-time features of large models are mainly based on the combination of Hive (a data warehouse tool based on Hadoop) and HBase (a distributed, column-oriented open source database). These solutions usually involve synchronizing business data to Hive, then cleaning and processing the data in Hive, and finally storing the processed data in HBase as a feature repository. However, these solutions have some shortcomings, mainly including: these solutions involve the integration and configuration of multiple components and technologies, requiring a high level of technology and maintenance costs, which makes data processing operations dependent on technical personnel and is not conducive to the use and operation of business personnel; these solutions usually have long delays when processing large-scale data. The delay in the data processing process will affect business decisions and response speed, and are not suitable for scenarios with high requirements for data real-time performance. Summary of the invention

[0004] In view of the above-mentioned deficiencies or shortcomings, the present application provides a real-time feature data processing method, apparatus, computer equipment and storage medium. The embodiments of the present application can improve the real-time performance of feature data, reduce the complexity and maintenance cost of the system, and improve the efficiency of feature data acquisition.

[0005] The present application provides a real-time feature data processing method according to a first aspect. In some embodiments, the method includes:

[0006] Synchronize the business data of multiple business libraries to the business data table of the distributed relational database TiDB, calculate the pre-calculated results of real-time features based on the business data in the business data table and write them into the pre-calculated feature result table of the distributed relational database TiDB; the table structure of the pre-calculated feature result table includes subject domain fields, feature identification fields and feature value fields;

[0007] When data changes occur in any business database, the business data table is updated synchronously, the corresponding change data when the business data table is updated is captured, the change data is converted into Kafka messages and sent to the specified topic of the Kafka message system, and the stream processing engine Flink is used to pull new Kafka messages from the specified topic of the Kafka message system, and the new Kafka messages are processed according to the preset feature data processing rules to obtain real-time feature update information, and the pre-calculated feature result table is updated based on the real-time feature update information;

[0008] In response to receiving a feature query request from an application, feature data matching the feature query request is obtained from a pre-computed feature result table and returned to the application.

[0009] In some embodiments, synchronizing the business data of multiple business libraries to the business data table of the distributed relational database TiDB includes:

[0010] Synchronize the full business data of each business database to the business data table;

[0011] When data changes occur in any business database, the business data table is updated synchronously, including:

[0012] Monitor the Binlog file of each business library;

[0013] When an update is detected in the Binlog file of any business library, the incremental content of the Binlog file of the business library is obtained and a business data update instruction is generated according to the incremental content of the Binlog file of the business library;

[0014] Execute business data update instructions in the distributed relational database TiDB to update the business data table.

[0015] In some embodiments, capturing the corresponding change data when the business data table is updated includes:

[0016] Monitor the Binlog file of the distributed relational database TiDB;

[0017] When an update is detected in the Binlog file of the distributed relational database TiDB, the change data corresponding to the update of the business data table is extracted from the incremental content of the Binlog file of the distributed relational database TiDB.

[0018] In some embodiments, the real-time feature update information includes the operation type, the subject domain information corresponding to the feature to be updated, the feature identification information, and the changed feature value; the real-time feature update information is obtained by processing the new Kafka message according to the preset feature data processing rules, including:

[0019] Parse the new Kafka message to obtain the changed data; the changed data includes the table ID, field ID, change type and changed field value corresponding to the changed business data;

[0020] When the change type indicates that the change operation is insert or update, the subject domain information and feature identification information corresponding to the feature to be updated are determined based on the table identifier and field identifier corresponding to the changed business data, and the changed feature value corresponding to the feature to be updated is determined based on the changed field value included in the change data.

[0021] In some embodiments, updating the pre-calculated feature result table based on the real-time feature update information includes:

[0022] Constructing a feature data update instruction based on the operation type, subject domain information, feature identification information and changed feature value included in the real-time feature update information;

[0023] Execute feature data update instructions in the distributed relational database TiDB to update the pre-computed feature result table.

[0024] In some embodiments, the feature query request includes subject domain information and feature identification information; acquiring feature data matching the feature query request from the pre-calculated feature result table includes:

[0025] Using the subject domain information and feature identification information included in the feature query request as query conditions to query the pre-calculated feature result table;

[0026] Feature data matching the query condition in the pre-calculated feature result table is obtained as feature data matching the feature query request.

[0027] According to a second aspect, the present application provides a real-time feature data processing device. In some embodiments, the device includes:

[0028] The initial processing module is used to synchronize the business data of multiple business libraries to the business data table of the distributed relational database TiDB, calculate the pre-calculated results of real-time features based on the business data in the business data table, and write them into the pre-calculated feature result table of the distributed relational database TiDB;

[0029] The incremental update module is used to synchronously update the business data table when data changes occur in any business database, capture the corresponding change data when the business data table is updated, convert the change data into Kafka messages and send them to the specified topic of the Kafka message system, use the stream processing engine Flink to pull new Kafka messages from the specified topic of the Kafka message system, process the new Kafka messages according to the preset feature data processing rules to obtain real-time feature update information, and update the pre-calculated feature result table based on the real-time feature update information;

[0030] The query request processing module is used to, in response to receiving a feature query request from an application, obtain feature data matching the feature query request from a pre-calculated feature result table and return the feature data to the application.

[0031] In some embodiments, the incremental update module includes:

[0032] The business data update submodule is used to synchronously update the business data table when data changes occur in any business database;

[0033] The data change message output submodule is used to capture the corresponding change data when the business data table is updated, convert the change data into Kafka messages and send them to the specified topic of the Kafka message system.

[0034] The stream processing computing submodule is used to use the stream processing engine Flink to pull new Kafka messages from the specified topic of the Kafka message system, process the new Kafka messages according to the preset feature data processing rules to obtain real-time feature update information, and update the pre-calculated feature result table based on the real-time feature update information.

[0035] According to a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the real-time feature data processing method provided in any of the above embodiments are implemented.

[0036] According to a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the real-time feature data processing method provided in any of the above embodiments are implemented.

[0037] In the above-mentioned embodiment of the present application, a new pre-calculated real-time feature data processing process is provided in combination with the distributed relational database TiDB, the distributed publish-subscribe messaging system Kafka and the stream processing engine Apache Flink. In this process, the business data of multiple business libraries are first synchronized to the business data table of the distributed relational database TiDB, and the pre-calculated results of the real-time features are calculated based on the business data in the business data table and written into the pre-calculated feature result table of the distributed relational database TiDB. After that, when data changes occur in any business library, the business data table is updated synchronously, the corresponding change data when the business data table is updated is captured, and the change data is converted into Kafka messages and sent to the specified topic of the Kafka messaging system. After that, message processing is performed in the stream processing engine Flink, that is, new Kafka messages are pulled from the specified topic of the Kafka messaging system. According to the preset feature data processing rules, new Kafka messages are processed to obtain real-time feature update information, and the pre-calculated feature result table is updated based on the real-time feature update information. This process can update the pre-calculated feature result table in real time when the business data is updated, thereby improving the real-time performance of the feature data. Moreover, when the application has feature data retrieval requirements, the matching feature data can be directly queried in the pre-calculated feature result table without accessing multiple business libraries, thus reducing the access frequency of the business library and improving the data retrieval efficiency. In addition, the table structure of the pre-calculated feature result table is very simple, and only includes the subject domain field, feature identification field, and feature value field, which is conducive to further improving the query efficiency of feature data and controlling the query speed within 10 milliseconds, thereby meeting actual business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a real-time feature data processing method provided by the present application according to one or more embodiments;

[0039] Figure 2 An application environment diagram of a real-time feature data processing method provided by the present application according to one or more embodiments;

[0040] Figure 3 A structural block diagram of a real-time feature data processing device provided by the present application according to one or more embodiments;

[0041] Figure 4 This is a diagram of the internal structure of a computer device provided by the present application according to one or more embodiments. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0043] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0044] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0045] In response to the shortcomings or defects of the relevant technologies, the present application provides a real-time feature data processing method, which combines the distributed relational database TiDB, the distributed publish-subscribe messaging system Kafka (referred to as the Kafka messaging system) and the stream processing engine Apache Flink (referred to as the stream processing engine Flink), and provides a new pre-calculated real-time feature data processing process, which can synchronously update the pre-calculated feature result table at a very fast speed when the business data is updated, thereby improving the real-time performance of the feature data in the pre-calculated feature result table; and, when the application has a feature data acquisition requirement, it can directly query the matching feature data in the pre-calculated feature result table without accessing multiple business libraries, thereby reducing the access frequency of the business library and improving the feature data acquisition efficiency; in addition, the table structure of the pre-calculated feature result table is very simple, and only includes the subject domain field, the feature identification field and the feature value field, which is conducive to further improving the query efficiency of the feature data and controlling the query speed within 10 milliseconds, thereby meeting actual business needs.

[0046] In some embodiments, the real-time feature data processing method includes: Figure 1 The steps shown below are applied to Figure 2 The application environment shown is used as an example for explanation.

[0047] S110: Synchronize the business data of multiple business libraries to the business data table of the distributed relational database TiDB, calculate the pre-calculated results of the real-time features according to the business data in the business data table, and write them into the pre-calculated feature result table of the distributed relational database TiDB.

[0048] The execution subject of the real-time feature data processing method can be Figure 2 The data processing system in the application environment shown in the figure includes the distributed relational database TiDB, the distributed publish-subscribe messaging system Kafka, and the stream processing engine Flink. TiDB is an open source distributed relational database that has the advantage of being compatible with the MySQL protocol and supports online horizontal expansion. It has excellent high availability and can effectively cope with the challenges of large-scale data and complex data structures, ensuring efficient management and reliable storage of data. Kafka is an open source distributed publish-subscribe messaging system used to build real-time data stream processing pipelines and applications. It has the characteristics of high throughput and low latency. It can process hundreds of thousands of messages per second with a latency of only a few milliseconds. It also supports horizontal hot expansion and high concurrency, supporting thousands of clients to read and write simultaneously. Flink is a framework and distributed stream processing engine for stateful computing of unbounded and bounded data streams. It can run in all common cluster environments and perform computing at memory execution speed and at any scale.

[0049] Before updating the feature data in real time, the data processing system first connects with multiple business libraries specified by the user based on actual business needs. The business library refers to the database of the business system, such as the database of the user center, order center, and other business systems. In some embodiments, the business data of multiple business libraries are synchronized to the business data table of the distributed relational database TiDB, including: synchronizing the full amount of business data of each business library to the business data table.

[0050] Storing the data of each business library in the business data table of the distributed relational database TiDB first helps to simplify the subsequent data processing logic and avoid the complexity of reading data directly from multiple heterogeneous systems. In addition, in the process of writing the business data of each business library into the business data table, the business data can be cleaned and format converted based on actual business needs to ensure the quality of messages transmitted to the Kafka message system. When storing the business data of multiple business libraries in the distributed relational database TiDB, different strategies can be used to organize the data according to actual business needs and different data structures. For example, the business data of different business libraries can be stored in different business data tables. For another example, all business data can be stored in one business data table. For another example, the business data table can be designed according to different business fields, such as storing user information, order information, payment information, etc. in different business data tables.

[0051] When synchronizing the business data of each designated business database to the distributed relational database TiDB, the business data of each business database can be processed according to actual business needs. Specifically, the data preprocessing logic can be configured, such as filtering out data in certain fields in the business data, processing the field values ​​of one or some fields in the business data according to preset calculation rules (such as addition, subtraction, division, etc.) to obtain the field value of the new field (referring to the field that does not exist in the business database) (for example, subtracting the user's date of birth field value in the user information table from the current date to obtain the user's age, and the user's date of birth is the new field).

[0052] After the data in the business data table is initialized, all existing business data in the business data table are fully analyzed to calculate the corresponding feature data, and these feature data are written in batches to the pre-calculated feature result table, and then the feature data in the pre-calculated feature result table can be updated in real time through step 120. Here, update refers to insert, update and delete.

[0053] The table structure of the pre-calculated feature result table adopts a simple format. The structure of the table only includes the subject domain field, feature identification field and feature value field, which helps to improve the efficiency of feature extraction.

[0054] S120: When data changes occur in any business database, the business data table is updated synchronously, the corresponding change data when the business data table is updated is captured, the change data is converted into Kafka messages and sent to the specified topic of the Kafka message system, and the stream processing engine Flink is used to pull new Kafka messages from the specified topic of the Kafka message system, and the new Kafka messages are processed according to the preset feature data processing rules to obtain real-time feature update information, and the pre-calculated feature result table is updated based on the real-time feature update information.

[0055] By using the Kafka messaging system for real-time data transmission and the stream processing engine Flink for efficient stream processing, the real-time performance of data processing has been significantly improved. The entire process of data from the business database to the pre-calculated feature result table is completed almost in real time, meeting the needs of specific business scenarios for high-real-time data, thereby supporting faster business decisions and responses. In addition, TiDB, Kafka, and Flink are all technologies with high scalability and high availability, which can ensure that the data processing system can cope with the processing needs of large-scale data. TiDB's online horizontal expansion capabilities and Kafka's high throughput and low latency characteristics can also ensure the stability and efficiency of the data processing system when processing large-scale and complex data.

[0056] In some embodiments, when data changes occur in any business library, the business data table is updated synchronously, including: monitoring the Binlog file of each business library; when an update is detected in the Binlog file of any business library, obtaining the incremental content of the Binlog file of the business library and generating a business data update instruction based on the incremental content of the Binlog file of the business library; executing the business data update instruction in the distributed relational database TiDB to update the business data table.

[0057] Each business database needs to enable the Binlog function, so that the business database will record the relevant data in the Binlog file after performing specific database operations (such as insert, update, delete, etc.). The data synchronization tool will keep the data of the business database consistent with the data of the business data table based on the Binlog files of each business database.

[0058] The incremental content of the Binlog file of the business library records the operation information of the latest database operation performed by the business library. The operation information may include the operation type (such as insert, update, delete, etc.), table information, field information, field values ​​of the field corresponding to the field information before and after the operation, etc. The latest database operation performed by the business library may be one or more times. The data synchronization tool will obtain the operation information of each database operation from the incremental content, and generate operation instructions for updating the business data table applicable to the distributed relational database TiDB, i.e., business data update instructions, based on the operation information of each database operation. These instructions are then executed in the distributed relational database TiDB to keep the data of the business library consistent with the data of the business data table. For example, assuming that the total number of loan orders of users is recorded in both business library A and business data table B, if business library A performs one database operation at a certain moment, specifically updating the total number of loan orders of user Xiao Ming from 10 to 11, then the data synchronization tool needs to generate an operation instruction for updating the total number of loan orders of Xiao Ming in business data table B to 11 based on the operation information of the database operation, and then execute the operation instruction to synchronously update the total number of loan orders of Xiao Ming in the business data table.

[0059] In some embodiments, capturing the change data corresponding to an update of a business data table includes: monitoring the Binlog file of a distributed relational database TiDB; upon monitoring an update of the Binlog file of the distributed relational database TiDB, extracting the change data corresponding to the update of the business data table from the incremental content of the Binlog file of the distributed relational database TiDB.

[0060] You can use a Kafka producer (which can be a custom program or an off-the-shelf tool such as Canal, Debezium, etc.) to listen to the Binlog file of the distributed relational database TiDB. When the Binlog file of TiDB is updated, get the incremental content of the file, and then extract the change data related to the update of the business data table, that is, the change data corresponding to the update of the business data table.

[0061] The change data at least includes the table identifier, field identifier, change type and field value after the change corresponding to the business data that has changed. The change type is used to indicate the type of database operation that has been performed, which can be insert, update, delete and other types. The change data can also include the field value before the change. If the change type is insert, the field value before the change is empty, and the field value after the change is the latest field value of the relevant field; if the change type is update, the field value before the change is the field value before the relevant field is updated, and the field value after the change is the field value after the relevant field is updated (specifically the latest field value of the relevant field); if the change type is delete, the field value before the change is the historical field value of the relevant field (that is, the field value before being deleted), and the field value after the change is empty.

[0062] After obtaining the change data, the Kafka producer converts the change data into a Kafka message in a format suitable for the Kafka message system and sends it to the specified topic (i.e. the specified Topic) of the Kafka message system. There can be multiple topics, and different topics correspond to different business areas. There is no special limit on the specific number of topics, and the number of topics can be determined or adjusted according to actual business needs.

[0063] In some embodiments, real-time feature update information includes operation type, subject domain information corresponding to the feature to be updated, feature identification information and changed feature value; accordingly, the new Kafka message is processed according to preset feature data processing rules to obtain real-time feature update information, including: parsing the new Kafka message to obtain change data; the change data includes the table identifier, field identifier, change type and changed field value corresponding to the changed business data; when the change type indicates that the change operation is insert or update, the subject domain information (which can be the subject domain identifier) ​​and feature identification information corresponding to the feature to be updated are determined according to the table identifier and field identifier corresponding to the changed business data, and the changed feature value corresponding to the feature to be updated is determined according to the changed field value included in the change data.

[0064] Kafka messages in a specified topic can be consumed by Kafka consumers, where multiple Kafka consumers can be set, and different Kafka consumers correspond to different specified topics. The data processing logic of Kafka messages in different specified topics can be different. The data processing logic refers to the operation rules for calculating feature data based on business data. For example, the data processing logic corresponding to a specified topic includes adding the field values ​​of field A and field B of the business data table to obtain the field value of feature C, such as adding the number of successful and failed borrowing times of a user to obtain the total number of borrowing times of the user.

[0065] Determining the subject domain information and feature identification information corresponding to the feature to be updated based on the table identifier and field identifier corresponding to the changed business data can be to use the subject domain corresponding to the specified topic where the Kafka message is located as the subject domain corresponding to the feature to be updated, and determine the feature identifier corresponding to the table identifier and field identifier of the business data according to the preset mapping rules.

[0066] The feature data in the pre-calculated feature result table is calculated based on the business data in the business data table. Therefore, when the data in the business data table changes, the pre-calculated feature result table also needs to be updated synchronously to ensure the accuracy of the feature data.

[0067] The change data may include the change information of the latest one or more data changes in the business data table, and the change information includes the table identifier, field identifier, change type (corresponding to the operation type, such as insert, update, delete, etc.), and post-change field value of the business data table corresponding to the changed business data (business data is the field value of a certain field). The change information may also include the field value before the change.

[0068] Kafka consumers generate operation instructions for updating the pre-calculated feature result table, i.e., feature data update instructions, applicable to the distributed relational database TiDB based on each change information in the change data. These instructions are then executed in the distributed relational database TiDB to synchronously update the data in the pre-calculated feature result table after the data in the business data table changes.

[0069] Accordingly, the pre-calculated feature result table is updated based on the real-time feature update information, including: constructing a feature data update instruction based on the operation type, subject domain information, feature identification information and changed feature value included in the real-time feature update information; and executing the feature data update instruction in the distributed relational database TiDB to update the pre-calculated feature result table.

[0070] For example, a business data table records the number of successful and failed borrowing times of a user, and the pre-calculated feature result table records the total number of borrowing times of the user. Assume that at time A, the number of successful and failed borrowing times of user Xiao Ming in the business data table are 5 and 1 respectively, and the total number of borrowing times of user Xiao Ming in the pre-calculated feature result table is 6. At time B, the number of successful borrowing times of user Xiao Ming in the business data table is updated to 6. Then, Kafka consumption can obtain the changed field value (6) of the number of successful borrowing times of user Xiao Ming and query the latest field value (1) of the number of failed borrowing times of user Xiao Ming in the business data table. The two field values ​​are added together to obtain the latest feature value (7) of the total number of borrowing times of user Xiao Ming. Then, a feature data operation instruction is generated to update the total number of borrowing times of user Xiao Ming in the pre-calculated feature result table to 7. The instruction is executed in the distributed relational database TiDB to update the total number of borrowing times of user Xiao Ming in the pre-calculated feature result table to 7.

[0071] S130: In response to receiving a feature query request from an application, acquiring feature data matching the feature query request from a pre-computed feature result table and returning the feature data to the application.

[0072] The feature data acquisition application refers to an application program that needs to query feature data. The feature query request includes subject domain information and feature identification information; accordingly, the feature data matching the feature query request is obtained from the pre-calculated feature result table, including: using the subject domain information and feature identification information included in the feature query request as query conditions to query the pre-calculated feature result table; obtaining the feature data matching the query conditions in the pre-calculated feature result table as the feature data matching the feature query request.

[0073] After extracting the subject domain information and feature identification information from the feature query request, you can use this information as query conditions to construct a query statement, and then execute the query statement in TiDB to query for matching feature data. For example, the query statement can be as follows:

[0074] "Select feature_key,feature_value

[0075] From feature_result

[0076] Where identifier = 'xxx'

[0077] And feature_key in('a','b','c')".

[0078] The above query statement is used to select the feature_key and feature_value fields from the feature_result table. The query condition is that the identifier field value is xxx and the feature_key field value is any one of a, b, or c. feature_result represents the pre-calculated feature result table, identifier represents the subject domain field, feature_key represents the feature identifier field, and feature_value represents the feature value field.

[0079] By pre-calculating the feature values ​​of real-time features and storing them in a simple TiDB pre-calculated feature result table, the feature extraction process of the application can be optimized. When extracting features, the application does not need to access multiple business libraries, nor does it need to execute specific calculation logic (for example, when it is necessary to obtain the total number of loans of a user, the total number of loans of the specified user can be directly queried from the pre-calculated feature result table, without obtaining the number of successful and failed loans of the specified user from the business library, and then adding the two to get the total number of loans). It only needs to access the distributed relational database TiDB to directly obtain the required feature data, which reduces the frequent access to the business library and improves the efficiency of feature extraction.

[0080] It should be noted that, in the embodiments of the present application, the data related to user information or user data (for example, historical search behavior, etc.) need to be obtained and processed after the user's authorization and consent. When the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions. It should also be noted that, with respect to the various steps included in the real-time feature data processing method provided in any of the above embodiments, unless there is a clear description in this article, the execution of these steps is not strictly restricted in order, and these steps can be executed in other orders. Moreover, at least a part of these steps can include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequentially, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0081] Based on the same inventive concept, the present application also provides a real-time feature data processing device. In some embodiments, Figure 4 As shown, the real-time feature data processing device includes the following modules:

[0082] The initial processing module 110 is used to synchronize the business data of multiple business libraries to the business data table of the distributed relational database TiDB, calculate the pre-calculated results of the real-time features according to the business data in the business data table, and write them into the pre-calculated feature result table of the distributed relational database TiDB;

[0083] The incremental update module 120 is used to synchronously update the business data table when data changes occur in any business library, capture the corresponding change data when the business data table is updated, convert the change data into Kafka messages and send them to the specified topic of the Kafka message system, use the stream processing engine Flink to pull new Kafka messages from the specified topic of the Kafka message system, process the new Kafka messages according to the preset feature data processing rules to obtain real-time feature update information, and update the pre-calculated feature result table based on the real-time feature update information;

[0084] The query request processing module 130 is used to, in response to receiving a feature query request from an application, obtain feature data matching the feature query request from a pre-calculated feature result table and return the feature data to the application.

[0085] In some embodiments, the incremental update module 120 includes:

[0086] The business data update submodule is used to synchronously update the business data table when data changes occur in any business database;

[0087] The data change message output submodule is used to capture the corresponding change data when the business data table is updated, convert the change data into Kafka messages and send them to the specified topic of the Kafka message system.

[0088] The stream processing computing submodule is used to use the stream processing engine Flink to pull new Kafka messages from the specified topic of the Kafka message system, process the new Kafka messages according to the preset feature data processing rules to obtain real-time feature update information, and update the pre-calculated feature result table based on the real-time feature update information.

[0089] In some embodiments, the operation of the initial processing module 110 synchronizing the business data of multiple business libraries to the business data table of the distributed relational database TiDB includes: synchronizing the full amount of business data of each business library to the business data table.

[0090] The business data update submodule synchronously updates the business data table when data changes occur in any business library, including: monitoring the Binlog file of each business library; when an update is detected in the Binlog file of any business library, obtaining the incremental content of the Binlog file of the business library and generating business data update instructions based on the incremental content of the Binlog file of the business library; executing the business data update instructions in the distributed relational database TiDB to update the business data table.

[0091] In some embodiments, the data change message output submodule captures the operation of the corresponding change data when the business data table is updated, including: monitoring the Binlog file of the distributed relational database TiDB; when monitoring the update of the Binlog file of the distributed relational database TiDB, extracting the corresponding change data when the business data table is updated from the incremental content of the Binlog file of the distributed relational database TiDB.

[0092] In some embodiments, real-time feature update information includes operation type, subject domain information corresponding to the feature to be updated, feature identification information and changed feature value; accordingly, the stream processing calculation submodule processes the new Kafka message according to the preset feature data processing rules to obtain the operation of real-time feature update information, including: parsing the new Kafka message to obtain the change data; the change data includes the table identifier, field identifier, change type and changed field value corresponding to the changed business data; when the change type indicates that the change operation is insert or update, the subject domain information and feature identification information corresponding to the feature to be updated are determined according to the table identifier and field identifier corresponding to the changed business data, and the changed feature value corresponding to the feature to be updated is determined according to the changed field value included in the change data.

[0093] In some embodiments, the stream processing computing submodule updates the operation of the pre-calculated feature result table based on the real-time feature update information, including: constructing a feature data update instruction based on the operation type, subject domain information, feature identification information and changed feature value included in the real-time feature update information; executing the feature data update instruction in the distributed relational database TiDB to update the pre-calculated feature result table.

[0094] In some embodiments, the feature query request includes subject domain information and feature identification information; the query request processing module 130 includes:

[0095] A query submodule, used to query the pre-calculated feature result table using the subject domain information and feature identification information included in the feature query request as query conditions;

[0096] The matching data acquisition submodule is used to obtain the feature data matching the query condition in the pre-calculated feature result table as the feature data matching the feature query request.

[0097] For the specific definition of the real-time feature data processing device, please refer to the definition of the real-time feature data processing method above, which will not be repeated here. Each module in the above-mentioned real-time feature data processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0098] The present application also provides a computer device. In some embodiments, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the clinical efficacy characteristic enhancement method provided in any of the above embodiments can be implemented.

[0099] Furthermore, in some embodiments, the internal structure diagram of the computer device may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store business data, real-time feature data and other data. The specific stored data can also refer to the definition in the above method embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a real-time feature data processing method is implemented.

[0100] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0101] The present application also provides a computer-readable storage medium. In some embodiments, a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the real-time feature data processing method provided in any of the above embodiments are implemented.

[0102] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0103] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink), DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0104] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A real-time feature data processing method, characterized in that: The method comprises: Synchronize the business data of multiple business libraries to the business data table of the distributed relational database TiDB, calculate the pre-calculated results of the real-time features according to the business data in the business data table and write them into the pre-calculated feature result table of the distributed relational database TiDB; the table structure of the pre-calculated feature result table includes a subject domain field, a feature identifier field and a feature value field; When data changes occur in any of the business libraries, the business data table is updated synchronously, the change data corresponding to the update of the business data table is captured, the change data is converted into a Kafka message and sent to a specified topic of the Kafka message system, a stream processing engine Flink is used to pull a new Kafka message from the specified topic of the Kafka message system, the new Kafka message is processed according to a preset feature data processing rule to obtain real-time feature update information, and the pre-calculated feature result table is updated based on the real-time feature update information; In response to receiving a feature query request from an application, feature data matching the feature query request is acquired from the pre-calculated feature result table and returned to the application.

2. The method according to claim 1, characterized in that Synchronize the business data of multiple business libraries to the business data tables of the distributed relational database TiDB, including: Synchronize the full amount of business data of each of the business libraries into the business data table; When data changes occur in any of the business libraries, the business data table is synchronously updated, including: Monitor the Binlog file of each business library; When an update is detected in the Binlog file of any of the business libraries, the incremental content of the Binlog file of the business library is obtained and a business data update instruction is generated according to the incremental content of the Binlog file of the business library; The business data update instruction is executed in the distributed relational database TiDB to update the business data table.

3. The method according to claim 1, characterized in that Capturing the corresponding change data when the business data table is updated, including: Monitor the Binlog file of the distributed relational database TiDB; When it is monitored that the Binlog file of the distributed relational database TiDB is updated, the change data corresponding to the update of the business data table is extracted from the incremental content of the Binlog file of the distributed relational database TiDB.

4. The method according to claim 1, characterized in that The real-time feature update information includes the operation type, the subject domain information corresponding to the feature to be updated, the feature identification information and the changed feature value; Processing the new Kafka message according to the preset feature data processing rules to obtain real-time feature update information includes: Parse the new Kafka message to obtain the change data; the change data includes the table identifier, field identifier, change type and changed field value corresponding to the changed business data; When the change type indicates that the change operation is insert or update, the subject domain information and feature identification information corresponding to the feature to be updated are determined based on the table identifier and field identifier corresponding to the changed business data, and the changed feature value corresponding to the feature to be updated is determined based on the changed field value included in the change data.

5. The method according to claim 1, characterized in that Updating the pre-calculated feature result table based on the real-time feature update information includes: Constructing a feature data update instruction based on the operation type, subject domain information, feature identification information and changed feature value included in the real-time feature update information; The feature data update instruction is executed in the distributed relational database TiDB to update the pre-calculated feature result table.

6. The method according to claim 5, characterized in that The feature query request includes subject domain information and feature identification information; Acquiring feature data matching the feature query request from the pre-calculated feature result table includes: Using the subject domain information and feature identification information included in the feature query request as query conditions to query the pre-calculated feature result table; Feature data matching the query condition in the pre-calculated feature result table is acquired as feature data matching the feature query request.

7. A real-time feature data processing device, characterized in that: The system comprises: An initial processing module, used to synchronize the business data of multiple business libraries to the business data table of the distributed relational database TiDB, calculate the pre-calculated results of real-time features based on the business data in the business data table, and write them into the pre-calculated feature result table of the distributed relational database TiDB; An incremental update module is used to synchronously update the business data table when data changes occur in any of the business libraries, capture the corresponding change data when the business data table is updated, convert the change data into Kafka messages and send them to the specified topic of the Kafka message system, use the stream processing engine Flink to pull new Kafka messages from the specified topic of the Kafka message system, process the new Kafka messages according to preset feature data processing rules to obtain real-time feature update information, and update the pre-calculated feature result table based on the real-time feature update information; The query request processing module is used to, in response to receiving a feature query request from an application, obtain feature data matching the feature query request from the pre-calculated feature result table and return the feature data to the application.

8. The system of claim 1, wherein: The incremental update module includes: A business data update submodule, used to synchronously update the business data table when data changes occur in any of the business libraries; The data change message output submodule is used to capture the corresponding change data when the business data table is updated, convert the change data into Kafka messages and send them to the specified topic of the Kafka message system. The stream processing computing submodule is used to use the stream processing engine Flink to pull new Kafka messages from the specified topic of the Kafka message system, process the new Kafka messages according to preset feature data processing rules to obtain real-time feature update information, and update the pre-calculated feature result table based on the real-time feature update information.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images