A real-time labeling method and device based on Flink

Abstract

The present invention discloses a real-time labeling method and device based on Flink, which belongs to the field of big data technology, and includes the following steps: creating an entity and an auxiliary table associated with the entity according to business data, and abstracting the entity into an object to be operated with an atomic label; dynamically configuring the label calculation type of the object to be operated, and setting the upstream data processing task that the object to be operated and the auxiliary table depend on; configuring label rules according to business requirements, auxiliary tables and atomic labels, and generating a structured query statement for real-time labeling according to the label rules and the label calculation type; submitting the structured query statement to the Yarn cluster to execute and obtain the target label corresponding to the entity, and storing the target label in a designated storage system. Compared with the existing technology, the present application has higher real-time performance, and basically the business data can be quickly labeled with the corresponding label after being sent down, so as to quickly respond to the changing market demand.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a real-time labeling method and device based on Flink. Background Art

[0002] In the current field of data processing and analysis, with the rapid development and complexity of business systems, the generation and updating of data labels has become an important foundation for supporting key business functions such as personalized recommendations, user behavior analysis, and precision marketing. Traditionally, these labels are often generated through offline batch processing, that is, data is regularly extracted from various business systems. After a series of complex calculations, the label results are finally generated and stored in the database or file system for subsequent business analysis or system use.

[0003] However, with the continuous expansion of business scenarios and the increasing requirements of users for real-time data, the traditional offline label generation method has gradually exposed its limitations. First, the offline processing mode means that there is a time delay in the update of labels, which cannot promptly reflect the latest changes in user or business status. This is particularly insufficient in scenarios that require highly real-time responses, such as real-time recommendations and risk control warnings. Secondly, with the growth of business volume and the explosive growth of data volume, the computing cost and storage requirements of offline processing have also increased dramatically, bringing a heavy burden to the system. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time labeling method and device based on Flink, so as to solve the problem in the prior art that offline generated labels cannot timely reflect the latest changes of user or business status due to update delays, and the problem that computing cost and storage requirements increase sharply with the growth of business volume.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] A real-time labeling method based on Flink in this application includes the following steps:

[0007] Create an entity and an auxiliary table associated with the entity according to the business data, and abstract the entity into an object to be operated with an atomic tag;

[0008] Dynamically configure the label calculation type of the object to be operated, and set the upstream data processing task that the object to be operated and the auxiliary table depend on;

[0009] Configure tag rules according to business requirements, the auxiliary table and the atomic tags, and generate structured query statements for real-time tagging according to the tag rules and the tag calculation type;

[0010] Submit the structured query statement to the Yarn cluster for execution to obtain the target tag corresponding to the entity, and store the target tag in a designated storage system.

[0011] Preferably, before creating an entity and an auxiliary table associated with the entity according to the business data, the method further includes:

[0012] Subscribe to a specific topic in Kafka, which contains business data pushed by the upstream business system.

[0013] Preferably, the label rules include label rules for generating derived rule labels, label rules for generating combined labels, and label rules for generating custom labels;

[0014] The configuration tag rule also includes selecting a synchronous marking object, where the synchronous marking object is an entity with a transfer attribute.

[0015] Preferably, generating a structured query statement for real-time labeling according to the label rule and the label calculation type includes:

[0016] Call the getCurrentDaoConnector method in the data access object connector factory to obtain the calculation engine used by the current application;

[0017] Based on the calculation engine, label rules and label calculation types, the getTaskBuildSql method in the row-level permission processing service interface is called to dynamically generate a structured query statement for real-time labeling.

[0018] Preferably, the dynamically generating a structured query statement for real-time tagging includes:

[0019] When the label rule includes a synchronous labeling object, a left join operation is performed using a temporal table based on processing time of Flink SQL and a corresponding structured query statement is generated;

[0020] When the label rules are used to generate combined labels, each source rule is used as an independent source table and the non-source rules are used as dimension tables to generate a multi-segment structured query statement.

[0021] Preferably, submitting the structured query statement to the Yarn cluster for execution includes:

[0022] Call the startTask method in the Flink task service to submit the structured query statement to the Yarn cluster for execution.

[0023] Preferably, the method further comprises:

[0024] The startTask method sets the task to run in perjob mode and supports starting, rerunning, retrying, and continuing the task.

[0025] A real-time labeling device based on Flink, comprising:

[0026] A construction module is used to create an entity and an auxiliary table associated with the entity according to business data, and abstract the entity into an object to be operated with an atomic tag;

[0027] A configuration module, used to dynamically configure the label calculation type of the object to be operated, and set the upstream data processing tasks that the object to be operated and the auxiliary table depend on;

[0028] A generation module, configured to configure label rules according to business requirements, the auxiliary table and the atomic label, and to generate a structured query statement for real-time labeling according to the label rules and the label calculation type;

[0029] The execution module is used to submit the structured query statement to the Yarn cluster for execution to obtain the target label corresponding to the entity, and store the target label in a designated storage system.

[0030] An electronic device comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a real-time labeling method based on Flink as described in any one of the above.

[0031] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement any one of the above-mentioned real-time labeling methods based on Flink when executed.

[0032] The present invention has the following beneficial effects:

[0033] In this application, the upstream business system pushes business data to a specific topic in Kafka through mogodb cdc. Flink consumes the business data in the Kafka topic in real time, calculates the results in real time according to the configured label rules, and writes the results to starrocks and kafka, which can be used by other business parties and downstream composite label configurations of the label system. Compared with existing technologies, it has higher real-time performance. Basically, the business data can be quickly tagged with the corresponding labels after it is sent down, so as to quickly respond to changing market demands. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of a real-time labeling method based on Flink provided in an embodiment of the present application;

[0035] Figure 2 is another flow chart of a real-time labeling method based on Flink provided in an embodiment of the present application;

[0036] Figure 3 It is a structural diagram of a real-time labeling device based on Flink provided in an embodiment of the present application;

[0037] Figure 4 It is a schematic diagram of an electronic device for implementing a real-time labeling method based on Flink provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] To make the technical solution of the present application clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The terms "first", "second", etc. in the claims and specification of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is only to describe the distinction method used when describing objects with the same attributes in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that a process, method, system, product or device containing a series of units is not necessarily limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] In some embodiments, Figure 1 As shown in the figure, a real-time labeling method based on Flink is provided, including the following steps:

[0040] S110, creating an entity and an auxiliary table associated with the entity according to the business data, and abstracting the entity into an object to be operated with an atomic tag;

[0041] S120, dynamically configure the label calculation type of the object to be operated, and set the upstream data processing task that the object to be operated and the auxiliary table depend on;

[0042] S130, configuring tag rules according to business requirements, auxiliary tables and atomic tags, and generating structured query statements for real-time tagging according to the tag rules and tag calculation types;

[0043] S140: Submit the structured query statement to the Yarn cluster for execution to obtain the target label corresponding to the entity, and store the target label in a designated storage system.

[0044] like Figure 2As shown in the figure, first create a topic in Kafka. After the creation is completed, the upstream business system can push business data to this specific topic through MongoDB CDC. MongoDB CDC is a change data capture mechanism of MongoDB database. It allows users to capture changes in data in the database in real time, such as insert, update, and delete operations, and publish these change events to a special collection or expose them to the outside through other means such as integrated event streams. The tag system can consume the business data in it by subscribing to this specific topic.

[0045] The tag system creates entities based on business data and abstracts the entities into an object to be operated. For example, a consumer group entity is defined, which contains all consumer records in the upstream business system, such as age, gender, purchase history, and average purchase amount, etc., corresponding to which a consumer group object is obtained. At the same time, a tag page is created for the consumer group object on the tag system. On this page, different tag calculation types can be checked and configured according to business needs to generate tags. The tag calculation types here include real-time tag calculation and offline tag calculation. Therefore, through the configured tag calculation type, the background will automatically generate the corresponding offline or streaming tag tasks, and once the tag calculation type is configured, it will not change later. In addition, a series of atomic tags will be generated based on the original data of the consumer, such as "Age: 25-30 years old", "Gender: Female", " Purchase history: electronic products" and "average purchase amount: high", etc. These atomic tags are entities and original dimensions of the objects to be operated, which can be used for the generation of subsequent derived rule tags and combined tags. For example, the derived rule tag "high-value electronic product consumers" is derived by combining the two atomic tags "purchase history: electronic products" and "average purchase amount: high", and the combined tag "25-30 years old female electronic product enthusiasts" is generated according to "age: 25-30 years old", "gender: female" and "purchase history: electronic products". Of course, the combined tag can also be composed of atomic tags and derived rule tags. For example, in this embodiment, the combined tag "25-30 years old female high-value electronic product consumers" is generated according to "age: 25-30 years old", "gender: female", "purchase history: electronic products" and "high-value electronic product consumers".

[0046] Creating an entity means creating a primary table. The data in the primary table supports two update methods: one is offline update based on Starroks, and the other is real-time update based on Kafka. Therefore, when creating an entity, an auxiliary table associated with the entity will also be created synchronously. The auxiliary table is a further modification and enhancement of the primary table. For example, if the primary table is a consumer, the auxiliary table can be the consumer membership information. That is, the auxiliary table can enrich the label selection dimension and application scope, and the auxiliary table is essentially the same as the entity primary table, which is a Starrocks or hive table.

[0047] Then, configure the upstream data processing tasks that the objects to be operated and the auxiliary tables depend on, and ensure that the upstream business system data is placed in the table before the data processing of the label system in the task dependency chain, where the task dependency chain is used to ensure the consistency and timeliness of the data. For example, the upstream business system in this embodiment is an order processing system, which is responsible for processing consumer orders and generating related transaction data. The label system relies on the data generated by the order processing system to generate consumer portraits. First, define Task A as an order processing task, which is responsible for receiving consumer order information, processing order logic such as inventory deduction and payment verification, and writing order data to the order database table; define Task B as a consumer data synchronization task, which regularly reads the latest order data from the order database table and extracts consumer-related data such as purchase history and purchase amount; define Task C as a consumer portrait generation task, which relies on the consumer data output by Task B, and further processes and generates a consumer portrait table, including the consumer's basic information and purchase preferences. Then configure the task dependencies in the task scheduling system as follows: Task A -> Task B: After the order processing task is completed, the consumer data synchronization task is triggered to pull the latest data from the order table; Task B -> Task C: After the consumer data synchronization task is completed, the consumer profile generation task is triggered to use the synchronized consumer data to generate the profile table. Finally, deploy and start the task scheduling system to ensure that all tasks can be automatically executed according to the predefined dependencies.

[0048] Next, configure label rules according to business needs, auxiliary tables, and atomic labels. The label rules include label rules for generating derived rule labels, label rules for generating combined labels, and label rules for generating custom labels. Among them, an entity relationship model can be built in advance to determine the relationship between the entities in the main table and the auxiliary table. Assuming that the label rule for generating derived rule labels is that the product production location is not equal to the sales location, and the production location and sales location are both atomic labels of the objects to be operated, the products whose production location is equal to Luzhou and whose sales location is not equal to Luzhou will be labeled as cross-regional goods. This cross-regional goods label is the derived rule label, and in the derived and complex When complex business scenarios cannot be configured, you can choose to configure custom SQL tags through custom structured query language (SQL). What's more, you can generate combined tags based on atomic tags, derived rule tags and custom tags to further select more accurate group portraits. At the same time, by adding a benchmark table to store historical cumulative data, for example, if the cumulative sales volume of a product calculated before July 15 is 100, write this result to the benchmark table. If you want to calculate the overall accumulation in July, you can start counting from the 16th and add the historical accumulation of 100, thereby reducing repeated calculations before the 16th and preventing excessive operator states.

[0049] When configuring labeling rules, you can also select objects for synchronous labeling. For example, if a box of milk is produced before 2024-04-25, it will be labeled "expired". If more than three bottles of milk in a box of milk are expired, the box of milk will also be labeled "expired". Usually, synchronous labeling objects are relationship entities with transitive attributes. The transitive attribute means that if A has relationship R with B, and B has relationship R with C, then it can be inferred that there is also relationship R between A and C. Of course, this embodiment also supports synchronous labeling of superior, peer or subordinate entities.

[0050] Finally, a structured query statement SQL for real-time labeling is generated according to the label rules and label calculation type, and the startTask method in the Flink task service (FlinkTaskService) is called to submit the SQL to the Yarn cluster for execution, thereby generating target labels. The target labels are stored in the starroks primary key table or Kafkatopic according to business needs for use by downstream business systems, and can also be used for the downstream composite label configuration of the label system. The startTask method also sets the task to run in perjob mode, and the method also supports task rerunning, retrying, and continuing. At the same time, it realizes task continuation by dynamically obtaining the latest checkpoint, and realizes task retry based on the latest SQL of the published task.

[0051] Compared with the existing technology, this embodiment has higher real-time performance. Basically, the business data can be quickly tagged with corresponding tags after being sent down, so as to quickly respond to changing market demands.

[0052] In some embodiments, a structured query statement for real-time tagging is generated according to a tag rule and a tag calculation type, including:

[0053] Call the getCurrentDaoConnector method in the data access object connector factory to obtain the calculation engine used by the current application;

[0054] Based on the calculation engine, tag rules, and tag calculation type, the getTaskBuildSql method in the row-level permission processing service interface is called to dynamically generate structured query statements for real-time tagging.

[0055] In this embodiment, the computing engine used by the current application can be obtained by calling the getCurrentDaoConnector method in the data access object connector factory DaoConnectorFactory, such as Kafka as the computing engine for real-time tags and starrocks as the computing engine for offline tags. Then, based on the obtained computing engine, the configured tag rules and the tag calculation type, the getTaskBuildSql method in the row-level permission processing service RowLevelPermissionProcessService interface is called to dynamically generate the SQL for tagging. In the process of generating SQL, because the amount of dimension table data is large, the memory resources occupied are very large, and the dimension table data often changes, the conventional left join operation cannot be used to achieve synchronous tagging. Instead, the Flink SQL temporal table based on processing time is used to perform a left join operation to achieve synchronous tagging of synchronous tagging objects. At the same time, when generating real-time combined tags, in order to prevent the data of one stream from arriving while the data of another stream has not arrived, and the current stream data from being discarded, each source rule is used as an independent source table and the non-source rule is used as a dimension table to generate a multi-segment SQL statement to achieve combined tagging.

[0056] In some embodiments, Figure 3 As shown, a real-time labeling device based on Flink is provided, including:

[0057] A construction module 10 is used to create an entity and an auxiliary table associated with the entity according to the business data, and abstract the entity into an object to be operated with an atomic tag;

[0058] Configuration module 20, used to dynamically configure the label calculation type of the object to be operated, and set the upstream data processing tasks that the object to be operated and the auxiliary table depend on;

[0059] A generation module 30 is used to configure tag rules according to business requirements, auxiliary tables and atomic tags, and to generate structured query statements for real-time tagging according to the tag rules and tag calculation types;

[0060] The execution module 40 is used to submit the structured query statement to the Yarn cluster for execution to obtain the target label corresponding to the entity, and store the target label in a designated storage system.

[0061] One implementation method of the above-mentioned device may be: the construction module 10 creates an entity and an auxiliary table associated with the entity according to business data, and abstracts the entity into an object to be operated with an atomic label; the configuration module 20 dynamically configures the label calculation type of the object to be operated, and sets the upstream data processing tasks that the object to be operated and the auxiliary table depend on; the generation module 30 configures label rules according to business requirements, auxiliary tables and atomic labels, and generates a structured query statement for real-time labeling according to the label rules and label calculation type; the execution module 40 submits the structured query statement to the Yarn cluster for execution to obtain the target label corresponding to the entity, and stores the target label in the specified storage system.

[0062] In some embodiments, Figure 4 As shown, an electronic device is provided, including a memory 401 and a processor 402, wherein the memory 401 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 402 to implement the above-mentioned real-time labeling method based on Flink.

[0063] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0064] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement the above-mentioned real-time labeling method based on Flink when executed.

[0065] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 401 and executed by the processor 402, and the I / O interface transmission of data is completed by the input interface 405 and the output interface 406 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0066] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a memory 401 and a processor 402. Those skilled in the art may understand that this embodiment is only an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include an input device 407, a network access device, a bus, etc.

[0067] The processor 402 may be a central processing unit (CPU), or other general-purpose processors 402, digital signal processors 402 (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 402 may be a microprocessor 402 or the processor 402 may also be any conventional processor 402, etc.

[0068] The memory 401 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. The memory 401 may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 401 may include both an internal storage unit of the computer device and an external storage device. The memory 401 is used to store computer programs and other programs and data required by the computer device. The memory 401 may also be used to temporarily store in the output device 408, and the aforementioned storage media include various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory ROM403, a random access memory RAM404, a disk or an optical disk.

[0069] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A real-time labeling method based on Flink, characterized in that: The following steps are involved: Create an entity and an auxiliary table associated with the entity according to the business data, and abstract the entity into an object to be operated with an atomic tag; Dynamically configure the label calculation type of the object to be operated, and set the upstream data processing task that the object to be operated and the auxiliary table depend on; According to business requirements, the auxiliary table and the atomic tag, the tag rules are configured, wherein the tag rules include tag rules for generating derived rule tags, tag rules for generating combined tags and tag rules for generating custom tags, and a structured query statement for real-time tagging is generated according to the tag rules and the tag calculation type; Submit the structured query statement to the Yarn cluster for execution to obtain the target label corresponding to the entity, and store the target label in a designated storage system; The configuration tag rule further includes selecting a synchronous marking object, wherein the synchronous marking object is an entity having a transfer attribute; Generating a structured query statement for real-time labeling according to the label rule and the label calculation type includes: Call the getCurrentDaoConnector method in the data access object connector factory to obtain the calculation engine used by the current application; Based on the calculation engine, label rules and label calculation type, the getTaskBuildSql method in the row-level permission processing service interface is called to dynamically generate a structured query statement for real-time labeling, including: When the label rule includes a synchronous labeling object, a left join operation is performed using a temporal table based on processing time of Flink SQL and a corresponding structured query statement is generated; When the label rules are used to generate combined labels, each source rule is used as an independent source table and the non-source rules are used as dimension tables to generate a multi-segment structured query statement.

2. According to the real-time labeling method based on Flink in claim 1, it is characterized in that: Before creating entities and auxiliary tables associated with the entities based on business data, it also includes: Subscribe to a specific topic in Kafka, which contains business data pushed by the upstream business system.

3. According to the real-time labeling method based on Flink in claim 1, it is characterized in that: Submitting the structured query statement to the Yarn cluster for execution includes: Call the startTask method in the Flink task service to submit the structured query statement to the Yarn cluster for execution.

4. According to the real-time labeling method based on Flink in claim 3, it is characterized in that: The method further comprises: The startTask method sets the task to run in perjob mode and supports starting, rerunning, retrying, and continuing the task.

5. A real-time labeling device based on Flink, characterized in that: include: A construction module is used to create an entity and an auxiliary table associated with the entity according to business data, and abstract the entity into an object to be operated with an atomic tag; A configuration module, used to dynamically configure the label calculation type of the object to be operated, and set the upstream data processing tasks that the object to be operated and the auxiliary table depend on; A generation module, configured to configure label rules according to business requirements, the auxiliary table and the atomic label, wherein the label rules include label rules for generating derived rule labels, label rules for generating combined labels and label rules for generating custom labels, and generate structured query statements for real-time labeling according to the label rules and label calculation types; An execution module, used for submitting the structured query statement to the Yarn cluster for execution to obtain a target tag corresponding to the entity, and storing the target tag in a designated storage system; The configuration tag rule further includes selecting a synchronous marking object, wherein the synchronous marking object is an entity having a transfer attribute; Generating a structured query statement for real-time labeling according to the label rule and the label calculation type includes: Call the getCurrentDaoConnector method in the data access object connector factory to obtain the calculation engine used by the current application; Based on the calculation engine, label rules and label calculation type, the getTaskBuildSql method in the row-level permission processing service interface is called to dynamically generate a structured query statement for real-time labeling, including: When the label rule includes a synchronous labeling object, a left join operation is performed using a temporal table based on processing time of Flink SQL and a corresponding structured query statement is generated; When the label rules are used to generate combined labels, each source rule is used as an independent source table and the non-source rules are used as dimension tables to generate a multi-segment structured query statement.

6. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a real-time labeling method based on Flink as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to implement a real-time labeling method based on Flink as claimed in any one of claims 1 to 4 when executed.

Images