Real-time pushing method, device, equipment and medium

Abstract

The application provides a real-time pushing method, device, equipment and medium. The method comprises the following steps: determining a subscription user, storing user data corresponding to the subscription user into a data lake, and obtaining a user data storage table; determining subscription data of the subscription user according to a preset kafka processing platform, and performing data association on the user data storage table and the subscription data according to a preset flink program interface, to obtain a data association table; determining a pushing time period and a pushing threshold of the subscription data according to a preset mysql rule table, filtering the subscription data according to the pushing time period and the pushing threshold, and obtaining target subscription data; determining target user data corresponding to the target subscription data and a target user of the target subscription data according to the data association table; determining corresponding data pushing information according to the target user data, and pushing the data pushing information to the target user. In this way, real-time data reminding is performed on the subscription user, and the pushing function is more perfect and flexible.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of push of subscription service, and particularly to a real-time push method, device, equipment and medium. BACKGROUND

[0002] Voice reminder is an application result of intelligent technology. Voice reminder is more intuitive and universal than text and picture reminder in the use of Internet devices and public facilities.

[0003] The industry has done a lot of work on the push of voice reminder, including call duration reminder, missed call reminder, etc.

[0004] However, the existing subscription service push method has the problem of slow running of push service. SUMMARY

[0005] The present application provides a real-time push method, device, equipment and medium to solve the problem of slow running of push service in the existing subscription service push method.

[0006] In a first aspect, the present application provides a real-time push method, which comprises:

[0007] determining a subscription user and storing user data corresponding to the subscription user into a data lake to obtain a user data storage table;

[0008] determining subscription data of the subscription user according to a preset kafka processing platform, and performing data association on the user data storage table and the subscription data according to a preset flink program interface to obtain a data association table;

[0009] determining a push period and a push threshold of the subscription data according to a preset mysql rule table, and filtering the subscription data according to the push period and the push threshold to obtain target subscription data;

[0010] determining target user data and a target user corresponding to the target subscription data according to the data association table;

[0011] determining corresponding data push information according to the target user data, and pushing the data push information to the target user.

[0012] In this embodiment, determining a subscription user and storing user data corresponding to the subscription user into a data lake to obtain a user data storage table comprises:

[0013] determining a preset hive data table, and determining historical subscription users and user data corresponding thereto according to the hive data table;

[0014] storing the user data into the data lake to obtain an initial user data storage table;

[0015] obtain platform users and platform user data sent by the preset kafka processing platform, and perform data matching on the platform user data and user data in an initial user data storage table to obtain a matching result;

[0016] According to the matching result, determine the user data storage table.

[0017] In this embodiment, the user data is stored in the data lake to obtain the initial user data storage table, including:

[0018] Determine the preset flink programming statement, the data storage table corresponding to the data lake;

[0019] According to the flink programming statement, store the user data in the data lake, and write the user data into the data storage table to obtain the initial user data storage table.

[0020] In this embodiment, according to the matching result, the user data storage table is determined, including:

[0021] Determine the matching result;

[0022] If the matching result is that the platform user data is not stored in the initial user data storage table, the platform user data is determined as new user data;

[0023] Store the new user data in the initial user data storage table to obtain the user data storage table.

[0024] In this embodiment, according to the preset kafka processing platform, determine the subscription data of the subscription user, and according to the preset flink program interface, perform data association on the user data storage table and the subscription data to obtain a data association table, including:

[0025] Obtain the subscription data sent by the kafka processing platform, and determine the identification information of the subscription user in the subscription data;

[0026] Determine the identification information in the user data storage table;

[0027] According to the identification information, perform data association to obtain the data association table.

[0028] In this embodiment, according to the preset mysql rule table, determine the push period and push threshold of the subscription data, and according to the push period and push threshold, filter the subscription data to obtain target subscription data, including:

[0029] Determine the mysql rule table, and according to the table structure of the mysql rule table, determine the push period and push threshold;

[0030] Compare the acquisition period of the subscription data with the push period to obtain a comparison result;

[0031] If the comparison result is that the acquisition period is within the push period, it is determined that the subscription data is to-be-filtered subscription data.

[0032] If the comparison result is that the acquisition period is not within the push period, it is determined that the subscription data is non-filtered subscription data.

[0033] According to the push threshold, the to-be-filtered subscription data is filtered to obtain target subscription data.

[0034] In this embodiment, according to the push threshold, the to-be-filtered subscription data is filtered to obtain target subscription data, including:

[0035] The data value of the to-be-filtered subscription data is determined, and the data value is compared with the push threshold to obtain a comparison result.

[0036] If the comparison result is that the data value does not exceed the push threshold, it is determined that the to-be-filtered subscription data is target subscription data.

[0037] If the comparison result is that the data value exceeds the push threshold, it is determined that the to-be-filtered subscription data is non-target subscription data.

[0038] In a second aspect, the present application provides a real-time push device, the device comprising:

[0039] A data storage module is configured to determine a subscription user and store user data corresponding to the subscription user in a data lake to obtain a user data storage table.

[0040] A data association module is configured to determine subscription data of the subscription user according to a preset kafka processing platform, and perform data association between the user data storage table and the subscription data according to a preset flink program interface to obtain a data association table.

[0041] A filtering module is configured to determine a push period and a push threshold of the subscription data according to a preset mysql rule table, and filter the subscription data according to the push period and the push threshold to obtain target subscription data.

[0042] A data determination module is configured to determine target user data corresponding to the target subscription data and a target user of the target subscription data according to the data association table.

[0043] An information push module is configured to determine corresponding data push information according to the target user data, and push the data push information to the target user.

[0044] In a third aspect, the present application provides a computer device, comprising a processor and a memory in communication with the processor.

[0045] The memory stores computer execution instructions.

[0046] The processor executes computer-executed instructions stored in the memory to implement the method of the present application.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium storing program code, the program code being executed by a processor to implement the method of the present application.

[0048] The real-time push method, device, equipment and medium provided by the present application determine the subscription user, store the user data corresponding to the subscription user into the data lake to obtain a user data storage table; determine the subscription data of the subscription user according to a preset kafka processing platform, and perform data association on the user data storage table and the subscription data according to a preset flink program interface to obtain a data association table; determine the push period and the push threshold of the subscription data according to a preset mysql rule table, and filter the subscription data according to the push period and the push threshold to obtain target subscription data; determine the target user data corresponding to the target subscription data and the target user according to the data association table; determine the corresponding data push information according to the target user data, and push the data push information to the target user.

[0049] In this way, the voice saturation reminding push of the online short message subscription user can be realized by the technology combining flink and StreamPark, the data lake Paimon flow batch integrated read-write is used to efficiently update the data in real time, and the flow batch integrated storage data is used. The multi-table association of hundreds of millions of single-table data is realized, the intermediate result is real-time searchable and reusable, the data consistency can be well guaranteed, and various forms of data service capabilities are realized. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0051] Figure 1 A flowchart of a real-time push method provided by an embodiment of the present application;

[0052] Figure 2 A flowchart of another real-time push method provided by an embodiment of the present application;

[0053] Figure 3 A structure diagram of a real-time push device provided by an embodiment of the present application;

[0054] Figure 4 A structure block diagram of equipment for executing the real-time push method according to the embodiments of the present application.

[0055] The specific embodiments of the application have been shown and described in considerable detail in the drawings and specification made this application, for purposes of illustrative discussion. The drawings and description are not intended to limit the scope of this application in any way. DETAILED DESCRIPTION

[0056] The exemplary embodiments will be described in detail herein with reference to the attached drawings; Figures and description are not intended to limit the scope of this application in any way. Rather, they are included to describe particular embodiments in accordance with the concepts of this application as detailed herein.

[0057] Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed stream data flow engine for stateful computation on unbounded and bounded data streams. Flink can run in all common cluster environments and perform in-memory speed and arbitrary scale computation. The flink runtime consists of two types of processes: a JobManager and one or more TaskManagers.

[0058] Flink's stateful computation can simply match data correlation based on event rules. When there are multiple table associations in business scenarios, it is difficult to achieve real-time query of intermediate results through simple methods, and data consistency cannot be guaranteed. Based on this, the Paimon data lake technology is selected through the data lake solution to solve the above problems.

[0059] Apache Paimon is a stream data lake storage technology that can provide users with high throughput, low latency data ingestion, stream subscription, and real-time query capabilities. Paimon uses open data formats and technical concepts, and can be connected to Apache Flink / Spark / Trino and other industry-leading computing engines to promote the popularization and development of the StreamingLakehouse architecture.

[0060] Paimon manages metadata based on a distributed file system in a lake storage manner, and uses open ORC, Parquet, and Avro file formats to support major computing engines, including Flink, Spark, Hive, Trino, and Presto. In the future, more engines will be connected, including Doris and Starrocks.

[0061] In the prior art, the existing subscription service push method has done a lot of work on voice reminders, including call duration reminders, missed call reminders, etc., but the work on mobile phone voice saturation reminders, especially online SMS subscription of this function and real-time voice saturation reminders, is still not perfect.

[0062] To solve the above problems, the embodiments of the present application provide a real-time push method, which can realize real-time voice saturation reminder push of a subscription user through the combination of flink and Paimon. At the same time, by writing an independent flink task to complete the landing of topics such as hive tables, subsequent data monitoring and troubleshooting can be realized.

[0063] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0064] Figure 1 A flowchart of a real-time push method provided by an embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the real-time push method can include the following steps:

[0065] S110, determine a subscription user and store user data corresponding to the subscription user in a data lake to obtain a user data storage table.

[0066] The subscription user is a user who has subscribed to a real-time push service and can be a user who has subscribed to the service on an online SMS platform.

[0067] The user data can be understood as basic data of the user, such as basic information such as the user's phone number.

[0068] The data lake can be understood as a system or storage that stores data in a natural or original format, usually as object blocks or files; the data lake is usually a single storage of full data in an enterprise, and the full data includes a copy of original data generated by the original system and converted data generated for various tasks.

[0069] The user data storage table is a storage table in the data lake that stores user basic data and can be a Paimon table.

[0070] Based on this, by determining a user who has subscribed to a real-time push service, the corresponding user data is stored in a data lake to obtain a corresponding user data storage table, so as to associate the user data with the subscription data according to the user data in the user data storage table, thereby determining the subscription user who needs real-time reminders.

[0071] In S120, subscription data of the subscription user is determined according to the preset kafka processing platform, and the user data storage table and the subscription data are data-associated according to a preset flink program interface, to obtain a data association table.

[0072] The preset kafka processing platform is a pre-set kafka platform. Kafka is a high-throughput distributed publish-subscribe message system, which can process all action flow data of consumers in a website. Kafka can be used to process real-time subscription data of a subscription user. The subscription data is business data of a user subscription service, for example, if a user subscribes to a real-time reminder service of voice saturation, the subscription data is a voice saturation value of the user.

[0073] The preset flink program interface is a pre-set Data Stream API of flink. In the overall system architecture of flink, support for stream computing is one of the most important functions. A Data Stream API is defined in flink to enable users to flexibly and efficiently write flink stream applications. In this way, the preset Data Stream API can be used to associate user voice saturation data pushed by Kafka and a Paimon table.

[0074] The data association table can be a user-level real-time wide table, which is used to store data.

[0075] Based on this, subscription data of a subscription user pushed by a kafka platform is obtained, and a user data storage table in a data lake and the subscription data are connected according to a Data Stream API of flink, so as to determine a user corresponding to to-be-pushed subscription data according to a data association table obtained by association.

[0076] In S130, a push time period and a push threshold of the subscription data are determined according to a preset mysql rule table, and the subscription data is filtered according to the push time period and the push threshold, to obtain target subscription data.

[0077] The preset mysql rule table is a pre-set mysql data table. Mysql is a relational database management system. Data is saved in different tables instead of being placed in a large warehouse, which increases speed and improves flexibility. The rule table is a special table in mysql for defining and managing database rules and constraints. It contains a set of rules for constraining data in the database, such as data type restrictions, uniqueness constraints, integrity constraints, etc.

[0078] The push period is the execution period of the table rule corresponding to the mysql rule table, that is, if the acquisition period of the subscription data is within the push period, the rule of the mysql rule table takes effect on the subscription data; the push threshold is the push limit value of the user subscription data, for example, if the user subscribes to the voice saturation push prompt, a voice saturation threshold can be set, and if the real-time acquired user voice saturation value exceeds the threshold, it indicates that the current needs to remind the subscription user.

[0079] The target subscription data is the subscription data that meets the push period and the push threshold in the mysql rule table, which can be understood as the subscription data that needs to be pushed and reminded, for example, the user voice saturation data that exceeds the push threshold in the push period.

[0080] Therefore, by the mysql rule table, the corresponding push period and push threshold are determined, so that the real-time acquired subscription data is filtered to obtain the target subscription data, so as to determine the user corresponding to the target subscription data and push the reminder to the user.

[0081] S140, determining the target user data corresponding to the target subscription data and the target user according to the data association table.

[0082] The target user is the subscription user corresponding to the target subscription data, and the target user data is the user data of the subscription user.

[0083] Therefore, by the data association table, the user information corresponding to the target subscription data is determined, so as to push the subscription data and the reminder information to the subscription user in the future.

[0084] S150, determining the corresponding data push information according to the target user data, and pushing the data push information to the target user.

[0085] The data push information is information determined according to the target user data for pushing and reminding the user, for example, it can include mobile phone number, user home province, user home city, voice tariff type, voice saturation value and other key data.

[0086] Therefore, by determining the data push information including the target user data, the data push information is pushed to the corresponding subscription user, and the subscription service of the subscription user is pushed and reminded.

[0087] On the basis of the feasible implementation mode of S110, the application further provides a process of storing the user data of the historical subscription user and the new subscription user in the data lake by determining the historical subscription user through the hive data table and the subscription data sent by the kafka processing platform.

[0088] determine a preset hive data table, and determine historical subscription users and corresponding user data of the historical subscription users according to the hive data table;

[0089] store the user data into a data lake to obtain an initial user data storage table;

[0090] obtain platform users and platform user data sent by a preset kafka processing platform, and perform data matching between the platform user data and the user data in the initial user data storage table to obtain a matching result;

[0091] determine a user data storage table according to the matching result.

[0092] The preset hive data table is a preset hive table, and the hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading. It is a mechanism that can store, query and analyze large-scale data stored in Hadoop. The hive data warehouse tool can map structured data files into a database table and provide SQL query functions, and can convert SQL statements into MapReduce tasks for execution.

[0093] The historical subscription users are users stored in the hive data table, which can be understood as users who have historically subscribed to related businesses and have stored corresponding user data of the users, or can be understood as historical data of initialization subscription users stored in the hive.

[0094] The data matching can be understood as matching the platform user data with the user data in the initial user data storage table, so as to determine the data in the platform user data that is not stored in the initial user data storage table, that is, the user data of the new users. The matching result is that there is data in the platform user data that is not stored in the initial user data storage table, or there is no data in the platform user data that is not stored in the initial user data storage table.

[0095] Based on this, the historical subscription users and the corresponding user data of the historical subscription users are determined based on the user data stored in the hive data table, and are stored into the data lake to obtain the initial user data storage table. At the same time, the subscription user data pushed in real time from the upstream is read from the kafka, so that the newly obtained user data is stored into the initial user data storage table, so as to obtain the user data storage table. In this way, the rate and convenience of user data into the lake are improved, so as to realize real-time push reminding of user subscription business.

[0096] On the basis of the feasible implementation manner of S110, the application further provides a process of storing user data in a hive data table into a data lake through a flink programming statement:

[0097] Determine the preset flink programming statement and the data storage table corresponding to the data lake;

[0098] According to the flink programming statement, the user data is stored into the data lake, and the user data is written into the data storage table, to obtain an initial user data storage table.

[0099] The preset flink programming statement is a pre-set flink sql, and the flink sql is an upper layer library based on flink core, which uses SQL semantics for structured data processing.

[0100] The initial user data storage table is a data table storing user data, which can be a Paimon table.

[0101] Therefore, by determining the preset flink sql, data parsing and data lake are performed, and a Paimon table is generated, so as to store user data into the Paimon table, and realize the data lake of the user data.

[0102] On the basis of the feasible implementation manner of S110, the application further provides a process of storing user data in a hive data table into a data lake through a flink programming statement:

[0103] Determine the matching result;

[0104] If the matching result is that the platform user data is not stored in the initial user data storage table, the platform user data is determined as new user data;

[0105] Store the new user data into the initial user data storage table to obtain a user data storage table.

[0106] Therefore, by determining the matching result of data matching, the subscription data in the subscription data sent by the kafka platform and not stored in the initial user data storage table is determined as new user data, and the new user data is stored into the initial user data storage table to obtain a user data storage table.

[0107] On the basis of the feasible implementation manner of S120, the application further provides a process of associating user data storage table and subscription data through user identification information:

[0108] Acquire the subscription data sent by the kafka processing platform, and determine the identification information of the subscription user in the subscription data;

[0109] Determine the identification information in the user data storage table;

[0110] According to the identification information, data correlation is performed to obtain a data correlation table.

[0111] The identification information can be understood as identification information, which is used to determine the subscription user corresponding to the user data, for example, it can be the user's mobile phone number.

[0112] Based on this, the identification information in the user data storage table and the subscription data is determined, and data correlation is performed according to the identification information.

[0113] In this embodiment, in order to realize the storage of the user data of the subscription user into the lake, so as to realize the purpose of real-time subscription data reminding and pushing of the subscription user according to the combination of flink and paimon, the historical data of the online short message subscription user stored by hive and the real-time subscription data pushed by kafka can be used to realize the storage of the user data into the lake, and the data correlation between the user data and the subscription data can be realized according to the flink sql, so as to obtain a data correlation table. At the same time, the subscription data is filtered through a preset mysql rule table to obtain target subscription data that meets the pushing threshold within a pushing period, and the corresponding target subscription user is determined according to the data correlation table, so as to push the target subscription data to the target subscription user.

[0114] In this way, the online short message subscription user voice saturation real-time reminding method based on stream computing is realized, Paimon is used as the data lake technology to replace the original Flink state or Hbase technology, and the intermediate results of the pushing can be tracked.

[0115] Figure 2 Another flowchart of a real-time pushing method provided by the embodiment of the application is shown in the figure. Figure 2 As shown in the figure, the real-time pushing method can include the following steps:

[0116] S210, determine the subscription user, and store the user data corresponding to the subscription user into the data lake to obtain a user data storage table.

[0117] S220, according to the preset kafka processing platform, determine the subscription data of the subscription user, and according to the preset flink program interface, perform data correlation between the user data storage table and the subscription data to obtain a data correlation table.

[0118] In this embodiment, the specific implementation of steps S210 to S220 can refer to the content in the foregoing embodiments, which will not be described here.

[0119] S230, determine the mysql rule table, and determine the push period and the push threshold according to the table structure of the mysql rule table.

[0120] The table structure of the mysql rule table is the table header structure. By pre-setting the rule of the table header structure, data filtering can be realized through the mysql rule table. For example, the specific push period and the push threshold can be set in the table structure of the mysql rule table, so that after the subscription data is obtained, the target subscription data in the push period and meeting the push threshold can be determined according to the table structure of the mysql rule table.

[0121] Therefore, the push period and the push threshold are determined by determining the table structure of the mysql rule table, so as to filter the subscription data subsequently.

[0122] S240, compare the acquisition period of the subscription data with the push period to obtain a comparison result.

[0123] Therefore, by comparing the acquisition period of the subscription data sent by the kafka platform with the preset push period in the mysql rule table, the data in the push period in the subscription data can be determined according to the comparison result subsequently.

[0124] S250, if the comparison result is that the acquisition period is in the push period, it is determined that the subscription data is the to-be-filtered subscription data.

[0125] Therefore, by determining the to-be-filtered subscription data in the push period, the to-be-filtered subscription data can be filtered according to the push threshold subsequently, and the target subscription data that needs to be pushed to the user for reminding is obtained.

[0126] S260, if the comparison result is that the acquisition period is not in the push period, it is determined that the subscription data is the non-filtered subscription data.

[0127] Therefore, by determining the non-filtered subscription data that is not in the push period, the non-filtered subscription data does not need to be filtered according to the push threshold subsequently, so as to improve the rate and real-time performance of data filtering and reduce unnecessary waste of resources.

[0128] S270, filtering the to-be-filtered subscription data according to the push threshold to obtain the target subscription data.

[0129] Therefore, by the preset push threshold in the mysql rule table, the to-be-filtered subscription data is filtered to obtain the target subscription data, so as to push the corresponding subscription user for reminding subsequently.

[0130] S280. Based on the data association table, determine the target user data and its target users corresponding to the target subscription data.

[0131] S290. Based on the target user data, determine the corresponding data push information and push the data push information to the target user.

[0132] In this embodiment, the specific implementation of steps S280 to S290 can be found in the content of the foregoing embodiments, and will not be repeated here.

[0133] Based on the feasible implementation of S270 described above, this application further provides a process for determining target subscription data based on the comparison results by comparing data values ​​and push thresholds:

[0134] Determine the data value of the subscription data to be filtered, and compare the data value with the push threshold to obtain the comparison result;

[0135] If the comparison result shows that the data value does not exceed the push threshold, then the subscription data to be filtered is determined to be the target subscription data;

[0136] If the comparison result shows that the data value exceeds the push threshold, then the subscription data to be filtered is determined to be non-target subscription data.

[0137] Among them, the data value is the business data value corresponding to the user's subscribed service. For example, if the user subscribes to the voice saturation push reminder service, the data value is the user's voice saturation value. If the voice saturation value exceeds the preset push threshold, it indicates that a push reminder needs to be sent to the subscribed user.

[0138] Based on this, by comparing the data values ​​of the subscription data to be filtered with the preset push threshold, the target subscription data exceeding the push threshold is determined according to the comparison results. In order to subsequently determine the corresponding subscriber based on the target subscription data, and push reminder information including the target subscription data to the user.

[0139] In practical applications, a whitelist of subscribers can be set through a MySQL table, and then the whitelist data can be dynamically read from MySQL in real time using FlinkCDC technology to achieve the filtering of subscribers.

[0140] Furthermore, after identifying the target subscription data and pushing it to the corresponding subscribers, fuzzy matching can be used to push all Topic data corresponding to Topic names with voice saturation identifiers to a Hive table. The data can be partitioned by time and province, and the expiration time can be set to one month to facilitate subsequent data tracking and troubleshooting.

[0141] In the embodiment, in order to realize the preset push period and push threshold in the table structure according to the mysql rule table, so as to filter the subscription data and obtain the target subscription data, the subscription data obtained in the push period can be determined by comparing the acquisition period of the subscription data with the push period, and the target subscription data exceeding the push threshold can be determined by comparing the data value of the subscription data with the push threshold.

[0142] In this way, the online short message subscription user voice saturation real-time reminding method based on stream computing is realized, and the real-time voice saturation reminding and pushing of the subscription user are realized through the combination of flink and Paimon. Meanwhile, the subscription user real-time user voice saturation reminding sets a user white list, the user voice saturation real-time pushing combines the FlinkCDC technology, the database update is updated in real time, the user range is issued, the pushing function is more perfect and flexible; the topic is written into a separate flink task, such as a hive table, and subsequent data issuing monitoring and troubleshooting are realized.

[0143] Figure 3 A structure diagram of a real-time pushing device 300 provided in the embodiment of the application is shown in FIG. 1. Figure 3 As shown in the figure, the real-time pushing device 300 includes a data storage module 310, a data association module 320, a filtering module 330, a data determination module 340, and an information pushing module 350.

[0144] The data storage module 310 is configured to determine a subscription user and store user data corresponding to the subscription user into a data lake to obtain a user data storage table.

[0145] The data association module 320 is configured to determine subscription data of the subscription user according to a preset kafka processing platform, and perform data association between the user data storage table and the subscription data according to a preset flink program interface to obtain a data association table.

[0146] The filtering module 330 is configured to determine a push period and a push threshold of the subscription data according to a preset mysql rule table, and filter the subscription data according to the push period and the push threshold to obtain target subscription data.

[0147] The data determination module 340 is configured to determine target user data and a target user corresponding to the target subscription data according to the data association table.

[0148] The information pushing module 350 is configured to determine corresponding data pushing information according to the target user data, and push the data pushing information to the target user.

[0149] In the embodiment of the application, the data storage module 310 can be further configured to:

[0150] determine the historical subscription user and the corresponding user data according to the hive data table;

[0151] store the user data into the data lake to obtain an initial user data storage table;

[0152] obtain platform users and platform user data sent by the preset kafka processing platform, and perform data matching on the platform user data and the user data in the initial user data storage table to obtain a matching result;

[0153] determine the user data storage table according to the matching result.

[0154] In the embodiment of the present application, the data storage module 310 can also be specifically used for:

[0155] determine a preset flink programming statement and a data storage table corresponding to the data lake;

[0156] store the user data into the data lake according to the flink programming statement, and write the user data into the data storage table to obtain an initial user data storage table.

[0157] In the embodiment of the present application, the data storage module 310 can also be specifically used for:

[0158] determine the matching result;

[0159] if the matching result is that the platform user data is not stored in the initial user data storage table, determine the platform user data as new user data;

[0160] store the new user data into the initial user data storage table to obtain the user data storage table.

[0161] In the embodiment of the present application, the data association module 320 can also be specifically used for:

[0162] obtain subscription data sent by the kafka processing platform, and determine the identification information of the subscription user in the subscription data;

[0163] determine the identification information in the user data storage table;

[0164] perform data association according to the identification information to obtain a data association table.

[0165] In the embodiment of the present application, the filtering module 330 can also be specifically used for:

[0166] determine a mysql rule table, and determine a push period and a push threshold according to the table structure of the mysql rule table;

[0167] The acquisition time period and the push time period are compared to obtain a comparison result.

[0168] If the comparison result is that the acquisition time period is within the push time period, the subscription data is determined as the to-be-filtered subscription data.

[0169] If the comparison result is that the acquisition time period is not within the push time period, the subscription data is determined as the non-filtered subscription data.

[0170] According to the push threshold, the to-be-filtered subscription data is filtered to obtain target subscription data.

[0171] In the embodiments of the present application, the filtering module 330 can also be specifically configured to:

[0172] determine a data value of the to-be-filtered subscription data, and compare the data value with the push threshold to obtain a comparison result;

[0173] If the comparison result is that the data value does not exceed the push threshold, the to-be-filtered subscription data is determined as the target subscription data.

[0174] If the comparison result is that the data value exceeds the push threshold, the to-be-filtered subscription data is determined as the non-target subscription data.

[0175] Figure 4 The structural schematic diagram of the device provided in the embodiments of the present application is shown in FIG. 4. Figure 4 As shown in FIG. 4, the device 400 includes:

[0176] The device 400 can include a processor 401 with one or more processing cores, a memory 402 with one or more computer readable storage media, a communication component 403, and the like. The processor 401, the memory 402, and the communication component 403 are connected through a bus 404.

[0177] In the specific implementation process, the at least one processor 401 executes the computer execution instructions stored in the memory 402, so that the at least one processor 401 performs the real-time push method as described above.

[0178] The specific implementation process of the processor 401 can refer to the method embodiments described above, which has similar implementation principles and technical effects, and will not be described here in detail.

[0179] In the above Figure 4In the illustrated embodiment, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the present application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0180] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0181] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0182] In some embodiments, a computer program product is also provided, which includes a computer program or instructions, which, when executed by a processor, implements the steps of any of the real-time push methods described above.

[0183] The specific implementation of each operation can refer to the foregoing embodiments, which will not be repeated here.

[0184] Those of ordinary skill in the art can understand that all or part of the steps of the various methods of the above embodiments can be completed by instructions, or by relevant hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0185] To this end, the embodiments of the present application provide a computer readable storage medium, which stores a plurality of program codes. The program codes can be loaded by a processor to execute the steps of any of the real-time push methods provided by the embodiments of the present application.

[0186] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0187] According to an aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium.

[0188] Since the instructions stored in the storage medium can execute the steps in any real-time pushing method provided by the embodiments of the present application, the beneficial effects that can be achieved by any real-time pushing method provided by the embodiments of the present application can be achieved. Details are described in the foregoing embodiments, which will not be repeated here.

[0189] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the following claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0190] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the appended claims.

Claims

1. A real-time push method, characterized in that, The method includes: Identify the subscribers and store the user data corresponding to the subscribers in the data lake Paimon to obtain the user data storage table; Based on the preset Kafka processing platform, the subscription data of the subscriber is determined, and based on the preset Flink program interface, the user data storage table and the subscription data are associated with a stream table in a stream computing manner to obtain a data association table; wherein, through the read and write characteristics of the Paimon data lake that integrates stream and batch processing, the intermediate results of the stream table association are stored to achieve real-time queryability of the intermediate results and data consistency guarantee. Based on a preset MySQL rule table, the push period and push threshold of the subscription data are determined, and the subscription data is filtered according to the push period and push threshold to obtain the target subscription data; Based on the data association table, determine the target user data and its target users corresponding to the target subscription data; Based on the target user data, determine the corresponding data push information and push the data push information to the target user; The step involves determining the push period and push threshold for the subscription data based on a preset MySQL rule table, and filtering the subscription data according to the push period and push threshold to obtain the target subscription data, including: The MySQL rule table is determined, and based on the table structure of the MySQL rule table, the push period and the push threshold are determined; The comparison results are obtained by comparing the acquisition period of the subscription data with the push period; If the comparison result indicates that the acquisition time period is within the push time period, then the subscription data is determined to be subscription data to be filtered. If the comparison result indicates that the acquisition period is not within the push period, then the subscription data is determined to be unfiltered subscription data. The target subscription data is obtained by filtering the subscription data to be filtered according to the push threshold. The process of determining the subscriber and storing the corresponding user data in the data lake Paimon to obtain the user data storage table includes: Determine the preset Hive data table, and based on the Hive data table, determine the historical subscribers and their corresponding user data; The user data is stored in the data lake to obtain the initial user data storage table; Obtain the platform users and their data sent by the preset Kafka processing platform, and match the platform user data with the user data in the initial user data storage table to obtain the matching result; Based on the matching results, the user data storage table is determined; The step of filtering the subscription data to be filtered according to the push threshold to obtain the target subscription data includes: Determine the data value of the subscription data to be filtered, and compare the data value with the push threshold to obtain the comparison result; If the comparison result indicates that the data value does not exceed the push threshold, then the subscription data to be filtered is determined to be the target subscription data; If the comparison result indicates that the data value exceeds the push threshold, then the subscription data to be filtered is determined to be non-target subscription data.

2. The method according to claim 1, characterized in that, The step of storing the user data in the data lake to obtain an initial user data storage table includes: Determine the preset Flink programming statements and the data storage table corresponding to the data lake; According to the Flink programming statements, the user data is stored in the data lake and written into the data storage table to obtain the initial user data storage table.

3. The method according to claim 1, characterized in that, The step of determining the user data storage table based on the matching result includes: Determine the matching result; If the matching result indicates that the platform user data is not stored in the initial user data storage table, then the platform user data is identified as new user data. The new user data is stored in the initial user data storage table to obtain the user data storage table.

4. The method according to claim 1, characterized in that, The process involves determining the subscription data of the subscriber based on a preset Kafka processing platform, and then, according to a preset Flink program interface, associating the user data storage table with the subscription data to obtain a data association table, including: Obtain the subscription data sent by the Kafka processing platform, and determine the identification information of the subscribing user in the subscription data; Determine the identification information in the user data storage table; Based on the identification information, the data is associated to obtain the data association table.

5. A real-time push device, characterized in that, The device includes: The data storage module is used to identify the subscribers and store the user data corresponding to the subscribers in the data lake Paimon to obtain the user data storage table; The data association module is used to determine the subscription data of the subscriber based on the preset Kafka processing platform, and to perform a stream table association between the user data storage table and the subscription data in a stream computing manner according to the preset Flink program interface to obtain a data association table; wherein, through the read and write characteristics of the Paimon data lake that integrates stream and batch processing, the intermediate results of the stream table association are stored to achieve real-time queryability of the intermediate results and data consistency guarantee. The filtering module is used to determine the push period and push threshold of the subscription data according to a preset MySQL rule table, and to filter the subscription data according to the push period and push threshold to obtain the target subscription data; The data determination module is used to determine the target user data and its target users corresponding to the target subscription data based on the data association table. The information push module is used to determine the corresponding data push information based on the target user data, and push the data push information to the target user. The filtering module is specifically used to determine the MySQL rule table and, based on the table structure of the MySQL rule table, determine the push period and the push threshold. The comparison results are obtained by comparing the acquisition period of the subscription data with the push period; If the comparison result indicates that the acquisition time period is within the push time period, then the subscription data is determined to be subscription data to be filtered. If the comparison result indicates that the acquisition period is not within the push period, then the subscription data is determined to be unfiltered subscription data. The target subscription data is obtained by filtering the subscription data to be filtered according to the push threshold. The data storage module is specifically used to determine a preset Hive data table, and based on the Hive data table, determine historical subscribers and their corresponding user data; store the user data in the data lake to obtain an initial user data storage table; obtain platform users and their platform user data sent by a preset Kafka processing platform, and match the platform user data with the user data in the initial user data storage table to obtain a matching result; and determine the user data storage table based on the matching result. When filtering the subscription data to be filtered according to the push threshold to obtain the target subscription data, the filtering module is specifically used to determine the data value of the subscription data to be filtered, and compare the data value with the push threshold to obtain a comparison result; If the comparison result indicates that the data value does not exceed the push threshold, then the subscription data to be filtered is determined to be the target subscription data; If the comparison result indicates that the data value exceeds the push threshold, then the subscription data to be filtered is determined to be non-target subscription data.

6. A device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 4.

Images