Information push method, device, equipment and storage medium based on Kafka

Through the information push method based on kafka, the user data is accurately analyzed and pushed using a distributed system and computing engine, which solves the problem of low accuracy of information push in the existing technology, and improves the efficiency and conversion rate of information recommendation.

CN111666490BActive Publication Date: 2024-11-01CHINA PING AN PROPERTY INSURANCE CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202010350127.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-04-28
Publication Date
2024-11-01
Estimated Expiration
2040-04-28

AI Technical Summary

Technical Problem

The existing information recommendation system analyzes and pushes all user data, resulting in low accuracy of information push, low conversion rate of recommended information, and low efficiency.

Method used

Using kafka-based information push method, group categories are generated by preset user operation events and dynamic filtering attributes, distributed search engine ElasticSearch and distributed publishing and subscription message system kafka for data collection and analysis, combined with distributed computing engine Spark for data Filter analysis to achieve accurate push to users.

Benefits of technology

It improves the accuracy and efficiency of information push, ensures that information push is more in line with user needs, and improves the conversion rate of recommended information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111666490B_ABST
    Figure CN111666490B_ABST
Patent Text Reader

Abstract

The embodiments of the present application belong to the field of data analysis, and relate to a method, device, computer equipment and storage medium for information push based on kafka, wherein the method includes: obtaining a group category based on preset user operation events and dynamic screening condition attributes; constructing a buried event, and collecting user data of the user through the buried event, and storing the user data in the distributed search engine ElasticSearch; by configuring the distributed publish-subscribe message system kafka and using the distributed computing engine Spark to process the user data, the group category corresponding to each user is obtained, and according to the business information, the group category to be recommended is determined from the group category, and the business information is pushed to the users included in the group category to be recommended. The present application realizes accurate push for users of different group categories and improves the efficiency of business information push. The present application also relates to blockchain technology, and the user data can be stored in a blockchain node.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to a Kafka-based information push method, device, equipment and storage medium. Background Art

[0002] Information recommendation is a method for pushing data information that users are interested in to users based on their preferences. Accurate information recommendation is beneficial for data providers to push their own data information to users on the one hand, and for users to obtain the information they want on the other hand. In recent years, recommendation systems have become more and more popular and are used in various fields, including movies, music, news, books, etc. For example, e-commerce platforms also have their own dedicated recommendation systems to provide customers with products they may like. Under reasonable settings, it can effectively increase profits, click-through rates, conversion rates, etc., and provide users with a better experience.

[0003] The existing information recommendation system collects user data by setting up tracking points, obtains user preference information by analyzing the user data, and pushes the push information to the user based on the user preference information. However, the user data collected is analyzed based on all user information and pushed to all users. This method of information push has a low accuracy, and the recommended information is not what the user actually needs, resulting in a low conversion rate of the recommended information, which leads to low efficiency of information recommendation. Summary of the invention

[0004] The purpose of the embodiment of the present application is to propose an information push method based on Kafka to improve the efficiency of information recommendation.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for pushing information based on Kafka, including:

[0006] Generate group categories based on preset user operation events and dynamic screening condition attributes;

[0007] Construct a tracking event, collect user data of the user through the tracking event, and store the user data in the distributed search engine ElasticSearch;

[0008] Configure the parameter information of the topic in the distributed publish-subscribe messaging system kafka, and based on the configured topic, perform data analysis on the user data stored in the distributed search engine ElasticSearch, and store the obtained analysis results in the messaging middleware;

[0009] Using the distributed computing engine Spark, the analysis results are obtained from the message middleware, and the analysis results are filtered and analyzed, and the obtained filtered analysis results are written back to the message middleware;

[0010] The filtering analysis results in the message middleware are written into the distributed publish-subscribe message system Kafka, and the filtering analysis results are parsed and processed by the distributed publish-subscribe message system Kafka to obtain the group category corresponding to each of the users;

[0011] According to the service information, a group category to be recommended is determined from the group categories, and the service information is pushed to users included in the group category to be recommended.

[0012] Further, the configuration of the parameter information of the topic in the distributed publish-subscribe messaging system kafka, and based on the configured topic, performing data analysis on the user data stored in the distributed search engine ElasticSearch, and storing the obtained analysis results in the message middleware includes:

[0013] Configure parameter information of the topic in the distributed publish-subscribe messaging system kafka, and perform data analysis on the user data based on the configured topic to obtain analysis results;

[0014] The analysis result is encapsulated into a json string to obtain an analysis result json string, and the analysis result json string is written into the message middleware.

[0015] Further, the adopting of the distributed computing engine Spark to obtain the analysis result from the message middleware, filtering and analyzing the analysis result, and writing the obtained filtering and analysis result back to the message middleware includes:

[0016] Encapsulating the analysis results into SQL commands;

[0017] The SQL command is executed by the distributed computing engine Spark, and the table made of the hive table is searched to obtain the number of users and user information of the group category, and the number of users and the user information are stored in the message middleware.

[0018] Further, executing the SQL command through the distributed computing engine Spark, searching the table made of the hive table, obtaining the number of users and user information of the group category, and saving the number of users and the user information in the message middleware includes:

[0019] Traversing the user data in the table created by the hive library through the SQL command, filtering and deleting the duplicated user data, and obtaining a filtering result;

[0020] The number of users in the group category of the filtering result is counted, and the user corresponding information is matched to obtain the number of users and the user information, and the number of users and the user information are saved in the subject middleware.

[0021] Further, the adopting of the distributed computing engine Spark to obtain the analysis result from the message middleware, filtering and analyzing the analysis result, and writing the obtained filtering and analysis result back to the message middleware also includes:

[0022] Encapsulate the filtering analysis result into a json string to obtain a filtering analysis result json string;

[0023] The filtering and analysis result json string is written back to the message middleware.

[0024] Furthermore, after determining the group category to be recommended from the group category according to the service information, and pushing the service information to users included in the group category to be recommended, the method further includes:

[0025] The group categories are traversed periodically through the scheduler Quartz, and the users corresponding to the group categories are updated.

[0026] Furthermore, after determining the group category to be recommended from the group category according to the service information, and pushing the service information to users included in the group category to be recommended, the method further includes:

[0027] According to the generated attributes of the group category, a corresponding label is assigned to the user corresponding to the group category.

[0028] In order to solve the above technical problems, the embodiment of the present application provides an information push device based on Kafka, including:

[0029] A group category generation module is used to generate group categories based on preset user operation events and dynamic screening condition attributes;

[0030] A user data collection module is used to construct a point-of-view event, collect user data of users through the point-of-view event, and store the user data in a distributed search engine ElasticSearch;

[0031] The user data analysis module is used to configure the parameter information of the topic in the distributed publish-subscribe messaging system kafka, and based on the configured topic, perform data analysis on the user data stored in the distributed search engine ElasticSearch, and store the obtained analysis results in the message middleware;

[0032] A filtering and analysis result module is used to obtain the analysis result from the message middleware by using the distributed computing engine Spark, and to filter and analyze the analysis result, and write the obtained filtering and analysis result back to the message middleware;

[0033] A group category determination module is used to write the filtering analysis results in the message middleware into the distributed publish-subscribe message system Kafka, and parse the filtering analysis results through the distributed publish-subscribe message system Kafka to obtain the group category corresponding to each user;

[0034] The business information push module is used to determine the group category to be recommended from the group category according to the business information, and push the business information to users included in the group category to be recommended.

[0035] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors can implement any one of the above-mentioned Kafka-based information push solutions.

[0036] In order to solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned Kafka-based information push solutions is implemented.

[0037] In the above scheme, a kafka-based information push method generates group categories and constructs point-of-view events according to preset user operation events and dynamic screening condition attributes, and collects user data through point-of-view events, which is conducive to obtaining different user data according to different group categories; through data analysis of the distributed publish-subscribe message system kafka and the distributed computing engine Spark, the group category user corresponding to each user is obtained, so as to realize accurate push to users and improve the efficiency of information push. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a schematic diagram of the application environment of the Kafka-based information push method provided in an embodiment of the present application;

[0040] Figure 2 A flowchart of an implementation method of a Kafka-based information push method provided in an embodiment of the present application;

[0041] Figure 3 This is a flowchart for implementing step S3 in the Kafka-based information push provided in an embodiment of the present application;

[0042] Figure 4 This is a flowchart for implementing step S4 in the Kafka-based information push method provided in an embodiment of the present application;

[0043] Figure 5 This is a flowchart for implementing step S42 in the Kafka-based information push method provided in an embodiment of the present application;

[0044] Figure 6 It is a schematic diagram of a backend management device of a database provided in an embodiment of the present application;

[0045] Figure 7 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0047] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0049] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0050] See also Figure 1 , the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0051] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, search applications, instant messaging tools, etc.

[0052] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0053] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0054] It should be noted that the Kafka-based information push method provided in the embodiment of the present application is generally executed by a server, and accordingly, the Kafka-based information push device is generally set in the server.

[0055] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0056] See also Figure 2 , Figure 2A specific implementation of the information push method based on Kafka is shown.

[0057] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 2 The process sequence shown is limited to the following steps:

[0058] S1: Generate group categories based on preset user operation events and dynamic screening condition attributes.

[0059] Specifically, when a user browses a page on the client and interacts with the page, a user operation event is triggered. The client collects the user operation event and feeds it back to the server. The server distinguishes the received operation events through the preset user operation event and dynamic screening condition attributes, thereby establishing group categories.

[0060] Among them, configure dynamic filtering condition attributes, as well as the number of occurrences of the filtering event (>, =, <, ! =), and the attributes under the filtering event (>, =, <, ! =, include, not include); the relationship between various events can be intersection and union; the time of the event supports fixed time periods (such as 2019 / 07 / 02-2019 / 07 / 25) and dynamic time (nearly n days, take the current time as the end time, and n days ago as the start time). For example, if you select the event "unpaid", the condition is ">", and the number is "5", it means "the number of unpaid times is greater than 5 times". Exception For the attributes under the event, there are two types of attributes, one is numeric type, and the corresponding filtering conditions are greater than, equal to, less than, not equal to, and the number of occurrences; the other is character type attributes, and the corresponding conditions are include, not include, and then fill in the characters you want to filter; for example, select the attribute name "product name", the condition is "include", and the character is "insurance". Combined with the event, the combination is "the number of unpaid times of the product is greater than 5 times, including insurance".

[0061] Among them, the group category is used to distinguish different user groups based on preset user operation events and dynamic screening condition attributes.

[0062] S2: Build tracking events, collect user data through tracking events, and store the user data in the distributed search engine ElasticSearch.

[0063] Specifically, a compass tracking event is set for each operation event. When a user browses a page on the client and triggers a user operation event when interacting with the page, the tracking point corresponding to the user operation event is obtained. That is, through the tracking point event, the client collects the user data and feeds it back to the server. When the user clicks or browses the operation event, the user data is collected through the tracking point method and stored in the distributed search engine ElasticSearch.

[0064] Among them, the distributed search engine ElasticSearch is a search server based on Lucene, which is a distributed, highly scalable, and highly real-time search and data analysis engine. It can easily enable large amounts of data to have the ability to search, analyze, and explore. Taking full advantage of the horizontal scalability of the distributed search engine ElasticSearch can make data more valuable in a production environment. In the present invention, since the user data collected through the buried point events is distributed user data, and the distributed search engine ElasticSearch has the characteristics of distribution and high scalability, it is stored through the distributed search engine ElasticSearch.

[0065] Among them, user data is the user behavior data collected by the server when the user clicks or browses by building embedded data.

[0066] S3: Configure the parameter information of the topic in the distributed publish-subscribe messaging system Kafka, and based on the configured topic, perform data analysis on the user data stored in the distributed search engine ElasticSearch, and store the analysis results in the messaging middleware.

[0067] Specifically, since the user data collected through the tracking events in the above step S2 is stored in the distributed search engine ElasticSearch, by configuring the parameter information of the topic in the distributed publish-subscribe message system Kafka, after storing a type of information, data analysis is performed on the user data, including parsing and packaging of the user data, and stored in the message middleware, it is convenient for further analysis of the user data later.

[0068] Among them, the distributed publish-subscribe messaging system kafka is an open source stream processing platform developed by the Apache Software Foundation and is written in Scala and Java. The distributed publish-subscribe messaging system kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on the website. These data are usually solved by processing logs and log aggregation due to throughput requirements. The purpose of the distributed publish-subscribe messaging system kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time messages through clusters. In the present invention, the collected user data is stored in the message middleware of the distributed publish-subscribe messaging system kafka in an encapsulated form by configuring the parameter information of the topic through the server, providing a basis for further analysis of the user data in the future.

[0069] Among them, topic is a component of the distributed publish-subscribe messaging system kafka, which is a logical concept for storing messages, that is, a message collection. Each message sent to the distributed publish-subscribe messaging system kafka cluster has a topic. Messages corresponding to different topics are stored separately. Each topic can have multiple producers sending messages to it, and multiple consumers consuming the messages. In the present invention, by configuring the parameter information of the topic, a certain type of message is stored, and then the user data is stored in the message middleware through the distributed publish-subscribe messaging system kafka.

[0070] Among them, message middleware is suitable for distributed environments that require reliable data transmission. In a system using a message middleware mechanism, different objects activate each other's events by passing messages to complete corresponding operations. Message middleware can communicate between different platforms. It is often used to shield the characteristics between various platforms and protocols and realize the collaboration between applications. Its advantage is that it can provide synchronous and asynchronous connections between clients and servers, and messages can be transmitted or stored and forwarded at any time, which is why it is a step further than remote procedure calls. In the present invention, the analysis results after data analysis are stored by the message middleware.

[0071] The analysis result is user data obtained after preliminary parsing and encapsulation of the user data, and the analysis result can be written into the message middleware.

[0072] S4: Use the distributed computing engine Spark to obtain the analysis results from the message middleware, filter and analyze the analysis results, and write the filtered analysis results back to the message middleware.

[0073] Specifically, in step S3, the user data has been analyzed, and the analysis results are written into the message middleware. The distributed computing engine Spark generates SQL names, filters and analyzes the analysis results stored in the message middleware, and queries the table generated in hive to return the number of users in the group category. By matching the corresponding user information, the corresponding user information of each user is obtained, and the filtering and analysis results are written back to the message middleware. The purpose of writing the filtering and analysis results back to the message middleware is that the distributed publish-subscribe message system kafka can obtain the filtering and analysis results by writing back to the middleware.

[0074] Among them, the distributed computing engine Spark (Apache Spark) is a fast and general computing engine designed for large-scale data processing. It has the advantages that the intermediate output results of the job can be stored in the memory, so there is no need to read and write HDFS, so the distributed computing engine Spark is better suitable for MapReduce algorithms that require iterations such as data mining and machine learning. In the present invention, the distributed computing engine Spark is used to filter and analyze the analysis results, filter out redundant and abnormal data, identify the number of users and user information, and write the filtered analysis results back to the message middleware.

[0075] S5: Write the filtering analysis results in the message middleware into the distributed publish-subscribe message system Kafka, and parse and process the filtering analysis results through the distributed publish-subscribe message system Kafka to obtain the group category corresponding to each user.

[0076] Specifically, by writing the filtering analysis results in the message middleware into the distributed publish-subscribe messaging system Kafka, the distributed publish-subscribe messaging system Kafka can consume the data of the message middleware, and then parse the filtering analysis results according to the parameter information of the topic in the distributed publish-subscribe messaging system Kafka to obtain the group category corresponding to each user.

[0077] S6: Determine the group category to be recommended from the group categories according to the business information, and push the business information to users included in the group category to be recommended.

[0078] Specifically, different business information is matched to different group categories. Since the users included in the group categories have been determined, the corresponding business information is pushed to the users included in the corresponding group categories.

[0079] For example, the business information is about the introduction and preferential policies of a certain cosmetic product, so we need to find the group category that is interested in the cosmetic product, and the group category already contains users who are interested in it. Therefore, the business information is pushed to users in the group category who are interested in cosmetic products, so as to improve the push efficiency.

[0080] Furthermore, the means of push include batch synchronization, single synchronization and daily incremental synchronization. Among them, the daily incremental method can select the incremental data cycle (1, 30, 60, 90) days; when daily incremental synchronization is selected, the task will synchronize and calculate the number of users included in the group category at regular intervals every day to ensure data consistency. After delivery, the data is written to the message middleware; after the program consumes data from the distributed publish-subscribe message system kafka, it writes the calculated data to Hbase (request access) or Tidb (batch synchronization) according to the delivery method, and writes the number of users corresponding to each channel into kafka, thereby obtaining the number of users under each channel.

[0081] In this embodiment, group categories are generated based on preset user operation events and dynamic screening condition attributes, and point-of-view events are constructed. User data of users are collected through point-of-view events, which is conducive to obtaining different user data according to different group categories. Through data analysis of the distributed publish-subscribe message system kafka and the distributed computing engine Spark, the group category corresponding to each user is obtained, accurate push to users is achieved, and the efficiency of information push is improved.

[0082] See also Figure 3 , Figure 3 A specific implementation method of step S3 is shown. In step S3, parameter information of the topic in the distributed publish-subscribe messaging system kafka is configured, and based on the configured topic, data analysis is performed on user data stored in the distributed search engine ElasticSearch, and the specific implementation process of storing the obtained analysis results in the message middleware is described in detail as follows:

[0083] S31: Configure the parameter information of the topic in the distributed publish-subscribe messaging system Kafka, and perform data analysis on the user data based on the configured topic to obtain the analysis results.

[0084] Specifically, since the parameter information of the topic must be configured before sending and consuming messages using the distributed publish-subscribe messaging system Kafka, the parameter information of the topic is configured through the distributed publish-subscribe messaging system Kafka, mainly by configuring the type, name, and parameters of the topic, etc. After setting the topic, perform data analysis on the user data to obtain the analysis results.

[0085] S32: Encapsulate the analysis result into a json string to obtain the analysis result json string, and write the analysis result json string into the message middleware.

[0086] Specifically, the analysis results are written into the message middleware in the form of a json string, so that the subsequent distributed computing engine Spark can obtain the analysis results.

[0087] In this embodiment, the user data is analyzed through the parameter information of the topic in the distributed publish-subscribe message system Kafka, and the obtained analysis results are encapsulated into a JSON string and written into the information message middleware, which is conducive to the analysis of user data and facilitates the subsequent distributed computing engine Spark to obtain the analysis results, which is conducive to improving the analysis of user groups and thus improving the push efficiency.

[0088] See also Figure 4 , Figure 4 A specific implementation of step S4 is shown. In step S4, the distributed computing engine Spark is used to obtain the analysis results from the message middleware, and the analysis results are filtered and analyzed. The specific implementation process of writing the obtained filtered analysis results back to the message middleware is described in detail as follows:

[0089] S41: Encapsulate the analysis results into SQL commands.

[0090] Specifically, the above step S32 has converted the analysis results into the format of a json string. By obtaining the analysis results from the message and converting them into SQL commands of the structured query language, the encapsulated SQL commands are convenient for querying the user data analysis table generated in hive, and obtaining the number of users and user information through corresponding relationships.

[0091] S42: Execute SQL commands through the distributed computing engine Spark, search the table made of hive table, obtain the number of users and user information of the group category, and save the number of users and user information in the message middleware.

[0092] Specifically, the analysis results consumed from the message middleware are packaged into SQL commands in the form of JSON strings through the distributed computing engine Spark, and the SQL commands are executed to search the tables made by hive and filter and analyze the number of users and user information.

[0093] Among them, hive is a data warehouse tool based on Hadoop, which is used for data extraction, conversion and loading. It is a mechanism that can store, query and analyze large-scale data stored in Hadoop. The hive data warehouse tool can map structured data files into a database table, and provide SQL query function, which can convert SQL statements into MapReduce tasks for execution. In the present invention, since hive provides SQL query function, the analysis results are encapsulated into SQL commands to query the table made by hive through the distributed computing engine Spark. Hive can record and store the operation events performed by users in group categories in real time, and generate tables for storage. By encapsulating the analysis results in the form of SQL commands and comparing the analysis results with the data in the table generated by hive, the number of users and user information of group categories can be obtained.

[0094] The number of users refers to the number of users under the group information, and is the number of users who have clicked or browsed under the filtering conditions set for the group category.

[0095] Among them, user information is the personal information corresponding to each user under the group category, including each user's phone number, email and other information.

[0096] In this embodiment, since hive can record and store the operation events performed by users in group categories in real time and save them in the form of tables, the analysis results are encapsulated into SQL commands, and the SQL commands are executed through the distributed computing engine Spark, the tables made of hive tables are searched to obtain the number of users and user information in the group information, so as to determine the number of pushes and push target information for subsequent business information push, thereby improving the push efficiency.

[0097] See also Figure 5 , Figure 5 A specific implementation of step S42 is shown. In step S42, the SQL command is executed by the distributed computing engine Spark to search the table made of the hive table, obtain the number of users and user information of the group category, and save the number of users and user information in the message middleware. The specific implementation process is described in detail as follows:

[0098] S421: traverse the protected data in the table created in the hive library through SQL commands, filter and delete duplicate user data, and obtain the filtering result.

[0099] Specifically, since there may be repeated information in the user data, this part of the repeated information is equivalent to redundancy, and it can be filtered and deleted to obtain the filtering result.

[0100] For example, if a user repeatedly clicks on the same operation event in different time periods, the information recorded in the group category is considered duplicate information. In order to improve data processing efficiency, it can be filtered and deleted.

[0101] S422: Count the number of users in the grouping information of the filtering result, match the corresponding information of the users, obtain the number of users and the user information, and save the number of users and the user information in the theme middleware.

[0102] Specifically, since the SQL command is generated based on user data, by searching the table made by hive, the user information of the duplicate grouping information can be deleted. After statistics, the user data is obtained, and then the number of users and user information are obtained by matching the corresponding information of the user.

[0103] In this embodiment, duplicate grouping information is filtered and deleted, the number of users in the grouping information after filtering is counted, and the corresponding information of the users is matched to obtain the number of users and user information, thereby effectively improving the data processing of group categories, achieving the efficiency of obtaining the number of users and user information, improving the push accuracy, and thus improving the push efficiency.

[0104] Another specific implementation of step S4 is described in detail as follows:

[0105] Encapsulate the filtering analysis result into a json string to obtain the filtering analysis result json string.

[0106] Specifically, the filtering analysis results are encapsulated into a json string to facilitate the storage of the filtering analysis results and subsequent analysis thereof.

[0107] Write the filtering and analysis result json string back to the message middleware.

[0108] Specifically, the filtering and analysis results are repackaged into a JSON string and written back to the message middleware, providing a basis for the data of the Kafka consumption middleware in the subsequent step.

[0109] In this embodiment, by encapsulating the filtering and analysis results into a JSON string and writing the filtering and analysis result JSON string back to the message middleware, the processing efficiency of user data is improved, providing a basis for the data of the subsequent kafka consumption middleware, and then determining the group category of users, thereby improving the push efficiency.

[0110] After step S6, the kafka-based information push method further includes:

[0111] Through the scheduler Quartz, group categories are traversed regularly and the users corresponding to the group categories are updated.

[0112] Specifically, by setting a scheduled task and using the scheduler Quartz, group categories are traversed at preset times. If daily updates are set for group categories, the server writes the group categories and delivery channels into the distributed publish-subscribe messaging system kafka component, and then obtains the number of users and user information, thereby updating the users corresponding to the group categories.

[0113] The scheduler Quartz is an open source job scheduling framework written entirely in Java. In the present invention, the scheduler Quartz regularly updates the users corresponding to the group categories.

[0114] In this embodiment, the group categories are traversed regularly through the scheduler Quartz, and the users corresponding to the group categories are updated, which is beneficial to the update of group category data, the push of business information, and the improvement of the efficiency of business push.

[0115] After step S6, the kafka-based information push method further includes:

[0116] According to the generated attributes of the group category, the users included in the group category are assigned corresponding labels.

[0117] Specifically, according to the filtering condition attributes set when the group category is established, corresponding labels are assigned to the users included in the group category.

[0118] In a specific embodiment: the configured event name is "product added to shopping cart but not paid for", and the screening condition is "occurred 5 times". Through the data processing end, a user package is generated, which includes the users of this event. At this time, tags are processed and generated, and the users of this user package are labeled as "having purchase intention". Then, various channels can be used to recommend the product to the user, providing efficiency in information push.

[0119] In this implementation, by assigning corresponding labels to users included in the group categories according to the filtering condition attributes set when the group categories are established, the users included in different group categories are identified, which is convenient for later identification and classification.

[0120] like Figure 6 As shown, the Kafka-based information push device of this embodiment includes: a group category generation module 51, a user data collection module 52, a user data analysis module 53, a filtering and analysis result module 54, a group category determination module 55 and a business information push module 56, wherein:

[0121] A group category generating module 51 is used to generate group categories according to preset user operation events and dynamic screening condition attributes;

[0122] The user data collection module 52 is used to construct a buried event, collect user data of the user through the buried event, and store the user data in the distributed search engine ElasticSearch;

[0123] The user data analysis module 53 is used to configure the parameter information of the topic in the distributed publish-subscribe messaging system kafka, and based on the configured topic, perform data analysis on the user data stored in the distributed search engine ElasticSearch, and store the obtained analysis results in the messaging middleware;

[0124] The filtering analysis result module 54 is used to obtain the analysis results from the message middleware by using the distributed computing engine Spark, and to filter and analyze the analysis results, and write the obtained filtering analysis results back to the message middleware;

[0125] The group category determination module 55 is used to write the filtering analysis results in the message middleware into the distributed publish-subscribe message system Kafka, and parse the filtering analysis results through the distributed publish-subscribe message system Kafka to obtain the group category corresponding to each user;

[0126] The business information push module 56 is used to determine the group category to be recommended from the group categories according to the business information, and push the business information to the users included in the group category to be recommended.

[0127] Furthermore, the user data analysis module 53 includes:

[0128] The topic setting unit is used to configure the parameter information of the topic in the distributed publish-subscribe messaging system Kafka, and analyze the user data based on the configured topic to obtain the analysis results;

[0129] The analysis result encapsulation unit is used to encapsulate the analysis result into a json string, obtain the analysis result json string, and write the analysis result json string into the message middleware.

[0130] Furthermore, the filtering analysis result module 54 includes:

[0131] SQL command generation unit, used to encapsulate the analysis results into SQL commands;

[0132] The user data acquisition unit is used to execute SQL commands through the distributed computing engine Spark, search the table made of the hive table, obtain the number of users and user information of the grouping information, and save the number of users and user information in the message middleware.

[0133] Furthermore, the user data acquisition unit includes:

[0134] The filtering result acquisition subunit is used to traverse the user data in the table created by the hive library through SQL commands, filter and delete duplicate user data, and obtain the filtering results;

[0135] The user matching subunit is used to count the number of users in the group category of the filtering result, match the corresponding information of the users, obtain the number of users and user information, and save the number of users and user information in the theme middleware.

[0136] The filtering analysis result module 54 also includes:

[0137] The string acquisition unit is used to encapsulate the filtering analysis result into a json string to obtain the filtering analysis result json string;

[0138] The string writing unit is used to write the json string of the filtering and analysis results back to the message middleware.

[0139] Furthermore, the information push device based on kafka also includes:

[0140] The group category update unit module is used to traverse the group categories regularly through the scheduler Quartz and update the users corresponding to the group categories.

[0141] Furthermore, the kafka-based information push device also includes:

[0142] The label acquisition module is used to assign corresponding labels to users included in the group category according to the generated attributes of the group category.

[0143] To solve the above technical problems, the present application also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0144] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 having three components: a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0145] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0146] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as the program code of the information push method based on kafka, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0147] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the program code stored in the memory 61 or process data, such as running a program code of a kafka-based information push method.

[0148] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0149] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores a server maintenance program. The server maintenance program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned Kafka-based information push method.

[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0151] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0152] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for pushing information based on Kafka, characterized in that: include: Generate group categories based on preset user operation events and dynamic screening condition attributes; Construct a tracking event, collect user data of the user through the tracking event, and store the user data in the distributed search engine ElasticSearch; Configure the parameter information of the topic in the distributed publish-subscribe messaging system kafka, and based on the configured topic, perform data analysis on the user data stored in the distributed search engine ElasticSearch, and store the obtained analysis results in the messaging middleware; Using the distributed computing engine Spark, the analysis results are obtained from the message middleware, and the analysis results are filtered and analyzed, and the obtained filtered analysis results are written back to the message middleware; The filtering analysis results in the message middleware are written into the distributed publish-subscribe message system Kafka, and the filtering analysis results are parsed and processed by the distributed publish-subscribe message system Kafka to obtain the group category corresponding to each of the users; According to the service information, a group category to be recommended is determined from the group categories, and the service information is pushed to users included in the group category to be recommended.

2. The information push method based on kafka according to claim 1 is characterized in that , the configuration of the parameter information of the topic in the distributed publish-subscribe message system kafka, and based on the configured topic, performing data analysis on the user data stored in the distributed search engine ElasticSearch, and storing the obtained analysis results in the message middleware includes: Configure parameter information of the topic in the distributed publish-subscribe messaging system kafka, and perform data analysis on the user data based on the configured topic to obtain analysis results; The analysis result is encapsulated into a json string to obtain an analysis result json string, and the analysis result json string is written into the message middleware.

3. The information push method based on kafka according to claim 1 is characterized in that: The adopting of the distributed computing engine Spark to obtain the analysis result from the message middleware, filtering and analyzing the analysis result, and writing the obtained filtering and analysis result back to the message middleware includes: Encapsulating the analysis results into SQL commands; The SQL command is executed by the distributed computing engine Spark, and the table made of the hive table is searched to obtain the number of users and user information of the group category, and the number of users and the user information are stored in the message middleware.

4. The information push method based on kafka according to claim 3 is characterized in that: The step of executing the SQL command through the distributed computing engine Spark, searching the table made from the hive table, obtaining the number of users and user information of the group category, and storing the number of users and the user information in the message middleware includes: Traversing the user data in the hive table through the SQL command, filtering and deleting duplicate user data, and obtaining a filtering result; The number of users in the group category of the filtering result is counted, and the user corresponding information is matched to obtain the number of users and the user information, and the number of users and the user information are saved in the middleware of the topic.

5. The information push method based on kafka according to claim 1, characterized in that: The method of using the distributed computing engine Spark to obtain the analysis result from the message middleware, filtering and analyzing the analysis result, and writing the obtained filtering and analysis result back to the message middleware further includes: Encapsulate the filtering analysis result into a json string to obtain a filtering analysis result json string; The filtering and analysis result json string is written back to the message middleware.

6. The kafka-based information push method according to any one of claims 1 to 5, characterized in that: After determining the group category to be recommended from the group categories according to the service information, and pushing the service information to users included in the group category to be recommended, the method further includes: The group categories are traversed periodically through the scheduler Quartz, and the users corresponding to the group categories are updated.

7. The kafka-based information push method according to any one of claims 1 to 5, characterized in that: After determining the group category to be recommended from the group categories according to the service information, and pushing the service information to users included in the group category to be recommended, the method further includes: According to the generated attributes of the group category, corresponding labels are assigned to the users included in the group category.

8. A kafka-based information push device, characterized in that: include: A group category generation module is used to generate group categories based on preset user operation events and dynamic screening condition attributes; A user data collection module is used to construct a point-of-view event, collect user data of users through the point-of-view event, and store the user data in a distributed search engine ElasticSearch; The user data analysis module is used to configure the parameter information of the topic in the distributed publish-subscribe messaging system kafka, and based on the configured topic, perform data analysis on the user data stored in the distributed search engine ElasticSearch, and store the obtained analysis results in the message middleware; A filtering and analysis result module is used to obtain the analysis result from the message middleware by using the distributed computing engine Spark, and to filter and analyze the analysis result, and write the obtained filtering and analysis result back to the message middleware; A group category determination module is used to write the filtering analysis results in the message middleware into the distributed publish-subscribe message system Kafka, and parse the filtering analysis results through the distributed publish-subscribe message system Kafka to obtain the group category corresponding to each user; The business information push module is used to determine the group category to be recommended from the group category according to the business information, and push the business information to users included in the group category to be recommended.

9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the kafka-based information push method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the Kafka-based information push method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Movie recommendation system and method based on Spark

    CN110717093A

  • Business information processing method and system and computer readable storage medium

    CN110913000A

  • Control method and apparatus for publish / subscribe systems

    WO2015027853A1