Method, system, terminal and storage medium for population subscription based on kafka

Abstract

The application discloses a crowd subscription method and system based on kafka, a terminal and a storage medium. The method comprises the following steps: acquiring crowd data of different crowds, controlling a crowd production program to send the crowd data to kafka, receiving state data returned by the kafka, and storing the state data into a relational database management system; when a request sent by a subscription page is received, the state data is displayed on the subscription page, and a crowd is subscribed in the subscription page; based on the crowd, target state data corresponding to the crowd in the relational database management system is acquired, and the target state data is sent to the kafka for consumption, so as to complete the subscription of the crowd. When each team uses the user portrait system, the required crowd can be acquired for data extraction in a subscription mode, and the subscription can be performed before or after the crowd is executed.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a Kafka-based method, system, terminal, and storage medium for user subscription. Background Technology

[0002] User personas are virtual representations of real users. They are target user models built on a series of real data (marketing data, usability data). By conducting user research to understand users, they are divided into different types based on differences in their goals, behaviors, and opinions. Then, typical characteristics are extracted from each type and given a name, photo, some demographic elements, and scene descriptions to form a persona.

[0003] User profiling systems are currently extremely common in existing technologies, with various teams using company user profiling systems. However, each team only needs the user information of its own team. Secondly, it is not possible to send all user data to all teams. Most companies store user information through a storage system, and each team can query and extract it. This results in low timeliness, and user data can only be extracted after the process is completed.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a Kafka-based method, system, terminal, and storage medium for audience subscription, aiming to solve the problems in the prior art where data can only be extracted after execution and the extraction timeliness is not high.

[0006] To achieve the above objectives, this invention provides a Kafka-based audience subscription method, which includes the following steps:

[0007] Acquire population data from different population groups, control the population generation program to send the population data to Kafka, receive the status data returned by Kafka, and store the status data in a relational database management system;

[0008] When a request is received from the subscription page, the status data is displayed on the subscription page, and the subscribed audience is displayed on the subscription page.

[0009] Based on the population, the target status data corresponding to the population in the relational database management system is obtained, and the target status data is sent to the Kafka for consumption to complete the subscription of the population.

[0010] Optionally, the method for crowd subscription based on kafka, wherein the crowd data of different crowds is acquired, a crowd production program is controlled to send the crowd data to kafka, and state data returned by the kafka is received and stored in a relational database management system, and the method specifically comprises the following steps:

[0011] The crowd data of different crowds is acquired, and the crowd production program is controlled to send the crowd data to kafka according to production specifications and data specifications.

[0012] When the kafka stores the received crowd data, state data of different crowds returned by the kafka is received and stored in the relational database management system after being marked.

[0013] Optionally, the method for crowd subscription based on kafka, wherein the state data comprises Topic, partition, start offset and end offset.

[0014] Optionally, the method for crowd subscription based on kafka, wherein when a request sent by a subscription page is received, the state data is displayed on the subscription page, and a crowd is subscribed in the subscription page, and the method specifically comprises the following steps:

[0015] When the subscription page sends a request for viewing state data within a preset time, the relational database management system is controlled to display state data of all crowds on the subscription page.

[0016] Based on the state data, a crowd is selected on the subscription page for subscription.

[0017] Optionally, the method for crowd subscription based on kafka, wherein based on the crowd, target state data corresponding to the crowd in the relational database management system is acquired, and the target state data is sent to the kafka for consumption to complete subscription of the crowd, and the method specifically comprises the following steps:

[0018] Based on the subscribed crowd, target state data corresponding to the crowd is acquired from the relational database management system according to an identifier of the crowd.

[0019] Based on the target state data, a crowd consumption program is started, and the crowd consumption program is controlled to send a request for acquiring crowd data of the crowd to the kafka.

[0020] When the crowd consumption program receives the crowd data, the crowd consumption program is controlled to send a state of successful crowd subscription to the subscription page.

[0021] Optionally, the method for crowd subscription based on kafka, wherein the crowd data comprises detailed data and end symbol.

[0022] Optionally, the method for crowd subscription based on kafka, wherein the preset time is 5-20s.

[0023] Optionally, the method for crowd subscription based on kafka, wherein the method for crowd subscription based on kafka comprises:

[0024] a subscription module, configured to subscribe to a crowd, view execution status of the crowd and status data of the crowd in Kafka;

[0025] a crowd production module, configured to send crowd data to Kafka and return status data of the crowd according to production specification and data specification;

[0026] a kafka module, configured to store the crowd data in partition;

[0027] a crowd consumption module, configured to obtain status data in the subscribed crowd and perform crowd data consumption on the status data.

[0028] In addition, to achieve the above-mentioned purposes, the application further provides a terminal, wherein the terminal comprises a memory, a processor and a crowd subscription program based on kafka stored in the memory and executable on the processor, and the crowd subscription program based on kafka implements the steps of the method for crowd subscription based on kafka when executed by the processor.

[0029] In addition, to achieve the above-mentioned purposes, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a crowd subscription program based on kafka, and the crowd subscription program based on kafka implements the steps of the method for crowd subscription based on kafka when executed by a processor.

[0030] The present application obtains crowd data of different crowds, controls a crowd production procedure to send the crowd data to a kafka, receives state data returned by the kafka, stores the state data into a relational database management system, when receiving a request sent by a subscription page, displays the state data on the subscription page, subscribes a crowd in the subscription page, obtains target state data corresponding to the crowd in the relational database management system based on the crowd, and sends the target state data to the kafka for consumption, so as to complete the subscription of the crowd. When each team uses the user portrait system, the required crowd can be obtained for data extraction by adopting the subscription mode, and the subscription can be performed before or after the crowd is executed. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a flowchart of a preferred embodiment of the crowd subscription method based on the kafka in the present application;

[0032] Figure 2 is a flowchart of step S10 in the preferred embodiment of the crowd subscription method based on the kafka in the present application;

[0033] Figure 3 is a timing diagram of production and subscription of the crowd subscription method based on the kafka in the present application;

[0034] Figure 4 is a flowchart of step S20 in the preferred embodiment of the crowd subscription method based on the kafka in the present application;

[0035] Figure 5 is a flowchart of step S30 in the preferred embodiment of the crowd subscription method based on the kafka in the present application;

[0036] Figure 6 is a timing diagram of consumption and subscription of the crowd subscription method based on the kafka in the present application;

[0037] Figure 7 is a principle diagram of the preferred embodiment of the crowd subscription system based on the kafka in the present application;

[0038] Figure 8 is a running environment diagram of the preferred embodiment of the terminal in the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present application more clear and definite, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0040] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship, motion condition, etc. between components in a certain specific posture (as shown in the drawings), and if the specific posture changes, the directional indications will also change accordingly.

[0041] In addition, if the embodiments of the present application involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of a person skilled in the art, and when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0042] The population subscription method based on kafka provided by the preferred embodiment of the present application comprises the following steps: Figure 1 As shown in the figure, the population subscription method based on kafka comprises the following steps:

[0043] Step S10, obtain population data of different populations, control the population production program to send the population data to kafka, receive state data returned by the kafka, and store the state data into a relational database management system.

[0044] For specific processes, please refer to Figure 2 which is the flowchart of step S10 in the population subscription method based on kafka provided by the present application.

[0045] As shown in the figure, the step S10 comprises: Figure 2

[0046] Step S11, obtain population data of different populations, control the population production program to send the population data to kafka according to the production specification and data specification;

[0047] Step S12, when the kafka stores the received population data, receive state data of different populations returned by the kafka, and store the state data into a relational database management system after marking.

[0048] Specifically, as shown in the figure, Figure 3 ​As shown, the user starts a crowd production program, and sends the crowd data (including detailed data and end symbols) of different crowds obtained through the crowd production program to a kafka (message queue tool) according to production specifications and data specifications; when the kafka stores the received crowd data, the kafka returns state data (Topic, partition, start Offset (start offset) and end Offset (end offset)) of different crowds to the crowd production program, and the crowd production program stores the Topic, partition, start Offset and end Offset to a Mysql (relational database management system) after marking; when a subscription page sends an instruction to the relational database management system to request the Topic, partition, start Offset and end Offset of all crowds, the relational database management system displays the Topic, partition, start Offset and end Offset of all crowds on the subscription page, and the user can obtain the Topic, partition, start Offset and end Offset of the crowd through an interface (for example, an HTTP interface).

[0049] Step S20, when receiving the request sent by the subscription page, display the state data on the subscription page and subscribe to the crowd in the subscription page.

[0050] For specific processes, please refer to Figure 4 which is a flowchart of step S20 in the crowd subscription method based on kafka provided by the application.

[0051] As shown in Figure 4 , the step S20 includes:

[0052] Step S21, when the subscription page sends a request to view the state data within a preset time, control the relational database management system to display the state data of all crowds on the subscription page.

[0053] Step S22, based on the state data, select a crowd to subscribe to on the subscription page.

[0054] Specifically, when the subscription page sends a request to view the Topic, partition, start Offset and end Offset of the crowd to the relational database management system within a preset time (for example, 5-20s, 10s in the embodiment of the application), the relational database management system displays the Topic, partition, start Offset and end Offset of all crowds on the subscription page and selects a crowd to subscribe to on the subscription page.

[0055] Step S30, based on the crowd, obtaining target state data corresponding to the crowd in the relational database management system, and sending the target state data to the kafka for consumption, to complete the subscription of the crowd.

[0056] For specific process, please refer to Figure 5 , which is the flow chart of step S30 in the crowd subscription method based on kafka provided by the application.

[0057] As shown in Figure 5 , the step S20 includes:

[0058] Step S31, based on the subscribed crowd, obtaining target state data corresponding to the crowd from the relational database management system according to the identification of the crowd;

[0059] Step S32, starting a crowd consumption program based on the target state data, and controlling the crowd consumption program to send a crowd data request for obtaining the crowd to the kafka;

[0060] Step S33, when the crowd consumption program receives the crowd data, controlling the crowd consumption program to send a state of successful crowd subscription to the subscription page.

[0061] Specifically, as shown in Figure 6 , a user subscribes to a crowd M in the subscription page, the subscription page queries the relational database management system according to the identification of the crowd M to obtain the Topic, partition, start OffsetH and end Offset corresponding to the crowd M, and starts a crowd consumption program, the crowd consumption program starts to read the crowd M, and sends a crowd data request for obtaining the crowd M to the kafka; after accepting the request, the kafka sends the crowd data of the crowd M to the crowd consumption program, the crowd consumption program starts to receive the information of the crowd M, and after reading all the information of the crowd M, sends a state of successful subscription of the crowd M to the subscription page to be seen by the user.

[0062] Further, as shown in Figure 7 , based on the above crowd subscription method based on kafka, the application also correspondingly provides a crowd subscription system based on kafka, the crowd subscription system based on kafka includes:

[0063] The subscription module 51 is used for subscribing to a crowd, viewing the execution state of the crowd and the state data of the crowd in the kafka;

[0064] The production population module 52 is configured to send population data to Kafka according to the production specification and the data specification, and return state data of the population;

[0065] The kafka module 53 is configured to partition storage of the population data.

[0066] The consumption population module 54 is configured to acquire state data of a subscribed population, and perform data consumption of the population based on the state data.

[0067] Further, as shown in Figure 8 Based on the above-mentioned population subscription method based on Kafka, the application further provides a terminal, which comprises a processor 10, a memory 20 and a display 30. Figure 8 Only some components of the terminal are shown, but it should be understood that all the shown components are not required to be implemented, and more or less components can be alternatively implemented.

[0068] The memory 20 can be an internal storage unit of the terminal, such as a hard disk or a memory of the terminal, in some embodiments. The memory 20 can also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., in other embodiments. Further, the memory 20 can include both the internal storage unit and the external storage device of the terminal. The memory 20 is configured to store application software and various data installed on the terminal, such as program codes of the terminal, etc. The memory 20 can also be configured to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores a population subscription program based on Kafka 40, which can be executed by the processor 10, so as to implement the population subscription method based on Kafka.

[0069] The processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip, which is configured to run program codes or process data stored in the memory 20, such as to execute the population subscription method based on Kafka, etc., in some embodiments.

[0070] The display 30 can be, in some embodiments, an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. The display 30 is used to display information at the terminal and to display a visualized user interface. The components 10-30 of the terminal communicate with each other through a system bus.

[0071] In an embodiment, the following steps are implemented when the processor 10 executes the interface display program 40 of the split window in the memory 20:

[0072] Obtain crowd data of different crowds, send the crowd data to a kafka by a crowd production program, receive state data returned by the kafka, and store the state data in a relational database management system;

[0073] When receiving a request sent by a subscription page, display the state data on the subscription page and subscribe to a crowd in the subscription page;

[0074] Based on the crowd, obtain target state data corresponding to the crowd in the relational database management system, and send the target state data to the kafka for consumption to complete subscription of the crowd.

[0075] The obtaining of the crowd data of different crowds, the sending of the crowd data to the kafka by the crowd production program, the receiving of the state data returned by the kafka, and the storing of the state data in the relational database management system specifically include:

[0076] The crowd production program sends the crowd data to the kafka according to a production specification and a data specification.

[0077] When the kafka stores the received crowd data, receive state data of different crowds returned by the kafka, and store the state data in a relational database management system after marking.

[0078] The state data includes a topic, a partition, a start offset, and an end offset.

[0079] The displaying of the state data on the subscription page and the subscribing to a crowd in the subscription page when receiving a request sent by a subscription page specifically include:

[0080] When the subscription page sends a request for viewing state data within a preset time, the relational database management system is controlled to display state data of all crowds on the subscription page.

[0081] Based on the state data, a crowd is selected for subscription on the subscription page.

[0082] Among them, based on the crowd, the target state data corresponding to the crowd in the relational database management system is obtained, and the target state data is sent to the kafka for consumption to complete the subscription of the crowd, specifically including:

[0083] Based on the subscribed crowd, the target state data corresponding to the crowd is obtained from the relational database management system according to the identification of the crowd;

[0084] Based on the target state data, a crowd consumption program is started, and the crowd consumption program is controlled to send a crowd data request for obtaining the crowd to the kafka;

[0085] When the crowd consumption program receives the crowd data, the crowd consumption program is controlled to send a state of successful crowd subscription to the subscription page.

[0086] Among them, the crowd data includes detailed data and an end symbol.

[0087] Among them, the preset time is 5-20s.

[0088] The application also provides a computer readable storage medium, wherein the computer readable storage medium stores a kafka-based crowd subscription program, and the kafka-based crowd subscription program is executed by a processor to realize the steps of the above-mentioned kafka-based crowd subscription method.

[0089] In summary, the application provides a kafka-based crowd subscription method, system, terminal and storage medium, the method comprising: obtaining crowd data of different crowds, controlling a crowd production program to send the crowd data to a kafka, and receiving state data returned by the kafka and storing the state data in a relational database management system; when receiving a request sent by a subscription page, displaying the state data on the subscription page and subscribing a crowd in the subscription page; based on the crowd, obtaining target state data corresponding to the crowd in the relational database management system, and sending the target state data to the kafka for consumption to complete the subscription of the crowd. Through the use of the user portrait system by each team, the required crowd can be obtained for data extraction in a subscription mode, and the subscription can be performed before or after the crowd is executed.

[0090] It should be noted that, in the present document, the terms "comprising", "comprising" or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article, or apparatus that comprises a list of elements not only includes those elements, but also includes other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0091] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer readable computer readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The computer readable storage medium can be a memory, a magnetic disc, an optical disc, etc.

[0092] It should be understood that the application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall fall within the protection scope of the appended claims of the present application.

Claims

1. A kafka-based crowd subscription method, characterized in that, The crowd subscription method based on kafka comprises the following steps: Obtaining crowd data of different crowds, controlling a crowd production program to send the crowd data to kafka, receiving state data returned by the kafka, and storing the state data into a relational database management system; When receiving a request sent by a subscription page, displaying the state data on the subscription page and subscribing to a crowd in the subscription page; The step of displaying the state data on the subscription page and subscribing to a crowd in the subscription page when receiving a request sent by a subscription page comprises the following steps: When the subscription page sends a request for viewing state data within a preset time, controlling the relational database management system to display state data of all crowds on the subscription page; Based on the state data, selecting a crowd for subscription on the subscription page; Based on the crowd, obtaining target state data corresponding to the crowd in the relational database management system, and sending the target state data to the kafka for consumption to complete the subscription of the crowd; The step of obtaining target state data corresponding to the crowd in the relational database management system based on the crowd and sending the target state data to the kafka for consumption to complete the subscription of the crowd comprises the following steps: Based on the subscribed crowd, obtaining target state data corresponding to the crowd from the relational database management system according to the identification of the crowd; Based on the target state data, starting a crowd consumption program, and controlling the crowd consumption program to send a request for obtaining crowd data of the crowd to the kafka; When the crowd consumption program receives the crowd data, controlling the crowd consumption program to send a state of successful crowd subscription to the subscription page; The required crowd is obtained through subscription for data extraction, and the subscription time can be before or after the execution of the crowd.

2. The kafka-based crowd subscription method of claim 1, wherein, The step of obtaining crowd data of different crowds, controlling a crowd production program to send the crowd data to kafka, and receiving state data returned by the kafka and storing the state data into a relational database management system comprises the following steps: Obtaining crowd data of different crowds, controlling a crowd production program to send the crowd data to kafka according to production specifications and data specifications; When the kafka stores the received crowd data, receiving state data of different crowds returned by the kafka, and storing the state data into a relational database management system after marking.

3. The kafka-based crowd subscription method of claim 2, wherein, The state data comprises Topic, partition, start offset and end offset.

4. The kafka-based crowd subscription method of claim 1, wherein, The crowd data comprises detailed data and end symbol. 5.The kafka-based crowd subscription method of claim 1, wherein, The preset time is 5-20s. 6.A kafka-based crowd subscription system, characterized in that, The crowd subscription system based on kafka is applied to the crowd subscription method based on kafka in any one of claims 1-5, and the crowd subscription system based on kafka comprises: The subscription module is configured to subscribe to the crowd, view the execution status of the crowd and the status data of the crowd in Kafka. The crowd production module is configured to send the crowd data to Kafka and return the status data of the crowd according to the production specification and the data specification. The kafka module is configured to store the crowd data in a partition. The crowd consumption module is configured to obtain the status data of the subscribed crowd and perform crowd data consumption on the status data.

7. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a kafka-based crowd subscription program stored in the memory and executable on the processor, and the processor executes the kafka-based crowd subscription program to implement the steps of the kafka-based crowd subscription method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a kafka-based crowd subscription program, and the processor executes the kafka-based crowd subscription program to implement the steps of the kafka-based crowd subscription method according to any one of claims 1-5.

Images