A user grouping method, device and storage medium
By separating hot and cold tags and utilizing different data warehouse storage and query schemes, the problems of complex queries, slow speed, and large storage space consumption in existing user segmentation schemes have been solved, achieving second-level user segmentation and improving query efficiency and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TENCENT MUSIC ENTERTAINMENT TECH (SHENZHEN) CO LTD
- Filing Date
- 2023-05-19
- Publication Date
- 2026-07-24
AI Technical Summary
Existing user segmentation solutions suffer from problems such as complex query logic, slow query speed (minutes), unstable query performance, and excessive storage space consumption.
By separating hot and cold tags, different data warehouses are used to store user data of different tag types, and different query schemes are adopted. Specifically, hot tag data is stored in the real-time data warehouse and queried using bitmap technology, while cold tag data is stored in the offline data warehouse and queried using ordinary SQL.
This approach achieves a significant improvement in user segmentation speed by an order of magnitude, reaching sub-second segmentation, while reducing storage space usage, thus enhancing query efficiency and user experience.
Smart Images
Figure CN116595106B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a user grouping method, device and storage medium. Background Technology
[0002] Currently, various software applications store user data. To analyze large numbers of users and understand their needs to better drive subsequent business, users are typically tagged and segmented. For example, various tags can be generated for music software users, such as age, gender, VIP status, number of times they listened to music in the past week, and number of times they sang karaoke in the past week. Business personnel can then select user groups that meet certain requirements and push promotional activities to these groups. This process of selecting users is called user segmentation. However, existing user segmentation solutions suffer from drawbacks such as complex query logic, slow query speed (minutes), unstable query performance, and high storage space consumption.
[0003] Therefore, the aforementioned technical problems urgently need to be solved by those skilled in the art. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a user segmentation method, device, and storage medium that can achieve second-level segmentation while reducing storage space occupancy. The specific solution is as follows:
[0005] The first aspect of this application provides a user segmentation method, including:
[0006] Obtain the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags;
[0007] If multiple preset tags are all hot tags, then the user groups that match the tag rules are queried in the real-time data warehouse; the real-time data warehouse stores user data whose tag type is hot tag.
[0008] If any of the preset tags contain a tag of type "cold tag", then the user group that matches the tag rules is queried in the offline data warehouse; the offline data warehouse stores user data of all tag types.
[0009] Optionally, the user segmentation method further includes:
[0010] Real-time statistics on the frequency with which each preset tag is selected by the client;
[0011] The label type of each preset label is determined based on its frequency; the label type is either a cold label or a hot label.
[0012] Optionally, the user segmentation method further includes:
[0013] The user data corresponding to the preset tag with the tag type of hot tag in the offline data warehouse is synchronized to the real-time data warehouse in real time, so that the real-time data warehouse stores user data with the tag type of hot tag.
[0014] Optionally, after synchronizing the user data corresponding to the preset tag with the tag type of hot tag in the offline data warehouse to the real-time data warehouse in real time, the method further includes:
[0015] The synchronized user data is populated into the target vertical table constructed in the real-time data warehouse, so that the real-time data warehouse stores user data with the tag type hot tag; each row of the target vertical table stores at least one preset tag, a tag value of the preset tag, and a sequence containing a user identifier with the preset tag and the tag value.
[0016] Optionally, the user identifier stored in each row of the target vertical table is stored in bitmap format.
[0017] Optionally, querying the user group that matches the tag rules in the real-time data warehouse includes:
[0018] The user identifiers that match the tag rules are retrieved from the target vertical table in the real-time data warehouse.
[0019] Optionally, querying the user group that matches the tag rules in the real-time data warehouse includes:
[0020] The first database engine is used to transform the tag rules into a first query statement, and the first query statement is executed in the real-time data warehouse to obtain the user groups that conform to the tag rules;
[0021] The process of querying user groups that match the tag rules in the offline data warehouse includes:
[0022] The tag rules are transformed into a second query statement using a second database engine, and the second query statement is executed in the offline data warehouse to obtain the user groups that conform to the tag rules.
[0023] Optionally, the user segmentation method further includes:
[0024] The user groups that conform to the aforementioned tag rules are packaged to obtain the corresponding user groups;
[0025] The crowd data package is uploaded to a cloud storage platform so that the client can download the crowd data package from the cloud storage platform.
[0026] A second aspect of this application provides an electronic device including a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the aforementioned user grouping method.
[0027] A third aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the aforementioned user grouping method.
[0028] In this application, the tag rules created by the client are first obtained. These tag rules include multiple preset tags selected by the client and the logical relationships between them. If all preset tags are hot tags, a user group matching the tag rules is queried in the real-time data warehouse. The real-time data warehouse stores user data with hot tags. If there are cold tags among the preset tags, a user group matching the tag rules is queried in the offline data warehouse. The offline data warehouse stores user data for all tag types. Therefore, this application reduces storage space usage by separating hot and cold tags and using different data warehouses to store user data for different tag types. Furthermore, different query schemes are used for different tag types, thereby increasing user segmentation speed by an order of magnitude, achieving second-level segmentation. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0030] Figure 1 A schematic diagram of the hardware framework applicable to the user segmentation method provided in this application;
[0031] Figure 2 A flowchart of a user segmentation method provided in this application;
[0032] Figure 3 An example diagram is created for a specific labeling rule provided in this application;
[0033] Figure 4 A flowchart illustrating a specific user segmentation method provided in this application;
[0034] Figure 5 A flowchart illustrating a specific user segmentation method provided in this application;
[0035] Figure 6 An example diagram of a specific tag storage structure in a real-time data warehouse provided in this application;
[0036] Figure 7 An example diagram is created for a specific labeling rule provided in this application;
[0037] Figure 8 An example diagram of a specific tag storage structure in an offline data warehouse provided in this application;
[0038] Figure 9 A flowchart illustrating a specific user segmentation method provided in this application;
[0039] Figure 10 A schematic diagram illustrating a specific user segmentation method provided in this application;
[0040] Figure 11 A specific user segmentation method provided in this application is illustrated with an effect diagram.
[0041] Figure 12 A specific user segmentation method provided in this application is illustrated with an effect diagram.
[0042] Figure 13 This is a schematic diagram of a user grouping device provided in this application. Detailed Implementation
[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0044] Existing user segmentation schemes suffer from drawbacks such as complex query logic, slow query speed (minutes on the order of minutes), unstable query performance, and high storage space consumption. To address these shortcomings, this application provides a user segmentation scheme that separates tags into hot and cold tags, utilizing different data warehouses to store user data of different tag types to reduce storage space consumption. Furthermore, different query schemes are employed for different tag types, thereby increasing user segmentation speed by an order of magnitude, achieving segmentation within seconds.
[0045] To facilitate understanding, we will first introduce the hardware framework applicable to the user segmentation method scheme in this application. See [link to relevant documentation]. Figure 1 ,in, Figure 1 It shows a schematic diagram of the hardware framework to which a user grouping method of this application applies.
[0046] Depend on Figure 1 It can be seen that the hardware framework may include an electronic device 10, wherein the electronic device 10 may include a processor 11, a memory 12, a communication interface 13, an input / output interface 14, and a communication bus 15. The processor 11, memory 12, communication interface 13, and input / output interface 14 all communicate with each other through the communication bus 15.
[0047] In this embodiment, the processor 11 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, an off-the-shelf programmable gate array, or other programmable logic devices. The processor can call programs stored in the memory 12. Specifically, the processor can perform operations executed on the computer device side in the following embodiments of the user grouping method.
[0048] The memory 12 stores one or more programs, which may include program code, including computer operation instructions. The program code can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as one or more of Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. In this embodiment, the memory stores at least a program for implementing the following functions:
[0049] Install and run an instrumentation tool on the web platform to collect tracking data when users access web pages on the web platform.
[0050] The tracking data is saved in the local storage of the web client, and a data index is created for each piece of tracking data;
[0051] Obtain the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags;
[0052] If multiple preset tags are all hot tags, then the user groups that match the tag rules are queried in the real-time data warehouse; the real-time data warehouse stores user data whose tag type is hot tag.
[0053] If any of the preset tags contain a tag of type "cold tag", then the user group that matches the tag rules is queried in the offline data warehouse; the offline data warehouse stores user data of all tag types.
[0054] In one possible implementation, the memory 12 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as sound playback function, image playback function, etc.). The data storage area may store data created during the use of the electronic device, such as tag data, user data, etc.
[0055] The communication interface 13 can be an interface for a communication module, such as the interface for a GSM module.
[0056] Input / output interface 14 provides an interface between processor 11 and other interface modules, such as keyboards, mice, and buttons. These buttons can be virtual or physical. Communication interface 13 is used for wired or wireless communication between electronic device 10 and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 3G, 4G, or 5G, or a combination thereof. Therefore, the corresponding communication component 105 may include a Wi-Fi component, a Bluetooth component, and an NFC component.
[0057] Electronic device 10 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute user grouping methods.
[0058] certainly, Figure 1 The structure of the electronic device 10 shown does not constitute a limitation on the computer device in the embodiments of this application. In practical applications, the electronic device may include more than Figure 1 More or fewer components as shown, or combinations of certain components.
[0059] in, Figure 1 The electronic device 10 can be a terminal (such as a mobile terminal like a mobile phone or tablet computer, or a fixed terminal like a PC), a server, or a smart electronic device.
[0060] Figure 2 A flowchart illustrating a user segmentation method provided in an embodiment of this application. See also... Figure 2 As shown, the user segmentation method includes:
[0061] S11: Obtain the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags.
[0062] In this embodiment, the entity performing user segmentation can be a data processing platform with user segmentation functionality. This step first obtains the tag rules created by the client. Here, the client is the user terminal using the aforementioned data processing platform for user segmentation. The tag rules include multiple preset tags selected by the client and the logical relationships between the tags.
[0063] In this context, the preset tags refer to tags already assigned to users, and the client needs to select a subset of these preset tags to create the tag rules. The tag rules can be set according to user characteristics, and this embodiment does not limit this. For example, for music software users, the tag rules could be age, membership status, whether they are fans of a particular singer, listening time in the past month, listening frequency in the past month, etc. Furthermore, in addition to including the multiple preset tags selected by the client, the tag rules can also include logical relationships between these preset tags, including but not limited to "AND" and "OR". Taking the platform's user segmentation function as an example, the client can select different preset tags on the platform and then connect these preset tags using "AND" and "OR", thus creating different tag rules. The backend can then select all users who meet the rule conditions based on these tag rules. Figure 3 The image shows a specific example of creating a tag rule.
[0064] S12: If the tag type of multiple preset tags is hot tag, then query the user group that matches the tag rules in the real-time data warehouse; the real-time data warehouse stores user data with tag type hot tag.
[0065] In this embodiment, after obtaining the tag rules created by the client, the types of tags contained in the tag rules are determined, and different query schemes are adopted according to different tag types. The tag types here are mainly divided into hot tags and cold tags. Hot tags are tags used more frequently, while cold tags are tags used less frequently.
[0066] In this embodiment, if multiple preset tags are all hot tags, user groups matching the tag rules can be queried in the real-time data warehouse. This embodiment does not limit the specific query method; relevant technologies of real-time data warehouses can be referenced. For example, a first database engine can be used to convert the tag rules into a first query statement, and the first query statement can be executed in the real-time data warehouse to obtain user groups matching the tag rules. Here, the first database engine corresponds to the real-time data warehouse; the first query statement is an SQL query statement generated according to the tag rules and recognizable and processed by the real-time data warehouse; and the real-time data warehouse stores user data with hot tags. Taking the platform as an example again, after the client creates the tag rules, the backend will check whether the types of the preset tags included in the tag rules are all hot tags. If so, user groups are segmented based on the real-time data warehouse through real-time queries.
[0067] It is understood that the real-time data warehouse is a data warehouse capable of real-time analysis, such as the columnar database management system Clickhouse, the new query system Impala, or the distributed SQL query engine Presto. This embodiment does not limit the real-time data warehouse. Clickhouse uses columnar storage, ensuring consistent data types, higher compression performance, and higher hardware utilization. Presto is an open-source distributed SQL query engine that supports data volumes from GB to PB, primarily used to handle sub-second query scenarios. Compared to offline data warehouses like Hive, the real-time data warehouse better meets query requirements. It is worth noting that since the real-time data warehouse does not store user data with cold tags, it significantly reduces the amount of data that needs to be stored, thereby significantly improving the query speed of the real-time data warehouse.
[0068] S13: If there is a tag of type cold tag among the multiple preset tags, then query the offline data warehouse for user groups that match the tag rules; the offline data warehouse stores user data of all tag types.
[0069] In this embodiment, if there are cold tags among the multiple preset tags, user groups matching the tag rules can be queried in the offline data warehouse. The offline data warehouse stores both hot and cold tag user data; that is, it stores user data for all tag types. Specifically, if the backend detects that not all preset tags included in the tag rules are hot tags (e.g., some are cold tags or all are cold tags), user segmentation can be performed offline based on the offline data warehouse. This embodiment does not limit the specific query method; relevant offline data warehouse technologies can be referenced. For example, a second database engine can be used to convert the tag rules into a second query statement, and the second query statement can be executed in the offline data warehouse to obtain user groups matching the tag rules. The second database engine corresponds to the offline data warehouse; the second query statement is an SQL query statement generated according to the tag rules and recognizable and processed by the offline data warehouse.
[0070] Generally, there are thousands of tags. If all of them are stored in the real-time data warehouse, a lot of storage will be wasted. However, based on historical data analysis, the client only uses a few dozen tags most often. Therefore, it is only necessary to store these few dozen tags in the real-time data warehouse, which can save costs and greatly reduce the time for clustering.
[0071] As can be seen, this embodiment first obtains the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags; if the tag types of the multiple preset tags are all hot tags, then the user groups that conform to the tag rules are queried in the real-time data warehouse; the real-time data warehouse stores user data with the tag type of hot tags; if there are tags with the tag type of cold tags among the multiple preset tags, then the user groups that conform to the tag rules are queried in the offline data warehouse; the offline data warehouse stores user data of all tag types. This embodiment reduces storage space occupancy by separating hot and cold tags and using different data warehouses to store user data of different tag types. On this basis, different query schemes are adopted for different tag types, thereby improving the user grouping speed by an order of magnitude and achieving second-level grouping.
[0072] Figure 4 A flowchart illustrating a specific user segmentation method provided in this application embodiment. See also... Figure 4 As shown, the user segmentation method includes:
[0073] S21: Real-time statistics on the frequency of each preset label being selected by the client.
[0074] S22: Determine the label type of each preset label according to its frequency; the label type is either a cold label or a hot label.
[0075] S23: Synchronize the user data corresponding to the preset tag with the tag type of hot tag in the offline data warehouse to the real-time data warehouse in real time, so that the real-time data warehouse stores user data with the tag type of hot tag.
[0076] In this embodiment, the tag type for the same preset tag is dynamically set, meaning the user data stored in the real-time data warehouse also changes in real time. Specifically, the frequency of each preset tag being selected by the client is first counted in real time, and then the tag type of each preset tag is determined based on the frequency. A high selection frequency means that the preset tag is frequently used, and a low selection frequency means that the preset tag is infrequently used. The frequency threshold here can be set according to business needs, and this embodiment does not limit it. For frequently used preset tags, their tag type is identified as hot tags, and they need to be stored in the real-time data warehouse. For infrequently used preset tags, their tag type is identified as cold tags, and they need to be stored in the offline data warehouse.
[0077] Furthermore, in this embodiment, the user data in the real-time data warehouse is synchronized from the offline data warehouse. Once it is determined which preset tags in the offline data warehouse have hot tags, the user data corresponding to those preset tags is synchronized from the offline data warehouse to the real-time data warehouse. That is, the user data corresponding to the preset tags with hot tags in the offline data warehouse is synchronized to the real-time data warehouse in real time, so that the real-time data warehouse stores user data with hot tags. It should be noted that the user data includes user identifiers, the preset tags possessed by the user, and their corresponding tag values.
[0078] S24: Obtain the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags.
[0079] S25: If the tag type of multiple preset tags is a hot tag, the tag rules are converted into a first query statement using the first database engine, and the first query statement is executed in the real-time data warehouse to obtain the user group that conforms to the tag rules; the real-time data warehouse stores user data with tag type hot tags.
[0080] S26: If there is a tag of type cold tag among the multiple preset tags, the tag rule is converted into a second query statement using the second database engine, and the second query statement is executed in the offline data warehouse to obtain the user group that conforms to the tag rule; the offline data warehouse stores user data of all tag types.
[0081] In this embodiment, the specific process of steps S24 to S26 can be referred to the corresponding content disclosed in the previous embodiments, and will not be repeated here.
[0082] As can be seen, this embodiment first counts the frequency of each preset tag being selected by the client in real time, then determines the tag type of each preset tag based on the frequency, and then synchronizes the user data corresponding to the preset tags with the tag type "hot tag" in the offline data warehouse to the real-time data warehouse in real time, so that the real-time data warehouse stores user data with the tag type "hot tag". On this basis, the tag rules created by the client are obtained. If the tag type of multiple preset tags is "hot tag", the tag rules are converted into a first query statement using a first database engine, and the first query statement is executed in the real-time data warehouse to obtain the user group that conforms to the tag rules; the real-time data warehouse stores user data with the tag type "hot tag". If there are tags with the tag type "cold tag" among the multiple preset tags, the tag rules are converted into a second query statement using a second database engine, and the second query statement is executed in the offline data warehouse to obtain the user group that conforms to the tag rules; the offline data warehouse stores user data with all tag types. In this embodiment, the frequency of each preset tag being selected by the client is counted in real time. Based on the counted frequency, it is determined whether the tag is a cold tag or a hot tag. By synchronizing the hot tags to the real-time data warehouse, the cold and hot tags are separated, thereby improving the efficiency of subsequent grouping.
[0083] Figure 5 A flowchart illustrating a specific user segmentation method provided in this application embodiment. See also... Figure 5 As shown, the user segmentation method includes:
[0084] S31: Real-time statistics on the frequency of each preset label being selected by the client.
[0085] S32: Determine the label type of each preset label according to its frequency; the label type is either a cold label or a hot label.
[0086] S33: Synchronize the user data corresponding to the preset tag with the tag type of hot tag in the offline data warehouse to the real-time data warehouse in real time, so that the real-time data warehouse stores user data with the tag type of hot tag.
[0087] In this embodiment, the specific processes of steps S31 to S33 can be referred to the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0088] S34: Populate the synchronized user data into the target vertical table constructed in the real-time data warehouse, so that the real-time data warehouse stores user data with the tag type hot tag; each row of the target vertical table stores at least one preset tag, a tag value of the preset tag, and a sequence of user identifiers with the preset tag and the tag value.
[0089] In this embodiment, to further improve query efficiency, user data synchronized from the offline data warehouse is stored in the real-time data warehouse in the form of a vertical table. Specifically, the synchronized user data is filled into a target vertical table constructed in the real-time data warehouse, so that the real-time data warehouse stores user data with the tag type "hot tag". Each row of the target vertical table stores at least one preset tag, a tag value of the preset tag, and a sequence containing user identifiers with the preset tag and the tag value. Furthermore, the user identifiers stored in each row of the target vertical table are stored in bitmap format.
[0090] Figure 6 The diagram shows an example of the tag storage structure in the real-time data warehouse. Each row represents a bitmap containing a set of user identifiers (user IDs) corresponding to one tag value of a preset tag. Under this architecture, assuming a preset tag has N different tag values, only N rows are needed for storage. For example, for the gender tag, only 2 rows are needed (one for male "1" and one for female "2"), and for the age tag, only about 100 rows (over 100 years old) are needed. Furthermore, the bitmap structure is very space-efficient; a user ID that originally required 32 bits can be stored in a bitmap with only 1 bit. Also, bitmaps inherently deduplicate user IDs, eliminating the need for separate deduplication.
[0091] S35: Obtain the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags.
[0092] In this embodiment, the specific process of step S35 can be referred to the corresponding content disclosed in the previous embodiments, and will not be repeated here.
[0093] S36: If the tag type of multiple preset tags is a hot tag, the tag rules are converted into a first query statement using the first database engine, and the first query statement is executed in the real-time data warehouse to obtain the user group that conforms to the tag rules; the real-time data warehouse stores user data with tag type hot tags.
[0094] In this embodiment, when using bitmap technology for user segmentation, the specific process of querying user groups matching the tag rules in the real-time data warehouse involves retrieving user identifiers matching the tag rules from the target vertical table in the real-time data warehouse. Specifically, the first query statement can be executed in the real-time data warehouse to retrieve user identifiers matching the tag rules from the target vertical table. Figure 7 The bitmap SQL generated by the label rules shown is as follows:
[0095] with(select groupBitmapMergeState(uids_bitmap)
[0096] from tagstar_bitmap_features
[0097] where(ftime='2022-11-19')and app_id=3and feature_id='age'and(feature_value=30))as bitmap_0,
[0098] (select groupBitmapMergeState(uids_bitmap)from tagstar_bitmap_features
[0099] where(ftime='2022-11-19')and app_id=3and feature_id='last31_play_cnt'and(feature_value>100))as bitmap_1,
[0100] (select groupBitmapMergeState(uids_bitmap)from tagstar_bitmap_features
[0101] where(ftime='2022-11-19')and app_id=3and feature_id='is_green_luxury'and feature_value IN(1))as bitmap_2
[0102] select toString(arrayJoin(bitmapToArray(bitmapAnd(bitmap_2,bitmapOr(bitmap_1,bitm ap_0)))))as uid
[0103] limit 100000
[0104] S37: If there is a tag of type cold tag among the multiple preset tags, the tag rule is converted into a second query statement using the second database engine, and the second query statement is executed in the offline data warehouse to obtain the user group that conforms to the tag rule; the offline data warehouse stores user data of all tag types.
[0105] In this embodiment, user segments are generated using a real-time data warehouse bitmap technology. If cold tags are included, the original offline warehouse SQL query is used to generate user segments. Based on the separation of hot and cold tags, storage costs can be reduced. However, since the tag bitmap is stored separately in the real-time data warehouse, only the bitmaps of the dozens of most frequently used tags need to be synchronously stored there, while the other thousands of types of cold tags continue to be stored in the offline data warehouse. It should be noted that the offline data warehouse stores user data in the form of wide tables, where one row represents one user and each column represents one of the user's preset tags. Figure 8 For the tag storage structure in this case, please refer to the SQL implementation in related technologies.
[0106] As can be seen, this embodiment first counts the frequency of each preset tag being selected by the client in real time, and then determines the tag type of each preset tag based on the frequency. The tag type is either a cold tag or a hot tag. Next, the user data corresponding to the preset tags with the hot tag type in the offline data warehouse is synchronized to the real-time data warehouse in real time, so that the real-time data warehouse stores user data with the hot tag type. Further, the synchronized user data is filled into a target vertical table constructed in the real-time data warehouse, so that the real-time data warehouse stores user data with the hot tag type. Each row of the target vertical table stores at least one preset tag, a tag value of the preset tag, and a sequence containing a user identifier with the preset tag and the tag value. Based on this, the tag rules created by the client are obtained. If the tag types of multiple preset tags are all hot tags, the first database engine is used to convert the tag rules into a first query statement, and the first query statement is executed in the real-time data warehouse to obtain the user group that conforms to the tag rules. The real-time data warehouse stores user data with the hot tag type. If any of the preset tags contain a tag of type "cold tag," the second database engine is used to convert the tag rules into a second query statement, which is then executed in the offline data warehouse to obtain user groups that conform to the tag rules. The offline data warehouse stores user data for all tag types. This embodiment uses bitmap technology for user segmentation for preset tags of type "hot tag," and uses the original method based on ordinary SQL queries for user segmentation for cold tags. This solution, under controllable cost conditions, increases the speed of user segmentation by an order of magnitude, achieving user segmentation in seconds in most cases, greatly improving the user experience.
[0107] Figure 9 A flowchart illustrating a user segmentation method provided in an embodiment of this application. See also... Figure 9 As shown, the user segmentation method includes:
[0108] S41: Obtain the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags.
[0109] S42: If the tag type of multiple preset tags is hot tag, the tag rules are converted into a first query statement using the first database engine, and the first query statement is executed in the real-time data warehouse to obtain the user group that conforms to the tag rules; the real-time data warehouse stores user data with tag type hot tag.
[0110] S43: If there is a tag of type cold tag among the multiple preset tags, the tag rule is converted into a second query statement using the second database engine, and the second query statement is executed in the offline data warehouse to obtain the user group that conforms to the tag rule; the offline data warehouse stores user data of all tag types.
[0111] In this embodiment, the specific processes of steps S41 to S43 can be referred to the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0112] S44: Package the user groups that meet the label rules to obtain the corresponding user groups.
[0113] S45: Upload the crowd package to the cloud storage platform so that the client can download the crowd package from the cloud storage platform.
[0114] In this embodiment, after user segmentation, the resulting user groups conforming to the tagging rules are further packaged into corresponding audience packages. Simultaneously, these audience packages are uploaded to a cloud storage platform so that the client can download them from the cloud storage platform. Specifically, the user segmentation data is generated into a text file and provided to the client. Alternatively, the final generated audience packages and other files can be uploaded to cloud platforms such as Tencent Cloud COS for users to download and use.
[0115] As can be seen, this embodiment first obtains the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags; if the tag types of the multiple preset tags are all hot tags, then the first database engine is used to convert the tag rules into a first query statement, and the first query statement is executed in the real-time data warehouse to obtain the user group that conforms to the tag rules; the real-time data warehouse stores user data with the tag type of hot tags; if there are tags with the tag type of cold tags among the multiple preset tags, then the second database engine is used to convert the tag rules into a second query statement, and the second query statement is executed in the offline data warehouse to obtain the user group that conforms to the tag rules; the offline data warehouse stores user data with all tag types. Finally, the obtained user group that conforms to the tag rules is packaged to obtain the corresponding audience package, and the audience package is uploaded to the cloud storage platform so that the client can download the audience package from the cloud storage platform, thereby improving the user experience.
[0116] The user segmentation scheme of this application will be described in its entirety below. Figure 10 This is a diagram illustrating the corresponding user segmentation methods.
[0117] After the client creates the tagging rules, the data processing platform checks in the background whether all the preset tags in the tagging rules are hot tags. If so, it uses Clickhouse's bitmap technology to segment users. If it contains cold tags, it uses the original offline warehouse SQL query to segment users. Finally, the generated file is uploaded to Tencent Cloud COS for users to download and use. It's important to note that without hot and cold tag separation, even with a real-time data warehouse for user segmentation, there are still drawbacks such as complex SQL, inability to achieve sub-second speeds, high storage space consumption, and complex maintenance. Similarly, without hot and cold tag separation, relying solely on offline data warehouse SQL query technologies like Hive still suffers from complex SQL, slow speed (minutes on the order of magnitude), unstable query performance, complex maintenance, and high storage space consumption. This embodiment uses bitmap technology for user segmentation for preset tags of type hot tags, and uses the original method based on ordinary SQL queries for user segmentation for cold tags. This solution improves user segmentation speed by an order of magnitude under controllable cost conditions, achieving sub-second user segmentation in most cases, greatly improving the user experience.
[0118] After implementing the user segmentation technology in this embodiment, the average processing time for each user segment on the data processing platform decreased significantly, from 500 seconds to an average of 100 seconds. Figure 11 As shown, bitmap clustering takes an order of magnitude less time on average compared to traditional TDW offline data warehouse clustering, such as... Figure 12 As shown, the bitmap-based solution takes an average of only 30 seconds, while the traditional TDW offline data warehouse solution takes an average of 200-700 seconds. Taking a music software as an example, after the existing users in the software were segmented and switched to the bitmap solution, the total processing time was reduced from more than 10 hours to 1.5 hours.
[0119] See Figure 13 As shown in the figure, this application also discloses a user grouping device, including:
[0120] The rule acquisition module 21 is used to acquire the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags;
[0121] The first grouping module 22 is used to query user groups that match the tag rules in the real-time data warehouse if multiple preset tags are all hot tags; the real-time data warehouse stores user data with tag type hot tags.
[0122] The second grouping module 23 is used to query the user groups that match the tag rules in the offline data warehouse if there is a tag of the cold tag type among the multiple preset tags; the offline data warehouse stores user data of all tag types.
[0123] As can be seen, this embodiment first obtains the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags; if the tag types of the multiple preset tags are all hot tags, then the user groups that conform to the tag rules are queried in the real-time data warehouse; the real-time data warehouse stores user data with the tag type of hot tags; if there are tags with the tag type of cold tags among the multiple preset tags, then the user groups that conform to the tag rules are queried in the offline data warehouse; the offline data warehouse stores user data of all tag types. This embodiment reduces storage space occupancy by separating hot and cold tags and using different data warehouses to store user data of different tag types. On this basis, different query schemes are adopted for different tag types, thereby improving the user grouping speed by an order of magnitude and achieving second-level grouping.
[0124] In some specific embodiments, the user grouping device may further include:
[0125] The statistics module is used to count in real time the frequency of each preset tag being selected by the client;
[0126] The determination module is used to determine the label type of each preset label based on its frequency; the label type is either a cold label or a hot label.
[0127] In some specific embodiments, the user grouping device may further include: a synchronization module, used to synchronize user data corresponding to the preset tag with the tag type of hot tag in the offline data warehouse to the real-time data warehouse in real time, so that the real-time data warehouse stores user data with the tag type of hot tag;
[0128] In some specific embodiments, the user grouping device may further include:
[0129] A filling module is used to fill synchronized user data into a target vertical table constructed in the real-time data warehouse, so that the real-time data warehouse stores user data with the tag type hot tag; each row of the target vertical table stores at least one preset tag, a tag value of the preset tag, and a sequence containing a user identifier with the preset tag and the tag value;
[0130] In some specific embodiments, the user identifiers stored in each row of the target vertical table can be stored in bitmap format; in some specific embodiments, the first grouping module 22 can also be used for:
[0131] The user identifiers that match the tag rules are retrieved from the target vertical table in the real-time data warehouse.
[0132] In some specific embodiments, the first grouping module 22 can also be used for:
[0133] The first database engine is used to transform the tag rules into a first query statement, and the first query statement is executed in the real-time data warehouse to obtain the user groups that conform to the tag rules;
[0134] The second grouping module 23 can also be used for:
[0135] The tag rules are transformed into a second query statement using a second database engine, and the second query statement is executed in the offline data warehouse to obtain the user groups that conform to the tag rules.
[0136] In some specific embodiments, the user grouping device may further include:
[0137] The packaging module is used to package the user groups that conform to the tag rules to obtain the corresponding user packages.
[0138] The upload module is used to upload the crowd data package to the cloud storage platform so that the client can download the crowd data package from the cloud storage platform.
[0139] On the other hand, this application also provides an electronic device that may include a processor and a memory. The relationship between the processor and the memory in this electronic device can be referenced... Figure 1 .
[0140] The processor of the electronic device is used to execute the program stored in the memory;
[0141] The memory of the electronic device is used to store a program, which is used at least for:
[0142] Obtain the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags;
[0143] If multiple preset tags are all hot tags, then the user groups that match the tag rules are queried in the real-time data warehouse; the real-time data warehouse stores user data whose tag type is hot tag.
[0144] If any of the preset tags contain a tag of type "cold tag", then the user group that matches the tag rules is queried in the offline data warehouse; the offline data warehouse stores user data of all tag types.
[0145] Optionally, the program is also used for:
[0146] Real-time statistics on the frequency with which each preset tag is selected by the client;
[0147] The label type of each preset label is determined based on its frequency; the label type is either a cold label or a hot label.
[0148] The user data corresponding to the preset tag with the tag type of hot tag in the offline data warehouse is synchronized to the real-time data warehouse in real time, so that the real-time data warehouse stores user data with the tag type of hot tag.
[0149] The synchronized user data is populated into the target vertical table constructed in the real-time data warehouse, so that the real-time data warehouse stores user data with the tag type hot tag; each row of the target vertical table stores at least one preset tag, a tag value of the preset tag, and a sequence containing a user identifier with the preset tag and the tag value.
[0150] Optionally, the user identifier stored in each row of the target vertical table is stored in bitmap format.
[0151] Optionally, querying the user group that matches the tag rules in the real-time data warehouse includes:
[0152] The user identifiers that match the tag rules are retrieved from the target vertical table in the real-time data warehouse.
[0153] Optionally, querying the user group that matches the tag rules in the real-time data warehouse includes:
[0154] The first database engine is used to transform the tag rules into a first query statement, and the first query statement is executed in the real-time data warehouse to obtain the user groups that conform to the tag rules;
[0155] The process of querying user groups that match the tag rules in the offline data warehouse includes:
[0156] The tag rules are transformed into a second query statement using a second database engine, and the second query statement is executed in the offline data warehouse to obtain the user groups that conform to the tag rules.
[0157] Optionally, the program is also used for:
[0158] The user groups that conform to the aforementioned tag rules are packaged to obtain the corresponding user groups;
[0159] The crowd data package is uploaded to a cloud storage platform so that the client can download the crowd data package from the cloud storage platform.
[0160] Of course, the electronic device may also include communication interfaces, input / output interfaces, etc., without any specific restrictions here.
[0161] Furthermore, embodiments of this application also disclose a storage medium storing a computer program, which, when loaded and executed by a processor, performs at least the following steps:
[0162] Obtain the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags;
[0163] If multiple preset tags are all hot tags, then the user groups that match the tag rules are queried in the real-time data warehouse; the real-time data warehouse stores user data whose tag type is hot tag.
[0164] If any of the preset tags contain a tag of type "cold tag", then the user group that matches the tag rules is queried in the offline data warehouse; the offline data warehouse stores user data of all tag types.
[0165] Optional, also includes:
[0166] Real-time statistics on the frequency with which each preset tag is selected by the client;
[0167] The label type of each preset label is determined based on its frequency; the label type is either a cold label or a hot label.
[0168] The user data corresponding to the preset tag with the tag type of hot tag in the offline data warehouse is synchronized to the real-time data warehouse in real time, so that the real-time data warehouse stores user data with the tag type of hot tag.
[0169] Optionally, after synchronizing the user data corresponding to the preset tag with the tag type of hot tag in the offline data warehouse to the real-time data warehouse in real time, the method further includes:
[0170] The synchronized user data is populated into the target vertical table constructed in the real-time data warehouse, so that the real-time data warehouse stores user data with the tag type hot tag; each row of the target vertical table stores at least one preset tag, a tag value of the preset tag, and a sequence containing a user identifier with the preset tag and the tag value.
[0171] Optionally, the user identifier stored in each row of the target vertical table is stored in bitmap format.
[0172] Optionally, querying the user group that matches the tag rules in the real-time data warehouse includes:
[0173] The user identifiers that match the tag rules are retrieved from the target vertical table in the real-time data warehouse.
[0174] Optionally, querying the user group that matches the tag rules in the real-time data warehouse includes:
[0175] The first database engine is used to transform the tag rules into a first query statement, and the first query statement is executed in the real-time data warehouse to obtain the user groups that conform to the tag rules;
[0176] The process of querying user groups that match the tag rules in the offline data warehouse includes:
[0177] The tag rules are transformed into a second query statement using a second database engine, and the second query statement is executed in the offline data warehouse to obtain the user groups that conform to the tag rules.
[0178] Optional, also includes:
[0179] The user groups that conform to the aforementioned tag rules are packaged to obtain the corresponding user groups;
[0180] The crowd data package is uploaded to a cloud storage platform so that the client can download the crowd data package from the cloud storage platform.
[0181] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0182] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0183] The user grouping method, device, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A user segmentation method, characterized in that, include: User data synchronized from the offline data warehouse is populated into the target vertical table built in the real-time data warehouse, so that the real-time data warehouse stores user data with the tag type hot tag; Each row of the target vertical table stores at least one preset label, a label value of the preset label, and a sequence containing user identifiers with the preset label and the label value. The user identifiers stored in each row of the target vertical table are stored in bitmap format, where 1 bit in the bitmap represents 1 user identifier. Obtain the tag rules created by the client; the tag rules include multiple preset tags selected by the client and the logical relationships between the tags; If multiple preset tags are all hot tags, then the user groups that match the tag rules are queried in the real-time data warehouse; the real-time data warehouse stores user data whose tag type is hot tag. If any of the preset tags are of the cold tag type, then query the offline data warehouse for user groups that match the tag rules; The offline data warehouse stores user data for all tag types; The process of querying user groups that match the tag rules in the real-time data warehouse includes: The first database engine is used to transform the tag rules into a first query statement, and the first query statement is executed in the real-time data warehouse to obtain the user groups that conform to the tag rules; The process of querying user groups that match the tag rules in the offline data warehouse includes: The tag rules are transformed into a second query statement using a second database engine, and the second query statement is executed in the offline data warehouse to obtain the user groups that conform to the tag rules.
2. The user segmentation method according to claim 1, characterized in that, Also includes: Real-time statistics on the frequency with which each preset tag is selected by the client; The label type of each preset label is determined based on its frequency. The tag type is either a cold tag or a hot tag.
3. The user segmentation method according to claim 1, characterized in that, Also includes: The user data corresponding to the preset tag with the tag type of hot tag in the offline data warehouse is synchronized to the real-time data warehouse in real time, so that the real-time data warehouse stores user data with the tag type of hot tag.
4. The user segmentation method according to claim 1, characterized in that, The process of querying user groups that match the tag rules in the real-time data warehouse includes: The user identifiers that match the tag rules are retrieved from the target vertical table in the real-time data warehouse.
5. The user segmentation method according to any one of claims 1 to 4, characterized in that, Also includes: The user groups that conform to the aforementioned tag rules are packaged to obtain the corresponding user groups; The crowd data package is uploaded to a cloud storage platform so that the client can download the crowd data package from the cloud storage platform.
6. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the user segmentation method as described in any one of claims 1 to 5.
7. A computer-readable storage medium, characterized in that, Used to store computer-executable instructions, which, when loaded and executed by a processor, implement the user grouping method as described in any one of claims 1 to 5.