A user label intelligent generation and analysis method and system based on big data analysis

Abstract

The application relates to the technical field of user label, and discloses a user label intelligent generation and analysis method and system based on big data analysis. The method comprises the following steps: subscribing to a benefit activation event stream, a benefit exercise event stream and a refund event stream through Kafka, and obtaining event data by analysis; performing Flink stream processing based on the event data, obtaining a user state variable, and calculating a benefit preference vector according to the user state variable; taking a benefit activation sequence as a document and a benefit category as a vocabulary to construct a co-occurrence matrix of the user and the benefit category, performing iterative sampling, and obtaining a topic probability vector; performing label activation analysis based on the benefit preference vector and the topic probability vector, obtaining a user label, and writing the user label into an HBase database in real time; and the application improves the accuracy, timeliness and expression ability of the user label, and provides high-quality user portrait support for a business scenario of benefit recommendation, accurate marketing and risk identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of user label, and in particular to a user label intelligent generation and analysis method and system based on big data analysis. BACKGROUND

[0002] The user label generation method of the traditional rights platform generally adopts a T+1 offline batch processing mode, and the rights order data is extracted from the database for statistical analysis through a timing task. This way causes a 24-hour delay in label updating, and cannot capture the real-time behavior changes of users in the rights activation, exercise, refund and other scenarios. When the user completes the rights activation operation, the user label needs to be updated the next day, missing the best conversion window for immediate recommendation of related rights, and seriously affecting the timeliness and conversion effect of rights recommendation. The existing label generation method directly counts the activation frequency of the rights type, without considering the time decay characteristics of the rights activation behavior, and cannot distinguish the weight difference between the recently activated rights and the historically activated rights in the preference judgment. At the same time, it also ignores the composite behavior mode of the user activating the rights at different times and in different scenarios, resulting in that the label cannot accurately reflect the current real preference state of the user. SUMMARY

[0003] The present application provides a user label intelligent generation and analysis method and system based on big data analysis, which improves the accuracy, timeliness and expression ability of the user label, and provides high-quality user portrait support for the business scenarios of rights recommendation, precise marketing and risk identification.

[0004] In a first aspect, the present application provides a user label intelligent generation and analysis method based on big data analysis, which comprises:

[0005] Subscribing to the rights activation event stream, the rights exercise event stream and the refund event stream through Kafka and parsing to obtain event data;

[0006] Performing Flink stream processing based on the event data to obtain user state variables, and calculating a rights preference vector according to the user state variables;

[0007] Iterative sampling of the co-occurrence matrix of the user and the rights category by taking the rights activation sequence as a document and the rights category as a vocabulary to obtain a topic probability vector;

[0008] Performing label activation analysis based on the rights preference vector and the topic probability vector to obtain a user label, and writing the user label into an HBase database in real time.

[0009] In a first implementation manner of the first aspect, the subscribing to the benefit activation event stream, the benefit exercise event stream and the refund event stream and parsing to obtain event data comprises:

[0010] The benefit activation topic, the benefit exercise topic and the refund topic of the Kafka are configured, the benefit activation topic, the benefit exercise topic and the refund topic are subscribed to by the Flink consumer instance respectively and are distributed to corresponding partitions in a user identification hash value modulo manner, and the benefit activation event stream, the benefit exercise event stream and the refund event stream are obtained;

[0011] The event messages received in the benefit activation event stream, the benefit exercise event stream and the refund event stream are parsed in a JSON format respectively, and event data is extracted.

[0012] In a second implementation manner of the first aspect, the Flink stream processing based on the event data to obtain a user state variable and calculating a benefit preference vector according to the user state variable comprises:

[0013] The event data is subjected to a KeyBy grouping operation, event data of a same user is routed to a same parallel processing instance by taking a user identification as a grouping key, and an event data stream is obtained;

[0014] The event data stream is divided into time sequence segments by applying a rolling event time window and a water level mechanism is configured to process delayed data, and an event data set in a window is obtained;

[0015] The event data set in the window is traversed when the window is triggered, a benefit category and an event timestamp are extracted in turn after being sorted in an ascending order of event timestamp, a benefit activation sequence is updated, a benefit category counter is accumulated with a number of activations of each benefit category, and a latest event timestamp is recorded as a latest event time, and the user state variable is obtained;

[0016] A product of a time decay term frequency and an inverse document frequency in each benefit category is calculated according to the benefit activation sequence and the benefit category counter in the user state variable, and a benefit preference vector is obtained.

[0017] In a third implementation manner of the first aspect, the KeyBy grouping operation on the event data, by taking a user identification as a grouping key, to route event data of a same user to a same parallel processing instance to obtain an event data stream comprises:

[0018] The event data is converted into a DataStream data stream to obtain stream data;

[0019] performing a KeyBy grouping operation on the stream data, extracting a user identifier as a grouping key, calculating a hash value according to the grouping key and taking modulo of a total number of parallel processing instances to obtain a target parallel processing instance number;

[0020] routing the stream data to a corresponding parallel processing instance according to the target parallel processing instance number to obtain an event data stream.

[0021] In a fourth implementation manner of the first aspect, the calculating of the product of the time decay term frequency and the inverse document frequency of each benefit category according to the benefit activation sequence and the benefit category counter in the user state variable to obtain the benefit preference vector comprises:

[0022] traversing the benefit category and the event timestamp of the benefit activation sequence in the user state variable, calculating an exponential decay weight according to the event timestamp and accumulating by benefit category to obtain the time decay term frequency of each benefit category;

[0023] obtaining a total number of cumulative users of a platform and a number of activated users of each benefit category, performing logarithmic operation on a ratio of the total number of cumulative users of the platform and the number of activated users to obtain the inverse document frequency of each benefit category;

[0024] multiplying the time decay term frequency of each benefit category and the corresponding inverse document frequency as a preference score of the benefit category, and arranging the preference scores in a category order after traversing all the benefit categories to obtain the benefit preference vector.

[0025] In a fifth implementation manner of the first aspect, the constructing of the co-occurrence matrix of users and benefit categories by taking the benefit activation sequence as a document and taking the benefit categories in the benefit activation sequence as words comprises:

[0026] taking the benefit activation sequence as a document and taking the benefit categories in the benefit activation sequence as words to count the number of occurrences of each benefit category;

[0027] constructing an original matrix by taking a user as a row index and taking a benefit category as a column index and filling the number of occurrences to obtain a co-occurrence matrix;

[0028] randomly assigning an initial topic label to each benefit category in the co-occurrence matrix and iteratively resampling the topic attribution of each benefit category according to a conditional probability to obtain a topic assignment result;

[0029] counting the number of benefits of a user under each topic in the topic assignment result, dividing the number of benefits of each topic by a total number of user benefits to calculate the attribution probability of each topic, and constructing a topic probability vector according to the attribution probability of each topic.

[0030] In a sixth implementation form of the first aspect, the initial topic labels are randomly assigned to each equity category in the co-occurrence matrix, and the topic assignment result is obtained by iteratively resampling the topic attribution of each equity category according to conditional probability, including:

[0031] A1: traversing each equity category in the co-occurrence matrix and randomly assigning an initial topic label, initializing a topic count matrix recording the number of equities of each user in each topic and the number of occurrences of each equity category in each topic;

[0032] A2: sequentially processing each equity category in a single iteration, removing the equity category from the assigned topic and updating the topic count matrix, calculating the conditional probability of the attribution of the equity category to each topic according to the number of equities of each user in each topic and the number of occurrences of each equity category in each topic in the updated topic count matrix, resampling the topic label of the equity category according to the conditional probability and adding 1 to the count of the corresponding topic in the topic count matrix;

[0033] A3: repeating the iteration process of step A2 until a preset iteration number is reached, obtaining the topic assignment result.

[0034] In a seventh implementation form of the first aspect, the label activation analysis is performed based on the equity preference vector and the topic probability vector to obtain a user label, and the user label is written into an HBase database in real time, including:

[0035] a topic and category mapping matrix is constructed to record the contribution weight of each topic to each equity category, the topic probability vector is subjected to matrix multiplication operation with the topic and category mapping matrix to obtain an extended topic vector;

[0036] a user activity is calculated according to the cumulative activation number of the user, and a fusion weight coefficient is determined based on the user activity;

[0037] the equity preference vector and the extended topic vector are subjected to weighted summation according to the fusion weight coefficient to obtain a fusion feature vector;

[0038] label activation analysis is performed according to each dimension score in the fusion feature vector and a preset threshold to obtain a user label, which is written into an HBase database in real time.

[0039] In an eighth implementation form of the first aspect, the label activation analysis is performed according to each dimension score in the fusion feature vector and a preset threshold to obtain a user label, which is written into an HBase database in real time, including:

[0040] Extract the dimension score corresponding to each equity category in the fusion feature vector, compare each dimension score with the preset threshold value of the corresponding equity category, and obtain a set of equity categories that meet the threshold condition;

[0041] Generate a preference class label according to the set of equity categories that meet the threshold condition, query the user historical behavior record in combination with the business rule database to generate a behavior class label, calculate the entropy value of the fusion feature vector and generate a risk class label in combination with the cross-scene activation frequency, and obtain a user label;

[0042] Encapsulate the user label as a data record with the user identifier as the row key and the label name as the column qualifier, and write the data record into the label column family of the user label table in the HBase database.

[0043] In a second aspect, the present application provides a user label intelligent generation and analysis system based on big data analysis, which comprises:

[0044] An event stream analysis module is configured to subscribe to the equity activation event stream, the equity exercise event stream and the refund event stream through Kafka and analyze the event data obtained;

[0045] A stream processing module is configured to perform Flink stream processing based on the event data, obtain a user state variable, and calculate an equity preference vector according to the user state variable;

[0046] An iterative sampling module is configured to construct a co-occurrence matrix of users and equity categories by taking the equity activation sequence as a document and the equity category as a vocabulary, and perform iterative sampling to obtain a topic probability vector;

[0047] A label activation analysis module is configured to perform label activation analysis based on the equity preference vector and the topic probability vector, obtain a user label, and write the user label into the HBase database in real time.

[0048] The technical scheme provided by the application replaces the traditional T+1 offline batch processing mode by a Kafka-Flink streaming architecture, realizes real-time subscription and streaming processing of the equity event, reduces the label generation delay from 24 hours to 2 seconds, and significantly improves the timeliness of the label; the TF-IDF algorithm is improved by introducing an exponential decay function, the weight is dynamically adjusted according to the time difference between the equity activation time and the current time, the recently activated equity occupies a higher weight in the preference calculation, and the time evolution law of the user interest is accurately captured; the LDA topic model is used for latent semantic analysis on the equity activation sequence, the Collapsed Gibbs sampling algorithm is used to mine the deep interest theme behind the user behavior, and the interest migration path of the user is identified; the dimension alignment of the heterogeneous feature space is realized by constructing a topic-category mapping matrix, the fusion weight of the TF-IDF preference feature and the LDA topic feature is adaptively adjusted according to the user activity, so that the fusion feature can capture the recent explicit preference of the user and mine the long-term potential interest, and the new user cold start scene and the personalized demand of the active user are considered; the difference generation strategy of the preference class, the behavior class and the risk class label is realized by combining the fusion feature score, the preset threshold and the business rule, and the generated label is written into the HBase database in real time, the accuracy, timeliness and expression ability of the user label are improved, and high-quality user portrait support is provided for the business scenes of equity recommendation, precise marketing and risk identification. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 The flowchart of the user label intelligent generation and analysis method based on big data analysis provided by the embodiment of the application;

[0051] Figure 2 The structural schematic diagram of the user label intelligent generation and analysis system based on big data analysis provided by the embodiment of the application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0053] The flowcharts shown in the drawings are merely illustrative and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further decomposed, combined or partially merged, so the actual execution order can be changed based on actual conditions.

[0054] It should also be understood that the terms used herein in the specification and the appended claims are for the purpose of describing particular embodiments and are not intended to limit the application. As used in this specification and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0055] It should be further understood that the term "and / or" used in the specification and the appended claims herein refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0056] Some embodiments of the present application will be described in detail below with reference to the drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0057] Please refer to Figure 1 , Figure 1 The flowchart of the user tag intelligent generation and analysis method based on big data analysis provided by the embodiments of the present application is shown in Figure 1 The user tag intelligent generation and analysis method based on big data analysis provided by the embodiments of the present application includes:

[0058] Step S100, subscribing to the benefit activation event stream, the benefit exercise event stream and the refund event stream through Kafka and parsing to obtain event data;

[0059] Specifically, three types of topic channels are configured through the Kafka message middleware system, respectively corresponding to the benefit activation topic, the benefit exercise topic, and the refund topic. Among them, the benefit activation topic is used to receive event messages generated when the user triggers opening behaviors such as clicking on opening, checking member services, etc., the benefit exercise topic carries the consumption behavior events generated by the user actually using the opened benefits (such as ordering, watching, etc.), and the refund topic is used to collect benefit refund event information applied by the user or triggered by the system. The Kafka topics all adopt a unified message structure definition to ensure the consistency and parsability of the data structure. In the Flink stream computing framework, an independent FlinkKafkaConsumer consumer instance is configured for each Kafka topic, and the consumer offset is set to the earliest mode to ensure complete consumption from the beginning of the event. The parallel assignment of events is based on the user unique identifier (user_id) for hash modulo operation, and the user events are distributed to multiple partitions of Kafka according to the hash value, so that the event data of the same user is routed to consistent parallel sub-task instances in the Flink task, and the state consistency maintenance is realized. After the event data stream enters the Flink DataStream formally, a JSON format parsing operation is performed for each event message, and the five-tuple field data including the user unique identifier user_id, the benefit code benefit_id, the benefit category benefit_category, the event occurrence time event_time, and the trigger scene scene_type are extracted, forming the benefit activation event stream, the benefit exercise event stream, and the refund event stream.

[0060] Step S200, performing Flink stream processing based on the event data to obtain a user state variable, and calculating a benefit preference vector according to the user state variable;

[0061] Specifically, KeyBy grouping operation is performed on event data to route all events of the same user into the same parallel processing sub-task instance in Flink by taking the user identifier as the grouping key, so that no cross-instance state confusion occurs in state maintenance, and an event data stream divided by user dimension is formed. A rolling event time window strategy is applied on the grouped data stream to divide the continuously arriving event data by hour-level granularity according to event time as the window time reference, so that all equity behavior events occurring within a specific time period are contained in each time window. The waterline mechanism is configured, and the maximum out-of-order degree is set to 60 seconds, so that events can arrive with delay within a limited time and still be classified into the correct window for calculation, while late events exceeding the delay threshold are output through a side output stream to avoid affecting the correctness of the main data path. When each window is triggered, Flink will collect all event data sets within the time window, and sort these events in ascending order according to the event time field in the trigger function. After sorting, the event data is traversed one by one, and the equity category and event timestamp are extracted from each record, and the corresponding record is added to the user's equity activation sequence list, while the value of the equity category counter is updated for each equity category to complete the accumulation statistics of the activation frequency, and the timestamp of the last event in the current window is updated as the user's latest behavior time field to build the user state variable containing the equity activation sequence, equity category activation statistics and latest event time. For each record in the user activation sequence, the time decay weight is calculated according to the time difference between the event occurrence time and the end time of the current time window, and the time decay weight is applied to the word frequency accumulation of the corresponding category to form a time-weighted word frequency vector. By querying the platform-level user-category active statistics, the inverse document frequency of each equity category is calculated, and the time-decayed word frequency and the inverse document frequency are multiplied dimension by dimension to construct a 20-dimensional equity preference vector reflecting the user's current preference intensity for different equity types such as video entertainment, takeout food, finance and insurance, etc.

[0062] Step S300, construct a co-occurrence matrix of users and equity categories by taking the equity activation sequence as a document and the equity category as a word, and iteratively sample the co-occurrence matrix to obtain a topic probability vector;

[0063] Specifically, the benefit activation sequence of each user in the window period is regarded as a virtual document, and the benefit categories contained in the sequence are regarded as the words in the document. The benefit categories corresponding to each activation event are extracted by traversing the benefit activation sequence, and the frequency statistics of the benefit categories are constructed. A user-benefit category co-occurrence matrix in the form of a sparse matrix is established with the user as the row index and the benefit category as the column index, where each element of the matrix represents the cumulative number of times that the corresponding user activates a certain benefit category. The matrix is directly filled with data extracted from the activate_count_map in the user state variable to form the original corpus input reflecting the user behavior structure. In the topic modeling stage, the LDA latent Dirichlet allocation model is used to process the co-occurrence matrix. An initial topic label is randomly assigned to each benefit category activation record of each user, and the initial assignment is subject to uniform distribution to ensure that different topics have similar priors at initialization. The CollapsedGibbs sampling iteration process is entered, and the theme resampling operation is performed on each activation record in turn. In each iteration, the current activation record is first removed from the original theme count, and the user-theme count matrix n_k^(u), the theme-term count matrix n_k^(c) and the theme total term count n_k^(·) are updated. The belonging probability of each theme k is recalculated according to the Gibbs conditional probability formula. A new theme belonging label is sampled in a multinomial manner according to the conditional probability distribution, and it is readded to the corresponding statistical matrix. This is repeated for multiple rounds of sampling to approximate the true theme distribution. After all sampling iterations are completed, the theme assignment results are counted, the number of benefit records of each user under each theme is calculated, and the number of benefits under each theme is divided by the total number of all benefit activation records of the user to obtain the normalized theme belonging probability. The theme belonging probability is summarized to form a theme probability vector containing multiple dimensions.

[0064] Step S400, based on the benefit preference vector and the theme probability vector, label activation analysis is performed to obtain the user label, and the user label is written into the HBase database in real time.

[0065] Specifically, a mapping mechanism for aligning the equity preference vector and the topic probability vector is constructed. A topic-category mapping matrix is constructed, with topic numbers as row indexes and equity categories as column indexes. Each matrix element represents the contribution weight of the corresponding topic in a certain equity category. The contribution is calculated according to φ_{kc}=n_k^(c) / n_k^(·) obtained in the LDA model training process, where n_k^(c) represents the frequency of the equity category c appearing in topic k, and n_k^(·) represents the total frequency of all equity categories in topic k. The ratio reflects the probability density of topic k in category c. The user topic probability vector obtained by Gibbs sampling is multiplied by the mapping matrix to realize the projection from the low-dimensional topic space to the high-dimensional equity category space, and the extended topic feature vector is obtained. The user activity index is calculated by accumulating and summing the user historical equity activation frequency. The user activity index is compressed by taking the logarithm to alleviate the numerical inflation problem of extreme high-frequency users. According to the activity result, a piecewise function is used to determine the fusion weight coefficient. If the user activity is low, that is, the cumulative activation frequency is less than the preset threshold, more emphasis is placed on the immediacy of the TF-IDF feature, and a higher weight (such as 0.7) is given to TF-IDF, while a lower weight (such as 0.3) is given to the extended topic feature. If the user activity is high, it means that the behavior sample is sufficient and the long-term interest is more stable and reliable, so the dependence weight of LDA topic feature is increased, for example, it is set to 0.6, and the weight of TF-IDF part is reduced to 0.4. The original equity preference vector and the extended topic vector are respectively normalized by L2 norm to ensure the uniformity of the vector length to eliminate the dimension effect. The two normalized vectors are weighted and summed element by element according to the set fusion weight coefficient to form the fusion feature vector. According to the scores of each dimension of the fusion feature vector, the label activation logic is executed combined with the preset judgment threshold of different label types. For single preference labels such as "video entertainment type users" or "financial security preference users", if the corresponding category dimension score exceeds the set threshold, it is considered to meet the activation condition. For behavior composite labels such as "repurchase users", the fusion feature condition and historical behavior rules must be met simultaneously. For risk labels, the double judgment of equity activation frequency concentration and vector Shannon entropy index is required. When the label is successfully activated, the user identification, label name, label confidence, generation time and expiration time are packaged as a five-tuple format, and written into the specified column family of the HBase database with the user ID as the row key RowKey. The label type is identified by the label name as the column qualifier, and the label data is stored in the corresponding cell in JSON format.

[0066] In a specific embodiment, the process of step S100 can specifically include the following steps:

[0067] The equity activation topic, the equity exercise topic and the refund topic of Kafka are configured, the equity activation topic, the equity exercise topic and the refund topic are respectively subscribed by Flink consumer instances and distributed to corresponding partitions in a user identification hash value modulo manner, and an equity activation event stream, an equity exercise event stream and a refund event stream are obtained;

[0068] The event messages received in the equity activation event stream, the equity exercise event stream and the refund event stream are respectively parsed in a JSON format, and event data is extracted.

[0069] Specifically, three message topics with clear business semantics are predefined in the Kafka cluster, namely Topic_Activate for receiving activation events generated by user-triggered activation actions, Topic_Usage for receiving usage behavior events generated by actual use of activated benefits, and Topic_Refund for receiving refund events generated by user-initiated refund requests or automatically triggered by the system. Each topic is configured with multiple partitions, for example, set to 6 partitions, and the replica factor is configured to 2, thereby forming data redundancy backup within the Kafka cluster. The partition strategy uses a hash calculation based on the user's unique identifier user_id and takes the modulus of the number of partitions to allocate events, so that all events of the same user are always routed to the same partition. Inside the Flink stream computing framework, a FlinkKafkaConsumer instance is built for each Kafka topic, and a separate data consumption source is created using the Kafka connector supported by Flink natively. Each consumer instance needs to be configured with the topic name, Kafka cluster address, consumer group ID, and consumption strategy, where the consumption strategy is set to earliest to ensure complete consumption of all event messages from the beginning of the topic. After the consumer receives the raw event stream data written in Kafka, it encapsulates it as a Flink DataStream structure, forming three types of parallel data stream channels in the stream computing pipeline, corresponding to the benefit activation event stream, the benefit exercise event stream, and the refund event stream. The Kafka production end needs to ensure message format standardization, and all events are encapsulated in JSON structured text format, containing five core fields: user ID (user_id), benefit ID (benefit_id), benefit category (benefit_category), event time (event_time), and trigger scenario (scene_type). In the Flink consumption end, the received event stream is structured decoded by a JSON parser, and each raw JSON string is parsed into an internally defined event data object (such as the POJO class BenefitEvent) through JSON parsing libraries such as Jackson and Gson. The parsing process ensures that the field name is consistent with the Kafka message body, and the timestamp field is uniformly converted to adapt to the event time processing mechanism of Flink. After parsing, the structured event data is input into the state computing module of Flink.

[0070] In a specific embodiment, the process of performing step S200 can specifically include the following steps:

[0071] performing a KeyBy grouping operation on the event data to route event data of the same user to the same parallel processing instance with the user identifier as the grouping key, to obtain an event data stream;

[0072] applying a rolling event time window to split the event data stream into time-ordered segments and configure a watermark mechanism to handle delayed data, to obtain an event data set within the window;

[0073] traversing the event data set within the window at a window trigger, sorting the event data set in ascending order of event timestamp, extracting the equity category and event timestamp in turn, and updating the equity activation sequence, accumulating the activation times of each equity category to the equity category counter, and recording the latest event timestamp to the latest event time, to obtain user state variables;

[0074] calculating the product of time decay term frequency and inverse document frequency in each equity category according to the equity activation sequence and the equity category counter in the user state variables, to obtain an equity preference vector.

[0075] Specifically, in the Flink stream processing engine, the standardized event data stream from Kafka is subjected to a KeyBy operation, grouping the event stream according to the user unique identifier user_id. The grouping operation is implemented through the keyBy operator of Flink, using the user identifier field as the hash key to route the event data to the same parallel processing instance. After grouping, a rolling event time window is configured on the data stream of each user dimension, for example, setting a fixed time period of 1 hour, dividing the continuously arriving data into fixed-size, non-overlapping time segments, and constructing the window boundaries based on the event occurrence time. To deal with the situation of event out-of-order arrival, a waterline mechanism is introduced to tolerate a certain range of event delay arrival, for example, setting the maximum out-of-order time to 60 seconds, indicating that events that arrive up to 60 seconds after the current waterline are still processed in the current window, otherwise they are marked as late data and processed through a side output stream. When the rolling window triggers, all event data sets within the time slice are collected and cached, and in the window calculation logic, the events in the set are sorted in ascending order according to their timestamp field, ensuring that the event order conforms to the user's real behavior timing. The sorted event set is traversed in sequence, and in each traversal, the equity category and event timestamp fields are extracted and encapsulated as equity activation records, which are appended to the user's equity activation sequence. The equity activation sequence is stored in an ArrayList structure; at the same time, the equity category counter mapping table is updated incrementally by category, and if the category does not exist in the mapping table, it is initialized to 1, otherwise it is incremented by 1 based on the original value. After each traversal, the current event time is compared with the last active time recorded in the user record, and if the current event time is later, it is updated to the new latest active time, to complete the update of the user state variable, which contains three key dimensions: complete activation sequence, equity category counter, and latest event time. Based on the equity preference modeling logic, the user state variable records the equity activation sequence and activation frequency mapping, and the calculation is performed. According to the time difference Δt between the timestamp of each equity activation event and the end time of the current window, the exponential time decay weight is calculated, which takes the form of an attenuation function exp(-λ·Δt / 3600), where λ is the adjustable attenuation factor, for example, 0.01 represents a 1% decay per hour. In each equity category dimension, the decay weights of all activation records that meet the category are accumulated to form the weighted term frequency value, reflecting the user's behavior intensity for the category in the current window and emphasizing the high weight of recent behavior. The platform global statistical variable is queried to obtain the total number of users U_activate_c that have activated the equity category, and combined with the total number of platform registered users U_total, the inverse document frequency IDF value of the category is calculated through the formula log((U_total+1) / (U_activate_c+1)), and the weighted preference score of the category is obtained by multiplying the two.The operation is performed on all preset equity category enumeration values (such as 20 categories), and an equity preference vector containing scores of all categories is formed.

[0076] The application of the rolling event time window divides the event data stream into time sequence segments and configures the water level mechanism to process delayed data, and obtains an event data set in the window, including: configuring an event time extractor for the event data stream, extracting an event timestamp field from the event data as the event time, setting a water level generation strategy and configuring a maximum out-of-order tolerance of 60 seconds, obtaining a data stream with event time and water level markers; applying the rolling event time window based on the data stream with event time and water level markers, dividing the event stream into non-overlapping time sequence windows according to a 1-hour time length, determining whether the window is triggered according to the water level timestamp, triggering window calculation when the water level time exceeds the window end time, obtaining a window trigger signal and event data arriving in the window; performing delay judgment on delayed event data still arriving after the window is triggered, calculating the time difference between the event time of the delayed event and the window end time, when the time difference is less than 60 seconds, the delayed event is classified into the corresponding historical window for re-calculation, when the time difference is greater than or equal to 60 seconds, the delayed event is output to the side output stream for separate processing, obtaining the event data set in the window and the timeout delayed event data in the side output; grouping the timeout delayed event data in the side output according to the user identifier, and directly merging into the end of the benefit activation sequence of the user state variable, and updating the benefit category counter and the latest event time, obtaining the user state variable.

[0077] In a specific embodiment, the execution step performs a KeyBy grouping operation on the event data, and routes the event data of the same user to the same parallel processing instance with the user identifier as the grouping key, and the process of obtaining the event data stream can specifically include the following steps:

[0078] Converting the event data into a DataStream data stream to obtain the stream data;

[0079] Performing a KeyBy grouping operation on the stream data, extracting the user identifier as the grouping key, calculating the hash value according to the grouping key and taking the modulus of the total number of parallel processing instances to obtain the target parallel processing instance number;

[0080] According to the target parallel processing instance number, the stream data is routed to the corresponding parallel processing instance to obtain the event data stream.

[0081] Specifically, a connection with the Kafka cluster is built based on the Flink stream computing framework, an FlinkKafkaConsumer component is used to read event messages encapsulated in JSON structure in Kafka, an original JSON string is parsed into a structured event object by a deserialization function and input into the DataStream environment of Flink, an initialization conversion of event data stream from Kafka to Flink is completed, and a stream data structure is obtained. The events in the stream are grouped by KeyBy operation to realize calculation isolation and state binding according to the user dimension, the user_id field in the event is extracted as a grouping key by calling the keyBy function of Flink, in the underlying execution process, Flink performs a hash function calculation on the grouping key, maps the user identifier to an integer value, and performs a modulo operation on the parallel degree of the current operator instance to determine which specific sub-task instance number the current event data should be routed to, for example, if the integer value of a user's user_id obtained by the hash function is 23, the event will be routed to the parallel processing sub-task numbered 23 mod 6 = 5. All event data of the same user is always allocated to the same parallel instance in the entire Flink computing graph, so that the user-dimension state variables (such as the activation sequence, the benefit category counter, etc.) can be continuously and stably accumulated and maintained without errors caused by partition drift or state cross-node, and the event data stream accurately routed according to the user identifier is obtained.

[0082] In a specific embodiment, the process of calculating the product of the time decay term frequency and the inverse document frequency in each benefit category according to the benefit activation sequence and the benefit category counter in the user state variable to obtain the benefit preference vector can specifically include the following steps:

[0083] Traverse the benefit categories and event timestamps of the benefit activation sequence in the user state variable, calculate the exponential decay weight according to the event timestamp and accumulate it by benefit category, to obtain the time decay term frequency of each benefit category;

[0084] Obtain the total number of users and the number of activated users of each benefit category, perform logarithmic operation on the ratio of the total number of users and the number of activated users to obtain the inverse document frequency of each benefit category;

[0085] Multiply the time decay term frequency of each benefit category by the corresponding inverse document frequency to obtain the preference score of the benefit category, and arrange the preference scores in order after traversing all the benefit categories to obtain the benefit preference vector.

[0086] Specifically, the benefit activation sequence is extracted from the user state variable, which is a dynamic list structure sorted in ascending order of event time, and each element contains a benefit category and an event timestamp. The benefit activation sequence is traversed in turn, and for each activation record, the benefit category is extracted as the classification identifier, and the event timestamp is time-difference calculated with the current window end time or the current system time, and then the time decay weight of the behavior is calculated using the exponential decay function exp(-λ·Δt / 3600), where λ is a preset decay coefficient (such as 0.01), and Δt is the interval in seconds between the current time and the event occurrence time, which is converted to hours by 3600. The decay weight is used to emphasize the recency of the behavior and suppress the influence weight of historical activation behavior in preference modeling, ensuring the model's ability to respond to the evolution of user interest over time. In each benefit category dimension, the decay weights of all activation records belonging to the category are accumulated, that is, the exponential decay values of multiple activation events of the same category are summed up to obtain the time-weighted term frequency of the category, reflecting the behavior activity and time-related preference intensity of the user for the benefit in the current period. After completing the time decay term frequency calculation of all categories, the total number of registered users on the platform (U_total) and the number of independent users who have activated each benefit category (U_activate_c) are extracted from the global statistics center or Redis cache of the platform. The total number of registered users and the number of activated users are smoothed to avoid division by zero errors and logarithmic function domain problems, that is, 1 is added to the original value respectively, and the formula is log((U_total + 1) / (U_activate_c + 1)), and the logarithmic calculation is performed for each benefit category to obtain its inverse document frequency, which measures the scarcity or differentiation ability of a certain benefit category among all users on the platform, that is, the less frequently used benefit category will have a higher weight to enhance the personalized expression ability of the preference vector. In each benefit category dimension, the time decay term frequency and the corresponding inverse document frequency are multiplied to calculate the final preference score of the user in the current category. For all predefined benefit categories, such as video entertainment, takeout food, finance and insurance, travel and tourism, etc. 20 first-level categories, the same calculation process is performed, and all preference scores are arranged in order of category number to form a fixed-length floating-point number vector, which is the benefit preference vector, quantifying the user's current interest intensity and behavior tendency in different benefit types.

[0087] The time decay word frequency of each equity category is obtained by traversing the equity category and event timestamp of the equity activation sequence in the user state variable, calculating the exponential decay weight according to the event timestamp and accumulating by equity category, including: obtaining the end time of the current window as the reference time, traversing each equity event of the equity activation sequence in the user state variable, extracting the equity category and event timestamp of each equity event to obtain the equity category and timestamp correspondence list; for each equity event in the equity category and timestamp correspondence list, calculate the time difference between the reference time and the timestamp of the equity event and convert it to hour units, multiply the time difference by the decay coefficient 0.01 to get the negative value as the exponential term of the exponential function, and obtain the time decay weight of the equity event by performing exponential operation on the natural constant, wherein the closer the equity event to the reference time, the closer the decay weight to 1, and the farther the equity event from the reference time, the smaller the decay weight; initialize the accumulators of each equity category to zero, traverse the equity category and timestamp correspondence list, judge whether the equity category of the current equity event is the same as the target equity category, if the same, add the time decay weight of the equity event to the accumulator of the corresponding equity category, if not, skip the equity event, and obtain the accumulated value in the accumulators of each equity category after traversal; the accumulated value in the accumulators of each equity category is taken as the time decay word frequency of the equity category, the time decay word frequencies are arranged in order of equity category, and a time decay word frequency list containing the time decay word frequencies of all equity categories is obtained.

[0088] The equity preference stability evaluation step based on multiple time windows is further included after obtaining the equity preference vector and before fusing with the topic probability vector: a user's historical time window equity preference vector queue is maintained based on a Flink state backend, a new equity preference vector is appended to the end of the queue every time a new equity preference vector is generated for a current time window, while historical window data exceeding a retention period is removed, the last N time window equity preference vectors are stored in the queue, N is set to 5-10 windows, and an equity preference vector sequence including the current window and the historical window is obtained; each equity category dimension in the equity preference vector sequence is traversed, a time series numerical sequence of the preference score of the equity category in each time window is extracted, the mean and standard deviation of the time series numerical sequence are calculated, the standard deviation is divided by the mean to obtain a coefficient of variation as the preference volatility indicator of the equity category, when the preference volatility indicator is less than a preset stability threshold, the equity category is determined as a stable preference category, when the preference volatility indicator is greater than the preset stability threshold, the equity category is determined as a volatile preference category, and a stable preference category set and a volatile preference category set are obtained; an enhancement coefficient is calculated for each equity category in the stable preference category set, an enhancement amplitude is determined according to the product of the mean of the time series numerical sequence of the equity category and the inverse of the preference volatility indicator, the higher the preference mean and the smaller the volatility, the greater the enhancement amplitude, and the enhancement amplitude is normalized to map to an enhancement coefficient between 1.2 and 1.5; an inhibition coefficient is calculated for each equity category in the volatile preference category set, an inhibition strength is determined according to the value of the preference volatility indicator, the greater the volatility, the greater the inhibition strength, and the inhibition strength is mapped to an inhibition coefficient between 0.5 and 0.8; the preference score of each equity category in the equity preference vector of the current time window is multiplied by the corresponding enhancement coefficient or inhibition coefficient, the score of the stable preference category is amplified to highlight the long-term stable interest of the user, and the score of the volatile preference category is reduced to suppress temporary or noisy preference volatility, and the equity preference vector is obtained.

[0089] In a specific embodiment, the process of performing step S300 can specifically include the following steps:

[0090] The equity activation sequence is taken as a document, and the number of occurrences of each equity category in the equity activation sequence is counted as a word;

[0091] The original matrix is constructed with the user as the row index and the equity category as the column index, and the number of occurrences is filled in to obtain a co-occurrence matrix;

[0092] An initial topic label is randomly assigned to each equity category in the co-occurrence matrix, and the topic assignment of each equity category is iteratively resampled according to the conditional probability to obtain a topic assignment result;

[0093] The number of interests of the user under each theme in the statistical theme distribution result is divided by the total number of interests of the user to calculate the belonging probability of each theme, and a theme probability vector is constructed according to the belonging probability of each theme.

[0094] Specifically, the interest activation sequence is taken as a document in a text mining task, and each interest category appearing in the interest activation sequence is regarded as a word constituting the document. The number of times of each interest appearing in the user activation behavior is counted by traversing the interest activation sequence to form a word frequency distribution of the user. A sparse user-category co-occurrence matrix is constructed with the user ID as the row index and all platform-defined interest categories as the column index, and each element represents the cumulative activation number of a certain user on a certain interest. The Collapsed Gibbs sampling algorithm is used to perform latent theme modeling processing on the co-occurrence matrix. In the model initialization stage, each interest category instance in the co-occurrence matrix, i.e., each interest category activated by the user, is randomly assigned an initial theme label z i , and the assignment strategy is to uniformly sample from K preset themes to obtain, and K is 10, representing 10 potential interest themes preset by the platform, such as home life, travel, financial security, etc. After the initial assignment is completed, the iteration sampling stage is entered. In each Gibbs sampling iteration, theme resampling operations are performed on each activation record in turn, temporarily removing the record from the current theme, and updating three core count variables: n_k^(u) represents the number of interests of the user under each theme, n_k^(c) represents the cumulative number of times of a certain interest appearing in theme k, and n_k^(·) represents the total number of interests under theme k. The probability of the current activation record belonging to each theme k is calculated based on the Bayesian conditional probability formula, which is P(z i =k) ∝ (n_k^(u)+α)·(n_k^(c)+β) / (n_k^(·)+C·β), where α and β are smoothing hyperparameters, and C is the total number of interest categories. According to the conditional probability, z i is executed polynomial sampling, the theme assignment is reselected and the count is added back, and the process is repeated for multiple rounds (such as 1000 times) to converge to a stable theme distribution. After the sampling process is completed, the number of interest records assigned to each theme by the user is counted, i.e., the value corresponding to each theme in n_k^(u) is extracted, and the number of interests under each theme is divided by the total number of interest activations n_total^(u) of the user to normalize, to obtain the theme belonging probability of each user. The belonging probabilities of all themes are organized in order as a K-dimensional vector, which is the theme probability vector of the user, reflecting the preference degree of the user for different implicit interest themes.

[0095] In one specific embodiment, the process of randomly assigning initial topic labels to each equity category in the co-occurrence matrix and iteratively resampling the topic assignment of each equity category according to conditional probability to obtain the topic assignment result can specifically include the following steps:

[0096] A1: Traverse each benefit category in the co-occurrence matrix and randomly assign initial topic tags, and initialize a topic counting matrix that records the number of benefits for users in each topic and the number of times benefit categories appear in each topic;

[0097] A2: In a single iteration, process each benefit category in turn, remove the benefit category from its assigned topics and update the topic counting matrix. Calculate the conditional probability of the benefit category belonging to each topic based on the number of user benefits in each topic and the number of times the benefit category appears in each topic in the updated topic counting matrix. Resample the topic tags of the benefit category according to the conditional probability and increment the count of the corresponding topic in the topic counting matrix by 1.

[0098] A3: Repeat the iterative process of step A2 until the preset number of iterations is reached to obtain the topic assignment result.

[0099] Specifically, the user-benefit category co-occurrence matrix is ​​traversed, where each cell represents the activation frequency of a user for a particular benefit category. Each benefit category instance in the matrix is ​​treated as a "term" to be modeled, and an initial topic tag z is randomly assigned to it during the initialization phase. i This allocation process samples using a discrete uniform distribution from 1 to K, ensuring that each term has an equal probability of being assigned to any topic in the initial state. Simultaneously, three types of topic counting matrices are constructed and initialized: the first type is a user-topic matrix n_k^(u), recording the number of activated benefits for each user in each topic; the second type is a topic-category matrix n_k^(c), recording the frequency of each benefit category in each topic; and the third type is a topic total vector n_k^(·), recording the total number of benefit category instances in each topic. The initialization of the matrices depends on the initial topic tag allocation results, i.e., after each z-step... i The initial allocation is performed by incrementing the corresponding counting matrix by 1 to form the basic distribution. In a single iteration, a Gibbs sampling operation is performed, sequentially processing each equity category term w. i The process involves two stages. The first stage is "removing old tags," which involves... i The original topic k is temporarily removed from the counting matrix, creating a new counting state that does not include the current term. The second stage is "resampling new labels," which calculates the conditional probability that the current term belongs to each topic k based on the updated counting matrix. The conditional probability formula is: P(z i =k) ​​∝(n_k^(u)+α)·(n_k^(wi )+ β) / (n_k^(u) + C · β), where n_k^(u) represents the number of word items the user already has under topic k, n_k^(w i ) represents the number of word items the user already has under topic k, n_k^(w i The frequency under topic k, n_k^(·) is the total frequency under topic k, C is the total number of equity categories, and α and β are hyperparameters for smoothing topic assignment and word item weight. The conditional probability vector is normalized to construct a multinomial distribution, and a new topic label z i is obtained by random sampling, and it is immediately added to the corresponding topic count matrix, i.e., 1 is added to the corresponding positions of n_k^(u), n_k^(c), and n_k^(·). After completing a single iteration, the removal, resampling, and count updating process described in A2 is repeated, all equity category records are traversed, and the process is executed for a set number of iterations (such as 1000 iterations). The process continues until the sampling distribution tends to be stable or the preset convergence round is reached. The result formed is a set of topic labels z i for each user's equity activation record, i.e., the topic assignment result.

[0100] In a specific embodiment, the process of performing step S400 can specifically include the following steps:

[0101] A topic and category mapping matrix is constructed to record the contribution weight of each topic to each equity category. The topic probability vector is multiplied by the topic and category mapping matrix to obtain an extended topic vector;

[0102] The user activity is calculated according to the cumulative activation number of the user, and the fusion weight coefficient is determined based on the user activity;

[0103] The equity preference vector and the extended topic vector are weighted and summed according to the fusion weight coefficient to obtain a fusion feature vector;

[0104] According to the scores in each dimension of the fusion feature vector and the preset threshold, label activation analysis is performed to obtain user labels and write them into the HBase database in real time.

[0105] Specifically, a mapping matrix between topics and interest categories is constructed, the mapping matrix takes K topics as rows and C interest categories as columns, forming a KxC two-dimensional matrix structure, each matrix element represents the contribution weight of topic k to interest category c, the weight comes from the word-topic distribution parameter in the LDA model training process, its calculation method is to divide the number of occurrences of interest category c under topic k n_k^(c) by the total number of all category terms under topic k n_k^(·), reflecting the representative degree of a certain type of interest in the topic semantic space. Perform matrix multiplication operation on the topic probability vector and the mapping matrix, project the low-dimensional topic vector to the interest category space consistent with the TF-IDF vector, generate an extended topic feature vector with a length of C, which is used to describe the user's long-term interest distribution in each interest category. Calculate the activity index of the user according to the total number of cumulative active interests, the total number is summed up by the activate_count_map in the user state variable, and log(1 + total activation times) is used as the activity function to compress the original value, in order to avoid extreme high-frequency behavior causing deviation in weight allocation. Based on the activity value, it is divided into two stages to set the fusion weight coefficient: if the user activity is lower than the threshold (for example, log(1+6.4) ≈ 2), it means that the user's behavior data is limited, at this time, more rely on the explicit preference captured by TF-IDF, set the TF-IDF feature weight to 0.7 and the extended topic feature weight to 0.3; if the activity is not less than the threshold, it means that the user is a high active user, rely on the deep interest features revealed by topic modeling, set the TF-IDF feature weight to 0.4 and the extended topic feature weight to 0.6. Perform L2 norm normalization processing on the two input vectors respectively, so that their module lengths are standardized to 1, avoiding affecting the numerical stability of the weighting process due to different feature magnitudes. Perform weighted summation on the normalized TF-IDF vector and the extended topic vector at the element level, perform fusion operation on each dimension of the C interest categories, form a fusion feature vector, which is used to represent the comprehensive preference tendency of the current user in all interest category dimensions. Perform label activation analysis operation according to the fusion feature vector. For preference class labels, such as "video entertainment type users", set the index of the interest category dimension corresponding to each label and the threshold, when the user's score in the interest category dimension exceeds the threshold, it is considered that the label meets the activation condition; for behavior class labels, such as "keep repurchase users", in addition to the fusion feature score, it also needs to query the user's historical card state record in SQL or cache to check whether there is a refund, whether there is an expired state and no valid card, etc. For risk labels, whether the activation frequency of a single category exceeds the threshold, and whether the information entropy of the fusion vector is less than the risk threshold (such as 1.5) are combined to judge whether the behavior is too concentrated.The activated label constructs a data structure of five-tuple, wherein the confidence score is the fusion feature value of the corresponding dimension, and the label life cycle is set according to the label type (7 days for the preference class, 30 days for the behavior class, and 1 day for the risk class). The label data is written into the column family structure of HBase by using the HBase client interface with the user ID as the RowKey, and the label details in the JSON structure are stored with the label name as the column qualifier, so as to complete the real-time generation and persistent storage of the user label.

[0106] In the method, the user activity is calculated according to the cumulative activation number of the user, and the fusion weight coefficient is determined based on the user activity, including: traversing the activation number of each benefit category in the benefit category counter in the user state variable, summing the activation numbers of all benefit categories to obtain the cumulative activation number of the user, taking the natural logarithm of the cumulative activation number of the user plus 1 to obtain the user activity index; performing segmented threshold judgment based on the user activity index, determining that the user is a low-activity user or a new user when the user activity index is less than 2, and determining that the user is a high-activity user when the user activity index is greater than or equal to 2, to obtain the user activity classification result; determining the TF-IDF feature fusion weight coefficient and the LDA topic feature fusion weight coefficient according to the user activity classification result, setting the TF-IDF feature fusion weight coefficient to 0.7 and setting the LDA topic feature fusion weight coefficient to 0.3 when the user is a low-activity user or a new user to give priority to capturing recent explicit preferences, and setting the TF-IDF feature fusion weight coefficient to 0.4 and setting the LDA topic feature fusion weight coefficient to 0.6 when the user is a high-activity user to give priority to mining long-term potential interests, to obtain the adaptively determined fusion weight coefficient pair; calculating the square root of the square sum of each dimension element of the benefit preference vector and the extended topic vector as the vector module length, respectively dividing each dimension element of the benefit preference vector and the extended topic vector by the corresponding vector module length to perform L2 norm normalization, obtaining the normalized benefit preference vector and the normalized extended topic vector, and then performing weighted summation according to the fusion weight coefficient pair.

[0107] In a specific embodiment, the process of performing step of performing label activation analysis according to the dimension scores in the fusion feature vector and a preset threshold to obtain the user label and writing the user label into the HBase database in real time can specifically include the following steps:

[0108] Extracting the dimension scores corresponding to each benefit category in the fusion feature vector, comparing each dimension score with a preset threshold of the corresponding benefit category, and obtaining a set of benefit categories satisfying the threshold condition;

[0109] Generating a preference class label according to the set of benefit categories satisfying the threshold condition, generating a behavior class label by combining the user historical behavior record with the business rule database, calculating the entropy value of the fusion feature vector and generating a risk class label by combining the cross-scene activation frequency, and obtaining the user label;

[0110] The user label is encapsulated as a data record with the user identification as the row key and the label name as the column qualifier, and the data record is written into the label column family of the user label table in the HBase database.

[0111] Specifically, the score value corresponding to each benefit category defined by the platform is extracted from the fusion feature vector dimension by dimension. The fusion feature vector has a fixed dimension (for example, 20 dimensions), each dimension corresponding to a first-level benefit category. According to a preset threshold table, the score of each dimension is compared with the preference threshold of the corresponding category to determine whether the score of a certain category exceeds the threshold. If the score of a certain category exceeds the threshold, the category is marked as the active preference direction of the user. The set of all benefit categories that meet the threshold condition constitutes the explicit interest output set of the user. Based on this, preference category labels such as "video entertainment type user" and "financial preference user" are generated according to the mapping rules of the corresponding benefit categories to label definitions in the label rule library. The confidence of the label is directly assigned using the score value of the corresponding dimension to represent the preference intensity. The fusion vector result is jointly analyzed with the external business rule database, and further judgment is made on the behavior category label. For example, for the "keep repurchase user" label, based on the condition that the score of the "fitness and health" dimension in the fusion vector is higher than the threshold (such as 0.25), the member card state record of the user in the benefit platform is queried through SQL or cache to determine whether there is a history of refund behavior, whether there is an expired member card, and whether there is a valid member at present. When the behavior rule and the feature score jointly meet the condition, the behavior category label is activated. For risk category labels, cross-scene analysis and distribution index calculation are performed in parallel. The number of activations of the user in different benefit categories is counted to determine whether the activation frequency of a certain type of benefit exceeds the platform warning threshold (such as 10 times). At the same time, the Shannon entropy value of the entire fusion feature vector is calculated = -Σp i · log2(p i ), where p i is the normalized probability value of the fusion score of each category. The Shannon entropy value reflects the concentration degree of user preference. When the entropy value is lower than the set threshold (such as 1.5) and there is a phenomenon of high single-category activation frequency, it is determined that the user has a clear tendency to shear the wool in behavior, and the "risk category" label is activated and given a corresponding confidence. All generated label information is packaged as a five-tuple data structure in a structured manner, with the user identification as the row key in the HBase database and the label name as the column qualifier. The label details (including confidence, generation time, expiration time, etc.) are packaged in JSON format and written to the specified column family. The writing process is performed in batches through the HBase Table.put() API to ensure transaction consistency and high concurrency performance of the writing operation.

[0112] Please refer to Figure 2 , Figure 2A structural schematic block diagram of a user tag intelligent generation and analysis system 200 based on big data analysis provided by an embodiment of the present application is shown in Figure 2 As shown, the user tag intelligent generation and analysis system 200 based on big data analysis comprises:

[0113] An event stream analysis module 210 is configured to subscribe to a benefit activation event stream, a benefit exercise event stream and a refund event stream through Kafka and analyze the event data obtained;

[0114] A stream processing module 220 is configured to perform Flink stream processing based on the event data, obtain a user state variable, and calculate a benefit preference vector according to the user state variable;

[0115] An iterative sampling module 230 is configured to construct a co-occurrence matrix of a user and a benefit category by taking a benefit activation sequence as a document and a benefit category as a word, perform iterative sampling on the co-occurrence matrix, and obtain a topic probability vector;

[0116] A tag activation analysis module 240 is configured to perform tag activation analysis based on the benefit preference vector and the topic probability vector, obtain a user tag, and write the user tag into a HBase database in real time.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, system and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0118] The integrated unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0119] The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A user tag intelligent generation and analysis method based on big data analysis, characterized in that, The method comprises the following steps: Subscribing to the benefit activation event stream, the benefit exercise event stream and the refund event stream through Kafka and obtaining event data by parsing; Performing Flink stream processing based on the event data to obtain a user state variable and calculating a benefit preference vector according to the user state variable; Constructing a co-occurrence matrix of users and benefit categories by taking the benefit activation sequence as a document and the benefit category as a word, and iteratively sampling to obtain a topic probability vector; Performing label activation analysis based on the benefit preference vector and the topic probability vector to obtain a user label, and writing the user label into an HBase database in real time; Specifically, the method comprises the following steps: constructing a topic and category mapping matrix to record the contribution weight of each topic to each benefit category, performing matrix multiplication operation on the topic probability vector and the topic and category mapping matrix to obtain an extended topic vector, calculating a user activity degree according to the cumulative activation number of the user and determining a fusion weight coefficient based on the user activity degree, performing weighted summation on the benefit preference vector and the extended topic vector according to the fusion weight coefficient to obtain a fusion feature vector, extracting the dimension score corresponding to each benefit category in the fusion feature vector, comparing each dimension score with a preset threshold value of the corresponding benefit category to obtain a benefit category set satisfying the threshold condition, generating a preference class label according to the benefit category set satisfying the threshold condition, combining a business rule database to query the historical behavior record of the user to generate a behavior class label, calculating the entropy value of the fusion feature vector and combining the cross-scene activation frequency to generate a risk class label, and obtaining a user label; and encapsulating the user label as a data record with a user identifier as a row key and a label name as a column qualifier, and writing the data record into a label column family of a user label table in the HBase database. 2.The method of claim 1, wherein, The method comprises the following steps: Configuring a benefit activation topic, a benefit exercise topic and a refund topic of Kafka, subscribing to the benefit activation topic, the benefit exercise topic and the refund topic through a Flink consumer instance respectively, and distributing to corresponding partitions in a user identifier hash value modulo manner to obtain a benefit activation event stream, a benefit exercise event stream and a refund event stream; Performing JSON format parsing on event messages received in the benefit activation event stream, the benefit exercise event stream and the refund event stream respectively, and extracting event data. 3.The user tag intelligent generation and analysis method based on big data analysis of claim 1, wherein, The method comprises the following steps: Performing KeyBy grouping operation on the event data to route the event data of the same user to the same parallel processing instance with a user identifier as a grouping key to obtain an event data stream; Dividing the event data stream into time sequence segments by applying a rolling event time window and configuring a water level mechanism to process delayed data to obtain an event data set within a window; traversing the event data set in the window in a window trigger, extracting the equity category and event timestamp in turn after sorting in ascending order of event timestamp, and updating the equity activation sequence, accumulating the activation times of each equity category to the equity category counter, and recording the latest event timestamp to the latest event time, to obtain the user state variable; calculating the product of the time decay term frequency and the inverse document frequency of each equity category according to the equity activation sequence and the equity category counter in the user state variable, to obtain the equity preference vector. 4.The method of claim 3, wherein, The KeyBy grouping operation is performed on the event data, and event data of the same user is routed to the same parallel processing instance with the user identifier as the grouping key, to obtain an event data stream, including: converting the event data into a DataStream data stream to obtain streaming data; performing a KeyBy grouping operation on the streaming data, extracting the user identifier as the grouping key, calculating a hash value according to the grouping key and taking the modulus of the total number of parallel processing instances to obtain a target parallel processing instance number; routing the streaming data to the corresponding parallel processing instance according to the target parallel processing instance number to obtain an event data stream. 5.The method of claim 4, wherein, The KeyBy grouping operation is performed on the event data, and event data of the same user is routed to the same parallel processing instance with the user identifier as the grouping key, to obtain an event data stream, including: traversing the event data set in the window in a window trigger, extracting the equity category and event timestamp in turn after sorting in ascending order of event timestamp, and updating the equity activation sequence, accumulating the activation times of each equity category to the equity category counter, and recording the latest event timestamp to the latest event time, to obtain the user state variable; calculating the product of the time decay term frequency and the inverse document frequency of each equity category according to the equity activation sequence and the equity category counter in the user state variable, to obtain the equity preference vector. The KeyBy grouping operation is performed on the event data, and event data of the same user is routed to the same parallel processing instance with the user identifier as the grouping key, to obtain an event data stream, including: 6.The method of claim 1, wherein, traversing the event data set in the window in a window trigger, extracting the equity category and event timestamp in turn after sorting in ascending order of event timestamp, and updating the equity activation sequence, accumulating the activation times of each equity category to the equity category counter, and recording the latest event timestamp to the latest event time, to obtain the user state variable; calculating the product of the time decay term frequency and the inverse document frequency of each equity category according to the equity activation sequence and the equity category counter in the user state variable, to obtain the equity preference vector. The KeyBy grouping operation is performed on the event data, and event data of the same user is routed to the same parallel processing instance with the user identifier as the grouping key, to obtain an event data stream, including: traversing the event data set in the window in a window trigger, extracting the equity category and event timestamp in turn after sorting in ascending order of event timestamp, and updating the equity activation sequence, accumulating the activation times of each equity category to the equity category counter, and recording the latest event timestamp to the latest event time, to obtain the user state variable; calculating the product of the time decay term frequency and the inverse document frequency of each equity category according to the equity activation sequence and the equity category counter in the user state variable, to obtain the equity preference vector. 7.The user tag intelligent generation and analysis method based on big data analysis of claim 6, wherein, ​ A1: traversing each equity category in the co-occurrence matrix and randomly assigning an initial topic label, initializing a topic count matrix recording the number of equity of users in each topic and the number of occurrences of equity categories in each topic; A2: in a single iteration, each equity category is processed in turn, the equity category is removed from the topic it has been assigned and the topic count matrix is updated, the conditional probability of the equity category belonging to each topic is calculated according to the number of equity of users in each topic and the number of occurrences of equity categories in each topic in the updated topic count matrix, the topic label of the equity category is resampled according to the conditional probability and the count of the corresponding topic in the topic count matrix is incremented by 1; A3: repeat the iteration process of step A2 until a preset number of iterations is reached to obtain the topic assignment result. 8.A system for intelligent generation and analysis of user tags based on big data analysis, characterized in that, A method for performing the big data analysis-based intelligent generation and analysis of user labels as claimed in any one of claims 1-7, comprising: an event stream parsing module configured to subscribe to equity activation event streams, equity exercise event streams and refund event streams through Kafka and parse to obtain event data; a stream processing module configured to perform Flink stream processing based on the event data to obtain user state variables and calculate equity preference vectors according to the user state variables; an iterative sampling module configured to construct a co-occurrence matrix of users and equity categories by taking the equity activation sequence as a document and the equity category as a vocabulary, perform iterative sampling and obtain a topic probability vector; a label activation analysis module configured to perform label activation analysis based on the equity preference vector and the topic probability vector to obtain user labels and write the user labels into a HBase database in real time.

Images