User portrait generation method, device, equipment and storage medium based on thermal data

Abstract

The present invention relates to the field of big data technology, and discloses a method for generating a user portrait based on hot data, comprising the following steps: converting the user behavior data into a first data stream in the format of a data frame through a real-time streaming framework Structured Streaming; repartitioning the first data stream to obtain a second data stream, wherein the second data stream includes a user number; calculating the length of the user number in the second data stream, and determining whether the length of the user number meets a preset length; if so, determining whether the parameter of the second data stream is non-empty; if so, dividing the data into hot and cold data based on the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data; and generating a user portrait based on hot data. The present invention also provides a user portrait generation device, equipment, and storage medium based on hot data, and the user portrait generation method based on hot data provided by the present invention improves the timeliness of the user portrait.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method, device, equipment and storage medium for generating a user profile based on hot data. Background Art

[0002] With the development of mobile internet, the demand for personalization is increasing across various websites and mobile applications. Providing personalized services to users can more accurately meet their needs. Current user profiling systems primarily rely on offline processing to collect basic mobile user information (such as age and gender) or analyze user spending power through models.

[0003] The inventors realized that the offline user portrait system does not have real-time performance, and it is developed and deployed using traditional server methods, which has poor scalability and fault tolerance. Some real-time user portrait systems are not based on the current leading big data processing technology, which causes them to be unable to support gradually increasing massive data and cannot be effectively coupled with other systems. Recently, the real-time log processing and user portrait systems based on the batch processing framework Spark Streaming belong to the batch processing mode of mini batch (mini batch processing), which has a certain delay and a complex code structure, which is equivalent to the developer personally repeating the current standard data frame (dataframe) processing method, making it impossible for offline and real-time to share a set of codes, causing the problem of secondary development.

[0004] Furthermore, Spark Streaming was developed based on Spark 1.6, an older version that is not well compatible with some recently updated components. For example, Spark Streaming is based on Java 7, while Structured Streaming is based on Java 8. This prevents the use of some high-performance interfaces, resulting in low user profile timeliness. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method, device, equipment and storage medium for generating user portraits based on thermal data, aiming to solve the technical problem of low timeliness of existing user portraits.

[0006] Receive user behavior data through the log receiving tool Flume;

[0007] Converting the user behavior data into a first data stream in a data frame format through a real-time streaming framework StructuredStreaming;

[0008] Repartitioning the first data stream to obtain a second data stream, where the second data stream includes a user number;

[0009] Calculating the length of the user number in the second data stream, and determining whether the length of the user number meets a preset length;

[0010] If the user number length meets the preset length, determining whether the parameter of the second data stream is non-empty;

[0011] If the parameter of the second data stream is not empty, the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data are recorded, and based on the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data, the data are cached in a storage unit hfile of the distributed open source database HBase according to a preset rule, wherein the storage unit hfile is used to store hot data and cold data;

[0012] Obtaining hot data from the storage unit hfile, and calculating a real-time user interest score based on the hot data;

[0013] User interest tags are obtained according to the real-time user interest scores, and user portraits are generated based on the user interest tags.

[0014] Optionally, converting the user behavior data into a first data stream in a data frame format using a real-time streaming framework Structured Streaming includes the following steps:

[0015] Receiving the user behavior data through the real-time streaming framework StructuredStreaming;

[0016] The user behavior data is converted from string format data into a first data stream in data frame format through a stream processing module in the real-time stream framework StructuredStreaming.

[0017] Optionally, if the parameter of the second data stream is not empty, the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data are recorded, and based on the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data, the data are cached in a storage unit hfile of the distributed open source database HBase according to preset rules, wherein the storage unit hfile is used to store hot data and cold data, including the following steps:

[0018] Determine whether the time point for caching the second data stream into the distributed open source database HBase is within a preset time period;

[0019] If the time point when the second data stream is cached into the distributed open source database HBase is within a preset time period, the historical occurrence frequency of each data of the second data stream in the HBase is counted by polling, wherein the historical occurrence frequency includes: a first frequency appearing in the storage unit hfile storing hot data and a second frequency appearing in the storage unit hfile storing cold data;

[0020] determining whether the first frequency is greater than or equal to the second frequency;

[0021] If the first frequency is greater than or equal to the second frequency, caching the second data stream into a storage unit hfile for storing hot data in the storage unit HBase;

[0022] If the first frequency is less than the second frequency, the second data stream is cached in an hfile storage unit of HBase for storing cold data.

[0023] Optionally, obtaining hot data from the storage unit storage unit hfile and calculating the real-time user interest score based on the hot data includes the following steps:

[0024] Pull hot data from the storage unit hfile that stores hot data;

[0025] The real-time user interest score is calculated using the following formula:

[0026] Among them, S is the real-time user interest score, h i is the i-th type of thermal data, i≤n, W i is the preset hot data weight corresponding to the i-th hot data, where the hot data is data that appears within a preset time period and whose first frequency is greater than or equal to the second frequency.

[0027] Optionally, before receiving the user behavior data through the log receiving tool Flume, the following steps are also included:

[0028] Pulling relevant configuration files stored on the distributed file system HDFS on a driver node of the real-time streaming framework Structured Streaming, wherein the driver node is deployed with broadcast variables;

[0029] The information of the configuration file is broadcast to the memory of each execution node through broadcast variables.

[0030] Optionally, before the driver node of the real-time streaming framework Structured Streaming pulls the relevant configuration files stored on the distributed file system HDFS, the driver node further includes the following steps before deploying the broadcast variables:

[0031] Create a SparkSession instance using the real-time streaming framework StructuredStreaming, configure it based on the backpressure mechanism, and dynamically adjust the data reception rate of the receiver based on the job execution information fed back by the task scheduler.

[0032] The elastic distributed dataset is saved in a serialized format through the serialization library.

[0033] Optionally, repartitioning the first data stream to obtain a second data stream, where the second data stream includes a user number, includes the following steps:

[0034] Calling the partitioner HashPartitioner to re-partition the first data stream pre-distributed in the first partition area of ​​the elastic distributed dataset to obtain the first data stream distributed in the second partition area, and recording the number of the second partition areas;

[0035] determining whether the number of the second divided areas is greater than the number of the first divided areas;

[0036] If the number of the second divided areas is greater than the number of the first divided areas, a second data stream is obtained, and the second data stream includes a user number.

[0037] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a user portrait generation device based on thermal data, the user portrait generation device based on thermal data comprising:

[0038] The receiving module is used to receive user behavior data through the log receiving tool Flume;

[0039] A user behavior data conversion module is used to convert the user behavior data into a first data stream in a data frame format through a real-time streaming framework StructuredStreaming;

[0040] A first data stream partitioning module, configured to repartition the first data stream to obtain a second data stream, wherein the second data stream includes a user number;

[0041] A user number length calculation module, configured to calculate the length of the user number in the second data stream and determine whether the user number length meets a preset length;

[0042] a parameter determination module, configured to determine whether a parameter of the second data stream is non-empty if the length of the user number satisfies a preset length;

[0043] a storage module configured to, if the parameter of the second data stream is not empty, record the frequency of occurrence of each data item in the second data stream and the time interval between two occurrences of each data item, and cache the data in a storage unit hfile of the distributed open source database HBase according to preset rules based on the frequency of occurrence of each data item in the second data stream and the time interval between two occurrences of each data item, wherein the storage unit hfile is used to store hot data and cold data;

[0044] A user interest score calculation module is used to obtain hot data from the storage unit hfile and calculate a real-time user interest score based on the hot data;

[0045] A user portrait generation module is used to obtain user interest tags according to the real-time user interest scores and generate a user portrait based on the user interest tags.

[0046] Optionally, the user behavior data conversion module includes:

[0047] A receiving unit, configured to receive the user behavior data via a real-time streaming framework StructuredStreaming;

[0048] The user behavior data conversion unit is used to convert the user behavior data from string format data into a first data stream in data frame format through a stream processing module in the real-time stream framework StructuredStreaming.

[0049] Optionally, the storage module includes the following:

[0050] A preset time determination unit, configured to determine whether a time point for caching the second data stream into the distributed open source database HBase is within a preset time period;

[0051] a historical occurrence frequency statistics unit, configured to count the historical occurrence frequency of each data of the second data stream in the storage unit HBase by polling if the time point when the second data stream is cached in the distributed open source database HBase is within a preset time period, wherein the historical occurrence frequency includes: a first frequency appearing in the storage unit hfile storing hot data and a second frequency appearing in the storage unit hfile storing cold data;

[0052] a first frequency determination unit, configured to determine whether the first frequency is greater than or equal to the second frequency;

[0053] A hot data storage unit is configured to cache the second data stream into a storage unit hfile of the storage unit HBase for storing hot data if the first frequency is greater than or equal to the second frequency, and to cache the second data stream into a storage unit hfile of the storage unit HBase for storing cold data if the first frequency is less than the second frequency.

[0054] Optionally, the user interest score calculation module includes:

[0055] Hot data pulling unit, used to pull hot data from the storage unit hfile storing hot data;

[0056] The user interest score calculation unit is used to calculate the real-time user interest score using the following formula:

[0057] Among them, S is the real-time user interest score, h i is the i-th type of thermal data, i≤n, W i is the preset hot data weight corresponding to the i-th hot data, where the hot data is data that appears within a preset time period and whose first frequency is greater than or equal to the second frequency.

[0058] Optionally, the user profile generation device based on thermal data further includes:

[0059] A configuration file pulling module is used to pull relevant configuration files stored on the distributed file system HDFS on a driver node of the real-time streaming framework Structured Streaming, wherein the driver node is deployed with broadcast variables;

[0060] The information broadcast module is used to broadcast the information of the configuration file to the memory of each execution node through broadcast variables.

[0061] Optionally, the user profile generation device based on thermal data further includes:

[0062] Create a module for creating a SparkSession instance using the StructuredStreaming real-time streaming framework, configuring the backpressure mechanism and job execution information fed back by the task scheduler, and dynamically adjusting the data reception rate of the receiver.

[0063] The saving module is used to save the elastic distributed dataset in a serialized format through the serialization library.

[0064] Optionally, the first data stream partitioning module includes:

[0065] A calling unit is configured to call a partitioner HashPartitioner to re-partition a first data stream pre-distributed in a first partitioned area of ​​an elastic distributed dataset to obtain the first data stream distributed in a second partitioned area, and record the number of the second partitioned areas;

[0066] a second divided area number judging unit, configured to judge whether the number of the second divided areas is greater than the number of the first divided areas;

[0067] The second data stream acquiring unit is configured to acquire a second data stream if the number of the second divided areas is greater than the number of the first divided areas, where the second data stream includes a user number.

[0068] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a user portrait generation device based on thermal data, wherein the user portrait generation device based on thermal data includes a memory, a processor, and a user portrait generation program based on thermal data stored on the memory and executable on the processor. When the user portrait generation program based on thermal data is executed by the processor, the steps of the user portrait generation method based on thermal data as described in any one of the above-mentioned items are implemented.

[0069] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a user portrait generation program based on thermal data is stored. When the user portrait generation program based on thermal data is executed by a processor, the steps of the user portrait generation method based on thermal data as described in any one of the above-mentioned items are implemented.

[0070] The present invention receives user behavior data through the real-time streaming framework StructuredStreaming, and stores the user behavior data in different areas of HBase to achieve isolation of hot and cold data, and then generates user profiles based on the hot data. In this way, the timeliness of user profiles can be improved, and thus the accuracy of user profiles can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A schematic diagram of the structure of the operating environment of a device for generating a user profile based on thermal data according to an embodiment of the present invention;

[0072] Figure 2 This is a flow chart of a first embodiment of a method for generating a user profile based on thermal data according to the present invention;

[0073] Figure 3 for Figure 2 A detailed flow chart of an embodiment of step S20;

[0074] Figure 4 for Figure 2 A detailed flow chart of an embodiment of step S60;

[0075] Figure 5 for Figure 2 A detailed flow chart of an embodiment of step S70;

[0076] Figure 6 This is a flow chart of a second embodiment of the method for generating a user profile based on thermal data according to the present invention;

[0077] Figure 7 This is a flow chart of a third embodiment of a method for generating a user profile based on thermal data according to the present invention;

[0078] Figure 8 for Figure 2 Detailed flow chart of step S30;

[0079] Figure 9 This is a functional module diagram of an embodiment of a user portrait generation device based on thermal data of the present invention.

[0080] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0081] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0082] The thermal data-based user portrait generation method involved in the embodiment of the present invention is mainly applied to a thermal data-based user portrait generation device, which can be a PC, portable computer, mobile terminal or other device with display and processing functions.

[0083] Reference Figure 1 , Figure 1Schematic diagram of the hardware structure of the user portrait generation device based on thermal data involved in the embodiment of the present invention. In the embodiment of the present invention, the user portrait generation device based on thermal data may include a processor 1001 (such as a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard); the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface); the memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0084] Those skilled in the art will understand that Figure 1 The hardware structure shown in does not constitute a limitation on the user portrait generation device based on thermal data, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0085] Continue to refer to Figure 1 , Figure 1 The memory 1005 as a storage medium may include an operating system, a network communication module, and a user portrait generation program based on thermal data.

[0086] exist Figure 1 In the embodiment, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 can call the user portrait generation program based on thermal data stored in the memory 1005, and execute the user portrait generation method based on thermal data provided in an embodiment of the present invention.

[0087] An embodiment of the present invention provides a method for generating a user portrait based on thermal data.

[0088] Reference Figure 2 , Figure 2 1 is a flow chart of a first embodiment of a method for generating a user profile based on thermal data according to the present invention. In this embodiment, the method for generating a user profile based on thermal data includes the following steps:

[0089] Step S10, receiving user behavior data through the log receiving tool Flume;

[0090] In this embodiment, Flume is a tool for receiving user behavior data. Flume is a highly available, highly reliable, distributed system for massive log collection, aggregation, and transmission provided by Cloudera. Flume supports customization of various data senders in the log system for data collection. Therefore, user behavior data is collected through Flume to facilitate obtaining interest tags based on the user behavior data.

[0091] Step S20, converting the user behavior data into a first data stream in a data frame format through the real-time streaming framework StructuredStreaming;

[0092] In this embodiment, user behavior data includes log data and data frames, which are two-dimensional data structures, i.e., data arranged in a table format of rows and columns. This user behavior data is stream processed using the real-time streaming framework Structured Streaming. Stream processing is an abstract term for unbounded datasets. New data continuously arrives over time, thus enabling real-time processing of user behavior data.

[0093] Step S30: repartition the first data stream to obtain a second data stream, where the second data stream includes a user number;

[0094] In this embodiment, the partitioning operation is a repartition operation, which can divide the data more evenly. The partitioner HashPartitioner can be called to repartition the first data stream, thereby obtaining a second data stream with more even data distribution.

[0095] Step S40, calculating the length of the user number in the second data stream, and determining whether the length of the user number meets the preset length;

[0096] In this embodiment, the user numbers can be traversed in sequence. Each user has a unique number. For example, if the user's mobile phone number is the user number, the user number length is 11 digits. When the traversed user number is 11 digits, it means that the user number meets the preset length.

[0097] Step S50: If the user number length meets the preset length, determine whether the parameter of the second data stream is non-empty;

[0098] In this embodiment, each user number is associated with different user behavior data. When the user number meets the preset length, it only indicates that there is such a unique user. However, the user may have only registered an account on the specified platform, but not performed other operations. Therefore, the parameters of the second data stream are empty. Only when the parameters are not empty, it is indicated that the user has performed an operation and generated user behavior data. Only when user behavior data exists can user interest tags be generated based on the user behavior data.

[0099] Step S60: If the parameter of the second data stream is not empty, the frequency of occurrence of each data item in the second data stream and the time interval between two occurrences of each data item are recorded, and based on the frequency of occurrence of each data item in the second data stream and the time interval between two occurrences of each data item, the data items are cached in a storage unit hfile of the distributed open source database HBase according to a preset rule, wherein the storage unit hfile is used to store hot data and cold data;

[0100] In this embodiment, hfile is the smallest storage unit of the storage space HBase. Cold data refers to data that is relatively old and has a relatively low frequency of occurrence from the user's current behavior data, while hot data is relatively new and has a high frequency of occurrence. The judgment rules for cold data and hot data can be set in advance. For example, data within one month is hot data, and data older than one month is cold data. HBase data is ultimately stored in the distributed file system HDFS in the form of HFile. Different hfiles can be configured to store different data, which can isolate cold and hot data and call hot data at any time. In the prior art, user profiles are generally generated using data without any data differentiation, so user profiles have certain inaccuracies.

[0101] Step S70, obtaining hot data from the storage unit hfile, and calculating the real-time user interest score based on the hot data;

[0102] Step S80: Obtain user interest tags according to the real-time user interest score, and generate a user portrait based on the user interest tags.

[0103] In this embodiment, if there is a request to generate a user portrait, it is necessary to generate the user portrait based on hot data. In this way, the data that is close to the current user behavior data and has a higher frequency of occurrence can be used as the basis for generating the user portrait, thereby improving the accuracy of the user portrait.

[0104] User behavior data is received through the real-time streaming framework StructuredStreaming and stored in different areas of HBase to achieve isolation between hot and cold data. This solution has two innovations: 1. The use of the real-time streaming framework StructuredStreaming avoids the latency introduced by the batch processing framework StructuredStreaming, enabling real-time user profile generation; 2. Based on the frequency of user behavior data and the time interval between the data and the user's actions, different data are classified into different areas of HBase, and user profiles are generated from the hot data, which improves the accuracy of user profiles.

[0105] Reference Figure 3 , Figure 3 for Figure 2 Detailed flow diagram of an embodiment of step S20 in FIG. In this embodiment, step S20 converts user behavior data in the format of a data frame into a first data stream through the real-time streaming framework Structured Streaming, including the following steps:

[0106] Step S201, receiving user behavior data through the real-time streaming framework StructuredStreaming;

[0107] In this embodiment, Structured Streaming is a scalable, fault-tolerant stream processing engine built on the Spark SQL engine. When processing stream computations, it can be treated like batch computations on static data. The Spark SQL engine is responsible for continuously running the computations and updating the final results as streaming data continues to arrive.

[0108] Step S202: converting the user behavior data from a string format into a first data stream in a data frame format through a stream processing module in the real-time streaming framework StructuredStreaming.

[0109] In this embodiment, since the real-time streaming framework StructuredStreaming supports stream processing, there is no delay. The stream processing module is processStream, which converts the user behavior data from string format data into a first data stream in data frame format.

[0110] Reference Figure 4 , Figure 4 for Figure 2Detailed flow diagram of an embodiment of step S60 in FIG. In this embodiment, step S60, if the parameter of the second data stream is not empty, then caching the second data stream into an hfile of HBase for storing hot data and cold data according to the frequency and time interval of the second data stream, includes the following steps:

[0111] Step S601, determining whether the time point for caching the second data stream into the distributed open source database HBase is within a preset time period;

[0112] In this embodiment, the second data stream cached in the distributed open source database HBase can be timed by a timer in order to distinguish different data by different time intervals, for example, data within one day is hot data, and data within one year is cold data.

[0113] Step S602: If the time point when the second data stream is cached in the distributed open source database HBase is within a preset time period, historical occurrence frequencies of the second data stream in HBase are counted by polling, where the historical occurrence frequencies include: a first frequency of occurrence in the hfile storing hot data and a second frequency of occurrence in the hfile storing cold data;

[0114] In this embodiment, data is stored in HBase's hfiles, which are HBase storage units. To distinguish between hot and cold data, the frequency of data entering the hfiles is counted through polling. Different data can be pre-stored in different hfiles based on the time at which the data enters the hfiles. For example, the ratio of the number of times data a appears in the hfile storing hot data to the total number of times data a appears is used as the first frequency, and the ratio of the number of times data a appears in the hfile storing cold data to the total number of times data a appears is used as the second frequency.

[0115] Step S603, determining whether the first frequency is greater than or equal to the second frequency;

[0116] In this embodiment, the frequency of data entering the hfile is counted by polling, and it is determined whether the frequency of data appearing in the hfile for storing hot data is greater than the frequency of data appearing in the hfile for storing cold data.

[0117] Step S604: If the first frequency is greater than or equal to the second frequency, cache the second data stream into an hfile of HBase for storing hot data;

[0118] Step S605: If the first frequency is less than the second frequency, the second data stream is cached in an hfile of HBase for storing cold data.

[0119] In this embodiment, if the first frequency is greater than or equal to the second frequency, the second data stream is cached in the hfile of HBase for storing hot data. Otherwise, the second data stream is cached in the hfile of HBase for storing cold data. The purpose is to store cold and hot data in different hfiles.

[0120] Reference Figure 5 , Figure 5 for Figure 2 Detailed flow chart of an embodiment of step S70 in FIG. In this embodiment, step S70, obtaining hot data from the storage unit hfile and calculating the real-time user interest score based on the hot data, includes the following steps:

[0121] Step S701, pull hot data from the hfile storing hot data;

[0122] In this embodiment, if there is a request to generate a user profile, hot data is pulled from the hfile storing hot data. Since hot data is data that appears more frequently and is closer to the current user behavior data, the accuracy of the user profile can be improved.

[0123] Step S702: Calculate the real-time user interest score using the following formula:

[0124] Among them, S is the real-time user interest score, h i is the i-th type of thermal data, i≤n, W i is the preset hot data weight corresponding to the i-th hot data, where the hot data is data that appears within a preset time period and whose first frequency is greater than or equal to the second frequency.

[0125] In this embodiment, hot data is data that appears frequently and is close to the current user behavior data, rather than cold data that is far away from the current user behavior data. Therefore, the accuracy of the user portrait can be improved. Weights can be set in advance for different types of hot data. For example, some hot data have high weights, while other hot data have low weights. Some hot data with low weights account for a smaller proportion when calculating the real-time user interest score. Since the user interest score determines the user interest tag, for example, if the score of "running" is greater than the score of "eating", the user interest tag "running" can be obtained. Each user interest score corresponds to a user interest tag. User interest tags with high user interest scores are given priority, and then all user interest tags are combined together to generate a user portrait. Since the user interest score is calculated in real time based on real-time hot data, the user interest tag corresponding to the user interest score is also generated in real time, so the generated user portrait is also real-time, which can improve the accuracy of the user portrait.

[0126] User behavior data is received through the real-time streaming framework StructuredStreaming and stored in different areas of HBase to achieve isolation between hot and cold data. User profiles are then generated based on the hot data to improve the accuracy of user profiles.

[0127] Reference Figure 6 , Figure 6 This is a flow chart of the second embodiment of the method for generating a user profile based on thermal data of the present invention. Figure 2 Step S10, before receiving the user behavior data through the log receiving tool Flume, also includes the following steps:

[0128] Step S90: Pulling, on a driver node of the real-time streaming framework Structured Streaming, relevant configuration files stored on the distributed file system HDFS, wherein the driver node is deployed with broadcast variables;

[0129] In this embodiment, the driver node, or driver node, of the real-time streaming framework Structured Streaming pulls the relevant configuration files stored on the distributed file system HDFS. Broadcast variables allow a program to efficiently send a large, read-only value to all execution nodes for use by one or more Structured Streaming operations.

[0130] Step S100: broadcast the configuration file information to the memory of each execution node through broadcast variables.

[0131] In this embodiment, the execution node, namely the executor, broadcasts the information of the configuration file to the memory of each execution node through the broadcast variable.

[0132] Reference Figure 7 , Figure 7 This is a flow chart of the third embodiment of the method for generating a user profile based on thermal data of the present invention. Figure 6 In step S90, the driver node of the real-time streaming framework Structured Streaming pulls the relevant configuration files stored on the distributed file system HDFS. Before the driver node is deployed with the broadcast variables, the following steps are also included:

[0133] Step S110: Create a SparkSession instance through the real-time streaming framework StructuredStreaming, configure the execution information of the job based on the back pressure mechanism and feedback from the task scheduler, and dynamically adjust the data receiving rate of the receiver;

[0134] In this example, the SparkSession instance provides a unified entry point for users to access various Spark features. First, start the StructuredStreaming program to create a SparkSession instance. Configure the backpressure mechanism to feed job execution information to the JobScheduler. Dynamically adjust the receiver data reception rate based on the JobScheduler's feedback of job execution information. The backpressure mechanism is spark.streaming.backpressure.enabled.

[0135] Step S120: Save the elastic distributed dataset in a serialized format through a serialization library.

[0136] In this embodiment, the Java serialization library is used by default in the prior art. This solution uses the Kryo serialization library, which can reduce the memory space occupied. The serialization library is Kryo Serializer.

[0137] Resilient Distributed Datasets, or RDDs, are a special type of collection that supports multiple sources, has a fault-tolerant mechanism, can be cached, and supports parallel operations. An RDD represents a dataset in a partition.

[0138] Reference Figure 8 , Figure 8 for Figure 2Detailed flow diagram of step S30 in the embodiment. In this embodiment, step S30, repartitioning the first data stream to obtain a second data stream, wherein the second data stream includes a user number, includes the following steps:

[0139] Step S301: Call the partitioner HashPartitioner to re-partition the first data stream pre-distributed in the first partition area of ​​the elastic distributed dataset to obtain the first data stream distributed in the second partition area, and record the number of the second partition areas;

[0140] Step S302, determining whether the number of the second divided areas is greater than the number of the first divided areas;

[0141] Step S303: If the number of the second divided areas is greater than the number of the first divided areas, a second data stream is obtained, where the second data stream includes a user number.

[0142] In this embodiment, the repartition operation is used to repartition the data, using the HashPartitioner partitioner. After repartitioning the data, the number of some regions increases. To achieve a more even distribution of data, it is desirable to have a larger number of partitioned regions than the original number of distinct regions. This results in a more evenly distributed data distribution.

[0143] Reference Figure 9 , Figure 9 This is a functional module diagram of an embodiment of a user profile generation device based on thermal data of the present invention. In this embodiment, the user profile generation device based on thermal data includes:

[0144] Receiving module 10, used to receive user behavior data through the log receiving tool Flume;

[0145] The user behavior data conversion module 20 is used to convert the user behavior data into a first data stream in a data frame format through a real-time streaming framework StructuredStreaming;

[0146] A first data stream partitioning module 30 is configured to repartition the first data stream to obtain a second data stream, wherein the second data stream includes a user number;

[0147] A user number length calculation module 40 is configured to calculate the length of the user number in the second data stream and determine whether the user number length meets a preset length;

[0148] a parameter determination module 50, configured to determine whether a parameter of the second data stream is non-empty if the length of the user number satisfies a preset length;

[0149] a storage module 60 configured to, if the parameter of the second data stream is not empty, record the frequency of occurrence of each data item in the second data stream and the time interval between two occurrences of each data item, and cache the data in a storage unit hfile of the distributed open source database HBase according to a preset rule based on the frequency of occurrence of each data item in the second data stream and the time interval between two occurrences of each data item, wherein the storage unit hfile is used to store hot data and cold data;

[0150] A user interest score calculation module 70 is configured to obtain hot data from the storage unit hfile and calculate a real-time user interest score based on the hot data;

[0151] The user portrait generation module 80 is configured to obtain user interest tags according to the real-time user interest scores and generate a user portrait based on the user interest tags.

[0152] The present invention also provides a storage medium.

[0153] In this embodiment, the storage medium stores a user portrait generation program based on thermal data. When the user portrait generation program based on thermal data is executed by the processor, the steps of the user portrait generation method based on thermal data as described in any of the above embodiments are implemented.

[0154] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, or of course by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM) and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0155] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of the present invention, or directly or indirectly used in other related technical fields, all fall within the protection of the present invention.

Claims

1. A method for generating a user portrait based on thermal data, characterized in that: The method for generating a user portrait based on thermal data includes: Receive user behavior data through the log receiving tool Flume; The user behavior data is converted into a first data stream in a data frame format through a real-time streaming framework StructuredStreaming; Repartitioning the first data stream to obtain a second data stream, wherein the second data stream includes a user number; Calculating the length of the user number in the second data stream, and determining whether the length of the user number meets a preset length; If the user number length satisfies the preset length, determining whether the parameter of the second data stream is not empty; If the parameter of the second data stream is not empty, the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data are recorded, and based on the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data, the data is cached in a storage unit hfile of the distributed open source database HBase according to a preset rule, wherein the storage unit hfile is used to store hot data and cold data; Acquire hot data from the storage unit hfile, and calculate a real-time user interest score based on the hot data; User interest tags are obtained according to the real-time user interest scores, and user portraits are generated based on the user interest tags.

2. The method for generating a user profile based on thermal data according to claim 1, characterized in that: The step of converting the user behavior data into a first data stream in a data frame format through a real-time streaming framework Structured Streaming includes: Receiving the user behavior data through the real-time streaming framework StructuredStreaming; The user behavior data is converted from string format data into a first data stream in data frame format through a stream processing module in the real-time stream framework Structured Streaming.

3. The method for generating a user profile based on thermal data according to claim 1, characterized in that: If the parameter of the second data stream is not empty, the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data are recorded, and based on the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data, the data is cached in a storage unit hfile of the distributed open source database HBase according to a preset rule, wherein the storage unit hfile is used to store hot data and cold data, including: Determine whether the time point for caching the second data stream into the distributed open source database HBase is within a preset time period; If the time point of caching the second data stream into the distributed open source database HBase is within a preset time period, the historical occurrence frequency of each data of the second data stream in the HBase is counted by polling, wherein the historical occurrence frequency includes: a first frequency appearing in the storage unit hfile storing hot data and a second frequency appearing in the storage unit hfile storing cold data; Determining whether the first frequency is greater than or equal to the second frequency; If the first frequency is greater than or equal to the second frequency, cache the second data stream into a storage unit hfile of HBase for storing hot data; If the first frequency is less than the second frequency, the second data stream is cached in a storage unit hfile of HBase for storing cold data.

4. The method for generating a user profile based on thermal data according to claim 1, characterized in that: The step of obtaining hot data from the storage unit hfile and calculating a real-time user interest score according to the hot data includes: Pull hot data from the storage unit hfile that stores hot data; The real-time user interest score is calculated using the following formula: Among them, S is the real-time user interest score, h i is the i-th type of thermal data, i≤n, W i is a preset hot data weight corresponding to the i-th hot data, where the hot data is data that appears within a preset time period and whose first frequency is greater than or equal to the second frequency.

5. The method for generating a user profile based on thermal data according to claim 1, characterized in that: Before receiving the user behavior data through the log receiving tool Flume, the method further includes: Pull relevant configuration files stored on the distributed file system HDFS on a driver node of the real-time streaming framework StructuredStreaming, wherein the driver node is deployed with broadcast variables; The information of the configuration file is broadcast to the memory of each execution node through broadcast variables.

6. The method for generating a user profile based on thermal data according to claim 5, characterized in that: Before the driver node of the real-time streaming framework Structured Streaming pulls the relevant configuration files stored in the distributed file system HDFS, the driver node of the real-time streaming framework Structured Streaming further includes: Create a SparkSession instance through the real-time streaming framework StructuredStreaming, configure the back pressure mechanism, and the job execution information fed back by the task scheduler, and dynamically adjust the data receiving rate of the receiver. The resilient distributed dataset is saved in a serialized format through the serialization library.

7. The method for generating a user profile based on thermal data according to any one of claims 1 to 6, characterized in that: The repartitioning of the first data stream to obtain a second data stream, wherein the second data stream includes a user number, includes: Calling the partitioner HashPartitioner to re-partition the first data stream pre-distributed in the first partition area of ​​the elastic distributed data set to obtain the first data stream distributed in the second partition area, and recording the number of the second partition area; Determining whether the number of the second divided areas is greater than the number of the first divided areas; If the number of the second divided areas is greater than the number of the first divided areas, a second data stream is obtained, and the second data stream includes a user number.

8. A user portrait generation device based on thermal data, characterized in that: The user portrait generation device based on thermal data includes the following modules: The receiving module is used to receive user behavior data through the log receiving tool Flume; A user behavior data conversion module, used to convert the user behavior data into a first data stream in a data frame format through a real-time streaming framework StructuredStreaming; A first data stream partitioning module, used for repartitioning the first data stream to obtain a second data stream, wherein the second data stream includes a user number; A user number length calculation module, used to calculate the length of the user number in the second data stream, and determine whether the user number length meets a preset length; A parameter judgment module, configured to judge whether a parameter of the second data stream is not empty if the length of the user number satisfies a preset length; A storage module, configured to record the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data if the parameter of the second data stream is not empty, and cache the data in a storage unit hfile of a distributed open source database HBase according to a preset rule based on the frequency of occurrence of each data in the second data stream and the time interval between two occurrences of each data, wherein the storage unit hfile is used to store hot data and cold data; A user interest score calculation module, used to obtain hot data from the storage unit hfile, and calculate a real-time user interest score based on the hot data; The user portrait generation module is used to obtain user interest tags according to the real-time user interest scores and generate user portraits based on the user interest tags.

9. A user portrait generation device based on thermal data, characterized in that: The thermal data-based user portrait generation device includes a memory, a processor, and a thermal data-based user portrait generation program stored in the memory and executable on the processor. When the thermal data-based user portrait generation program is executed by the processor, the steps of the thermal data-based user portrait generation method as described in any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores a user portrait generation program based on thermal data, and when the user portrait generation program based on thermal data is executed by the processor, the steps of the user portrait generation method based on thermal data according to any one of claims 1 to 7 are implemented.

Images