Long-term and short-term user portrait generation method, device, equipment and readable storage medium

Abstract

The present invention relates to the field of big data technology, and discloses a method for generating long-term and short-term user portraits, including the following steps: obtaining Hbase information of a distributed storage system from broadcast variables; pulling user behavior data in the Hbase information according to the user number, calculating real-time interest scores and historical interest scores, and obtaining long-term and short-term user interest tags according to the preset mapping relationship between the real-time interest score and the user interest tag, and the preset mapping relationship between the historical interest score and the user interest tag; converting the long-term and short-term user interest tags into Jason json format, encrypting the user's sensitive information through the information digest algorithm MD5, and storing it in a preset storage system; obtaining long-term and short-term user interest tags, and generating long-term and short-term user portraits based on the long-term and short-term user interest tags. The present invention also provides a long-term and short-term user portrait generation device, equipment, and readable storage medium, and the long-term and short-term user portrait generation method provided by the present invention improves the accuracy of user portraits.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method, device, equipment and readable storage medium for generating long-term and short-term user portraits. Background Art

[0002] At present, with the rapid development of mobile Internet, personalized systems have become an important technical means for many enterprises to apply big data to mine potential commercial value. As a method of abstractly outlining target users, user interest portraits have the ability to accurately and instantly analyze user behavior, interest characteristics, and consumption behavior, so they are favored by more and more companies. The user portrait system is usually portrayed using a professional user tag system. However, the existing user portrait system mainly uses offline processing, which cannot guarantee real-time and instant feedback. In recent years, real-time systems are usually developed and deployed using traditional servers, with poor scalability and fault tolerance. Some real-time user portrait systems are not based on the current leading big data processing technology, which makes it impossible to efficiently process massive user log data and perform some complex calculations to meet the changing market needs. The real-time user portrait system can only obtain the user's current and immediate interest tags, and cannot effectively combine the user's historical behavior to form a relatively complete and accurate user portrait of the user. How to improve the accuracy of user portraits is a technical problem that needs to be solved urgently in this technical field. Summary of the invention

[0003] The main purpose of the present invention is to provide a method, device, equipment and readable storage medium for generating long-term and short-term user portraits, aiming to solve the technical problem of low accuracy of existing user portraits.

[0004] Obtaining Hbase information of a distributed storage system from broadcast variables, wherein the Hbase information of the distributed storage system includes: columns and column cluster information;

[0005] Pull the user behavior data in the Hbase information according to the user ID, and calculate the real-time interest score and the historical interest score through a preset formula, wherein the user behavior data includes: click data and purchase data; obtain the long-term and short-term user interest tags according to the preset mapping relationship between the real-time interest score and the user interest tag, and the preset mapping relationship between the historical interest score and the user interest tag;

[0006] Convert the long-term and short-term user interest tags into json format, encrypt the user's sensitive information through the information digest algorithm MD5, and store them in a preset storage system, wherein the preset storage system includes a key-value storage system Redis and a distributed storage system Hbase;

[0007] According to the acquired instruction, long-term and short-term user interest tags corresponding to the instruction are acquired from the preset storage system, and long-term and short-term user portraits are generated based on the long-term and short-term user interest tags.

[0008] Optionally, in the acquiring of the distributed storage system Hbase information from the broadcast variable, wherein the distributed storage system Hbase information includes: before the column and column cluster information, further includes:

[0009] Create a SparkConf object configuration instance through the stream computing framework SparkStreaming program, and load the changed data into the SparkConf object, wherein the SparkConf object is used to manage Spark configuration items, and the SparkStreaming includes a driver end;

[0010] The driver reads the configuration file stored in the distributed file system HDFS, where the configuration file comes from the open source stream processing platform Kafka, the distributed storage system Hbase, and the key-value storage system Redis port and storage table name;

[0011] The log data of the open source stream processing platform Kafka is received through the stream computing framework SparkStreaming to form an initial data stream in Dstream format, and the initial data stream is distributed in different areas of the elastic distributed data set;

[0012] Preprocessing the initial data stream to obtain a target data stream;

[0013] The target data stream is converted into an event Event class and cached in memory or hard disk, and the Event class is converted into a queue List and stored in the distributed storage system Hbase intermediate table.

[0014] Optionally, the preprocessing the initial data stream to obtain a target data stream includes:

[0015] Re-partition the initial data stream distributed in the first partition area of ​​the elastic distributed data set through the HashPartitioner function in the repartition operation, obtain the intermediate data stream distributed in the second partition area, and record the number of the second partition area;

[0016] Determining whether the number of the second divided areas is greater than the number of the first divided areas;

[0017] If the number of the second divided areas is greater than the number of the first divided areas, a target data stream is obtained.

[0018] Optionally, the SparkStreaming program of the stream computing framework is used to create a SparkConf object configuration instance, and the changed data is loaded into the SparkConf object, wherein the SparkConf object is used to manage the configuration items of Spark, and the SparkStreaming includes a driver end, including:

[0019] Create a SparkConf object instance through the stream computing framework SparkStreaming program, and broadcast the configuration file to each working node through broadcast variables, where the working node includes: the memory and hard disk of the executor end;

[0020] Determine whether the data in the memory or the data in the hard disk is changed;

[0021] If the data in the memory or the data on the hard disk is changed, the changed data is loaded into a SparkConf object, wherein the SparkConf object is used to manage configuration items of Spark, and the SparkStreaming includes a driver end.

[0022] Optionally, before creating a SparkConf object configuration instance through the stream computing framework SparkStreaming program and loading the changed data into the SparkConf object, wherein the SparkConf object is used to manage Spark configuration items, the SparkStreaming includes the following before the driver end:

[0023] Determine whether the Driver has crashed and restarted;

[0024] If the Driver crashes and restarts, the configuration information in the configuration file is saved through the internal event Checkpoint pre-deployed in Spark Streaming.

[0025] Optionally, the user behavior data in the Hbase information is pulled according to the user ID, and the real-time interest score and the historical interest score are calculated by a preset formula, wherein the user behavior data includes: click data and purchase data includes:

[0026] Pull the user behavior data in the Hbase information in chronological order according to the user number;

[0027] The real-time interest score and the historical interest score are calculated by the following formula, wherein the user behavior data includes: click data and purchase data;

[0028] S = ∑(Y*W);

[0029] H = ∑ [W - (T) * X)];

[0030] Among them, Y is the real-time user behavior data, W is the preset user behavior weight, T is the day difference, X is the time decay coefficient, S is the real-time interest score, and H is the historical interest score.

[0031] Optionally, after obtaining the long-term and short-term user interest tags according to the preset mapping relationship between the real-time interest score and the user interest tag, and the preset mapping relationship between the historical interest score and the user interest tag, before converting the long-term and short-term user interest tags into the json format, encrypting the user's sensitive information by the information digest algorithm MD5, and storing it in the preset storage system, wherein the preset storage system includes the key-value storage system Redis and the distributed storage system Hbase, it also includes:

[0032] Calculating the accuracy of the long-term and short-term user interest tags, and determining whether the accuracy of the long-term and short-term user interest tags is less than or equal to a preset accuracy;

[0033] If the accuracy of the long-term and short-term user interest tags is less than or equal to the preset accuracy, the user behavior data in the distributed storage system Hbase information is pulled according to the user number, the real-time interest score and the historical interest score are calculated, and the time attenuation coefficient is optimized by the gradient descent algorithm until convergence.

[0034] Furthermore, in order to achieve the above-mentioned purpose, the present invention also provides a long-term and short-term user portrait generation device, and the long-term and short-term user portrait generation device includes the following modules:

[0035] An acquisition module is used to acquire Hbase information of a distributed storage system from broadcast variables, wherein the Hbase information of the distributed storage system includes: columns and column cluster information;

[0036] The interest score calculation module is used to pull the user behavior data in the Hbase information according to the user number, and calculate the real-time interest score and the historical interest score by the following formula, wherein the user behavior data includes: click data and purchase data;

[0037] A long-term and short-term user interest tag acquisition module, used to obtain long-term and short-term user interest tags according to the preset mapping relationship between the real-time interest score and the user interest tag, and the preset mapping relationship between the historical interest score and the user interest tag;

[0038] A long-term and short-term user interest tag storage module is used to convert the long-term and short-term user interest tags into json format, encrypt the user's sensitive information through the information digest algorithm MD5, and store it in a preset storage system, wherein the preset storage system includes a key-value storage system Redis and a distributed storage system Hbase;

[0039] The long-term and short-term user portrait acquisition module is used to acquire the long-term and short-term user interest tags corresponding to the instructions from the preset storage system according to the acquired instructions, and generate long-term and short-term user portraits based on the long-term and short-term user interest tags.

[0040] Optionally, the long-term and short-term user portrait generation device includes:

[0041] A data loading module is used to create a SparkConf object configuration instance through a stream computing framework SparkStreaming program, and load the changed data into the SparkConf object, wherein the SparkConf object is used to manage Spark configuration items, and the SparkStreaming includes a driver end;

[0042] The configuration file reading module is used to read the configuration file stored in the distributed file system HDFS on the driver side, wherein the configuration file comes from the open source stream processing platform Kafka, the distributed storage system Hbase, and the key-value storage system Redis port and storage table name;

[0043] An initial data stream conversion module is used to receive log data from the open source stream processing platform Kafka through the stream computing framework SparkStreaming to form an initial data stream in Dstream format, and the initial data stream is distributed in different areas of the elastic distributed data set;

[0044] An initial data stream preprocessing module, used for preprocessing the initial data stream to obtain a target data stream;

[0045] The event conversion module is used to convert the target data stream into an event Event class and cache it in the memory or hard disk, and convert the Event class into a queue List and store it in the distributed storage system Hbase intermediate table.

[0046] Optionally, the initial data stream preprocessing module includes:

[0047] A repartitioning unit is used to repartition the initial data stream distributed in the first partitioned area of ​​the elastic distributed data set through the HashPartitioner function in the repartition operation repartition, obtain the intermediate data stream distributed in the second partitioned area, and record the number of the second partitioned areas;

[0048] a first divided area quantity determination unit, configured to determine whether the quantity of the second divided areas is greater than the quantity of the first divided areas;

[0049] The target data stream acquisition unit is configured to obtain the target data stream if the number of the second divided areas is greater than the number of the first divided areas.

[0050] Optionally, the data loading module includes:

[0051] The broadcast variable broadcast unit is used to create a SparkConf object instance through the stream computing framework SparkStreaming program, and broadcast the configuration file to each working node through the broadcast variable, wherein the working node includes: the memory and hard disk of the executor end;

[0052] A data changing unit, used for determining whether the data in the memory or the data in the hard disk has changed;

[0053] A change data loading unit is used to load the changed data into a SparkConf object if the data in the memory or the data on the hard disk is changed, wherein the SparkConf object is used to manage configuration items of Spark, and the SparkStreaming includes a driver end.

[0054] Optionally, the long-term and short-term user portrait generation device includes:

[0055] Crash restart judgment module, used to judge whether the Driver end crashes and restarts;

[0056] The configuration information saving module is used to save the configuration information in the configuration file through the internal event Checkpoint pre-deployed in Spark Streaming if the Driver crashes and restarts.

[0057] Optionally, the interest score calculation module includes:

[0058] A user behavior data pulling unit, used to pull the user behavior data in the Hbase information in chronological order according to the user number;

[0059] A real-time calculation unit, used to calculate the real-time interest score and the historical interest score by the following formula, wherein the user behavior data includes: click data and purchase data;

[0060] S = ∑(Y*W);

[0061] H = ∑ [W - (T) * X)];

[0062] Among them, Y is the real-time user behavior data, W is the preset user behavior weight, T is the day difference, X is the time decay coefficient, S is the real-time interest score, and H is the historical interest score;

[0063] Optionally, the long-term and short-term user portrait generation device includes:

[0064] An accuracy calculation module, used to calculate the accuracy of the long-term and short-term user interest tags, and determine whether the accuracy of the long-term and short-term user interest tags is less than or equal to a preset accuracy;

[0065] The optimization module is used to execute, if the accuracy of the long-term and short-term user interest tags is less than or equal to the preset accuracy, pulling the user behavior data in the distributed storage system Hbase information according to the user number, calculating the real-time interest score and the historical interest score, and optimizing the time attenuation coefficient through the gradient descent algorithm until convergence.

[0066] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a long-term and short-term user portrait generation device, which includes a memory, a processor, and a long-term and short-term user portrait generation program stored in the memory and executable on the processor, and when the long-term and short-term user portrait generation program is executed by the processor, the steps of the long-term and short-term user portrait generation method as described in any one of the above items are implemented.

[0067] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a readable storage medium, on which a long-term and short-term user portrait generation program is stored. When the long-term and short-term user portrait generation program is executed by a processor, the steps of the long-term and short-term user portrait generation method as described in any of the above items are implemented.

[0068] The real-time user profiling method based on Spark Streaming proposed in the present invention can obtain the user's interest tags in real time, and can form the user's long-term and short-term portraits by taking into account the user's historical behavior. The data is converted into a data stream, and the data stream is divided more evenly. When generating a user portrait, historical behavior data and real-time behavior data are added, so that long-term and short-term user portraits can be generated, which can increase the accuracy of the user portrait. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1A schematic diagram of the structure of the operating environment of the long-term and short-term user portrait generation device involved in the embodiment of the present invention;

[0070] Figure 2 It is a flowchart of the first embodiment of the method for generating long-term and short-term user portraits of the present invention;

[0071] Figure 3 It is a flow chart of the second embodiment of the method for generating long-term and short-term user portraits of the present invention;

[0072] Figure 4 for Figure 3 A detailed flow chart of step S90;

[0073] Figure 5 for Figure 3 A detailed flow chart of step S60;

[0074] Figure 6 It is a flowchart diagram of a third embodiment of the method for generating long-term and short-term user portraits of the present invention;

[0075] Figure 7 for Figure 2 Detailed flow chart of step S20;

[0076] Figure 8 It is a flowchart diagram of a fourth embodiment of the method for generating long-term and short-term user portraits of the present invention;

[0077] Fig. 9 This is a schematic diagram of the functional modules of an embodiment of a device for generating long-term and short-term user portraits of the present invention.

[0078] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0079] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0080] The long-term and short-term user portrait generation method involved in the embodiment of the present invention is mainly applied to the long-term and short-term user portrait generation device, which can be a PC, a portable computer, a mobile terminal and other devices with display and processing functions.

[0081] Reference Figure 1 , Figure 1Schematic diagram of the hardware structure of the long-term and short-term user portrait generation device involved in the embodiment of the present invention. In the embodiment of the present invention, the long-term and short-term user portrait generation device may include a processor 1001 (such as a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard); the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface); the memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory, and the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0082] Those skilled in the art will understand that Figure 1 The hardware structure shown in the figure does not constitute a limitation on the long-term and short-term user portrait generation device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0083] Continue to refer to Figure 1 , Figure 1 The memory 1005 as a readable storage medium may include an operating system, a network communication module, and a long-term and short-term user portrait generation program.

[0084] exist Figure 1 In the embodiment, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 can call the long-term and short-term user portrait generation program stored in the memory 1005, and execute the long-term and short-term user portrait generation method provided in the embodiment of the present invention.

[0085] An embodiment of the present invention provides a method for generating long-term and short-term user portraits.

[0086] Reference Figure 2 , Figure 2 1 is a flow chart of the first embodiment of the method for generating long-term and short-term user portraits of the present invention. In this embodiment, the method for generating long-term and short-term user portraits includes the following steps:

[0087] Step S10, obtaining Hbase information of the distributed storage system from the broadcast variable, wherein the Hbase information of the distributed storage system includes: column and column cluster information;

[0088] In this embodiment, if you want to distribute large objects in distributed computing, such as dictionaries, collections, blacklists, etc., these will be distributed by the Driver side. Generally speaking, if this variable is not a broadcast variable, then each task will be distributed with one copy. In the case of a large number of tasks, the bandwidth of the Driver will become a bottleneck of the system, and it will consume a large amount of resources on the task server. If this variable is declared as a broadcast variable, then only each executor has one copy, and the tasks started by this executor will share this variable, saving communication costs and server resources. In this embodiment, each column and each column cluster information of the distributed storage system Hbase is obtained from the broadcast variable, and the columns are sorted in the column cluster according to the dictionary. For example, the column cluster is the fruit category, and the column is the basic information: apple.

[0089] Step S20, pulling user behavior data from Hbase information according to the user ID, and calculating the real-time interest score and the historical interest score through a preset formula, wherein the user behavior data includes: click data and purchase data;

[0090] In this embodiment, different weights are marked for different behaviors. For example, the "click" behavior is recorded as one weight, that is, 1 point, and the "purchase" behavior is recorded as 3 weights, that is, 3 points. The time decay coefficient is pre-configured. For example, the time decay coefficient is 0.05. If the user's current click interest is 1 point, then the historical interest score = 1-(day difference)*0.05=0.5 points, that is, the closer the user's historical behavior is to his current behavior, the higher the score, and the total score of the user's interest is calculated.

[0091] When pulling user behavior data, the current user's behavior data within the preset recent time period can be pulled first, for example, data within the latest month. "Purchase" is established in advance. Since there is a mapping relationship between "Purchase" and "Click" data, the "Click" data closest to it and the "Click" data within the preset time can be traced back based on the "Purchase" data. For example, data within the latest month can be obtained. Similarly, the amount of data that can be pulled can also be limited. For example, the user's most recent 50 behavior records are pulled to calculate the user tag, instead of only considering the user's real-time interests. For example, the user bought lipstick in real time, and the lipstick category is beauty products. He clicked on the mobile phone 10 days ago, and the mobile phone category is electrical appliances. The user interest tag format obtained according to the above calculation method is: {['lipstick', 3 points, 'beauty products'], ['mobile phone', 0.5 points, 'electrical appliances']}, where 0.5 points is calculated as: 1-0.05*10=0.5. It can be seen that when generating user interest tags, real-time behavior and historical behavior are taken into consideration, and different scores are given according to the user's action time and action method (purchase, click), so that user interest tags can be calculated more reasonably.

[0092] Step S30, obtaining long-term and short-term user interest tags according to a preset mapping relationship between real-time interest scores and user interest tags, and a preset mapping relationship between historical interest scores and user interest tags;

[0093] In this embodiment, there is a corresponding relationship between the real-time interest score and the user interest tag, and there is also a corresponding relationship between the historical interest score and the user interest tag. Therefore, the user interest tag can be obtained based on the corresponding relationship. For example, if the interest score is between 0-2 points, the label "user low-frequency behavior" is obtained, and if the interest score is between 3-5 points, the label "user high-frequency behavior" is obtained. This can provide a basis for pushing products to users.

[0094] Step S40, converting the long-term and short-term user interest tags into json format, encrypting the user's sensitive information through the information digest algorithm MD5, and storing them in the key-value storage system Redis and the distributed storage system Hbase;

[0095] In this embodiment, in order to ensure that the user's privacy information is not stolen, the user's sensitive information is encrypted.

[0096] Step S50, according to the acquired instruction, the long-term and short-term user interest tags corresponding to the instruction are acquired from the preset storage system, and the long-term and short-term user portraits are generated based on the long-term and short-term user interest tags.

[0097] In this embodiment, after the human-computer interaction interface receives an instruction to obtain a user portrait, it will parse the user instruction, obtain user interest tags based on the parsed results, and then convert the user interest tags into a user portrait. In this embodiment, the user portrait is a collection of user interest tags.

[0098] Reference Figure 3 , Figure 3 This is a flow chart of the second embodiment of the method for generating long-term and short-term user portraits of the present invention. In this embodiment, before obtaining the distributed storage system Hbase information from the broadcast variable, the distributed storage system Hbase information includes: columns, column cluster information, and also includes:

[0099] Step S60, creating a SparkConf object configuration instance through the stream computing framework SparkStreaming program, and loading the changed data into the SparkConf object, wherein the SparkConf object is used to manage Spark configuration items, and SparkStreaming includes a driver end;

[0100] In this embodiment, StreamingContext will group the generated elastic distributed dataset RDD into a batch according to the set batch processing time interval, for example, batch processing is performed every 15 seconds. DStream is represented by a group of continuous RDDs in time series, so this batch of RDDs is a DStream, and DStream can be converted into another DStream through operator operations. When receiving multiple data, multiple DStreams can be created to receive different branches of the source data stream.

[0101] Step S70, reading the configuration file stored in the distributed file system HDFS on the driver side, wherein the configuration file comes from the open source stream processing platform Kafka, the distributed storage system Hbase, the key-value storage system Redis port and storage table name;

[0102] In this embodiment, the configuration file stored in the distributed file system HDFS is read on the Driver side, and the configuration file comes from the open source stream processing platform Kafka, the distributed storage system Hbase, and the key-value storage system Redis port and storage table name.

[0103] Step S80, receiving log data from the open source stream processing platform Kafka through the stream computing framework Spark Streaming, forming an initial data stream in the Dstream format, and the initial data stream is distributed in different areas of the elastic distributed data set;

[0104] Step S90, preprocessing the initial data stream to obtain a target data stream;

[0105] Step S100, convert the target data stream into an event Event class and cache it in the memory or hard disk, and convert the Event class into a queue List and store it in the distributed storage system Hbase intermediate table.

[0106] Reference Figure 4 , Figure 4 for Figure 3 In this embodiment, the initial data stream is preprocessed to obtain the target data stream, including:

[0107] Step S901, re-dividing the initial data stream distributed in the first partition area of ​​the elastic distributed data set through the HashPartitioner function in the repartition operation repartition, obtaining the intermediate data stream distributed in the second partition area, and recording the number of the second partition areas;

[0108] In this embodiment, when the data calculated by each task is relatively small, the calculation speed may be slow because the amount of data processed is small, but the scheduling time required for the task will be long, so it is necessary to set a more reasonable partition for the first data stream. The first data stream is repartitioned so that it can be fully distributed on the big data platform. By creating more or fewer partitions, the data is randomly scattered so that the data is relatively uniform between different partitions. This operation is often performed to scatter data over the network.

[0109] The repartition operation repartition is used to repartition the data, and the HashPartitioner function is used here.

[0110] Step S902, determining whether the number of the second divided areas is greater than the number of the first divided areas;

[0111] In this embodiment, after the data is re-partitioned, the number of some regions may increase. In order to make the data more evenly distributed, it is desired that the number of the multiple divided regions is greater than the number of the original different regions.

[0112] Step S903: If the number of the second divided areas is greater than the number of the first divided areas, a target data stream is obtained.

[0113] In this embodiment, if the number of the multiple divided areas is greater than the number of the original different areas, it means that the data is relatively uniform, so the second data stream is obtained, otherwise, it is necessary to continue partitioning.

[0114] Reference Figure 5 , Figure 5 for Figure 3In this embodiment, a SparkConf object configuration instance is created through the stream computing framework SparkStreaming program, and the changed data is loaded into the SparkConf object, wherein the SparkConf object is used to manage the configuration items of Spark, and SparkStreaming includes a driver end, including:

[0115] Step S601, create a SparkConf object instance through the stream computing framework SparkStreaming program, and broadcast the configuration file to each working node through the broadcast variable, where the working node includes: the memory and hard disk of the executor end;

[0116] In this embodiment, a SparkConf object instance is created through the stream computing framework SparkStreaming program, and the broadcast variables are broadcast to each working node.

[0117] Step S602, determining whether the data in the memory or the hard disk has been changed;

[0118] In this embodiment, it is possible to check whether the data in the memory and the hard disk have been changed by traversing.

[0119] Step S603: If the data in the memory or the data on the hard disk is changed, the changed data is loaded into the SparkConf object, where the SparkConf object is used to manage the Spark configuration item SparkStreaming including the driver Driver end.

[0120] In this embodiment, the changed data is loaded into the configuration of the Spark application.

[0121] Reference Figure 6 , Figure 6 The flowchart of the third embodiment of the method for generating long-term and short-term user portraits of the present invention is shown in FIG. In this embodiment, a SparkConf object configuration instance is created through the stream computing framework SparkStreaming program, and the changed data is loaded into the SparkConf object, wherein the SparkConf object is used to manage the configuration items of Spark. SparkStreaming includes the driver side and also includes:

[0122] Step S110, determining whether the Driver has crashed and restarted;

[0123] Step S120: If the Driver crashes and restarts, the configuration information in the configuration file is saved through the internal event Checkpoint pre-deployed in Spark Streaming.

[0124] In this embodiment, when a code error occurs, the Driver side will crash and restart. In order to avoid data loss when the Driver side crashes and restarts, the configuration information in the configuration file is saved through the internal event Checkpoint. The internal event Checkpoint can be applied in Spark Streaming. The internal event Checkpoint can be used to save the configuration information of the configuration file so that breakpoint processing can be performed when the Driver crashes and restarts.

[0125] Reference Figure 7 , Figure 7 for Figure 2 In this embodiment, the user behavior data in the distributed storage system Hbase information is pulled according to the user number, and the real-time interest score and the historical interest score are calculated by the following formula, wherein the user behavior data includes: click data and purchase data, including:

[0126] Step S201, according to the user number, pull the user behavior data in the Hbase information in chronological order;

[0127] Step S202, calculating the real-time interest score and the historical interest score by the following formula, wherein the user behavior data includes: click data and purchase data;

[0128] S = ∑(Y*W);

[0129] H = ∑ [W - (T) * X)];

[0130] Among them, Y is the real-time user behavior data, W is the preset user behavior weight, T is the day difference, X is the time decay coefficient, S is the real-time interest score, and H is the historical interest score.

[0131] In this embodiment, different weights are marked for different behaviors. For example, the "click" behavior is recorded as one weight, that is, 1 point, and the "purchase" behavior is recorded as 3 weights, that is, 3 points. The time decay coefficient is pre-configured. For example, the time decay coefficient is 0.05. If the user's current click interest is 1 point, then the historical interest score = 1-(day difference)*0.05=0.5 points, that is, the closer the user's historical behavior is to his current behavior, the higher the score, and the total score of the user's interest is calculated.

[0132] Pulling user behavior data, when pulling, give priority to pulling the current user behavior data, for example, "purchase". Since there is a mapping relationship between "purchase" and "click" data, the "click" data closest to it and the "click" data within the preset time can be traced back based on the "purchase" data. For example, data within the last month can be obtained. Similarly, the amount of data that can be pulled can also be limited. For example, the user's most recent 50 behavior records are pulled to calculate the user tag, rather than only considering the user's real-time interests. For example, the user bought lipstick in real time, and the lipstick category is beauty products. He clicked on the mobile phone 10 days ago, and the mobile phone category is electrical appliances. The user interest tag format obtained according to the above calculation method is: {['lipstick', 3 points, 'beauty products'], ['mobile phone', 0.5 points, 'electrical appliances']}, where 0.5 points is calculated as: 1-0.05*10=0.5. It can be seen that when generating user interest tags, real-time behavior and historical behavior are taken into consideration, and different scores are given according to the user's action time and action method (purchase, click), so that user interest tags can be calculated more reasonably.

[0133] Reference Figure 8 , Figure 8 This is a flow chart of the fourth embodiment of the method for generating long-term and short-term user portraits of the present invention. After obtaining long-term and short-term user interest tags according to the preset mapping relationship between the real-time interest score and the user interest tag, and the preset mapping relationship between the historical interest score and the user interest tag, the long-term and short-term user interest tags are converted into the Jason json format, and the user's sensitive information is encrypted by the information digest algorithm MD5 and stored in the preset storage system, wherein the preset storage system includes the key-value storage system Redis and the distributed storage system Hbase, and further includes:

[0134] Step S130, calculating the accuracy of the long-term and short-term user interest tags, and determining whether the accuracy of the long-term and short-term user interest tags is less than or equal to a preset accuracy;

[0135] Step S140, if the accuracy of the long-term and short-term user interest tags is less than or equal to the preset accuracy, the user behavior data in the distributed storage system Hbase information is pulled according to the user number, the real-time interest score and the historical interest score are calculated, and the time decay coefficient is optimized by the gradient descent algorithm until convergence.

[0136] In this embodiment, the time decay coefficient is used to adjust the weight of user behaviors at different time points. Initially, the value of the time decay coefficient may not meet the current scenario. Therefore, it is necessary to optimize the time decay coefficient through a gradient descent algorithm based on whether it is lower than the preset accuracy until convergence, that is, the rate at which the time decay coefficient changes over time is less than the predetermined value.

[0137] The real-time user portrait method based on the stream computing framework Spark Streaming can obtain the user's interest tags in real time, and taking into account the user's historical behavior, it can form the user's long-term and short-term portrait. The data is converted into a data stream and the data stream is divided more evenly. When generating a user portrait, historical behavior data and real-time behavior data are added, so that long-term and short-term user portraits can be generated, which can increase the accuracy of the user portrait.

[0138] Reference Fig. 9 , Fig. 9 This is a functional module diagram of an embodiment of a long-term and short-term user portrait generation device of the present invention. In this embodiment, the long-term and short-term user portrait generation device includes:

[0139] The acquisition module 10 is used to acquire the distributed storage system Hbase information from the broadcast variables, wherein the distributed storage system Hbase information includes: columns and column cluster information;

[0140] The interest score calculation module 20 is used to pull the user behavior data in the Hbase information according to the user ID, and calculate the real-time interest score and the historical interest score by the following formula, wherein the user behavior data includes: click data and purchase data;

[0141] The long-term and short-term user interest tag acquisition module 30 is used to obtain the long-term and short-term user interest tags according to the preset mapping relationship between the real-time interest score and the user interest tag, and the preset mapping relationship between the historical interest score and the user interest tag;

[0142] The long-term and short-term user interest tag storage module 40 is used to convert the long-term and short-term user interest tags into the json format, encrypt the user's sensitive information through the information digest algorithm MD5, and store it in the preset storage system, wherein the preset storage system includes the key-value storage system Redis and the distributed storage system Hbase;

[0143] The long-term and short-term user portrait acquisition module 50 is used to acquire long-term and short-term user interest tags corresponding to the instructions from a preset storage system according to the acquired instructions, and generate long-term and short-term user portraits based on the long-term and short-term user interest tags.

[0144] The invention also provides a readable storage medium.

[0145] In this embodiment, the readable storage medium stores a long-term and short-term user portrait generation program, and when the long-term and short-term user portrait generation program is executed by the processor, the steps of the long-term and short-term user portrait generation method described in any of the above embodiments are implemented.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM) and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0147] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.

Claims

1. A method for generating long-term and short-term user portraits, characterized in that: The long-term and short-term user portrait generation method comprises: Create a SparkConf object configuration instance through the stream computing framework SparkStreaming program, and load the changed data into the SparkConf object, wherein the SparkConf object is used to manage Spark configuration items, and the SparkStreaming includes a driver end; The driver reads the configuration file stored in the distributed file system HDFS, where the configuration file comes from the open source stream processing platform Kafka, the distributed storage system Hbase, and the key-value storage system Redis port and storage table name; The log data of the open source stream processing platform Kafka is received through the stream computing framework SparkStreaming to form an initial data stream in Dstream format, and the initial data stream is distributed in different areas of the elastic distributed data set; Preprocessing the initial data stream to obtain a target data stream; Convert the target data stream into an event class and cache it in memory or hard disk, and convert the Event class into a queue List and store it in the distributed storage system Hbase intermediate table; Obtaining Hbase information of a distributed storage system from broadcast variables, wherein the Hbase information of the distributed storage system includes: columns and column cluster information; Pull the user behavior data in the Hbase information according to the user ID, and calculate the real-time interest score and the historical interest score through a preset formula, wherein the user behavior data includes: click data and purchase data; obtain the long-term and short-term user interest tags according to the preset mapping relationship between the real-time interest score and the user interest tag, and the preset mapping relationship between the historical interest score and the user interest tag; Convert the long-term and short-term user interest tags into json format, encrypt the user's sensitive information through the information digest algorithm MD5, and store them in a preset storage system, wherein the preset storage system includes a key-value storage system Redis and a distributed storage system Hbase; According to the acquired instruction, long-term and short-term user interest tags corresponding to the instruction are acquired from the preset storage system, and long-term and short-term user portraits are generated based on the long-term and short-term user interest tags.

2. The method for generating long-term and short-term user portraits according to claim 1, characterized in that: The preprocessing of the initial data stream to obtain a target data stream includes: Re-partition the initial data stream distributed in the first partition area of ​​the elastic distributed data set through the HashPartitioner function in the repartition operation, obtain the intermediate data stream distributed in the second partition area, and record the number of the second partition area; Determining whether the number of the second divided areas is greater than the number of the first divided areas; If the number of the second divided areas is greater than the number of the first divided areas, a target data stream is obtained.

3. The method for generating long-term and short-term user portraits according to claim 1, characterized in that: The SparkStreaming program of the stream computing framework is used to create a SparkConf object configuration instance, and the changed data is loaded into the SparkConf object, wherein the SparkConf object is used to manage the configuration items of Spark, and the SparkStreaming includes a driver end, including: Create a SparkConf object instance through the stream computing framework SparkStreaming program, and broadcast the configuration file to each working node through broadcast variables, where the working node includes: the memory and hard disk of the executor end; Determine whether the data in the memory or the data in the hard disk is changed; If the data in the memory or the data in the hard disk is changed, the changed data is loaded into the SparkConf object, wherein the SparkConf object is used to manage the configuration items of Spark, and the SparkStreaming includes a driver end.

4. The method for generating long-term and short-term user portraits according to claim 1, characterized in that: In the step of creating a SparkConf object configuration instance through the stream computing framework SparkStreaming program, and loading the changed data into the SparkConf object, wherein the SparkConf object is used to manage the configuration items of Spark, the SparkStreaming includes the following before the driver end: Determine whether the Driver has crashed and restarted; If the Driver crashes and restarts, the configuration information in the configuration file is saved through the internal event Checkpoint pre-deployed in Spark Streaming.

5. The method for generating long-term and short-term user portraits according to any one of claims 1 to 4, characterized in that: After obtaining the long-term and short-term user interest tags according to the preset mapping relationship between the real-time interest score and the user interest tag, and the preset mapping relationship between the historical interest score and the user interest tag, before converting the long-term and short-term user interest tags into the json format, encrypting the user's sensitive information by the information digest algorithm MD5, and storing it in the preset storage system, wherein the preset storage system includes the key-value storage system Redis and the distributed storage system Hbase, it also includes: Calculating the accuracy of the long-term and short-term user interest tags, and determining whether the accuracy of the long-term and short-term user interest tags is less than or equal to a preset accuracy; If the accuracy of the long-term and short-term user interest tags is less than or equal to the preset accuracy, the user behavior data in the distributed storage system Hbase information is pulled according to the user number, the real-time interest score and the historical interest score are calculated, and the time attenuation coefficient is optimized by the gradient descent algorithm until convergence.

6. A device for generating long-term and short-term user portraits, characterized in that: The long-term and short-term user portrait generation device includes the following modules: A data loading module is used to create a SparkConf object configuration instance through a stream computing framework SparkStreaming program, and load the changed data into the SparkConf object, wherein the SparkConf object is used to manage Spark configuration items, and the SparkStreaming includes a driver end; The configuration file reading module is used to read the configuration file stored in the distributed file system HDFS on the driver side, wherein the configuration file comes from the open source stream processing platform Kafka, the distributed storage system Hbase, and the key-value storage system Redis port and storage table name; An initial data stream conversion module is used to receive log data from the open source stream processing platform Kafka through the stream computing framework SparkStreaming to form an initial data stream in Dstream format, and the initial data stream is distributed in different areas of the elastic distributed data set; An initial data stream preprocessing module, used for preprocessing the initial data stream to obtain a target data stream; An event conversion module is used to convert the target data stream into an event Event class and cache it in memory or hard disk, and convert the Event class into a queue List and store it in the distributed storage system Hbase intermediate table; An acquisition module is used to acquire Hbase information of a distributed storage system from broadcast variables, wherein the Hbase information of the distributed storage system includes: columns and column cluster information; The interest score calculation module is used to pull the user behavior data in the Hbase information according to the user number, and calculate the real-time interest score and the historical interest score by the following formula, wherein the user behavior data includes: click data and purchase data; A long-term and short-term user interest tag acquisition module, used to obtain long-term and short-term user interest tags according to the preset mapping relationship between the real-time interest score and the user interest tag, and the preset mapping relationship between the historical interest score and the user interest tag; A long-term and short-term user interest tag storage module is used to convert the long-term and short-term user interest tags into json format, encrypt the user's sensitive information through the information digest algorithm MD5, and store it in a preset storage system, wherein the preset storage system includes a key-value storage system Redis and a distributed storage system Hbase; The long-term and short-term user portrait acquisition module is used to acquire the long-term and short-term user interest tags corresponding to the instructions from the preset storage system according to the acquired instructions, and generate long-term and short-term user portraits based on the long-term and short-term user interest tags.

7. A long-term and short-term user portrait generation device, characterized in that: The long-term and short-term user portrait generation device includes a memory, a processor, and a long-term and short-term user portrait generation program stored in the memory and executable on the processor. When the long-term and short-term user portrait generation program is executed by the processor, the steps of the long-term and short-term user portrait generation method according to any one of claims 1 to 5 are implemented.

8. A readable storage medium, characterized in that: The readable storage medium stores a long-term and short-term user portrait generation program, which, when executed by a processor, implements the steps of the long-term and short-term user portrait generation method according to any one of claims 1 to 5.

Images