A sample data generation method and apparatus

Abstract

Embodiments of the present application provide a sample data generation method and device, for the received user behavior data, real-time pre-processing of user behavior data into user behavior objects, and the user behavior object is stored in association with the unique identifier corresponding to the user behavior data; for the received exposure data including the display content and its feature data, the display content in the exposure data is pre-processed into exposure data objects, and the exposure data objects are stored in association with the unique identifier corresponding to the exposure data; and, the feature data of the display content in the exposure data is stored in association with the unique identifier corresponding to the exposure data; at the end of each unique identifier corresponding time window, all user behavior objects, exposure data objects and feature data associated with the unique identifier in the time window are obtained, and all user behavior objects, exposure data objects and feature data obtained are spliced into sample data corresponding to the unique identifier associated with the time window.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular to the field of sample generation in content recommendation, and more particularly to a sample data generation method and device. BACKGROUND

[0002] In a content recommendation system, a candidate content set needs to be input into a model for scoring and sorting, the purpose being to filter out the content that is most matched with the user interest or the content that is most interesting to the user and recommend to the user, and the model needs to input a large amount of samples for training, so the sample is the cornerstone of the recommendation system. Currently, there are mainly two sample generation methods, one is an offline generation method, which is run once every certain time, such as 1 hour, and the specific method is to use big data tools such as Hive, Spark-Sql, etc. to process offline data, and then join according to the associated key, and then store the join result, that is, the sample, as offline data for subsequent training. The other method is a real-time sample splicing method, which is an attempt to improve the offline method. Currently, the mainstream is to read data from a message queue in real time through some real-time computing framework such as Storm, Spark-streaming, and then store the intermediate state in an external memory database such as Redis to complete splicing and output the final sample to the message queue so that the downstream can consume the training model in real time.

[0003] In the process of implementing the present application, the applicant found that there are at least the following problems in the prior art:

[0004] In the process of generating a large-scale sample, real-time and low resource consumption cannot be considered at the same time. SUMMARY

[0005] The sample data generation method and device provided by the embodiments of the present application are also a large-scale real-time sample splicing method and device, which solve the technical problem that real-time and low resource consumption cannot be considered at the same time in the process of generating a large-scale sample.

[0006] To achieve the above purpose, in a first aspect, the embodiments of the present application provide a sample data generation method, comprising:

[0007] The received user behavior data is preprocessed into a user behavior object in real time, and the user behavior object is stored in association with the unique identifier corresponding to the user behavior data;

[0008] The exposure data including the display content and the feature data of the display content is preprocessed into exposure data objects, and the exposure data objects are stored in association with the unique identifier corresponding to the exposure data; and the feature data of the display content in the exposure data is stored in association with the unique identifier corresponding to the exposure data.

[0009] For each unique identifier, in the case that the unique identifier is not associated with a time window or the time window associated with the unique identifier has ended, a time window with a preset time width is established with the earliest receiving time in the user behavior data and the exposure data corresponding to the unique identifier as the starting point of the time window, and the established time window is associated with the unique identifier.

[0010] At the end of the time window corresponding to each unique identifier, all user behavior objects, exposure data objects and feature data associated with the unique identifier in the time window are acquired, and the acquired all user behavior objects, exposure data objects and feature data are spliced into sample data corresponding to the unique identifier associated with the time window.

[0011] The time window is a window with a preset time length with the earliest receiving time in the user behavior data and the exposure data corresponding to the unique identifier as the starting point, and the unique identifiers corresponding to each user behavior data and each exposure data generated by the same user on the same display content are the same.

[0012] In a second aspect, an embodiment of the present application provides a sample data generation device, comprising:

[0013] The behavior data acquisition unit is configured to, for the received user behavior data, pre-process the user behavior data into user behavior objects in real time, and store the user behavior objects in association with the unique identifier corresponding to the user behavior data.

[0014] The exposure data acquisition unit is configured to, for the received exposure data including the display content and the feature data of the display content, pre-process the display content in the exposure data into exposure data objects, and store the exposure data objects in association with the unique identifier corresponding to the exposure data; and store the feature data of the display content in the exposure data in association with the unique identifier corresponding to the exposure data.

[0015] The sample splicing unit is configured to, at the end of the time window corresponding to each unique identifier, acquire all user behavior objects, exposure data objects and feature data associated with the unique identifier in the time window, and splice the acquired all user behavior objects, exposure data objects and feature data into sample data corresponding to the unique identifier associated with the time window.

[0016] The time window is a window of a preset time length with the earliest receiving time in the user behavior data and the exposure data corresponding to the unique identifier as a starting point, and each user behavior data and each exposure data generated by a same user on a same displayed content has a same unique identifier.

[0017] The above technical solution has the following beneficial effects: when the user behavior data and the exposure data are received in real time, the user behavior data is preprocessed into a user behavior object, the displayed content in the exposure data is preprocessed into an exposure data object, and the features of the user behavior object, the exposure data object, and the displayed content in the exposure data are respectively associated with the unique identifier and stored. At the end of the specified time window, the user behavior object, the exposure data object, and the feature data corresponding to the same user and the same displayed content in the time window are spliced into sample data. In this process, the user behavior data and the displayed content in the exposure data are received and processed in real time, achieving real-time. At the same time, since the feature data usually has a large scale, the feature data is stored separately from the user behavior object and the exposure data object, so that the storage resources can be allocated more reasonably, and the consumption of the memory is avoided. By setting the time window, the feature data is read out at the end of the time window and spliced with the user behavior object and the exposure data object to obtain the sample data. The high throughput and low delay characteristics are ensured in the large-scale feature scenario. Further, the user behavior object and the exposure data object are stored in the memory for real-time processing, and the features of the displayed content in the exposure data are stored in the external storage. The characteristics that the data amount of the user behavior object and the exposure data object is relatively small and the data amount of the feature data is relatively large are fully utilized. The user behavior object and the exposure data object are placed in the memory to provide better real-time performance, and the feature data is placed in the external storage to reduce the consumption of the memory resources, so as to balance the real-time performance and the resource consumption, further ensure the high throughput and low delay characteristics in the large-scale feature scenario, balance the real-time performance and the stability of the system, and reserve available resources for future functional expansion. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only show some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0019] Figure 1 is a flowchart of a sample data generation method according to an embodiment of the present application;

[0020] Figure 2 is a feature data storage process schematic diagram of one of the embodiments of the present application;

[0021] Figure 3 is a sample data distribution process schematic diagram of one of the embodiments of the present application;

[0022] Figure 4 is an abnormality monitoring process schematic diagram of one of the embodiments of the present application;

[0023] Figure 5 is a structural diagram of a sample data generation device of one of the embodiments of the present application;

[0024] Figure 6 is a structural diagram of another sample data generation device of one of the embodiments of the present application;

[0025] Figure 7 is a flowchart of another sample data generation method of one of the embodiments of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0027] The inventors analyzed two main ways of sample splicing in the prior art, namely, offline sample splicing and real-time computing framework combined with in-memory database. The inventors found that the two technologies have advantages and disadvantages. The advantage of offline splicing is that it saves resources, but the disadvantage is also obvious, that is, the real-time performance of the sample is poor, and the real-time performance has a great influence on the accuracy of model scoring, so this way is only suitable for recommendation systems that are not too sensitive to real-time performance. The second way has greatly improved real-time performance compared with the offline solution, but due to the reasons of the computing framework itself, when the data size is large, the throughput decreases and the computing delay increases. In addition, due to the help of in-memory database for splicing, when the feature size in the data is large, a large amount of server resources, especially memory resources, will be consumed, resulting in high sample splicing cost. In addition, this splicing method based on in-memory database has poor scalability, and when the data volume suddenly increases or hot spots arrive, memory overflow may occur, resulting in unavailable services.

[0028] In a first aspect, as shown in Figure 1 , the embodiment of the present application provides a sample data generation method applied to a sample data generation device, comprising:

[0029] Step S100, for the received user behavior data, pre-process the user behavior data into user behavior objects in real time, and store the user behavior objects in association with the unique identifier corresponding to the user behavior data;

[0030] Step S101, for the received exposure data including the display content and its feature data, pre-process the display content in the exposure data into exposure data objects, and store the exposure data objects in association with the unique identifier corresponding to the exposure data; and store the feature data of the display content in the exposure data in association with the unique identifier corresponding to the exposure data;

[0031] Step S102, at the end of each unique identifier corresponding time window, obtain all user behavior objects, exposure data objects and feature data associated with the unique identifier within the time window, and splice all the obtained user behavior objects, exposure data objects and feature data into sample data corresponding to the unique identifier associated with the time window;

[0032] Wherein, the time window is a window of a preset time length with the earliest receiving time in the user behavior data and the exposure data corresponding to the unique identifier as the starting point, and the unique identifiers corresponding to each user behavior data and each exposure data generated by the same user on the same display content are the same;

[0033] The unique identifier is used to identify each user behavior data and each exposure data generated by the same user on the same display content;

[0034] Preferably, the unique identifier is determined by the information of the user and the information of the display content displayed to the user, and the unique identifier obtained by the same user and the same display content is the same.

[0035] In some embodiments, the display content is content displayed to a user, including but not limited to microblog, news, advertisement, video, music, and other content composed of text, picture, and / or audio / video; the user can operate on the display content and generate corresponding user behavior data, including but not limited to data corresponding to operations such as browsing, liking, forwarding, and commenting. The user behavior data can be generated by a module of a system providing the display content and distributed to a sample data generation device to generate sample data in real time, for example, the user behavior data can be distributed to the sample data generation device for processing through a message queue, a publish / subscribe mechanism, or the like. The exposure data contains the display content and the feature data of the display content; generally, the data volume of the feature data of the display content is large. The exposure data can be distributed to the sample data generation device through a mechanism similar to the user behavior data. The unique identifier can identify the user behavior data and the exposure data; the unique identifier is used to identify each user behavior data and each exposure data generated based on the same user on the same display content; preferably, the unique identifier is generated according to information of the display content and information of a user receiving the display content, the information of the display content including an id of the display content, and the information of the user including a user id; when each user behavior data and each exposure data is received, the information of the display content corresponding to the user behavior data and the exposure data and the information of the user receiving the display content are also received. Preferably, the unique identifier (i.e., sid) is generated using the id of the display content and the user id; since the id of the display content and the user id are both unique, the unique identifier sid generated by the two is also unique. The user can generate multiple user behaviors on the same display content, so that one exposure data can correspond to one or more user behavior data. As long as the exposure data and the user behavior data obtained by the same user on the same display content use the same unique identifier (i.e., sid identifier), when the sample is spliced, the associated exposure data object, the feature data of the display content in the exposure data, and all user behavior objects can be obtained according to the value of the unique identifier. The user behavior data is usually from a log, and the format of the user behavior data may not meet the requirements when the sample is spliced, or there are abnormal fields, illegal fields, other log contents, etc. in the user behavior data. For each received user behavior data, pre-processing is needed, the data is encapsulated into a unified data format after data extraction, filtering, and conversion, a user behavior object is generated, and the user behavior object is stored in association with the unique identifier corresponding to the user behavior data. Each received exposure data includes the display content corresponding to the exposure data and the feature data of the display content. The feature data of the display content is usually large-scale data.The system providing the display content sends the exposure data to the user terminal, and the user terminal displays the display content in the exposure data to the user. The user needs a certain time to operate the display content. Therefore, in order to collect all user behavior data of the user for the display content, a time window is generated for each unique identifier (sid). The time width of the time window can be obtained according to the statistics of the historical operation logs of a large number of users. When the time window ends, it is considered that all user behavior data of the user for the display content has been collected. The determination of the starting point of the time window is described below. When the system providing the display content sends the exposure data to the user terminal, the exposure data and the user behavior data generated by the user for the display content are also sent to the sample data generation device in sequence through, for example, a message queue and the like. Due to the communication mechanism, the arrival time of the exposure data and the user behavior data associated with the same unique identifier at the sample data generation device may be uncertain. For example, the exposure data and the user behavior data are transmitted through respective corresponding message queues. The message queue may have a relative delay. The first user behavior data generated by the user may arrive at the sample data generation device earlier than the exposure data. At this time, the time when the first user behavior data arrives is used as the time of the starting point of the time window. In the usual case, the exposure data arrives at the sample data generation device earlier than the user behavior data. At this time, the arrival time of the exposure data is used as the time of the starting point of the time window. The system providing the display content faces multiple users, and each time can provide only one display content to each user, or can provide multiple display contents to each user at a time. That is, the sample data generation device may receive user behavior data and exposure data of a user or multiple users for one display content or multiple different display contents. The sample data generation device receives the user behavior data and the exposure data through the unique identifier, and generates a respective time window for each unique identifier. Therefore, when splicing the sample, the user behavior data and the exposure data of the same user for the same display content are spliced into one sample data as an identifier. The user behavior data and the exposure data corresponding to one unique identifier will not be mixed with the user behavior data and the exposure data corresponding to another unique identifier.

[0036] The embodiment of the present application has the following technical effects: when the user behavior data and the exposure data are received in real time, the user behavior data is preprocessed into user behavior objects in real time, the display content in the exposure data is preprocessed into exposure data objects, and the feature data of the user behavior objects, the exposure data objects, and the display content in the exposure data are respectively stored in association with unique identifiers, the user behavior objects, the exposure data objects, and the feature data corresponding to the same user and the same display content in a time window are spliced into sample data at the end of the specified time window, in this process, the user behavior data and the display content in the exposure data are received and processed in real time, the real-time purpose is achieved, and meanwhile, since the feature data usually has a large scale, the feature data is stored separately from the user behavior data and the exposure data, the storage resources can be more reasonably allocated, the consumption of a large amount of memory is avoided, the feature data is read out at the end of the time window and the user behavior objects and the exposure data objects are obtained to obtain sample data by setting the time window, and the high-throughput and low-delay characteristics are ensured under the large-scale feature scenario.

[0037] Further, the method further comprises: when the user behavior data or the exposure data is received, judging whether the unique identifier corresponding to the user behavior data or the exposure data is associated with a valid time window;

[0038] In the case that the unique identifier is not associated with a time window or the time window associated with the unique identifier has ended, a time window is established with the reception time in the user behavior data or the exposure data corresponding to the unique identifier as a starting point, and the established time window is associated with the unique identifier.

[0039] In some embodiments, when the user behavior data or the exposure data is received, it is judged whether the unique identifier corresponding to the user behavior data or the exposure data is associated with a valid time window; if a time window is associated and the time window has not ended, the time window is valid, and the time window corresponding to the unique identifier does not need to be re-created. If the unique identifier does not have an associated valid time window or the associated time window has ended at this time, a new time window needs to be created and associated with the unique identifier.

[0040] The embodiment of the present application has the following technical effects: when the user behavior data and the exposure data are received in real time, the user behavior data is preprocessed into a user behavior object, the display content in the exposure data is preprocessed into an exposure data object, and the feature data of the user behavior object, the exposure data object, and the display content in the exposure data are respectively stored in association with the unique identifier. At the end of the specified time window, the user behavior object, the exposure data object, and the feature data corresponding to the same user and the same display content in the time window are spliced into sample data. In this process, the user behavior data and the display content in the exposure data are received and processed in real time, achieving real-time. At the same time, since the feature data usually has a large scale, the feature data is stored separately from the user behavior object and the exposure data object, so that the storage resources can be allocated more reasonably, and the consumption of the memory is avoided. By setting the time window, the feature data is read out at the end of the time window and spliced with the user behavior object and the exposure data object to obtain the sample data. It is ensured that the high throughput and low delay characteristics are still maintained in the large-scale feature scenario.

[0041] Further, the user behavior object and the exposure data object are stored in association with the unique identifier in the memory.

[0042] The feature data of the display content is stored in association with the unique identifier in the external storage, and the external storage includes a solid state disk SSD.

[0043] In some embodiments, the scale of the user behavior data is usually small, and in order to ensure the real-time processing, the user behavior object is saved in the memory of the computing device. Each received exposure data includes the display content corresponding to the exposure data and the feature data of the display content. The feature data of the display content is usually large-scale data. The display content is encapsulated into an exposure data object and saved in the memory. Since the feature data of the display content contained in the exposure data has a large scale and is only needed when splicing the sample, the feature data is saved to the external storage to save the memory. The memory is used to temporarily store the operation data in the computing device, and the program of the computing device is executed in the memory, and the memory has the characteristics of fast reading and writing speed; the external storage refers to a storage device other than the memory and CPU cache of the computing device, and such storage device can still save data after power off; common external storage includes hard disk, floppy disk, optical disk, U disk, and SSD (Solid State Drives), and the external storage has the characteristics of large capacity; the SSD has the characteristics of large capacity and fast reading and writing speed. Preferably, the exposure data is read in real time, the feature data of the display content in the exposure data is stored in the KV database in the SSD, and the rest of the data is encapsulated into an exposure data object according to the exposure data filtering and encapsulation rule and identified by the sid. Figure 2As shown, the data of the sid is located in the storage position of the KV database through Sidhash (i.e. hashing the sid), and the access of the feature data according to the sid is realized.

[0044] The embodiment of the present application has the following technical effects: The user behavior object and the exposure data object are stored in the memory for real-time processing, and the feature data of the display content in the exposure data is stored in the external storage. The characteristics of relatively small data volume of the user behavior object and the exposure data object and the characteristics of relatively large data volume of the feature data are fully utilized, the user behavior object and the exposure data object are placed in the memory to provide better real-time performance, and the feature data is placed in the external storage to reduce the consumption of memory resources, so as to balance the real-time performance and resource consumption, further ensure high throughput and low delay characteristics in a large-scale feature scenario, balance the overall performance of the system in real-time performance and stability, and reserve available resources for future function expansion. Further, the SSD is used as the external storage, effectively solving the problem of huge resource consumption of sample splicing in a large-scale feature scenario, reducing the cost of sample splicing, and having high expansibility and fault tolerance.

[0045] Further, the user behavior data is pre-processed into a user behavior object in real time according to the received user behavior data, including:

[0046] The user behavior data is received in real time through the Flink stream processing framework;

[0047] The current received user behavior data is parsed using the FlatMap in the Flink stream processing framework to obtain corresponding parsed user behavior data;

[0048] The parsed user behavior data is filtered using the Filter in the Flink stream processing framework to obtain filtered user behavior data;

[0049] The filtered user behavior data is encapsulated using the Map in the Flink stream processing framework to obtain the user behavior object.

[0050] In some embodiments, the user behavior data is read from the message queue, after data extraction, filtering and conversion, the data is packaged into a unified data format, and sent to the sample splicing unit. The filtering and packaging of the behavior data: the filtering includes filtering the abnormal analysis, the log containing illegal fields, and the non-related business scene, and the qualified data is packaged into a behavior object, and the unique identifier of the behavior object is identified by the user id (i.e. uid) and the display content id (i.e. cid). The implementation based on Flink mainly passes through three stages, and the operators used are FlatMap, Filter and Map. FlatMap parses the original behavior log, Filter filters abnormal data, and Map encapsulates the processed data into a behavior object.

[0051] The embodiment of the application has the following technical effects: based on the Flink streaming computing framework, the user behavior data is obtained and preprocessed to improve the real-time performance of sample generation.

[0052] Further, the method further comprises:

[0053] According to the preset sample filtering rule, all sample data is filtered, and target sample data meeting the sample filtering rule is distributed to a preset sample receiver;

[0054] Preferably, the target sample data is distributed to the preset sample receiver through different Flink connectors in the Flink streaming processing framework.

[0055] The sample receiver includes: a system for using the sample data in a real-time online manner and / or a system for using the sample data in an offline manner; preferably, the system for using the sample data in an offline manner is a system for using the sample data in an offline batch manner.

[0056] In some embodiments, after the sample splicing is completed, the samples that do not meet the preset sample filtering rule are filtered, and the filtering rule includes complete features, effective browsing time, etc. Then distributed to the downstream data system, including the message queue for online learning training, and the offline storage system for offline batch training. The downstream data application corresponding to the sample data generally has multiple types, and different data storage systems are required after splicing. For example Figure 3 As shown, the embodiment ensures that the data is output to the main stream data storage system through different Flink connectors in the Flink streaming processing framework, including the message queue and the distributed file system, to meet the real-time and offline data application, and support the extension of new data storage systems.

[0057] The embodiment of the present application has the following technical effects: providing sample data for real-time online learning and training in real time, so that the whole system can evolve by itself to improve the performance of the system; automatically generating offline training data that can be used for offline training without human intervention, so that the offline training data is available at any time and is automatically updated, improving the efficiency of offline training and the timeliness of the obtained model.

[0058] Further, the method further comprises:

[0059] Monitoring the running state parameters in the sample data generation process, and issuing an alarm when the running state parameters are abnormal.

[0060] In some embodiments, for a perfect system, it is indispensable to acquire and display the running state of the system in real time, and to issue an alarm in time when the system runs abnormally. As shown in the figure, the embodiment of the present application generates system module monitoring data through Fink Metric, and reports the monitoring to a database (such as graphite, influxdb, etc.) for storage through Metric Reporter, and displays, issues an alarm, etc. through Grafana configuration. Figure 4

[0061] The monitoring data specifically includes the data volume (QPS) of the data preprocessing module, the data delay (Delay), the data output volume (QPS) of the sample distribution module, the average value (Rate) of each spliced field, the read / write data volume (QPS) of the feature storage module, the request delay (Latency), etc. The embodiment also supports the extension of different monitoring items.

[0062] The embodiment of the present application has the following technical effects: by monitoring the running state of large-scale real-time sample splicing and issuing an alarm for the occurred abnormality, the stability of the sample splicing process is improved. Specifically, using the Metric component based on the Flink stream computing framework, various indicators of system running can be easily generated and reported, and an effective Metric system is provided to ensure the normal operation of the system.

[0063] The second aspect, as shown in the figure, the embodiment of the present application provides a sample data generation device, comprising: Figure 5

[0064] The behavior data acquisition unit 500 is configured to, for the received user behavior data, pre-process the user behavior data into a user behavior object in real time, and store the user behavior object in association with a unique identifier corresponding to the user behavior data;

[0065] ​​The exposure data acquisition unit 501 is configured to, for the received exposure data including the display content and the feature data thereof, pre-process the display content in the exposure data into an exposure data object, and store the exposure data object in association with a unique identifier corresponding to the exposure data; and store the feature data of the display content in the exposure data in association with the unique identifier corresponding to the exposure data.

[0066] The sample splicing unit 502 is configured to, at the end of each time window corresponding to the unique identifier, acquire all the user behavior objects, exposure data objects and feature data associated with the unique identifier in the time window, and splice all the acquired user behavior objects, exposure data objects and feature data into sample data corresponding to the unique identifier associated with the time window.

[0067] The time window is a window with a preset time length, and the earliest reception time in the user behavior data and the exposure data corresponding to the unique identifier is the starting point. The unique identifiers corresponding to the user behavior data and the exposure data generated by the same user on the same display content are the same.

[0068] The unique identifier is used to identify the user behavior data and the exposure data generated by the same user on the same display content.

[0069] Preferably, the unique identifier is determined by the information of the user and the information of the display content displayed to the user, and the unique identifiers obtained by the same user and the same display content are the same.

[0070] Further, the apparatus further comprises:

[0071] The time window validity judgment unit is used to, when the user behavior data or the exposure data is received, judge whether the unique identifier corresponding to the user behavior data or the exposure data is associated with a valid time window.

[0072] The time window creation unit is configured to, in the case that the unique identifier is not associated with a time window or the time window associated with the unique identifier has ended, establish the time window with the reception time in the user behavior data or the exposure data corresponding to the unique identifier as the starting point, and associate the established time window with the unique identifier.

[0073] Further, the behavior data acquisition unit 500 is configured to store the user behavior objects and the unique identifiers, and the exposure data objects and the unique identifiers in the memory in association.

[0074] The exposure data acquisition unit 501 is configured to store feature data of the display content in association with the unique identifier in an external memory, and the external memory includes a solid state drive (SSD).

[0075] Further, the behavior data acquisition unit 500 includes:

[0076] A behavior data receiving module is configured to receive the user behavior data in real time through a Flink stream processing framework.

[0077] A behavior data parsing module is configured to parse the user behavior data received currently to obtain corresponding parsed user behavior data by using a FlatMap in the Flink stream processing framework.

[0078] A behavior data filtering module is configured to filter the parsed user behavior data to obtain filtered user behavior data by using a Filter in the Flink stream processing framework.

[0079] A behavior object generating module is configured to encapsulate the filtered user behavior data to obtain the user behavior object by using a Map in the Flink stream processing framework.

[0080] Further, the apparatus further includes:

[0081] A sample data distribution unit is configured to filter all sample data according to a preset sample filtering rule, and distribute target sample data meeting the sample filtering rule to a preset sample receiver.

[0082] Further, the apparatus further includes:

[0083] A monitoring unit is configured to monitor a running state parameter in a sample data generation process, and issue an alarm when the running state parameter is abnormal.

[0084] The above technical solution has the following beneficial effects: when the user behavior data and the exposure data are received in real time, the user behavior data is preprocessed into a user behavior object, the display content in the exposure data is preprocessed into an exposure data object, and the feature data of the user behavior object, the exposure data object, and the display content in the exposure data are respectively stored in association with a unique identifier. At the end of a specified time window, the user behavior object, the exposure data object, and the feature data corresponding to the same user and the same display content in the time window are spliced into sample data. In this process, the user behavior data and the display content in the exposure data are received and processed in real time, achieving real-time. At the same time, since the feature data usually has a large scale, the feature data is stored separately from the user behavior object and the exposure data object, so that the storage resources can be allocated more reasonably, and the consumption of the memory is avoided. By setting the time window, the feature data is read out at the end of the time window and spliced with the user behavior object and the exposure data object to obtain the sample data. The high throughput and low delay characteristics are ensured in the large-scale feature scenario. Further, the user behavior object and the exposure data object are stored in the memory for real-time processing, and the feature data of the display content in the exposure data is stored in the external storage. The characteristics that the data amount of the user behavior object and the exposure data object is relatively small and the data amount of the feature data is relatively large are fully utilized. The user behavior object and the exposure data object are placed in the memory to provide better real-time performance, and the feature data is placed in the external storage to reduce the consumption of the memory resources, so as to balance the real-time performance and the resource consumption, further ensure the high throughput and low delay characteristics in the large-scale feature scenario, balance the real-time performance and stability of the system, and reserve available resources for future functional expansion.

[0085] Further, the user behavior data is acquired based on the Flink streaming computing framework, and preprocessed to improve the real-time performance of sample generation. Further, the SSD is used as the external storage, which effectively solves the problem of huge resource consumption in sample splicing in the large-scale feature scenario, reduces the sample splicing cost, and has high scalability and fault tolerance. Further, the running state of large-scale real-time sample splicing is monitored, and an alarm is given for the occurred exception, so as to improve the stability of the sample splicing process. Specifically, the Metric component based on the Flink streaming computing framework can easily generate and report various indicators of system running, and provides an effective Metric system to ensure normal system running.

[0086] The above technical solution of the embodiments of the present application will be described in detail below in combination with specific application examples. Technical details not introduced in the implementation process can be referred to the related description in the foregoing.

[0087] The embodiment of the application provides a sample data generation method, which is also a large-scale real-time sample splicing method and a large-scale real-time sample splicing method based on Flink stream calculation technology and SSD. First, unlike traditional stream calculation technology (such as Storm, Spark-streaming), the Flink stream data processing framework improves the large-scale sample generation efficiency, guarantees the real-time performance of large-scale sample splicing, and has the characteristics of high throughput and low delay in the case of large data scale. Secondly, in order to save a large amount of state in the sample splicing process, an external storage database based on SSD is introduced to ensure the consistency and stability of the state, and the storage based on SSD can save a large amount of memory consumption and reduce the sample splicing cost.

[0088] In one aspect, a large-scale real-time sample splicing method is provided, and the method comprises the following steps.

[0089] The data processing method refers to different behavior log data extraction, filtering, conversion, encapsulation and other processing rules and logic.

[0090] The real-time sample splicing method refers to assembling behavior data and exposure feature data according to a unique identifier, and then completing sample splicing after sample data processing.

[0091] In another aspect, a large-scale real-time sample splicing device is provided, and the device mainly comprises the following five modules.

[0092] The real-time data processing module extracts, filters, converts and encapsulates the original data into corresponding data models according to different behavior log data processing rules.

[0093] The real-time sample splicing module assembles the data processing model and the data in the state storage module according to a unique identifier, filters out samples that do not meet the rules according to sample filtering logic, and completes sample splicing.

[0094] The sample state storage module stores state data in a splicing time window.

[0095] The real-time sample distribution module distributes the spliced samples, which can be sent to a real-time message queue for real-time training on one hand, and can be sent to an offline data storage system for offline training on the other hand.

[0096] The monitoring and alarm module monitors the sample splicing process from multiple dimensions and alarms when the process is abnormal.

[0097] Multiple (for example, dozens) of microblogs are issued to a user each time, each user has a user id, and each microblog (equivalent to display content) has a content id.

[0098] A sid (unique identifier) is generated by combining each user id and the content id corresponding to each microblog, and is used as a unique identifier of the corresponding relationship between the user and the microblog; the behavior data (i.e., user behavior data) and the exposure data are generated on the basis of the interaction between the user and the microblog, so the same sid is used to mark the behavior data and the exposure data of the same user for the same microblog; when the time window is established, each sid corresponds to its own time window, that is, the behavior data and the exposure data of different users or different microblogs are sent to the time window corresponding to each sid for processing, respectively, and the data of different sids will not be mixed.

[0099] The main processes in the recommendation system include: material, recall, sorting, display. Among them, the recall and sorting stages are essentially to score and sort the candidate content set through the model, and the model needs to be trained with uninterrupted input samples, so a large-scale real-time sample generation system needs to be constructed.

[0100] As shown in Figure 6 The sample splicing schematic diagram provided by the embodiment of the application, sample generation mainly includes five modules: real-time data preprocessing module, real-time sample splicing module, sample state storage module, real-time sample distribution module, monitoring alarm module.

[0101] Step S1: Obtain user behavior data in a recommendation scenario and perform preprocessing.

[0102] S11: The real-time data processing module reads user behavior data from the message queue, and after data extraction, filtering and conversion, encapsulates the data into a unified data format and sends it to the real-time sample splicing module. The filtering and encapsulation of behavior data: the filtering includes filtering of abnormal analysis, non-legal fields, and non-related business scenarios. Qualified data is encapsulated into a behavior object, and the unique identifier of the behavior object is identified by the user id (i.e., uid) and the content id (i.e., cid). Based on Flink, the implementation mainly goes through three stages, and the operators used are FlatMap, Filter, and Map. FlatMap parses the original behavior log, Filter filters abnormal data, and Map encapsulates the processed data into a behavior object.

[0103] S12: Real-time read exposure data, send the features in the exposure data to the sample state storage module for storage, and encapsulate the remaining data and send it to the real-time sample splicing module. The exposure data filtering and encapsulation rules encapsulate the exposure data object, which is uniquely identified by the sid, and store the features in the exposure data in the KV database.

[0104] Step S2: Read the encapsulated behavior object and exposure object, read the features in the KV database, and splice the sample.

[0105] S21: Real-time sample splicing module, according to different data sources and arrival time, taking the earliest arriving data as the starting time, waiting for a specific time window, the length of the time window is N minutes, N is a pre-set positive integer, the data arriving within the time window is spliced according to the unique identifier sid.

[0106] S22: After the time window ends, query the stored sample features (i.e. the feature data of the display content in the exposure data) from the sample state storage module, assemble the sample features according to the unique identifier sid, and then send them to the sample distribution module.

[0107] Real-time sample distribution module, after the sample splicing is completed, filter the samples that do not meet the sample rules, including complete features, valid browsing time, etc. Then distribute to the downstream data system, including the message queue for online learning training and the offline storage system for offline batch training.

[0108] Step S3: After the sample splicing is completed, filter the abnormal samples and distribute them to the downstream.

[0109] As shown in Figure 7 , a sample splicing flowchart is shown.

[0110] Interactive behavior includes user forwarding, commenting, and liking behavior on content; click behavior includes various click behavior of users on content, such as clicking on large images and personal homepages; browsing time includes the sum of browsing time of users on content in different views; feedback behavior includes negative feedback behavior of users on blog posts, etc.

[0111] Exposure data refers to a batch of content data issued to users by the recommendation server when the user refreshes the client. In the recommendation system, each data contains features related to the current content.

[0112] Splicing process: the arrival time of various behavior data at the splicing module is different, taking the earliest arrival time of each data at the splicing module as the starting splicing time, registering a timer for each sid, within the splicing time window, splicing according to the sid for each new behavior object or exposure object, after the timing window ends, judging whether each sample contains behavior data and exposure behavior data, if it does, reading the feature data and splicing it to the sample and then outputting.

[0113] The sample splicing process modularizes the sample splicing process, which uses Flink as a data processing framework for sample splicing to provide many operator APIs.

[0114] As shown in Figure 2 , a sample state storage module diagram is shown.

[0115] The external storage uses an SSD-based KV database such as RocksDB, Pika, etc. to extract feature data in exposure data and write the feature data to the external storage with sid as the key. After the sample splicing window ends, the feature data is read according to the sid and added to the sample. The external KV storage uses a sharding scheme, and the feature read and write need to be hashed according to the sid to obtain the storage address of the read and write data.

[0116] As shown in Figure 3 , a schematic diagram of a real-time sample distribution module is shown.

[0117] The downstream data application corresponding to the sample generally has multiple types, and different data storage systems are needed after splicing. This module ensures that the data is output to the mainstream data storage system through different Flink connectors, including message queues and distributed file systems, to meet real-time and offline data applications, and also supports the extension of new data storage systems.

[0118] As shown in Figure 4 , a schematic diagram of a monitoring alarm module is shown.

[0119] For a perfect system, it is an indispensable function to acquire and display the running state of the system in real time and to issue an alarm in time when the system runs abnormally. In this method, the monitoring data of each module of the system is generated through Fink Metric, and the monitoring is reported to the database (such as Graphite, InfluxDB, etc.) for storage through Metric Reporter, and Grafana is configured to display and alarm abnormally.

[0120] The monitoring data specifically includes the data volume (QPS) and data delay (Delay) of the data preprocessing module, the data output volume (QPS) of the sample distribution module, the average value (Rate) of each spliced field, the read and write data volume (QPS) of the feature storage module, and the request delay (Latency). This method also supports the extension of different monitoring items.

[0121] The embodiment of the present application has the following technical effects: the real-time sample generation method based on the Flink streaming calculation framework can improve the real-time performance of sample generation, and ensure that the system still has high throughput and low delay characteristics in a large-scale feature scenario. The use of KV external storage based on SSD effectively solves the problem of huge resource consumption in sample splicing in a large-scale feature scenario, reduces the sample splicing cost, and has high scalability and fault tolerance. The use of the Metric component based on the Flink streaming calculation framework can easily generate and report various indicators of system operation, and provides an effective Metric system to ensure normal system operation. The use of the Flink streaming calculation framework to integrate the entire sample splicing process involves Flink data processing, filtering rules, and the combination of Flink operators, providing a complete sample splicing idea and architecture. The use of the KV external storage feature based on SSD writes before sample splicing and retrieves after sample splicing. This technical solution enables the computing resources to focus on the splicing process, greatly saves sample splicing resources, and reduces sample splicing cost. For the entire sample splicing system, a complete evaluation and monitoring alarm system is provided.

[0122] It should be understood that the particular order or hierarchy of steps in the processes disclosed is an example. Based upon design preferences, it should be understood that the particular order or hierarchy of steps in the processes can be rearranged while remaining within the scope of the present disclosure. The accompanying method claims present elements of the various steps in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

[0123] In the above detailed description, various features are grouped together in single embodiments for the purpose of streamlining the disclosure. Such disclosed methods should not be interpreted as reflecting an intention that the claimed embodiments require more features than are explicitly recited in each claim. On the contrary, as indicated above, the inventiveness lies in less, the claimed subject matter than in the full scale of features disclosed. Therefore, the claims are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate preferred embodiment.

[0124] The disclosed embodiments are to enable any person skilled in the art to achieve or use the present application. Various modifications of the embodiments are obvious to those skilled in the art, and the general principles defined herein can also be applied to other embodiments without departing from the spirit and protection scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in the present application.

[0125] The above description includes examples of one or more embodiments. Of course, not all possible combinations of components or methods described above can be implemented to make an embodiment, but one of ordinary skill in the art will recognize that many such combinations are possible. Accordingly, the embodiments described herein are intended to embrace all such alterations, modifications and variations which fall within the scope of the appended claims. Further, in the description of the embodiments above, relative terms are used to describe the relationships between elements, including the use of the terms "apron", "including", and "including". Additionally, the use of the term "or" in any claim is intended to be used in the inclusive sense, and not in the exclusive sense.

[0126] Those of skill would further appreciate that the various illustrative logical blocks, modules, and steps described in connection with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present embodiments.

[0127] The various illustrative logical blocks, modules, and steps described in connection with the embodiments disclosed herein can be implemented or performed by a general purpose processor, a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the general purpose processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other such configuration.

[0128] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0129] In one or more exemplary designs, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or data

[0130] The above detailed description describes the purpose, technical solutions and advantages of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the present application.

Claims

1. A method of generating sample data, characterized by, The method comprises: real-time pre-processing of the received user behavior data into user behavior objects, and storing the user behavior objects in association with the unique identifier corresponding to the user behavior data in the memory; for the received exposure data including the display content and its feature data, pre-processing the display content in the exposure data into exposure data objects, and storing the exposure data objects in association with the unique identifier corresponding to the exposure data in the memory; and storing the feature data of the display content in the exposure data in association with the unique identifier corresponding to the exposure data in the external storage; at the end of each unique identifier corresponding time window, obtaining all user behavior objects, exposure data objects and feature data associated with the unique identifier within the time window, and splicing all obtained user behavior objects, exposure data objects and feature data into sample data corresponding to the unique identifier associated with the time window; wherein the time window is a window of a preset time length starting from the earliest received time in the unique identifier corresponding user behavior data and exposure data, and the unique identifiers corresponding to each user behavior data and each exposure data generated by the same user on the same display content are the same.

2. The sample data generating method of claim 1, wherein, The method further comprises: when receiving user behavior data or exposure data, determining whether the unique identifier corresponding to the user behavior data or exposure data is associated with a valid time window; in the case that the unique identifier is not associated with a time window or the time window associated with the unique identifier has ended, establishing the time window starting from the receiving time in the unique identifier corresponding user behavior data or exposure data currently received, and associating the established time window with the unique identifier.

3. The sample data generating method of claim 1, wherein, The external storage comprises a solid state drive (SSD).

4. The sample data generating method of claim 1, wherein, The real-time pre-processing of the received user behavior data into user behavior objects comprises: receiving the user behavior data in real time through a Flink stream processing framework; using a FlatMap in the Flink stream processing framework to analyze the currently received user behavior data to obtain corresponding analyzed user behavior data; using a Filter in the Flink stream processing framework to filter the analyzed user behavior data to obtain filtered user behavior data; using a Map in the Flink stream processing framework to encapsulate the filtered user behavior data to obtain the user behavior objects.

5. The sample data generating method of claim 1, wherein, The method further comprises: filtering all sample data according to a preset sample filtering rule, and distributing target sample data meeting the sample filtering rule to a preset sample receiver.

6. The sample data generating method of claim 1, wherein, The method further comprises: monitoring running state parameters in the sample data generation process, and issuing an alarm when the running state parameters are abnormal.

7. A sample data generating apparatus characterized by comprising: The method comprises: a behavior data obtaining unit configured to real-time pre-process the received user behavior data into user behavior objects, and store the user behavior objects in association with the unique identifier corresponding to the user behavior data in the memory; The exposure data acquisition unit is configured to, for the received exposure data including the display content and the feature data thereof, preprocess the display content in the exposure data into an exposure data object, and store the exposure data object in the memory in association with a unique identifier corresponding to the exposure data; and store the feature data of the display content in the exposure data in the external storage in association with the unique identifier corresponding to the exposure data. The sample splicing unit is configured to, at the end of each time window corresponding to the unique identifier, acquire all the user behavior objects, exposure data objects and feature data associated with the unique identifier in the time window, and splice all the acquired user behavior objects, exposure data objects and feature data into sample data corresponding to the unique identifier associated with the time window. The time window is a window of a preset time length with the earliest reception time in the user behavior data and the exposure data corresponding to the unique identifier as the starting point, and the unique identifiers corresponding to the user behavior data and the exposure data generated by the same user on the same display content are the same.

8. The sample data generating apparatus of claim 7, wherein The device further comprises: The time window validity judgment unit is configured to, when the user behavior data or the exposure data is received, judge whether the unique identifier corresponding to the user behavior data or the exposure data is associated with a valid time window. The time window creation unit is configured to, in the case that the unique identifier is not associated with a time window or the time window associated with the unique identifier has ended, establish the time window with the reception time in the user behavior data or the exposure data corresponding to the unique identifier as the starting point, and associate the established time window with the unique identifier.

9. The sample data generating apparatus of claim 7, wherein The external storage comprises a solid state disk (SSD).

10. The sample data generating apparatus of claim 7, wherein The behavior data acquisition unit comprises: The behavior data reception module is configured to receive the user behavior data in real time through a Flink stream processing framework. The behavior data analysis module is configured to use a FlatMap in the Flink stream processing framework to analyze the currently received user behavior data to obtain corresponding analyzed user behavior data. The behavior data filtering module is configured to use a Filter in the Flink stream processing framework to filter the analyzed user behavior data to obtain filtered user behavior data. The behavior object generation module is configured to use a Map in the Flink stream processing framework to encapsulate the filtered user behavior data to obtain the user behavior object.

Images