Log data processing method, device, storage medium and electronic device

Abstract

The present disclosure provides a log data processing method, device, storage medium and electronic device, which relate to the field of data processing technology. The log data processing method includes: dividing the log data into one or more primary data segments according to the event identifier in the log data to be processed; dividing the primary data segments into one or more secondary data segments; determining the bitmap of the secondary data segments according to the interval to which the event identifier in the secondary data segments belongs, so as to obtain the index information of the secondary data segments; each bit in the bitmap corresponds to an interval. The present disclosure reduces the storage cost of log data to a certain extent.

Description

Background Art

[0002] By analyzing the log data of user behavior types such as click log data and browsing log data, we can understand the user's preferences and provide personalized services to the user. In order to facilitate the analysis and utilization of these log data, it is inevitable to store these log data.

[0003] In the related art, when storing log data of user behavior types, the log data is usually cached in Redis memory rather than on a disk. Since the memory itself has a small storage capacity and high cost, the storage cost of the log data is high.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a log data processing method, a log data processing device, a computer-readable storage medium and an electronic device, thereby solving the problem of high storage cost of log data in the related art at least to a certain extent.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to the first aspect of the present disclosure, a log data processing method is provided, including: dividing the log data into one or more primary data segments according to event identifiers in the log data to be processed; dividing the primary data segments into one or more secondary data segments; determining a bitmap of the secondary data segments according to the interval to which the event identifiers in the secondary data segments belong, so as to obtain index information of the secondary data segments; each bit in the bitmap corresponds to an interval.

[0008] In an exemplary embodiment of the present disclosure, the method further includes: determining a plurality of the intervals according to a maximum value and a minimum value of the event identifier.

[0009] In an exemplary embodiment of the present disclosure, the determining the multiple intervals according to the maximum value and the minimum value of the event identifier includes: determining the multiple intervals according to the maximum value and the minimum value of the event identifier in the primary data segment.

[0010] In an exemplary embodiment of the present disclosure, when the data type of the event identifier is a string type, the method further includes: sorting the event identifiers with high bits first, and determining the maximum and minimum values ​​of the event identifiers according to the sorting result.

[0011] In an exemplary embodiment of the present disclosure, the method further includes: adding the maximum value and the minimum value of the event identifier to the index information of the secondary data segment.

[0012] In an exemplary embodiment of the present disclosure, determining the bitmap of the secondary data segment according to the interval to which the event identifier in the secondary data segment belongs includes: determining whether the number of event identifiers in the secondary data segment that fall into each interval is 0; when the number of event identifiers that fall into the interval is 0, setting the bit value corresponding to the interval in the bitmap to 0; when the number of event identifiers that fall into the interval is not 0, setting the bit value corresponding to the interval in the bitmap to 1.

[0013] In an exemplary embodiment of the present disclosure, determining the bitmap of the secondary data fragment according to the interval to which the event identifier in the secondary data fragment belongs includes: replacing the event identifier in the secondary data fragment with a mapping identifier; and determining the bitmap of the secondary data fragment according to the interval to which the mapping identifier belongs.

[0014] In an exemplary embodiment of the present disclosure, replacing the event identifier in the secondary data segment with a mapping identifier includes: when the event identifier in the secondary data segment meets a preset distribution condition, determining a discrete event identifier in the event identifier; and replacing the discrete event identifier with a mapping identifier.

[0015] In an exemplary embodiment of the present disclosure, the method further includes: storing a corresponding relationship between the event identifier and the mapping identifier.

[0016] In an exemplary embodiment of the present disclosure, the method also includes: when the amount of log data in any secondary data fragment is greater than a first preset threshold, splitting any secondary data fragment into at least two new secondary data fragments; determining the bitmap of the new secondary data fragment to obtain index information of the new secondary data fragment.

[0017] In an exemplary embodiment of the present disclosure, the method further includes: after splitting any of the secondary data fragments, updating version information in the primary data fragment to which any of the secondary data fragments belongs, so as to associate the version information with the new secondary data fragment obtained after the splitting.

[0018] In an exemplary embodiment of the present disclosure, the method also includes: obtaining an event identifier to be queried; determining candidate secondary data fragments among the secondary data fragments based on the event identifier to be queried and index information of each secondary data fragment; and searching for log data corresponding to the event identifier to be queried in the candidate secondary data fragments.

[0019] In an exemplary embodiment of the present disclosure, the candidate secondary data fragments are determined in each secondary data fragment based on the event identifier to be queried and the index information of each secondary data fragment, including: respectively judging whether the bit value corresponding to the target interval in the index information of each secondary data fragment is 1, and the target interval is the interval to which the event identifier to be queried belongs; when the bit value corresponding to the target interval in the index information is 1, determining that the secondary data fragment corresponding to the index information is a candidate secondary data fragment.

[0020] According to a second aspect of the present disclosure, a log data processing device is provided, including: a first partitioning module, used to divide the log data into one or more primary data segments according to event identifiers in the log data to be processed; a second partitioning module, used to divide the primary data segments into one or more secondary data segments; a bitmap determination module, used to determine the bitmap of the secondary data segments according to the interval to which the event identifier in the secondary data segments belongs, so as to obtain index information of the secondary data segments; each bit in the bitmap corresponds to an interval.

[0021] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned log data processing method is implemented.

[0022] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-mentioned log data processing method by executing the executable instructions.

[0023] The technical solution disclosed in this disclosure has the following beneficial effects:

[0024] In the above-mentioned log data processing process, the log data is divided into one or more primary data segments according to the event identifier in the log data to be processed; the primary data segment is divided into one or more secondary data segments; the bitmap of the secondary data segment is determined according to the interval to which the event identifier in the secondary data segment belongs, so as to obtain the index information of the secondary data segment; each bit in the bitmap corresponds to an interval. On the one hand, by dividing the log data to be processed, the distributed storage of the log data can be realized so that the storage load can be shared by multiple storage servers. On the other hand, by constructing an index using the interval to which the event identifier in the data segment belongs, the search efficiency of the log data can be ensured even if the log data is stored in a low-cost disk, which can reduce the storage cost while ensuring the search efficiency.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A flow chart showing a log data processing method in this exemplary embodiment;

[0028] Figure 2 An example diagram showing a data structure of a primary data segment in this exemplary embodiment;

[0029] Figure 3 A flowchart of determining a bitmap of a secondary data segment in this exemplary embodiment is shown;

[0030] Figure 4 A flowchart showing a method of replacing an event identifier with a mapping identifier in this exemplary embodiment is shown;

[0031] Figure 5 A flow chart showing a secondary data segment segmentation in this exemplary embodiment is shown;

[0032] Figure 6 A flowchart of searching log data in this exemplary embodiment is shown;

[0033] Figure 7 A flowchart for determining a candidate secondary data segment in this exemplary embodiment is shown;

[0034] Figure 8 A flowchart of querying click log data in this exemplary embodiment is shown;

[0035] Fig. 9 A flowchart of click log data processing in this exemplary embodiment is shown;

[0036] Fig.10 A structural block diagram of a log data processing device in this exemplary embodiment is shown;

[0037] Fig.11 An electronic device for implementing the above method in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0038] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0039] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0040] Herein, “first”, “second”, etc. are labels for specific objects, but do not limit the number or order of the objects.

[0041] In the related art, when storing log data of user behavior types, the log data is usually cached in a Redis memory. Redis is an in-memory database that stores data in memory rather than on disk. As the amount of log data continues to increase, the Redis memory occupied is also increasing. Since the memory itself has a small storage capacity and high cost, the storage cost of log data is high.

[0042] In view of one or more of the above problems, an exemplary embodiment of the present disclosure provides a log data processing method.

[0043] Figure 1 The schematic flow of the log data processing method in this exemplary embodiment is shown, including the following steps S110 to S130:

[0044] Step S110, dividing the log data into one or more primary data segments according to the event identifier in the log data to be processed;

[0045] Step S120, dividing the primary data segment into one or more secondary data segments;

[0046] Step S130, determining a bitmap of the secondary data segment according to the interval to which the event identifier in the secondary data segment belongs, so as to obtain index information of the secondary data segment; each bit in the bitmap corresponds to an interval.

[0047] In the above-mentioned log data processing process, the log data is divided into one or more primary data segments according to the event identifier in the log data to be processed; the primary data segment is divided into one or more secondary data segments; the bitmap of the secondary data segment is determined according to the interval to which the event identifier in the secondary data segment belongs, so as to obtain the index information of the secondary data segment; each bit in the bitmap corresponds to an interval. On the one hand, by dividing the log data to be processed, the distributed storage of the log data can be realized so that the storage load can be shared by multiple storage servers. On the other hand, by constructing an index using the interval to which the event identifier in the data segment belongs, the search efficiency of the log data can be ensured even if the log data is stored in a low-cost disk, which can reduce the storage cost while ensuring the search efficiency.

[0048] The following are Figure 1 Each step in the procedure is described in detail.

[0049] Step S110 , dividing the log data into one or more primary data segments according to the event identifier in the log data to be processed.

[0050] The log data to be processed can be log data of user behavior types such as click log data and browsing log data. The log data can include data fields of dimensions such as event identifier, URL, and IP. Event identifier refers to the identifier used to distinguish each event in the log data. It can be a numeric type or a string type and is unique, such as an event ID. The first-level data segment is a data segment composed of log data, which is obtained by dividing the log data to be processed.

[0051] In an optional implementation, dividing the log data into one or more primary data segments can be achieved by: obtaining a hash value of an event identifier in the log data to be processed; and dividing the log data into one or more data segments according to the hash value of the event identifier.

[0052] For example, multiple hash value ranges may be pre-set to divide log data whose hash values ​​of event identifiers fall within the same hash value range into a first-level data segment.

[0053] Dividing log data into multiple first-level data segments facilitates distributed storage of log data. Each first-level data segment can be stored on a different storage server, and the storage load can be shared by multiple storage servers.

[0054] In an optional implementation, the index information of the primary data segment may also be determined according to the features of the event identifier in the primary data segment.

[0055] The feature of the event identifier in the first-level data segment can be the hash value range interval to which the hash value of the event identifier belongs. By recording the correspondence between each first-level data segment and the hash value range interval, the correspondence is used as the index information of the first-level data segment, so that the first-level data segment to which the log data belongs can be determined by the hash value of the event identifier of the log data.

[0056] Step S120, dividing the first-level data segment into one or more second-level data segments.

[0057] The second-level data segments are data segments composed of log data, which are obtained by dividing the first-level data segments. Figure 2 The example diagram of the data structure of the first-level data segment is shown, and the first-level data segment is divided into n second-level data segments.

[0058] It should be noted that, in actual implementation, the data segments may be divided more finely, for example, the secondary data segments may be further divided into smaller data segments for storage.

[0059] In an optional implementation, multiple intervals may be determined according to the maximum and minimum values ​​of the event identifier.

[0060] The maximum and minimum values ​​of the above event identifiers can be the maximum and minimum values ​​of the event identifiers of the log data to be processed, or the maximum and minimum values ​​of the event identifiers in the first-level data segment. When determining the interval by the maximum and minimum values, the interval formed by the maximum and minimum values ​​can be divided equally, or the interval formed by the maximum and minimum values ​​can be divided unequally according to the distribution of the event identifiers, and the range with sparse event identifier distribution can be divided into larger intervals. For example, the event identifiers of 5 log data are 1, 2, 9, 10, and 100 respectively, and the range interval formed by the maximum and minimum values ​​of the event identifiers is [1,100]. This interval can be divided into [1,5], [6,10], and [11,100].

[0061] The data segments can be divided by the divided intervals, and the log data with event identifiers falling in the same interval can be divided into one data segment.

[0062] In an optional embodiment, when the maximum and minimum values ​​of the event identifier are the maximum and minimum values ​​of the event identifier in the first-level data segment, multiple intervals are determined based on the maximum and minimum values ​​of the event identifier. This can be achieved in the following way: multiple intervals are determined based on the maximum and minimum values ​​of the event identifier in the first-level data segment.

[0063] By determining multiple intervals according to the maximum and minimum values ​​of the event identifiers in the primary data segments, the log data in which the event identifiers in the primary data segments fall within the same interval can be divided into one secondary data segment. In the above process, the range interval formed by the maximum and minimum values ​​of the event identifiers in the primary data segments is divided into multiple intervals, so that the primary data segments are divided into one or more secondary data segments according to the divided intervals.

[0064] In an optional implementation, when the data type of the event identifier is a string type, the method further includes: sorting the event identifiers with high bits first, and determining the maximum and minimum values ​​of the event identifiers according to the sorting result.

[0065] High-order first sorting is a string sorting method that can use the first and last event identifiers after sorting as the minimum and maximum values, respectively. For example, the event identifiers of 5 log data are c, a, b, abc, and ab, and after high-order first sorting, they are a, abc, ab, b, and c. A can be used as the minimum value of the event identifier and c as the maximum value of the event identifier. When the event identifier is a string type, by determining the minimum and maximum values ​​of the event identifier, the data type of the event identifier is not limited to the numeric type, and can be applied in a wider range.

[0066] Step S130, determining a bitmap of the secondary data segment according to the interval to which the event identifier in the secondary data segment belongs, so as to obtain index information of the secondary data segment; each bit in the bitmap corresponds to an interval.

[0067] The above bitmap is implemented using a binary array. Each bit of the array is used to store a state. It is suitable for large-scale data, but the data state is not very large. It can be used to determine whether a certain data exists. Each bit in the bitmap corresponds to an interval, and the bit value of each bit in the bitmap indicates whether there is data in the interval. When the length of the bitmap is 5 bits, it can represent the distribution state of 2 to the power of 5 event identifiers. The index information of the secondary segment is composed of the interval to which the event identifier in the secondary data segment belongs and the bitmap.

[0068] In an optional implementation, the maximum value and the minimum value of the event identifier may be added to the index information of the secondary data segment.

[0069] The maximum and minimum values ​​of the event identifier in the secondary data segment are added to the index information of the secondary data segment. There is no need to store the index of each log data in the secondary data segment. Adding the maximum and minimum values ​​of the event identifier to the index information of the secondary data segment can reduce the size of the index file and facilitate loading.

[0070] In an optional implementation, multiple intervals can be further determined based on the maximum and minimum values ​​of the event identifier in the secondary segment, or multiple intervals can be further determined based on the maximum and minimum values ​​of the event identifier in the primary segment corresponding to the secondary segment, so as to determine the interval to which the event identifier in the secondary data segment belongs from the divided intervals. The interval can be formed by equally dividing the range interval formed by the maximum and minimum values ​​of the event identifier in the secondary segment; or it can be formed by unequally dividing the interval formed by the maximum and minimum values ​​of the event identifier in the secondary segment according to the distribution of the event identifier. For example, there are 5 log data in the secondary data segment, and the corresponding event identifiers are 1, 2, 9, 10, and 100, respectively. The range interval formed by the maximum and minimum values ​​of the event identifier is [1,100], and the interval can be divided into [1,5], [6,10], and [11,100]. By dividing the interval, a reference interval can be provided for determining the interval to which the event identifier in the secondary data segment belongs.

[0071] In an optional implementation, the bitmap of the secondary data segment is determined according to the interval to which the event identifier in the secondary data segment belongs. This can be achieved in the following way: determine whether the number of event identifiers in the secondary data segment that fall into each interval is 0; when the number of event identifiers that fall into the interval is 0, set the corresponding bit value of the interval in the bitmap to 0; when the number of event identifiers that fall into the interval is not 0, set the corresponding bit value of the interval in the bitmap to 1.

[0072] The bit value corresponding to the interval in the bitmap indicates whether the event identifier in the secondary data segment falls into the interval. A bit value of 0 indicates that no event identifier in the secondary data segment falls into the interval, and a bit value of 1 indicates that an event identifier in the secondary data segment falls into the interval. By setting the bit value, the event identifier can be indexed, which can effectively filter out irrelevant secondary data segments and improve search efficiency.

[0073] like Figure 2 As shown, each secondary data segment includes index information composed of a corresponding bitmap and an index, and the index information of the secondary data segment is stored in the secondary data segment. The index information structure of the secondary data segment can be shown in Table 1, wherein the position identifier of the bitmap indicates the identifier of each bit in the bitmap, the numerical range indicates the divided interval, a numerical range indicates an interval, the bit value indicates the bit value corresponding to each bit in the bitmap, the bit value of 0 indicates that there is no event identifier in the secondary data segment that falls into the corresponding data value range, and the bit value of 1 indicates that there is an event identifier in the secondary data segment that falls into the corresponding data value range.

[0074] Table 1

[0075] Bitmap location marker Data value range Place value 0 1-40 0 1 41-80 0 2 81-200 1 3 201-400 1 4 401-8000 0

[0076] In an optional implementation, the bitmap of the secondary data segment is determined according to the interval to which the event identifier in the secondary data segment belongs, which can be done as follows: Figure 3 The steps S310 to S320 shown in the figure are implemented, which specifically include the following steps:

[0077] Step S310: Replace the event identifier in the secondary data segment with the mapping identifier.

[0078] The mapping identifier is an identifier corresponding to a certain event identifier and can be used to replace the corresponding event identifier. The mapping identifier has the same data type as the event identifier.

[0079] Step S320: Determine the bitmap of the secondary data segment according to the interval to which the mapping identifier belongs.

[0080] The bitmap of the secondary data segment may be determined according to the interval to which the replaced mapping identifier in the secondary data segment belongs.

[0081] Figure 3 In the process shown, the event identifiers that need to be replaced can be determined based on the event identifier distribution. By replacing the event identifiers, the identifier distribution can be optimized, thereby improving the search efficiency of log data.

[0082] In an optional implementation, the event identifier in the secondary data segment is replaced with a mapping identifier, which can be achieved by Figure 4 The steps S410 to S420 shown in the figure are implemented, which specifically include the following steps:

[0083] Step S410: when the event identifiers in the secondary data segment meet a preset distribution condition, a discrete event identifier is determined in the event identifiers.

[0084] The preset distribution condition may be an event identifier discrete distribution condition preset by a developer based on experience. When the distribution of an event identifier satisfies the preset distribution condition, the event identifier is determined as a discrete event identifier.

[0085] Step S420: Replace the discrete event identifier with a mapping identifier.

[0086] For example, when the event identifiers in the secondary data segment are unevenly distributed, for example, most of the event identifiers are distributed in the range of 1-1000, and there is an event identifier of 9999, 9999 can be replaced with an identifier close to the range of 1-1000.

[0087] Figure 4 In the steps shown, by setting a preset distribution condition, the discretely distributed event identifiers in the secondary data segment can be determined. By replacing the discretely distributed event identifiers, the span of the event identifier range can be reduced, further improving the search efficiency of the log data.

[0088] In an optional implementation, a correspondence between event identifiers and mapping identifiers is stored.

[0089] The correspondence between the event identifier and the mapping identifier can be stored in the corresponding primary data segment, for example, the correspondence between the event identifier and the mapping identifier can be stored in Figure 2 In the metadata list of the primary data segment shown, the self-descriptiveness of a single primary data segment is ensured for easy search. In addition, the corresponding relationship of the mapping identifier of the event identifier can also be stored in the corresponding secondary data segment to ensure the self-descriptiveness of a single secondary data segment for easy search.

[0090] In an optional implementation, the data field in the log data may also be replaced, specifically including the following steps: when the frequency of occurrence of the data field in the log data is greater than a second preset threshold, the data field is replaced with a mapping field, and the correspondence between the data field and the mapping field is stored.

[0091] The correspondence between the stored data fields and the mapped fields can be stored in Figure 2 In the metadata of the primary data segment shown, one primary data segment adopts a set of replacement relationships; the correspondence between the stored data fields and the mapped fields may also be stored in the metadata of the secondary data segment, and one secondary data segment adopts a set of replacement relationships.

[0092] It should be noted that the data volume of the mapping field is smaller than that of the data field. By replacing data fields with larger data volume and higher frequency, the storage volume of log data can be effectively reduced.

[0093] In an optional embodiment, the Figure 5 Steps S510 to S520 shown in the figure implement segmentation of the secondary data segments to avoid excessive data volume of the secondary data segments, and specifically include the following steps:

[0094] Step S510: when the amount of log data in any secondary data segment is greater than a first preset threshold, split any secondary data segment into at least two new secondary data segments.

[0095] When splitting the secondary data fragment, the interval formed by the maximum and minimum values ​​corresponding to the event identifier in the secondary data fragment can be split into two intervals, and the original secondary data fragment can be split into two new secondary data fragments according to the intervals to which each event identifier in the original secondary data fragment belongs, and the original secondary data fragment is deleted; it can also be split according to the number of log data in the secondary data fragment to ensure that the amount of log data in each new secondary data fragment is the same or differs by 1. For example, a secondary data fragment contains 21 log data, and the log data in the secondary data fragment is sorted according to the corresponding event identifier, and can be split forward and backward from the position of the log data sorted as the 11th.

[0096] Step S520: determine the bitmap of the new secondary data segment to obtain index information of the new secondary data segment.

[0097] Figure 5 In the steps shown, by splitting the secondary data fragments, it is avoided that the content of a single secondary data fragment is too large and thus affects the search efficiency, so as to ensure the efficiency of searching for log data in the secondary data fragments.

[0098] In an optional implementation, after any secondary data segment is segmented, version information is updated in the primary data segment to which any secondary data segment belongs, so as to associate the version information with the new secondary data segment obtained after the segmentation.

[0099] like Figure 2 As shown, version information is also stored in the primary data segment. Updating the version information helps to understand the segmentation of the secondary data segment, and can associate the new secondary data segment obtained after segmentation with the corresponding primary data segment to clarify the ownership of the secondary data segment.

[0100] In an optional embodiment, if Figure 6 As shown, the log data can be found through the following steps S610 to S630.

[0101] Step S610, obtaining the event identifier to be queried.

[0102] You can get messages about orders or traffic waiting to be queried from a media platform (such as an advertising platform) through a message queue, and then extract the event identifier to be queried from the message to be queried.

[0103] Step S620: determining candidate secondary data segments in each secondary data segment according to the event identifier to be queried and the index information of each secondary data segment.

[0104] When determining candidate secondary data segments, first determine the candidate primary data segments corresponding to the event identifier to be queried from the log data to be processed based on the event identifier to be queried and the index information of the primary data segments, and then determine the candidate secondary data segments from the secondary data segments contained in the candidate primary data segments based on the event identifier to be queried and the index information of each secondary data segment.

[0105] In an optional implementation, step S620 may be performed as follows: Figure 7 The steps shown determine the candidate secondary data segments, specifically including step S710 and step S720:

[0106] Step S710 , determining whether the bit value corresponding to the target interval in the index information of each secondary data segment is 1, the target interval being the interval to which the event identifier to be queried belongs.

[0107] Step S720: When the bit value corresponding to the target interval in the index information is 1, the secondary data segment corresponding to the index information is determined to be a candidate secondary data segment.

[0108] Figure 7 In the steps shown, first locate the interval to which the event identifier to be queried belongs, without traversing all log data, and quickly filter out secondary data fragments that are not related to the event identifier to be queried. When the bit value corresponding to the target interval in the index information is 0, it means that there is no log data corresponding to the event identifier to be processed in the candidate interval, and there is no need to search further downward, avoiding unnecessary query operations. When the bit value corresponding to the target interval in the index information is 1, it means that there is log data corresponding to the event identifier to be processed in the candidate interval, and further search is performed downward.

[0109] Step S630: Search the candidate secondary data segments for log data corresponding to the event identifier to be queried.

[0110] After the candidate secondary data segment is determined, it is necessary to accurately search the candidate secondary data segment to find the log data corresponding to the event identifier to be queried.

[0111] In an optional implementation, a reverse index may be created for the secondary data segments, so as to accurately search for log data corresponding to the event identifier to be queried from the candidate secondary data through the index information of the reverse index, thereby ensuring the accuracy of the query.

[0112] When the log data is click log data, Figure 8 An implementation method for querying click log data is provided, which specifically includes the following steps:

[0113] Step S801, start;

[0114] Step S802, extracting the click ID from the order and traffic information in the message queue and obtaining its value, where the extracted click ID is used as the event identifier to be queried;

[0115] Step S803, performing range value matching on the bitmap index bit of the click ID in the block, where the block is used as the secondary data segment, and matching the bitmap of each block as index information to determine the target block file to which the click ID belongs, that is, the candidate secondary data segment;

[0116] Step S804: Find a block that matches the click ID value and obtain a target block file. In this process, the target block file found after matching in step S803 is used as a candidate secondary data segment;

[0117] Step S805, performing a reverse index matching query on the target block file to obtain the corresponding log data. This process uses the reverse index to accurately search for the log information corresponding to the click ID from the target block file;

[0118] Step S806, end.

[0119] When the log data is click log data, Fig. 9 An implementation method for processing click log data is shown, which is applied to the OCPX (Optimized Cost Per X) promotion bidding scenario, where "X" refers to the traditional settlement mode in different ways, including two processing stages: an access stage and an output stage.

[0120] The access phase includes the following steps:

[0121] Step S901, returning the user click log, where the user click log is used as the log data to be processed, and the advertising media returns the user click log to the advertiser's log core processing logic;

[0122] Step S902, click data structure construction layer. After receiving the user click log sent back by the advertising media, the advertiser log core processing logic click data structure construction layer converts the user click log into log data with a specific structure, such as n primary data segments and m secondary data segments;

[0123] Step S903, log distributed storage, the log data with a specific structure formed in step S902 is distributedly stored, and n primary data segments can be stored in different storage servers, and m secondary data segments can be stored in different blocks of a storage server disk;

[0124] After the access phase is completed, the output phase begins. The output phase specifically includes the following steps:

[0125] Step S904, obtaining real-time information such as traffic and orders;

[0126] Step S905, in the stream processing layer Flink / Storm, stream processing query is performed on the real-time message. This process is to query the log data matching the real-time message from the log data stored in the above step S903. Flink and Storm are two open source stream processing frameworks, which process large data streams through distributed stream data flow engines.

[0127] Step S906, Sink output layer output. Sink is the output end of the stream processing layer. The Sink output layer outputs the data processed by Flink / Storm. The output can be written to files, sockets, external systems, etc. The advertiser log core processing logic will output the Sink output layer and then return it to the advertising media;

[0128] Step S907, OCPX conversion. OCPX is a tool that optimizes conversion costs and adjusts intelligent dynamic bidding based on the click-through rate and conversion rate of a single flow. The advertising media performs OCPX conversion based on the output of the Sink output layer to achieve dynamic bidding.

[0129] Fig. 9 The steps shown are based on the promotion bidding model of OCPX, and real-time query is performed on the massive click log data in distributed storage to achieve real-time and rapid hits on the massive click log data. Real-time dynamic bidding is achieved based on the click rate and conversion rate of the hit click log data, which can ensure the timeliness and efficiency of OCPX hit conversion.

[0130] The exemplary embodiment of the present disclosure also provides a log data processing device. Fig.10 As shown, the log data processing device 1000 includes:

[0131] A first division module 1010, configured to divide the log data into one or more primary data segments according to event identifiers in the log data to be processed;

[0132] A second division module 1020, configured to divide the primary data segment into one or more secondary data segments;

[0133] The bitmap determination module 1030 is used to determine the bitmap of the secondary data segment according to the interval to which the event identifier in the secondary data segment belongs, so as to obtain index information of the secondary data segment; each bit in the bitmap corresponds to an interval.

[0134] In an optional implementation, the log data processing device 1000 further includes: an interval determination module, configured to determine a plurality of intervals according to a maximum value and a minimum value of an event identifier.

[0135] In an optional implementation, the interval determination module is configured to determine multiple intervals according to the maximum and minimum values ​​of the event identifiers in the primary data segment.

[0136] In an optional implementation, when the data type of the event identifier is a string type, the interval determination module is further configured to: sort the event identifiers with high bits first, and determine the maximum and minimum values ​​of the event identifiers according to the sorting result.

[0137] In an optional implementation, the log data processing device 1000 further includes: an adding module, configured to add the maximum value and the minimum value of the event identifier to the index information of the secondary data segment.

[0138] In an optional embodiment, the bitmap determination module 1030 is configured to: determine whether the number of event identifiers in the secondary data segment that fall into each interval is 0; when the number of event identifiers that fall into the interval is 0, set the corresponding bit value of the interval in the bitmap to 0; when the number of event identifiers that fall into the interval is not 0, set the corresponding bit value of the interval in the bitmap to 1.

[0139] In an optional implementation, the bitmap determination module 1030 further includes: a replacement module, used to replace the event identifier in the secondary data segment with a mapping identifier; and a bitmap determination module, used to determine the bitmap of the secondary data segment according to the interval to which the mapping identifier belongs.

[0140] In an optional implementation, the replacement module is further configured to: when the event identifier in the secondary data segment meets a preset distribution condition, determine a discrete event identifier in the event identifier; and replace the discrete event identifier with a mapping identifier.

[0141] In an optional implementation, the log data processing device 1000 further includes: a storage module, configured to store a correspondence between an event identifier and a mapping identifier.

[0142] In an optional embodiment, the log data processing device 1000 also includes: a data segmentation module, used to split any secondary data segment into at least two new secondary data segments when the amount of log data in any secondary data segment is greater than a first preset threshold; a bitmap determination submodule, used to determine the bitmap of the new secondary data segment to obtain index information of the new secondary data segment.

[0143] In an optional embodiment, the log data processing device 1000 also includes: a version update module, which is used to update the version information in the primary data segment to which any secondary data segment belongs after any secondary data segment is segmented, so as to associate the version information with the new secondary data segment obtained after the segmentation.

[0144] In an optional embodiment, the log data processing device 1000 also includes: an identification acquisition module, used to obtain the event identification to be queried; a candidate fragment determination module, used to determine the candidate secondary data fragments in each secondary data fragment based on the event identification to be queried and the index information of each secondary data fragment; a search module, used to search for log data corresponding to the event identification to be queried in the candidate secondary data fragments.

[0145] In an optional embodiment, the candidate fragment determination module is also configured to: determine whether the bit value corresponding to the target interval in the index information of each secondary data fragment is 1, and the target interval is the interval to which the event identifier to be queried belongs; when the bit value corresponding to the target interval in the index information is 1, determine that the secondary data fragment corresponding to the index information is a candidate secondary data fragment.

[0146] The specific details of each part of the above-mentioned log data processing device 1000 have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0147] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the log data processing method described above in this specification is stored. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to execute the steps described in the above “exemplary method” section of this specification according to various exemplary embodiments of the present disclosure. The program product may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device.

[0148] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0149] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0150] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0151] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0152] The exemplary embodiment of the present disclosure also provides an electronic device capable of implementing the above-mentioned log data processing method. Fig.11 1100 according to this exemplary embodiment of the present disclosure will be described. Fig.11 The electronic device 1100 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0153] like Fig.11 As shown, the electronic device 1100 may be in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one storage unit 1120, a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110), and a display unit 1140.

[0154] The storage unit 1120 stores program codes, which can be executed by the processing unit 1110, so that the processing unit 1110 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1110 can execute Figure 1 , Figures 3 to 9 Any one or more method steps.

[0155] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1121 and / or a cache memory unit 1122 , and may further include a read-only memory unit (ROM) 1123 .

[0156] The storage unit 1120 may also include a program / utility 1124 having a set (at least one) of program modules 1125, such program modules 1125 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0157] Bus 1130 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0158] The electronic device 1100 may also communicate with one or more external devices 1200 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1150. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0159] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary implementation of the present disclosure.

[0160] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0161] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0162] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0163] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A log data processing method, characterized in that: include: Dividing the log data into one or more primary data segments according to event identifiers in the log data to be processed; Determine a plurality of intervals according to the maximum value and the minimum value of the event identifier, and divide the primary data segment into one or more secondary data segments according to the divided intervals; According to the interval to which the event identifier in the secondary data segment belongs, a bitmap of the secondary data segment is determined to obtain index information of the secondary data segment; each bit in the bitmap corresponds to an interval.

2. The method according to claim 1, characterized in that The determining of the plurality of intervals according to the maximum value and the minimum value of the event identifier comprises: A plurality of intervals are determined according to the maximum and minimum values ​​of the event identifiers in the primary data segment.

3. The method according to claim 1, characterized in that When the data type of the event identifier is a string type, the method further includes: The event identifiers are sorted with high order first, and the maximum and minimum values ​​of the event identifiers are determined according to the sorting result.

4. The method according to claim 1, characterized in that The method further comprises: The maximum value and the minimum value of the event identifier are added to the index information of the secondary data segment.

5. The method according to claim 1, characterized in that: The step of determining the bitmap of the secondary data segment according to the interval to which the event identifier in the secondary data segment belongs includes: Determine whether the number of event identifiers in the secondary data segment falling within each interval is 0; When the number of event identifiers falling within the interval is 0, the bit value corresponding to the interval in the bitmap is set to 0; When the number of event identifiers falling within the interval is not 0, the bit value corresponding to the interval in the bitmap is set to 1.

6. The method according to claim 1, characterized in that The step of determining the bitmap of the secondary data segment according to the interval to which the event identifier in the secondary data segment belongs includes: Replacing the event identifier in the secondary data segment with a mapping identifier; A bitmap of the secondary data segment is determined according to the interval to which the mapping identifier belongs.

7. The method according to claim 6, characterized in that The replacing the event identifier in the secondary data segment with a mapping identifier includes: When the event identifiers in the secondary data segment meet a preset distribution condition, determining discrete event identifiers in the event identifiers; The discrete event identifier is replaced with a mapping identifier.

8. The method according to claim 6, characterized in that The method further comprises: The correspondence between the event identifier and the mapping identifier is stored.

9. The method according to claim 1, characterized in that: The method further comprises: When the amount of log data in any secondary data segment is greater than a first preset threshold, splitting the any secondary data segment into at least two new secondary data segments; A bitmap of the new secondary data segment is determined to obtain index information of the new secondary data segment.

10. The method according to claim 9, characterized in that The method further comprises: After any of the secondary data segments are segmented, the version information is updated in the primary data segment to which the any of the secondary data segments belongs, so as to associate the version information with the new secondary data segment obtained after the segmentation.

11. The method according to claim 1, characterized in that The method further comprises: Get the event ID to be queried; Determine candidate secondary data segments from among the secondary data segments according to the event identifier to be queried and index information of each secondary data segment; Search the candidate secondary data segments for log data corresponding to the event identifier to be queried.

12. The method according to claim 11, characterized in that The determining of candidate secondary data segments from each secondary data segment according to the event identifier to be queried and the index information of each secondary data segment includes: Determine whether the bit value corresponding to the target interval in the index information of each secondary data segment is 1, the target interval being the interval to which the event identifier to be queried belongs; When the bit value corresponding to the target interval in the index information is 1, the secondary data segment corresponding to the index information is determined to be a candidate secondary data segment.

13. A log data processing device, characterized in that: include: A first division module, configured to divide the log data into one or more primary data segments according to event identifiers in the log data to be processed; A second division module, configured to determine a plurality of intervals according to a maximum value and a minimum value of the event identifier, and divide the primary data segment into one or more secondary data segments according to the divided intervals; The bitmap determination module is used to determine the bitmap of the secondary data segment according to the interval to which the event identifier in the secondary data segment belongs, so as to obtain the index information of the secondary data segment; each bit in the bitmap corresponds to an interval.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 12 by executing the executable instructions.

Images