Link log sampling method and device, equipment and medium

Abstract

The invention discloses a link log sampling method and device, equipment and a medium. The method comprises the steps of obtaining a target log set, wherein a target log comprises at least one burying point and original business data; processing each target log according to each burying point in the target log to obtain a target sequence set, the target sequence set comprising at least one target sequence and a sequence type matched with the target sequence; processing each target sequence in the target sequence set based on a pre-trained sequence pattern mining algorithm to obtain at least one service sequence type; and obtaining a sampling log set, and performing screening operation on each sampling log in the sampling log set according to each service sequence type to obtain a result log set. Through the technical scheme of the invention, sampling of the link logs can be realized, the efficiency of log acquisition work is improved, and the comprehensiveness, accuracy and reliability of an acquisition result are improved at the same time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of log analysis, and in particular to a link log sampling method, device, equipment and medium. BACKGROUND

[0002] In a distributed system architecture, various business requests initiated by users will trigger a series of cross-service cooperative processing procedures, forming a complete business tracking link. The business tracking chain is composed of multiple internal operation units of services, which include function calls, database access, cache interaction, cross-service communication and other specific execution links, which jointly support the complete implementation of a single business request.

[0003] In order to monitor the system running state, troubleshoot the fault source, and optimize the business process, it is crucial to collect relevant log data of such business tracking links. However, in a large-scale distributed system, the daily business request volume can reach tens of millions or even hundreds of millions, and each tracking link contains a large number of operation units, and the operation units of some complex business links can even reach dozens. If all link logs are collected in full, it will generate a large amount of data, not only occupying a large amount of storage resources and network bandwidth, but also significantly increasing the cost of data transmission, processing and analysis, exceeding the carrying capacity of the system, so it is necessary to use sampling method to select part of the log data for collection. The commonly used sampling scheme in the prior art is simple fixed proportion sampling, which can quickly reduce the data volume and simplify the implementation process, but lacks pertinence and flexibility. In actual business scenarios, some key business scenarios may have low-probability abnormal situations, such as occasional call timeouts, data validation failures, resource preemption conflicts, etc. Such abnormalities have a significant impact on system stability and user experience, but they are easily missed due to the randomness of fixed proportion sampling. Such low-probability abnormalities often hide deep problems such as system architecture defects and resource scheduling bottlenecks, and if they are missed, it will prevent technical personnel from discovering the root cause of the abnormality in a timely manner, not only hindering fault troubleshooting, but also allowing potential risks to continue to accumulate, ultimately affecting business continuity, and cannot meet the precision and reliability requirements of distributed system link log collection.

[0004] In summary, the existing link log sampling method has the problems of low efficiency of log collection work, and poor comprehensiveness, accuracy and reliability of the collection results. SUMMARY

[0005] The present application provides a link log sampling method, device, equipment and medium, which can solve the problems of low efficiency of log collection work, and poor comprehensiveness, accuracy and reliability of the collection results of the existing link log sampling method.

[0006] In a first aspect, an embodiment of the present application provides a link log sampling method, which comprises:

[0007] obtain a target log set, the target log containing at least one buried point and original business data;

[0008] process each target log according to each buried point in the target log to obtain a target sequence set, the target sequence set containing at least one target sequence and a sequence category matched with the target sequence;

[0009] process each target sequence in the target sequence set based on a pre-trained sequence pattern mining algorithm to obtain at least one business sequence category;

[0010] obtain a sampling log set, and perform a filtering operation on each sampling log in the sampling log set according to each business sequence category to obtain a result log set.

[0011] In a second aspect, an embodiment of the present application provides a sampling device for link logs, and the device comprises:

[0012] a target log set obtaining module, configured to obtain a target log set, the target log containing at least one buried point and original business data;

[0013] a target sequence set obtaining module, configured to process each target log according to each buried point in the target log to obtain a target sequence set, the target sequence set containing at least one target sequence and a sequence category matched with the target sequence;

[0014] a business sequence category determining module, configured to process each target sequence in the target sequence set based on a pre-trained sequence pattern mining algorithm to obtain at least one business sequence category;

[0015] a log sampling module, configured to obtain a sampling log set, and perform a filtering operation on each sampling log in the sampling log set according to each business sequence category to obtain a result log set.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, and the electronic device comprises:

[0017] at least one processor; and

[0018] a memory in communication with the at least one processor; wherein

[0019] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform a sampling method for link logs according to any one of the embodiments of the present application.

[0020] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a processor execute a link log sampling method according to any one of the embodiments of the present application.

[0021] The technical scheme of the embodiment of the present application, by obtaining a target log set, the target log containing at least one buried point and original business data, then processing each target log according to each buried point in the target log to obtain a target sequence set, then processing each target sequence in the target sequence set based on a pre-trained sequence pattern mining algorithm to obtain at least one business sequence category, finally obtaining a sampling log set, and performing a screening operation on each sampling log in the sampling log set according to each business sequence category to obtain a result log set, solves the problem that the existing link log sampling method has low efficiency of log collection work, and the comprehensiveness, accuracy and reliability of the collection result are poor, realizes the sampling of the link log, improves the efficiency of the log collection work, and improves the comprehensiveness, accuracy and reliability of the collection result.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1 is a flowchart of a link log sampling method according to an embodiment of the present application;

[0025] Figure 2 is a flowchart of a link log sampling method according to an embodiment of the present application;

[0026] Figure 3 is a structural schematic diagram of a link log sampling device according to an embodiment of the present application;

[0027] Figure 4 is a structural schematic diagram of an electronic device for implementing a link log sampling method according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, any variations of the terms "comprising" and "having" are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a link log sampling method provided in Embodiment 1 of the present invention. This embodiment is applicable to the sampling of link logs. The method can be executed by a link log sampling device, which can be implemented in hardware and / or software. The link log sampling device can be configured in a terminal or server with link log sampling function.

[0032] like Figure 1 As shown, the method includes:

[0033] S110, Obtain the target log set.

[0034] The target log contains at least one tracking point and raw business data.

[0035] Specifically, the target log set is the link log generated during the execution of user requests in the distributed system, and each user request corresponds to a complete link record that constitutes the target log.

[0036] Furthermore, the target log consists of multiple spans, each span being the smallest recording unit for specific business operations such as internal service function calls, database accesses, and API calls. The tracking points in the target log are monitoring markers preset by the user at key nodes of each span, used to accurately capture the execution trajectory and status data of the business process. Furthermore, the original business data covers basic data directly related to business operations, such as user ID, order details, product information, payment methods, inventory status, and discount rules. These data are correlated with the tracking point information, ensuring that the target log set can comprehensively and completely reflect the entire business scenario from user request initiation to completion, providing reliable data support for subsequent sequence analysis.

[0037] S120. Process each target log according to each tracking point in the target log to obtain the target sequence set.

[0038] The target sequence set includes at least one target sequence and sequence types that match the target sequence.

[0039] In this embodiment, the target sequence is a DSS group, which refers to a business logic unit composed of multiple consecutive and strongly related tracking points and their corresponding spans. Each DSS group corresponds to a specific business process, such as a user verification process, an inventory operation process, a payment process, etc., which can accurately map the key links in the actual business scenario.

[0040] The process involves processing each target log based on its tracking points to obtain a target sequence set. This includes: obtaining a pre-set tracking point type table; dividing each target log into at least one target sequence based on the tracking points and their distribution locations; identifying the tracking point type of each tracking point according to the tracking point type table, and labeling the sequence type of each target sequence according to the tracking point type; and integrating each target sequence and its sequence type to obtain the target sequence set.

[0041] Specifically, the event tracking type table is preset by the user based on the system architecture, business scenarios and historical operational data. It includes information such as the unique identifier of the event tracking, the associated Span type and the type of event tracking, such as user verification event tracking, inventory operation event tracking, payment event tracking and other categories.

[0042] S130. Based on a pre-trained sequence pattern mining algorithm, each target sequence in the target sequence set is processed to obtain at least one business sequence type.

[0043] The process involves processing each target sequence in the target sequence set using a pre-trained sequence pattern mining algorithm to obtain at least one business sequence type. This includes: counting the occurrence frequency of target sequences of each sequence type in the target sequence set using the sequence pattern mining algorithm to obtain the frequency statistics of each sequence type; calculating the sequence type ratio of each sequence type based on the frequency statistics and the total number of target logs in the target log set; obtaining a preset ratio range; and determining the sequence types whose ratios fall within the ratio range as business sequence types.

[0044] S140. Obtain the sampled log set, and filter each sampled log in the sampled log set according to the type of each business sequence to obtain the result log set.

[0045] The sampling log set refers to a collection of link logs covering various business sequences obtained from the target log set through user-preset preliminary collection rules. Each sampling log is associated with a corresponding log type and contains complete tracking information and original business data, which can reflect the execution status of specific business processes.

[0046] The process includes: acquiring a sampled log set and filtering the sampled logs in the sampled log set according to each business sequence type to obtain a result log set; acquiring a preset global total sampling rate and calculating the total number of samples based on the global total sampling rate and the total number of sampled logs in the sampled log set; acquiring the sequence type ratio corresponding to each business sequence type and allocating the total number of samples to each business sequence type according to the sequence type ratio to obtain the number of business samples that match each business sequence type; and filtering the sampled logs in the sampled log set based on the total number of samples, each business sequence type, and each business sample number to obtain the result log set.

[0047] Furthermore, the global total sampling rate refers to the overall sampling ratio threshold preset in the log collection scenario to balance system storage pressure, data processing efficiency, and business analysis needs. Its setting is usually determined by the user in combination with factors such as the average daily log volume of the distributed system, the server storage limit, and the accuracy requirements of data analysis. It is the core parameter of the log sampling strategy, and this embodiment does not limit its specific quantity. Furthermore, the total sampling quantity is calculated based on the global total sampling rate and the total number of logs sampled in the sampling log set, that is, the total sampling quantity = total number of sampled logs × global total sampling rate. The scale of the log samples to be finally screened is determined by quantitative indicators.

[0048] Optionally, after filtering the sampled logs in the sampled log set according to each business sequence type to obtain the result log set, the method further includes: determining whether the proportion of each sequence type in the target sequence set is lower than the lower limit of a preset proportion range, and marking the sequence types with proportions lower than the lower limit as long-tail business types; obtaining a preset minimum sampling quantity, and filtering the sampled logs in the sampled log set based on the long-tail business types and the minimum sampling quantity to obtain logs of each long-tail type; merging the logs of each long-tail type with the result log set to obtain an updated result log set.

[0049] Furthermore, after determining whether the proportion of each sequence type in the target sequence set is lower than the lower limit of a preset proportion range, and marking the sequence types with proportions lower than the lower limit as long-tail business types, the process further includes: obtaining the tracking point type table and the sequence types of each target sequence in the target sequence set; determining whether there are any missing sequence types in the tracking point type table but not matching any business sequence type based on the sequence types in the target sequence set and the tracking point type table; if so, setting the missing sequence type as a long-tail business type.

[0050] Specifically, first, the sequence types of each target sequence in the target sequence set and the tracking point type table are obtained. Then, the tracking point type table is compared with each sequence type in the target sequence set to determine whether there are any missing sequence types in the tracking point type table but not matching any business sequence type. These sequences may not have been included in the target sequence set because the business scenario was not fully triggered or the data statistics were biased. If they exist, the missing sequence types are set as long-tail business types, and a minimum sampling strategy is performed on them to further improve the sampling coverage and avoid blind spots in business monitoring due to sequence omissions.

[0051] In a specific implementation scenario of this embodiment, for example, assuming a total of 100,000 sampled logs and a preset global sampling rate of 5%, the total number of samples = 100,000 × 5% = 5,000. The types and proportions of each business sequence are as follows: core type 40%, regular type 55%, and long-tail type 5%. The allocation of business sampling quantities is: core type 5,000 × 40% = 2,000 logs, regular type 5,000 × 55% = 2,750 logs, and long-tail type 5,000 × 5% = 250 logs. After filtering according to the corresponding quantities, the resulting log set is obtained. Subsequent checks revealed that the proportion of long-tail type (5%) is higher than the preset lower limit of 3%, and there are no missing sequences in the tracking type table. The final result log set contains 5,000 logs, covering various business paths.

[0052] The technical solution of this invention involves acquiring a target log set, which includes at least one data point and original business data. Then, each target log is processed based on its data point to obtain a target sequence set. Next, a pre-trained sequence pattern mining algorithm is used to process each target sequence in the target sequence set to obtain at least one business sequence type. Finally, a sampled log set is acquired, and each sampled log in the sampled log set is filtered according to its business sequence type to obtain a result log set. This solution addresses the problems of low efficiency and poor comprehensiveness, accuracy, and reliability of existing link log sampling methods. It achieves link log sampling, improves the efficiency of log collection, and enhances the comprehensiveness, accuracy, and reliability of the collection results.

[0053] Example 2

[0054] Figure 2 This is a flowchart of a link log sampling method provided in Embodiment 2 of the present invention. This embodiment is a refinement based on the above embodiment. Specifically, this embodiment refines the method of processing each target sequence in the target sequence set based on a pre-trained sequence pattern mining algorithm to obtain at least one type of business sequence.

[0055] like Figure 2 As shown, the method includes:

[0056] S210, Obtain the target log set.

[0057] The target log contains at least one tracking point and raw business data.

[0058] S220. Process each target log according to each tracking point in the target log to obtain the target sequence set.

[0059] The target sequence set includes at least one target sequence and sequence types that match the target sequence.

[0060] S230. The occurrence frequency of target sequences of each sequence type in the target sequence set is counted by using a sequence pattern mining algorithm, and the frequency statistics of each sequence type are obtained.

[0061] The sequence pattern mining algorithm is a data analysis algorithm used to discover frequently occurring ordered sequences in ordered datasets. Its core function is to accurately identify recurring sequence patterns from massive target sequences and eliminate random and isolated sequence interference. In specific execution, the target sequence set is first initially classified according to sequence type to ensure that sequences of the same category have the same business attributes and functional orientation. Then, the sequence pattern mining algorithm traverses the entire set of target sequences, counting the valid target sequences under each sequence type. During the counting process, incomplete sequences caused by abnormal data collection or business interruption are filtered out. Only logically coherent target sequences that conform to the business process are counted for their actual frequency of occurrence. Finally, the frequency statistics for each sequence type are generated, providing accurate data support for subsequent ratio calculations.

[0062] S240. Calculate the sequence type ratio of each sequence type based on the statistical results of each occurrence and the total number of target logs in the target log set.

[0063] Specifically, the calculation method for the sequence type ratio is as follows: Sequence type ratio of a certain sequence type = Number of occurrences of the sequence type ÷ Total number of target logs × 100%. This formula can quantify the distribution ratio of each sequence type in the total number of target logs. For example, if a certain sequence type appears 2000 times and the total number of target logs is 100,000, then its sequence type ratio is 2%, which clearly reflects the frequency of occurrence of each type of sequence.

[0064] S250. Obtain a preset ratio range, and determine the sequence types whose ratio falls within the ratio range as business sequence types.

[0065] The upper limit of the preset ratio range is less than 100%.

[0066] In this embodiment, the upper limit of the interval being less than 100% is used to exclude basic core sequences (such as system initialization-related sequences) that appear 100% in all target logs. These sequences do not need to be separately classified as business sequence types and can be included in the scope of global basic monitoring. By determining the sequence types whose proportion falls within this interval as business sequence types, it can be ensured that subsequent sampling strategies focus on sequences with actual business significance and reasonable frequency of occurrence, thereby improving sampling efficiency and the relevance of log analysis.

[0067] S260. Obtain the sampled log set, and filter each sampled log in the sampled log set according to the type of each business sequence to obtain the result log set.

[0068] The technical solution of this invention involves acquiring a target log set, which contains at least one data point and original business data. Then, each target log is processed based on its data points to obtain a target sequence set. A sequence pattern mining algorithm is used to count the occurrence frequency of target sequences of each sequence type in the target sequence set, resulting in a frequency statistics result for each sequence type. The proportion of each sequence type is calculated based on the frequency statistics and the total number of target logs in the target log set. A preset proportion range is then obtained, and sequence types whose proportions fall within this range are identified as business sequence types. Finally, a sampled log set is acquired, and each sampled log in the sampled log set is filtered according to its business sequence type to obtain a result log set. This solution addresses the problems of low efficiency and poor comprehensiveness, accuracy, and reliability of existing link log sampling methods. It achieves link log sampling, improving the efficiency of log collection while enhancing the comprehensiveness, accuracy, and reliability of the collection results.

[0069] Example 3

[0070] Figure 3 This is a schematic diagram of a link log sampling device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes:

[0071] The target log set acquisition module 310 is used to acquire a target log set, wherein the target logs contain at least one tracking point and raw business data.

[0072] The target sequence set acquisition module 320 is used to process each target log according to each embedding point in the target log to obtain a target sequence set, wherein the target sequence set includes at least one target sequence and sequence types that match the target sequence;

[0073] The business sequence type determination module 330 is used to process each target sequence in the target sequence set based on a pre-trained sequence pattern mining algorithm to obtain at least one business sequence type.

[0074] The log sampling module 340 is used to obtain a sampled log set and filter each sampled log in the sampled log set according to the type of each business sequence to obtain a result log set.

[0075] The technical solution of this invention involves acquiring a target log set, which includes at least one data point and original business data. Then, each target log is processed based on its data point to obtain a target sequence set. Next, a pre-trained sequence pattern mining algorithm is used to process each target sequence in the target sequence set to obtain at least one business sequence type. Finally, a sampled log set is acquired, and each sampled log in the sampled log set is filtered according to its business sequence type to obtain a result log set. This solution addresses the problems of low efficiency and poor comprehensiveness, accuracy, and reliability of existing link log sampling methods. It achieves link log sampling, improves the efficiency of log collection, and enhances the comprehensiveness, accuracy, and reliability of the collection results.

[0076] Based on the above embodiments, the target sequence set acquisition module 320 includes:

[0077] The event tracking unit is used to retrieve a pre-set event tracking type table;

[0078] The log splitting unit is used to split each target log into at least one target sequence based on the distribution location of each tracking point in the target log.

[0079] The category identification unit is used to identify the category of each tracking point according to the tracking point type table, and to label the sequence category of each target sequence according to the tracking point category of each tracking point.

[0080] The sequence integration unit is used to integrate the target sequences and the sequence types of each target sequence to obtain a target sequence set.

[0081] Based on the above embodiments, the business sequence type determination module 330 includes:

[0082] The frequency counting unit is used to count the occurrence frequency of target sequences of each sequence type in the target sequence set using a sequence pattern mining algorithm, and to obtain the frequency statistics results for each sequence type.

[0083] The proportion calculation unit is used to calculate the proportion of each sequence type based on the statistical results of each occurrence and the total number of target logs in the target log set;

[0084] An interval acquisition unit is used to acquire a preset ratio interval and determine the sequence types whose sequence type ratio falls within the ratio interval as business sequence types.

[0085] Based on the above embodiments, the log sampling module 340 includes:

[0086] The sampling quantity statistics unit is used to obtain the preset global total sampling rate and calculate the total number of samples based on the global total sampling rate and the total number of sampled logs in the sampling log set.

[0087] The sequence allocation unit is used to obtain the sequence type ratio corresponding to each service sequence type, and allocate the total sampling quantity to each service sequence type according to the sequence type ratio, so as to obtain the service sampling quantity that matches each service sequence type respectively.

[0088] The log filtering unit is used to filter each sampled log in the sampled log set based on the total number of samples, the type of each service sequence, and the number of samples for each service, to obtain a result log set.

[0089] Based on the above embodiments, the log sampling module 340 is further configured to: perform filtering operations on each sampled log in the sampled log set according to each business sequence type, and after obtaining the result log set, determine whether the proportion of each sequence type in the target sequence set is lower than the lower limit of a preset proportion range, and mark the sequence types with proportions lower than the lower limit of the range as long-tail business types; obtain a preset minimum sampling quantity, and filter each long-tail type log in the sampled log set based on the long-tail business types and the minimum sampling quantity; merge each long-tail type log with the result log set to obtain an updated result log set.

[0090] Based on the above embodiments, the log sampling module 340 is further configured to: determine whether the proportion of each sequence type in the target sequence set is lower than the lower limit of a preset proportion range, and mark the sequence types with proportions lower than the lower limit as long-tail business types; then obtain the tracking point type table and the sequence types of each target sequence in the target sequence set; determine whether there are any missing sequence types in the tracking point type table but not matching any business sequence type based on the sequence types in the target sequence set and the tracking point type table; if so, set the missing sequence type as a long-tail business type.

[0091] The link log sampling device provided in this embodiment of the invention can execute the link log sampling method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0092] Example 4

[0093] Figure 4A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0094] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0095] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0096] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a link log sampling method.

[0097] Accordingly, the method includes:

[0098] Obtain the target log set, which contains at least one tracking point and raw business data;

[0099] Each target log is processed based on the tracking points in the target log to obtain a target sequence set, which includes at least one target sequence and sequence types that match the target sequence.

[0100] The pre-trained sequence pattern mining algorithm processes each target sequence in the target sequence set to obtain at least one business sequence type.

[0101] Obtain the sampled log set, and filter each sampled log in the sampled log set according to the type of each business sequence to obtain the result log set.

[0102] In some embodiments, a link log sampling method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the link log sampling method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a link log sampling method by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0108] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0109] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

Claims

1. A method for sampling link logs, characterized in that, include: Obtain the target log set, which contains at least one tracking point and raw business data; Each target log is processed based on the tracking points in the target log to obtain a target sequence set, which includes at least one target sequence and sequence types that match the target sequence. The pre-trained sequence pattern mining algorithm processes each target sequence in the target sequence set to obtain at least one business sequence type. Obtain the sampled log set, and filter each sampled log in the sampled log set according to the type of each business sequence to obtain the result log set.

2. The method according to claim 1, characterized in that, Based on the tracking points in the target logs, each target log is processed to obtain a target sequence set, including: Retrieve the pre-set data point type table; Based on each tracking point in the target log, each target log is split into at least one target sequence according to the distribution location of each tracking point. Identify the type of each embedding point according to the embedding point type table, and label the sequence type of each target sequence according to the embedding point type. The target sequences and their sequence types are integrated to obtain the target sequence set.

3. The method according to claim 1, characterized in that, A pre-trained sequence pattern mining algorithm is used to process each target sequence in the target sequence set to obtain at least one business sequence type, including: The occurrence frequency of target sequences of each sequence type in the target sequence set is counted by using a sequence pattern mining algorithm, and the frequency statistics of each sequence type are obtained. The proportion of each sequence type is calculated based on the statistical results of each occurrence and the total number of target logs in the target log set. Obtain a preset ratio range, and determine the sequence types whose ratio falls within the ratio range as business sequence types.

4. The method according to any one of claims 1-3, characterized in that, Obtain the sampled log set, and filter each sampled log in the sampled log set according to the type of each business sequence to obtain the result log set, including: Obtain the preset global total sampling rate, and calculate the total number of samples based on the global total sampling rate and the total number of sampled logs in the sampling log set; Obtain the sequence type ratio corresponding to each service sequence type, and allocate the total sampling quantity to each service sequence type according to the sequence type ratio to obtain the service sampling quantity that matches each service sequence type respectively; Based on the total number of samples, the types of each service sequence, and the number of samples for each service, the sampled logs in the sampled log set are filtered to obtain the result log set.

5. The method according to any one of claims 1-3, characterized in that, After filtering the sampled logs in the sampled log set according to the type of each business sequence, the resulting log set also includes: Determine whether the proportion of each sequence type in the target sequence set is lower than the lower limit of the preset proportion range, and mark the sequence types with a proportion lower than the lower limit of the range as long-tail business types; Obtain a preset minimum sampling quantity, and filter the sampling log set based on the long-tail business type and the minimum sampling quantity to obtain logs of each long-tail type; The logs of each long-tail category are merged with the result log set to obtain the updated result log set.

6. The method according to claim 5, characterized in that, After determining whether the proportion of each sequence type in the target sequence set is lower than the lower limit of a preset proportion range, and marking the sequence types with proportions lower than the lower limit as long-tail business types, the process also includes: Obtain the tracking point type table and the sequence type of each target sequence in the target sequence set; Based on the sequence types in the target sequence set and the aforementioned tracking point type table, determine whether there are any missing sequence types that are in the tracking point type table but do not match any business sequence type. If it exists, the missing sequence type is set as a long-tail business type.

7. The method according to claim 3, characterized in that, The upper limit of the preset ratio range is less than 100%.

8. A sampling device for link logs, characterized in that, include: The target log set acquisition module is used to acquire the target log set, which contains at least one tracking point and raw business data; The target sequence set acquisition module is used to process each target log according to each tracking point in the target log to obtain a target sequence set, wherein the target sequence set includes at least one target sequence and sequence types that match the target sequence; The business sequence type determination module is used to process each target sequence in the target sequence set based on a pre-trained sequence pattern mining algorithm to obtain at least one business sequence type. The log sampling module is used to obtain a sampled log set and filter each sampled log in the sampled log set according to the type of each business sequence to obtain the result log set.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a link log sampling method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute a link log sampling method according to any one of claims 1-7.

Images