Burial point sampling method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a burying point sampling method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a system congestion index according to capacity and load characteristics acquired in real time, and determining a sampling pressure index according to the capacity and load characteristics and the system congestion index; determining a sampling decision score according to the capacity and load characteristics, and inputting the capacity and load characteristics and the system congestion index into a sampling strategy model to obtain a dynamic sampling rate output by the sampling strategy model; and determining a target sampling strategy according to the sampling pressure index, the sampling decision score and the dynamic sampling rate, and carrying out buried point sampling according to the target sampling strategy. According to the method, the target sampling strategy is reasonably determined according to the real-time dynamic operation state of the electronic equipment and the service priority condition of each piece of service burying point data, accurate burying point data acquisition is performed according to the strategy, and a comprehensive and accurate data basis is provided for subsequent data analysis.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data technology, and in particular to a method, apparatus, electronic device and storage medium for embedding sampling points. Background Technology

[0002] In the field of big data tracking (data sampling), after the front-end or client collects various tracking data, it will send these data to the back-end electronic devices. The electronic devices will preprocess the collected data and then use it for various purposes.

[0003] Currently, to cope with the pressure of high-concurrency data collection and massive data storage, the sampling of the aforementioned data points can directly use fixed sampling methods such as random sampling and header sampling. However, during periods of low system sampling activity, using some sampling methods such as random sampling and header sampling will result in the discarding of a large amount of valuable data, thereby affecting the comprehensiveness and accuracy of subsequent data analysis. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for collecting embedded data, which enables the reasonable determination of a target sampling strategy based on the real-time dynamic operating status of the electronic device and the business priority of each business embedded data point, and the accurate collection of embedded data points according to the strategy.

[0005] In a first aspect, embodiments of the present invention provide a method for embedding sampling points, the method comprising:

[0006] The system congestion index is determined based on the real-time acquired capacity and load characteristics, and the sampling pressure index is determined based on the capacity and load characteristics and the system congestion index.

[0007] The sampling decision score is determined based on capacity and load characteristics, and the capacity and load characteristics and system congestion index are input into the sampling strategy model to obtain the dynamic sampling rate output by the sampling strategy model.

[0008] The target sampling strategy is determined based on the sampling pressure index, sampling decision score, and dynamic sampling rate, and sampling points are carried out according to the target sampling strategy.

[0009] The sampling method provided in this invention determines system congestion indicators by acquiring real-time capacity and load characteristics, enabling accurate determination of the real-time congestion status of electronic devices processing data. Furthermore, by utilizing system congestion indicators and capacity and load characteristics reflecting the real-time congestion status of electronic devices, a basis is provided for determining different sampling methods based on the real-time sampling pressure of the electronic devices. On one hand, based on system load-related parameters and business priorities, sampling decision scores are determined to guide which business data should be retained, achieving reasonable prediction of the sampling importance of different business data based on business priorities and the actual carrying capacity of the electronic devices. On the other hand, by inputting capacity and load characteristics and system congestion indicators into the sampling strategy model to obtain a dynamic sampling rate, a selectable sampling strategy can be quickly obtained based on the trained model, providing a reference for subsequently determining the target sampling strategy. First, the sampling pressure index and dynamic sampling rate are used to determine whether the current sampling pressure of the electronic device has reached its upper limit, enabling the formulation of a target sampling strategy to reduce business pressure when the carrying capacity of the electronic device reaches its upper limit, thus protecting the electronic device and its upstream and downstream load pressure from exceeding the limit. Furthermore, when it is determined that the current carrying capacity of the electronic device has not reached its limit, the target sampling strategy is reasonably determined by using the sampling decision score and the quantified carrying capacity of the electronic device. This solves the problem that the current fixed sampling method still discards some valuable data during the system's low-end period, affecting subsequent data analysis. It realizes the reasonable determination of the target sampling strategy based on the real-time dynamic operating status of the electronic device and the business priority of each business data point, thereby realizing the reasonable collection of data points and providing a comprehensive and accurate data foundation for subsequent data analysis.

[0010] Secondly, embodiments of the present invention also provide a sampling device for embedded points, the device comprising:

[0011] The first determination module is used to determine the system congestion index based on the real-time acquired capacity and load characteristics, and to determine the sampling pressure index based on the capacity and load characteristics and the system congestion index.

[0012] The second determining module is used to determine the sampling decision score based on capacity and load characteristics, and input the capacity and load characteristics and system congestion index into the sampling strategy model to obtain the dynamic sampling rate output by the sampling strategy model.

[0013] The sampling module is used to determine the target sampling strategy based on the sampling pressure index, sampling decision score and dynamic sampling rate, and to perform embedded sampling according to the target sampling strategy.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory that is communicatively connected to at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the embedding sampling method of any embodiment of the present invention.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement the embedding sampling method of any embodiment of the present invention.

[0019] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the embedding sampling method of any embodiment of the present invention.

[0020] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the data tracking device, or it may be packaged separately from the processor of the data tracking device; this application does not impose any limitations on this.

[0021] The descriptions of the second, third, fourth, and fifth aspects in this application can be referenced to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, fourth, and fifth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0022] In this application, the name of the aforementioned embedded sampling device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear under other names. As long as the function of each device or functional module is similar to that of this application, it falls within the scope of the claims of this application and its equivalents.

[0023] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating a method for embedding sampling points according to an embodiment of the present invention;

[0026] Figure 2A flowchart illustrating another embedding sampling method provided in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of a buried sampling device provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0030] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0031] The terms “initial” and “target” in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0032] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0033] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc. Moreover, without conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0034] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0035] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0036] Figure 1 This is a flowchart illustrating a data tracking method according to an embodiment of the present invention. This embodiment is applicable to situations where the sampling method is dynamically adjusted based on the actual operating state of the electronic device. The method can be executed by a data tracking device, which can be implemented in hardware and / or software and can be configured within the electronic device. In this embodiment, the electronic device can be a computer or server used for preprocessing data tracking data (including sampling the data tracking data). (Continue to refer to...) Figure 1 This embodiment specifically includes the following steps:

[0037] S101. Determine the system congestion index based on the real-time acquired capacity and load characteristics, and determine the sampling pressure index based on the capacity and load characteristics and the system congestion index.

[0038] Capacity and load characteristics are used to characterize key indicators affecting the sampling strategy. In this embodiment, capacity and load characteristics mainly include system load-related parameters, upstream data traffic parameters, downstream processing capacity parameters, business priority parameters, and time / scenario parameters. System load-related parameters include real-time central processing unit (CPU) utilization, memory usage, and network bandwidth utilization. Upstream data traffic parameters include the number of data points collected per second (QPS) and the proportion of specific business events (such as the percentage of payment events in the total traffic). Downstream processing capacity parameters include the number of messages currently backlogged in the consumption queue, the average message processing latency, and the health status of the consumption components. Business priority parameters include whether the data points are marked as high priority. Time / scenario parameters include whether it is during peak business periods or special event periods. System congestion indicators are used to characterize the real-time congestion situation of electronic devices processing data. In this embodiment, system congestion indicators mainly characterize two dimensions: queue pressure and traffic pressure. The sampling pressure index is used to characterize the data collection pressure that electronic devices can still withstand.

[0039] Optionally, in this embodiment, collecting upstream data flow parameters can quickly identify data peaks, sudden traffic surges, or abnormal collection behaviors; collecting downstream processing capacity parameters can quickly determine whether the downstream can "digest" the collected data in a timely manner; and business priority parameters can ensure that important behavioral data is given priority retention during the sampling process.

[0040] Specifically, after the data is collected at the front end (upstream), the electronic device can receive this data. Simultaneously, the electronic device can monitor both the upstream and itself to obtain real-time capacity and load characteristics. Furthermore, these capacity and load characteristics can be preprocessed, such as deduplication (to prevent duplicate reporting), field validation (e.g., determining timestamp compliance), data anonymization (e.g., for sensitive user information), classification and tagging (e.g., distinguishing between new and old business), normalization, standardization, and data discarding, while eliminating differences in units of measurement. For example, some key business parameters can be encoded as Boolean values ​​or directly used as weighting coefficients. After preprocessing the capacity and load characteristics, some of these characteristics can be used to determine system congestion indicators. For example, real-time queue information can be used to calculate queue pressure-related indicators, and collection rate or query rate can be used to calculate real-time traffic-related indicators. Finally, a sampling pressure index can be determined based on the capacity and load characteristics and system congestion indicators. For example, based on processor usage, queue pressure-related indicators, and real-time traffic-related indicators, the current data collection pressure that the electronic device can still withstand (sampling pressure index) can be determined.

[0041] In this embodiment, by obtaining real-time capacity and load characteristics to determine system congestion indicators, the real-time congestion status of electronic devices processing data can be accurately determined. Furthermore, by utilizing system congestion indicators and capacity and load characteristics that reflect the real-time congestion status of electronic devices, a basis can be provided for determining different sampling methods based on the real-time sampling pressure of the electronic devices, while ensuring the remaining capacity of the electronic devices for data sampling.

[0042] S102. Determine the sampling decision score based on the capacity and load characteristics, and input the capacity and load characteristics and system congestion index into the sampling strategy model to obtain the dynamic sampling rate output by the sampling strategy model.

[0043] The sampling decision score is used to guide the retention of valuable business tracking data. In this embodiment, the sampling strategy model is a pre-trained lightweight model (such as extreme gradient boosting, lightweight gradient boosting machines, and shallow neural networks). It uses historical capacity and load characteristics and system congestion indicators as training samples to learn the nonlinear mapping relationship between these characteristics and indicators and the dynamic sampling strategy, and outputs the dynamic sampling rate as the output indicator. The dynamic sampling rate is used to characterize the optional sampling behavior obtained based on the actual operating conditions of the current electronic equipment; optional sampling behavior includes, for example, what percentage of sampling should be used or whether sampling should be temporarily stopped to protect downstream devices.

[0044] Specifically, the total load currently occupied by electronic devices can be calculated based on system load-related parameters in the capacity and load characteristics. Then, using service priorities and the total load currently occupied by electronic devices, a sampling decision score can be determined to guide which service data points should be retained. Simultaneously, the capacity and load characteristics and system congestion indicators are input into the sampling strategy model to obtain the dynamic sampling rate output by the model.

[0045] In this embodiment, on the one hand, the sampling decision score is determined based on system load-related parameters and business priorities to guide which business data points need to be retained. This enables reasonable prediction of the sampling importance of different business data based on business priorities and the actual capacity of electronic devices, ensuring that the importance of different business data can be determined while guaranteeing that electronic devices can process the data. On the other hand, by inputting capacity and load characteristics and system congestion indicators into the sampling strategy model, a dynamic sampling rate is obtained. This allows for the rapid development of an selectable sampling strategy based on the trained model, providing a reference for subsequently determining the target sampling strategy.

[0046] S103. Determine the target sampling strategy based on the sampling pressure index, sampling decision score and dynamic sampling rate, and perform embedded sampling according to the target sampling strategy.

[0047] The target sampling strategy refers to the sampling method and behavior for performing embedded point sampling.

[0048] Specifically, the sampling pressure index and dynamic sampling rate can be used to first determine whether the sampling pressure of the current electronic device is approaching its limit, ensuring that the pressure on the electronic device and its upstream and downstream loads does not exceed the limit. Furthermore, if the sampling pressure is determined not to be close to the limit based on the sampling pressure index and dynamic sampling rate, the sampling decision score can be used to determine which data points to sample first, and the sampling method for them, thus obtaining the target sampling strategy. Finally, data can be sampled based on the target sampling strategy.

[0049] In this embodiment, the sampling pressure of the current electronic device is first determined based on the sampling pressure index and dynamic sampling rate to determine whether the sampling pressure has reached its upper limit. This quantifies the carrying capacity of the current electronic device and enables the formulation of a target sampling strategy to reduce business pressure when the carrying capacity of the electronic device reaches its upper limit, thus protecting the electronic device and its upstream and downstream load pressure from exceeding the limit. Furthermore, when it is determined that the carrying capacity of the current electronic device has not reached its upper limit, a reasonable target sampling strategy is determined using the sampling decision score and the quantified carrying capacity of the electronic device. This solves the problem that current methods using only fixed sampling methods still discard some valuable data during system downtime, affecting subsequent data analysis. It achieves the reasonable determination of the target sampling strategy based on the real-time dynamic operating status of the electronic device and the business priority of each business data point, thereby enabling the reasonable collection of data points and providing a comprehensive and accurate data foundation for subsequent data analysis.

[0050] It is worth noting that the event tracking data and its subsequent processing in this embodiment are obtained and processed through legal means and with the consent of the data source (such as the user or data source). The event tracking data in this embodiment can be data captured by the front-end or back-end program and recorded as event tracking events, such as user behavior (clicks, browsing, purchases) on terminals such as applications, web pages, and mini-programs. This embodiment only focuses on the process of sampling the event tracking data after it is obtained; the specific process of obtaining the event tracking data is not the focus of this embodiment.

[0051] The sampling method provided in this invention determines system congestion indicators by acquiring real-time capacity and load characteristics, enabling accurate determination of the real-time congestion status of electronic devices processing data. Furthermore, by utilizing system congestion indicators and capacity and load characteristics reflecting the real-time congestion status of electronic devices, a basis is provided for determining different sampling methods based on the real-time sampling pressure of the electronic devices. On one hand, based on system load-related parameters and business priorities, sampling decision scores are determined to guide which business data should be retained, achieving reasonable prediction of the sampling importance of different business data based on business priorities and the actual carrying capacity of the electronic devices. On the other hand, by inputting capacity and load characteristics and system congestion indicators into the sampling strategy model to obtain a dynamic sampling rate, a selectable sampling strategy can be quickly obtained based on the trained model, providing a reference for subsequently determining the target sampling strategy. First, the sampling pressure index and dynamic sampling rate are used to determine whether the current sampling pressure of the electronic device has reached its upper limit, enabling the formulation of a target sampling strategy to reduce business pressure when the carrying capacity of the electronic device reaches its upper limit, thus protecting the electronic device and its upstream and downstream load pressure from exceeding the limit. Furthermore, when it is determined that the current carrying capacity of the electronic device has not reached its limit, the target sampling strategy is reasonably determined by using the sampling decision score and the quantified carrying capacity of the electronic device. This solves the problem that the current fixed sampling method still discards some valuable data during the system's low-end period, affecting subsequent data analysis. It realizes the reasonable determination of the target sampling strategy based on the real-time dynamic operating status of the electronic device and the business priority of each business data point, thereby realizing the reasonable collection of data points and providing a comprehensive and accurate data foundation for subsequent data analysis.

[0052] Figure 2 This is a flowchart illustrating another data collection and sampling method provided by an embodiment of the present invention. This embodiment, based on the above embodiments, specifies the steps for determining system congestion indicators, determining sampling pressure indices, determining sampling decision scores, and determining target sampling strategies. In this embodiment, the method may include:

[0053] S201. Determine the queue pressure score based on the current queue length and the maximum queue capacity threshold in the capacity and load characteristics.

[0054] The current queue length represents the number of event tracking points waiting to be processed in the data acquisition, processing, or reporting pipeline. The maximum queue capacity threshold is the maximum allowed length used to limit the queue length. The queue pressure score represents the current backlog in the queue.

[0055] Specifically, the queue pressure score can be obtained by dividing the current queue length by the maximum queue capacity threshold.

[0056] S202. Determine the traffic pressure score based on the historical average query rate and the real-time query rate in the capacity and load characteristics.

[0057] The historical query rate average is the average query rate over a specific historical period (a period corresponding to the current moment, or a historical time period that better represents the current moment). The traffic pressure score is used to quantify the collection rate of the collected data points.

[0058] Specifically, the average historical query rate is first determined based on the historical query rate. The selected historical query rate can be on a different date but within the same time interval as the current time (e.g., 10-11 AM on December 20, 2025 and 10:30 AM on December 25, 2025), or it can be a specific day, such as a holiday. Then, the traffic pressure score can be obtained by dividing the real-time query rate by the average historical query rate.

[0059] S203. The queue pressure score and the flow pressure score are determined as system congestion indicators.

[0060] Specifically, after calculating the queue pressure score and the flow pressure score, the two can be identified as system congestion indicators.

[0061] In this embodiment, not only is the load information of the electronic device acquired in real time, but also the current data processing information of the upstream and downstream. Based on these capacity and load characteristics, the system congestion index is further calculated, which realizes the accurate determination of the real-time operation status between the electronic device and the upstream and downstream, and provides a basis for the subsequent precise control of the target sampling strategy.

[0062] S204. Determine the weighting indicators based on the business priority and business attribute information of real-time services.

[0063] In this context, business priority is used to characterize the importance and priority of each business tracking data point during sampling. In this embodiment, business attribute information is used to characterize the device capabilities that need to be considered when sampling business tracking data. The weighting index is used to characterize the weights corresponding to different parameters in different capacity and load characteristics, i.e., to characterize the importance of different parameters.

[0064] Specifically, weighted indicators can be assigned to different parameters based on real-time business needs. For example, the business priority of a real-time business can characterize the necessity of the corresponding tracking data, thus allowing for the allocation of more weighted indicators. Similarly, business attribute information can characterize the data size, frequency, and business attributes of the tracking data, predicting the sampling parameters that the tracking data might require. For instance, a larger data size might result in higher CPU utilization, thus allowing for a larger weighted indicator for CPU utilization.

[0065] Optionally, the weighting index for business priority can be in the range of [0, 1], with a higher value indicating a more important business.

[0066] S205. Determine the sampling pressure index based on processor usage information, queue pressure score, flow pressure score, and allocation weight index in the capacity and load characteristics.

[0067] Specifically, the sampling pressure index can be calculated using the following formula:

[0068] L = w1 × processor usage information + w2 × queue pressure score + w3 × traffic pressure score + w4 × (1 - business priority weight).

[0069] Wherein, L is the sampling pressure index; w1 is the weight index corresponding to the processor usage information (i.e., the standardized value of CPU utilization) determined above in the weight allocation index; w2 is the weight index corresponding to the queue pressure score determined above in the weight allocation index; w3 is the weight index corresponding to the traffic pressure score determined above in the weight allocation index; the service priority weight is the weight index corresponding to the service priority determined above in the weight allocation index; w4 is the weight determined by the service priority weight in the weight allocation index. In practice, this weight can be preset or determined according to the value of the service priority weight and different weight ranges.

[0070] S206. Determine the comprehensive system load value based on the system load information in the capacity and load characteristics.

[0071] Among them, the system load information is a composite index obtained by aggregating all system load information.

[0072] Specifically, methods such as weighted summation, fuzzy logic, worst-case dimensionality method, and statistical algorithms can be used to aggregate and calculate parameters such as CPU utilization, memory usage, disk input / output, and network bandwidth in system load information to obtain a comprehensive system load value.

[0073] S207. Determine the sampling decision score based on the overall system load value and the service priority of real-time services.

[0074] Specifically, the sampling decision score can be calculated using the following formula:

[0075] S = α × (1 - System load value) + β × Service priority value;

[0076] The business priority value is a quantified value obtained by mapping business priorities to numerical form based on preset rules, quantifying them using a multi-dimensional scorecard method, or quantifying them using a historical data statistical model. α and β can be weights in the assigned weighting indicators, or they can be pre-set weights.

[0077] In this embodiment, the allocation weight index is determined based on business priority and business attribute information, and the sampling pressure index is determined based on the allocation weight index and capacity and load characteristics. This enables the quantification of the sampling pressure based on the business situation corresponding to the subsequent data to be sampled. The sampling decision score is determined based on the comprehensive system load value and the business priority of real-time business. This enables the correlation between the real-time system load and business priority, providing a basis for determining the sampling priority of the embedded data and the overall sampling rate range.

[0078] S208. Input the capacity and load characteristics and system congestion indicators into the sampling strategy model to obtain the dynamic sampling rate output by the sampling strategy model.

[0079] Specifically, a sampling strategy model can be pre-trained, and in actual use, the real-time capacity and load characteristics as well as system congestion indicators can be input into the sampling strategy model, which can then output a dynamic sampling rate.

[0080] S209. Determine the initial sampling strategy based on the dynamic sampling rate and sampling pressure index, and determine the target sampling points based on the sampling decision score.

[0081] Among them, the target sampling points are the data points corresponding to high-priority services, or data points that are of high importance and should be sampled first regardless of the current sampling pressure of the electronic device.

[0082] Specifically, an initial sampling strategy can be determined based on the dynamic sampling rate and sampling pressure index; at the same time, target sampling points with higher business priority can be determined based on the sampling decision score.

[0083] For example, determining the initial sampling strategy based on the dynamic sampling rate and the sampling pressure index includes:

[0084] (a) Determine whether the dynamic sampling rate is within the threshold of the stop sampling interval.

[0085] Among them, the stop sampling interval threshold is used to analyze whether the obtained dynamic sampling rate is sufficient for the current electronic device to stop collecting sample data.

[0086] Specifically, in one implementation, the dynamic sampling rate is expressed as a floating-point number (e.g., between 0 and 1) or a sampling ratio (0%-100%), and the stop sampling interval threshold is in numerical form, such as [0, 0.01) or [0%, 1%). In another implementation, the dynamic sampling rate is expressed in text form, such as "dynamic sampling decision is to pause sampling", and the stop sampling interval threshold is also in text form.

[0087] (ii) If the dynamic sampling rate is at the threshold of the stop sampling interval, the initial sampling strategy is determined to be to pause sampling.

[0088] Specifically, if the dynamic sampling rate is at the threshold of the stop sampling interval, it indicates that downstream consumption is severely blocked, and sampling needs to be temporarily stopped to avoid the collapse of the entire system (including upstream, electronic devices, and downstream). Therefore, the initial sampling strategy is to pause sampling.

[0089] Optionally, after pausing sampling, the system can return to S201 to obtain capacity and load characteristics in real time until it is determined that the dynamic sampling rate is not within the threshold of the pausing sampling interval.

[0090] (iii) If the dynamic sampling rate is not at the threshold of the stop sampling interval, the initial sampling strategy shall be determined based on the dynamic sampling rate, the threshold of the dynamic sampling interval and the sampling pressure index.

[0091] Among them, the dynamic sampling interval threshold is used to determine whether the obtained dynamic sampling rate indicates that the current electronic device is sampling according to different sampling methods.

[0092] Specifically, if the dynamic sampling rate is not at the threshold of the stop sampling interval, it means that the system pressure has not reached the level of severe blockage. Therefore, the preferred sampling method can be determined first based on the dynamic sampling rate and the threshold of the dynamic sampling interval, and then the initial sampling strategy can be determined based on the sampling pressure index.

[0093] For example, dynamic sampling interval thresholds are pre-set: the first interval threshold indicates high system pressure, requiring a reduction in sampling amplitude to protect the backend (downstream); the second interval threshold indicates low system load and unobstructed downstream operation, allowing for full sampling. Then, determining which interval the dynamic sampling rate belongs to: if it falls within the first interval threshold, the sampling method can be initially set to partial sampling to alleviate real-time system pressure. Furthermore, since the current sampling is partial, to ensure the integrity of high-priority event data, a temporary full sampling strategy can be determined when the sampling pressure index meets the pressure threshold. Simultaneously, for event data with low business priority, the initial sampling strategy can be directly set to partial sampling. If it falls within the second interval threshold, the sampling method can be directly set to full sampling to protect data integrity and continuity. Therefore, as long as the sampling pressure index meets the pressure threshold during sampling, the initial sampling strategy can be set to full sampling; if the sampling pressure index does not meet the pressure threshold, the initial sampling strategy can be temporarily set to partial sampling.

[0094] In this embodiment, the system first determines whether it is in the sampling halt interval based on dynamic sampling. When it is, sampling is paused, which protects the interaction between electronic devices and downstream devices when downstream consumption is severely blocked, and prevents the entire system from collapsing due to exceeding the system's load-bearing capacity. Furthermore, when the dynamic sampling rate is not in the sampling halt interval, the initial sampling strategy can be further determined based on the sampling pressure index, providing a basis for determining the target sampling strategy later.

[0095] S210. When the initial sampling strategy is non-pause sampling, the target sampling strategy is determined based on the number of target sampling points and the sampling attributes.

[0096] Among them, the sampling attributes are the sampling-related attributes of the electronic device when sampling data; in this embodiment, the sampling attributes include historical time series attributes and business importance; the historical time series attributes are used to characterize the sampling traffic trend of the electronic device during the sampling process in the past, such as "the average query rate per second in the past N hours / days, historical peak, and traffic trend in the recent period (stable, rising, falling, etc.)"; the business importance is used to characterize the degree of importance attached to the sampling of the embedded data when sampling different business priorities.

[0097] Specifically, if the initial sampling strategy is non-pause sampling, it means that the system pressure has not reached a critical state. Therefore, the target sampling strategy can be determined based on the number of target sampling points and the collection attributes.

[0098] For example, determining the target sampling strategy based on the number of target sampling points and sampling attributes includes:

[0099] (i) If the initial sampling strategy is full sampling, then all target sampling points are sorted according to the historical time sequence attribute in the sampling attribute to obtain the first sorting order, and the target sampling strategy is determined according to the first sorting order and the number of target sampling points.

[0100] Specifically, if the initial sampling strategy is full sampling, then the focus is on sorting all target sampling points based on the sampling timing of the data points corresponding to different priority services. That is, based on service priority, data points with higher priority and shorter sampling times are collected first, resulting in a first sorting order. Then, the first sorting order and the number of target sampling points can be used to determine whether data points can be collected in parallel, thus obtaining the target sampling strategy.

[0101] (ii) If the initial sampling strategy is non-full sampling, then all target sampling points are sorted according to business priority and business importance in the sampling attributes to obtain a second sorting order, and the target sampling strategy is determined according to the second sorting order and the number of target sampling points.

[0102] Specifically, if the initial sampling strategy is not full sampling, such as partial sampling or downsampling, then all target sampling points need to be sorted according to the business priority and the pre-defined business importance of each business in the sampling attributes, resulting in a second sorting order set only based on business priority. Then, the target sampling strategy can be determined based on the second sorting order and the number of target sampling points.

[0103] For example, all target sampling points are first grouped based on business priority and business importance. Target sampling points whose product of business priority and business importance exceeds a preset business threshold are directly added to a mandatory sampling list, and then sorted according to the degree of exceeding the threshold. Next, based on the sorting in the mandatory sampling list, non-full sampling methods (such as stratified sampling, weighted probability sampling, and cluster sampling) that maximize data representativeness and integrity are used to sample the sampling points in the mandatory sampling list. Furthermore, for target sampling points whose product of business priority and business importance does not exceed the preset business threshold, they are directly sorted according to the size of the product of business priority and business importance to obtain a second sorting order. Then, the sampling quantity can be continuously reduced according to the second sorting order. For example, a sampling rate of 10% is set for the first 10 sampling points in the second sorting order, an 8% sampling rate is set for the 11th to 20th sampling points in the second sorting order, and so on. Finally, the sampling method or sampling rate determined according to the above method is the final target sampling strategy.

[0104] Optionally, for the above-mentioned non-full sampling method, some data will definitely be discarded. However, the data discarded in this embodiment is not randomly discarded, but strategically discarded. The data to be discarded will be determined based on the importance, data type and function of each data in the tracking data, and important tracking data will be sampled.

[0105] In this embodiment, depending on whether the initial sampling strategy is full sampling, different sampling strategies can be specifically selected to sample the target sampling points. When the initial sampling strategy is full sampling, the current system load is low and the downstream is smooth. Therefore, the target sampling points can be directly sorted and sampled according to historical time series attributes, which can ensure faster and more comprehensive sampling of all sampling points. When the initial sampling strategy is non-full sampling, the system pressure is not low. Therefore, it is necessary to reduce sampling to protect the backend. Therefore, the sampling points can be sorted according to the business priority and business importance of the data points to ensure that, in the case of non-full sampling, key data can be sampled more completely.

[0106] S211. Perform embedded point sampling according to the target sampling strategy.

[0107] Specifically, once the target sampling strategy is obtained, the embedded data can be sampled according to the target sampling strategy.

[0108] In this embodiment, the target sampling strategy is determined in real time for the sampling points of different services by continuously monitoring capacity and load characteristics. The whole process is automated and can realize real-time self-adjustment according to the actual operating status. While ensuring the stability of services, it intelligently balances the integrity, representativeness and system consumption of data collection.

[0109] Figure 3 This is a schematic diagram of a buried sampling device provided in an embodiment of the present invention. Figure 3 As shown, the device includes:

[0110] The first determining module 301 is used to determine the system congestion index based on the real-time acquired capacity and load characteristics, and to determine the sampling pressure index based on the capacity and load characteristics and the system congestion index.

[0111] The second determining module 302 is used to determine the sampling decision score based on the capacity and load characteristics, and input the capacity and load characteristics and system congestion index into the sampling strategy model to obtain the dynamic sampling rate output by the sampling strategy model.

[0112] The sampling module 303 is used to determine the target sampling strategy based on the sampling pressure index, sampling decision score and dynamic sampling rate, and to perform embedded sampling according to the target sampling strategy.

[0113] Based on the above embodiments, the system congestion index is determined according to the real-time acquired capacity and load characteristics. The first determining module 301 is specifically used for:

[0114] The queue pressure score is determined based on the current queue length and the maximum queue capacity threshold in the capacity and load characteristics; the traffic pressure score is determined based on the historical average query rate and the real-time query rate in the capacity and load characteristics; the queue pressure score and the traffic pressure score are used as system congestion indicators.

[0115] Based on the above embodiments, the sampling pressure index is determined according to capacity and load characteristics and system congestion indicators. The first determining module 301 is specifically used for:

[0116] The allocation weight index is determined based on the service priority and service attribute information of real-time services; the sampling pressure index is determined based on the processor usage information, queue pressure score, traffic pressure score and allocation weight index in the capacity and load characteristics.

[0117] Based on the above embodiments, the sampling decision score is determined according to capacity and load characteristics, and the second determining module 302 is specifically used for:

[0118] The overall system load value is determined based on the system load information in the capacity and load characteristics; the sampling decision score is determined based on the overall system load value and the service priority of real-time services.

[0119] Based on the above embodiments, the target sampling strategy is determined according to the sampling pressure index, sampling decision score, and dynamic sampling rate. The sampling module 303 is specifically used for:

[0120] The initial sampling strategy is determined based on the dynamic sampling rate and sampling pressure index, and the target sampling points are determined based on the sampling decision score. When the initial sampling strategy is non-pause sampling, the target sampling strategy is determined based on the number of target sampling points and the sampling attributes.

[0121] Based on the above embodiments, the initial sampling strategy is determined according to the dynamic sampling rate and the sampling pressure index. The sampling module 303 is specifically used for:

[0122] Determine whether the dynamic sampling rate is within the stop sampling interval threshold; if the dynamic sampling rate is within the stop sampling interval threshold, determine the initial sampling strategy as pause sampling; if the dynamic sampling rate is not within the stop sampling interval threshold, determine the initial sampling strategy based on the dynamic sampling rate, the dynamic sampling interval threshold, and the sampling pressure index.

[0123] Based on the above embodiments, the target sampling strategy is determined according to the number of target sampling points and sampling attributes. The sampling module 303 is specifically used for:

[0124] If the initial sampling strategy is full sampling, all target sampling points are sorted according to the historical time sequence attribute in the sampling attributes to obtain a first sorting order, and the target sampling strategy is determined according to the first sorting order and the number of target sampling points. If the initial sampling strategy is non-full sampling, all target sampling points are sorted according to the business priority and the business importance in the sampling attributes to obtain a second sorting order, and the target sampling strategy is determined according to the second sorting order and the number of target sampling points.

[0125] The embedded sampling device provided in this embodiment of the invention can execute the embedded sampling method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0126] It is worth noting that in the embodiments of the above-mentioned embedded sampling device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0127] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 4 A block diagram is shown of an exemplary electronic device 11 suitable for implementing embodiments of the present invention. Figure 4 The electronic device 11 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0128] like Figure 4 As shown, the electronic device 11 is represented in the form of a general-purpose computing electronic device. The components of the electronic device 11 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0129] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0130] Electronic device 11 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 11, including volatile and non-volatile media, removable and non-removable media.

[0131] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 11 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 As not shown, disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disc drives for reading and writing to removable non-volatile optical discs (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0132] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0133] Electronic device 11 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 11, and / or with any device that enables electronic device 11 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 11 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. Figure 4 As shown, network adapter 20 communicates with other modules of electronic device 11 via bus 18. It should be understood that, although... Figure 4 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 11, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0134] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28, such as implementing the data tracking method provided in this embodiment. Of course, those skilled in the art will understand that the processor can also implement the technical solutions of the data tracking method provided in any embodiment of this invention.

[0135] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements, for example, the embedded sampling method provided in this invention.

[0136] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0137] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0138] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0139] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the embedding sampling method as provided in any embodiment of this invention.

[0140] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0142] Furthermore, the acquisition, storage, use, and processing of data in the technical solution of this invention all comply with the relevant provisions of national laws and regulations.

[0143] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for embedding sampling points, characterized in that, The method includes: The system congestion index is determined based on the real-time acquired capacity and load characteristics, and the sampling pressure index is determined based on the capacity and load characteristics and the system congestion index. The sampling decision score is determined based on the capacity and load characteristics, and the capacity and load characteristics and the system congestion index are input into the sampling strategy model to obtain the dynamic sampling rate output by the sampling strategy model. The target sampling strategy is determined based on the sampling pressure index, the sampling decision score, and the dynamic sampling rate, and the embedded sampling is performed according to the target sampling strategy.

2. The method according to claim 1, characterized in that, The process of determining system congestion indicators based on real-time acquired capacity and load characteristics includes: The queue pressure score is determined based on the current queue length and the maximum queue capacity threshold in the capacity and load characteristics. The traffic pressure score is determined based on the historical average query rate and the real-time query rate in the capacity and load characteristics. The queue pressure score and the flow pressure score are determined as the system congestion indicators.

3. The method according to claim 2, characterized in that, The step of determining the sampling pressure index based on the capacity and load characteristics and the system congestion index includes: The weighting indicators are determined based on the business priority and business attribute information of real-time services. The sampling pressure index is determined based on the processor usage information in the capacity and load characteristics, the queue pressure score, the flow pressure score, and the allocation weight index.

4. The method according to claim 1, characterized in that, The step of determining the sampling decision score based on the capacity and load characteristics includes: The overall system load value is determined based on the system load information in the capacity and load characteristics. The sampling decision score is determined based on the overall system load value and the service priority of real-time services.

5. The method according to claim 1, characterized in that, The step of determining the target sampling strategy based on the sampling pressure index, the sampling decision score, and the dynamic sampling rate includes: The initial sampling strategy is determined based on the dynamic sampling rate and the sampling pressure index, and the target sampling points are determined based on the sampling decision score. When the initial sampling strategy is non-pause sampling, the target sampling strategy is determined based on the number of target sampling points and the sampling attributes.

6. The method according to claim 5, characterized in that, The step of determining the initial sampling strategy based on the dynamic sampling rate and the sampling pressure index includes: Determine whether the dynamic sampling rate is within the threshold of the stop sampling interval; If the dynamic sampling rate is within the threshold of the stop sampling interval, then the initial sampling strategy is determined to be to pause sampling; If the dynamic sampling rate is not within the threshold of the sampling stop interval, then the initial sampling strategy is determined based on the dynamic sampling rate, the dynamic sampling interval threshold, and the sampling pressure index.

7. The method according to claim 5, characterized in that, The step of determining the target sampling strategy based on the number of target sampling points and sampling attributes includes: If the initial sampling strategy is full sampling, then all target sampling points are sorted according to the historical time sequence attribute in the sampling attribute to obtain a first sorting order, and the target sampling strategy is determined according to the first sorting order and the number of target sampling points. If the initial sampling strategy is non-full sampling, then all target sampling points are sorted according to business priority and business importance in the sampling attributes to obtain a second sorting order, and the target sampling strategy is determined according to the second sorting order and the number of target sampling points.

8. A sampling device for embedded points, characterized in that, The device includes: The first determining module is used to determine the system congestion index based on the real-time acquired capacity and load characteristics, and to determine the sampling pressure index based on the capacity and load characteristics and the system congestion index. The second determining module is used to determine the sampling decision score based on the capacity and load characteristics, and input the capacity and load characteristics and the system congestion index into the sampling strategy model to obtain the dynamic sampling rate output by the sampling strategy model; The sampling module is used to determine the target sampling strategy based on the sampling pressure index, the sampling decision score and the dynamic sampling rate, and to perform embedded sampling according to the target sampling strategy.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data collection sampling method as described in any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the embedding sampling method as described in any one of claims 1 to 7.

Images