A link log storage method, device and electronic equipment

Abstract

Embodiments of the present application disclose a link log storage method and device and electronic equipment, comprising: obtaining all original link logs corresponding to a target object, and extracting state features, service features, performance features and topology features in each original link log; determining comprehensive scores corresponding to each original link log according to the state features, service features, performance features and topology features corresponding to each original link log; determining target link logs in each original link log according to the comprehensive scores corresponding to each original link log, and storing each target link log, thereby improving the storage efficiency of original link logs and saving storage space.

Description

Technical Field

[0001] This invention relates to the field of log management technology, and in particular to a link log storage method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of distributed systems and microservice architectures, various business scenarios typically involve multiple service nodes interacting through protocols such as Hypertext Transfer Protocol (HTTP) to form a complete business chain. As crucial data recording the interaction process and status information of each service node in the business chain, the storage method of the chain log directly impacts the efficiency and accuracy of troubleshooting, performance optimization, data tracing, and auditing.

[0003] Currently, most link logs adopt a general storage solution, which directly stores the link logs generated by all business links in memory, resulting in low link log storage efficiency and wasted storage space.

[0004] Therefore, given the low storage efficiency and resource waste of existing link log storage methods, there is an urgent need for an optimized link log storage method to address the shortcomings of current technologies. Summary of the Invention

[0005] This invention provides a link log storage method, apparatus, and electronic device, which improves the storage efficiency of raw link logs and saves storage space.

[0006] In a first aspect, embodiments of the present invention provide a link log storage method, including: Obtain all raw link logs corresponding to the target object, and extract the status characteristics, business characteristics, performance characteristics, and topology characteristics from each raw link log; Based on the status characteristics, business characteristics, performance characteristics, and topology characteristics corresponding to each original link log, a comprehensive score corresponding to each original link log is determined. Based on the comprehensive score corresponding to each original link log, the target link log is determined from each original link log and stored.

[0007] Secondly, embodiments of the present invention also provide a link log storage device, comprising: The feature extraction module is used to obtain all raw link logs corresponding to the target object and extract the status features, business features, performance features and topology features from each raw link log. The comprehensive scoring module is used to determine the comprehensive score corresponding to each original link log based on the status characteristics, service characteristics, performance characteristics, and topology characteristics corresponding to each original link log respectively. The log storage module is used to determine the target link log from each original link log based on the comprehensive score corresponding to each original link log, and to store each target link log.

[0008] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the link log storage method provided in any embodiment of the present invention.

[0009] The technical solution of this invention determines the comprehensive score corresponding to each original link log by using the status characteristics, service characteristics, performance characteristics, and topology characteristics corresponding to each original link log respectively; and determines the target link log in each original link log according to the comprehensive score corresponding to each original link log respectively, and stores each target link log. This solves the problem of low storage efficiency and wasted storage space caused by directly storing all original link logs in memory, thereby improving the storage efficiency of original link logs and saving storage space.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a link log storage method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of another link log storage method provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of a link log storage device according to Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Example 1 Figure 1 This is a flowchart of a link log storage method according to Embodiment 1 of the present invention. This embodiment is applicable to the sampling and storage of link logs. The method can be executed by a link log storage device, which can be implemented in hardware and / or software and can be configured in an electronic device such as a computer.

[0016] like Figure 1 As shown in this embodiment, a link log storage method includes: S110. Obtain all original link logs corresponding to the target object, and extract the status characteristics, business characteristics, performance characteristics and topology characteristics from each original link log.

[0017] In this embodiment, the target object can be understood as the object to be stored as the raw link log, such as a system and an application. The raw link log can be used to record various types of information generated along a complete business link. A business link can be understood as a business flow path composed of multiple service nodes connected sequentially, used to record the interaction process and status information between each service node. Status characteristics, business characteristics, performance characteristics, and topology characteristics can be used to record the basic status, business characteristics, performance characteristics, and topology characteristics of the business link, respectively. Specifically, status characteristics may include the root service node name, the number of service nodes, the total latency, and whether an error occurred in the business link. Business characteristics may include the business type, user level, user ID, and product value. Performance characteristics may include the number of error-reporting nodes, query performance, and service extension list. Topology characteristics may include the node call path, node call depth, and whether there are parallel calls.

[0018] In one specific implementation, an Application Performance Management (APM) probe can be used to collect all raw link logs corresponding to the target object. Then, each raw link log can be scanned in real time to extract the status characteristics, service characteristics, performance characteristics, and topology characteristics from each raw link log.

[0019] Taking the scenario of high-priority user Zhang San purchasing a mobile phone worth 2999 yuan on an e-commerce platform at 10:00 AM on January 1, 2024 as an example, an APM probe can be used to collect a raw link log describing this scenario. Specifically, since the business link corresponding to this raw link log is Gateway Service → Order Service → Payment Service → Inventory Service, the raw link log consists of basic status field groups, gateway service field groups, order service field groups, payment service field groups, and inventory service field groups. The basic status field group includes the raw link number, current timestamp, root service node name, total latency, and whether an error occurred in the business link. The gateway service field group includes the gateway service name, gateway service start time, gateway service end time, gateway service execution time, gateway service execution status, HTTP request method, HTTP interface path, HTTP response status code, user ID, user level, business type, and product value. The order service field group includes the order service name, order service execution time, order service execution status, database operation statement, and product value. The payment service field group includes the payment service name, payment service execution time, payment service execution status, payment method, and product value. The inventory service field group includes the inventory service name, inventory service execution time, inventory service execution status, inventory number, and inventory quantity.

[0020] Then, the status characteristics, business characteristics, performance characteristics, and topology characteristics can be extracted from the original link logs. Specifically, the status characteristics are as follows: the root service node name of the original link is "Gateway Service," the number of service nodes is 4, the total latency is 333 milliseconds, and no errors occurred in the business link. The business characteristics are as follows: the business type is order creation, the user level is high-priority user, the user ID is "user_vip_001," and the product value is 2999 yuan. The performance characteristics are as follows: the number of error nodes is 0, the query performance is non-slow query, and the service latency list is [333, 330, 200, 240] milliseconds. The topology characteristics are as follows: the node call path is ["gateway-service", "order-service", "payment-service"], the node call depth is 3 levels, and there are no parallel calls.

[0021] Finally, while the original link logs are still stored in memory, structured feature vectors corresponding to state features, business features, performance features, and topology features can be generated.

[0022] The advantage of this setup is that by extracting the status characteristics, service characteristics, performance characteristics, and topology characteristics from each original link log, redundant labels and duplicate information in the original link log can be removed, thereby improving the efficiency of subsequent comprehensive score determination.

[0023] S120. Based on the status characteristics, service characteristics, performance characteristics, and topology characteristics corresponding to each original link log, determine the comprehensive score corresponding to each original link log.

[0024] In this step, specifically, based on the status characteristics, business characteristics, performance characteristics, and topology characteristics corresponding to each original link log, a traffic score, business importance score, performance anomaly score, and dependency score can be determined for each original link log. Then, based on the traffic score, business importance score, performance anomaly score, and dependency score corresponding to each original link log, a comprehensive score corresponding to each original link log is determined.

[0025] Alternatively, the structured feature vectors corresponding to each original link log can be directly input into a pre-built comprehensive scoring model to obtain a comprehensive score corresponding to each original link log. The comprehensive scoring model is a deep learning model trained using multiple structured feature vectors as training samples.

[0026] S130. Based on the comprehensive score corresponding to each original link log, determine the target link log in each original link log and store each target link log.

[0027] In this step, specifically, the original link logs can be sorted in descending order based on their respective comprehensive scores, and the top predetermined number of original link logs in the descending sort result can be used as target link logs. Alternatively, the target score corresponding to each original link log can be determined based on the storage space utilization corresponding to the target object, as well as the comprehensive score, user level, and latency time corresponding to each original link log. The target link logs can then be identified from the original link logs based on these target scores. After identifying the target link logs, they can be stored losslessly or compressed.

[0028] The technical solution of this embodiment solves the problem of low storage efficiency and wasted storage space caused by directly storing all original link logs in memory, which improves the storage efficiency of original link logs and saves storage space. This is achieved by obtaining all original link logs corresponding to the target object and extracting status features, service features, performance features, and topology features from each original link log; determining the comprehensive score corresponding to each original link log based on the status features, service features, performance features, and topology features corresponding to each original link log; determining the target link log in each original link log based on the comprehensive score corresponding to each original link log; and storing each target link log.

[0029] Example 2 Figure 2 This is a flowchart of another link log storage method provided by Embodiment 2 of the present invention. This embodiment is a further optimization and extension based on the above embodiments, and can be combined with various optional technical solutions in the above embodiments.

[0030] like Figure 2 As shown in this embodiment, a link log storage method includes: S210. Obtain all original link logs corresponding to the target object, and extract the status characteristics, business characteristics, performance characteristics and topology characteristics from each original link log.

[0031] In this embodiment, the status characteristics include at least the root service node name. The business characteristics include at least the business type, user level, and product value. The performance characteristics include at least the number of error-reporting nodes and query performance. The topology characteristics include at least the node call path.

[0032] S220. Based on the status characteristics, service characteristics, performance characteristics, and topology characteristics corresponding to each original link log, determine the traffic score, service importance score, performance anomaly score, and dependency score corresponding to each original link log.

[0033] In this embodiment, traffic score can be used to characterize the frequency of the original link log being invoked. Business importance score can be used to characterize the importance of the business link corresponding to the original link log. Performance anomaly score can be used to characterize the degree of anomaly of the business link corresponding to the original link log. Dependency score characterizes the degree to which the business link corresponding to the original link log is depended upon.

[0034] Specifically, in this step, based on the state characteristics corresponding to each original link log, a traffic score is determined. Based on the service characteristics corresponding to each original link log, a service importance score is determined. Based on the performance characteristics corresponding to each original link log, a performance anomaly score is determined. Based on the topology characteristics corresponding to each original link log, a dependency score is determined.

[0035] Optionally, based on the status characteristics corresponding to each original link log, determine the traffic score corresponding to each original link log, including: obtaining the root service node corresponding to each original link log based on the root service node name, and counting the number of times each root service node was called within a preset historical time period; obtaining all service nodes corresponding to the target object, and counting the cumulative number of times all service nodes were called within the historical time period; determining the traffic percentage corresponding to each original link log based on the number of times it was called and the cumulative number of times it was called; and determining the traffic score corresponding to each original link log based on the traffic percentage corresponding to each original link log.

[0036] Specifically, the traffic percentage corresponding to each original link log can be obtained by dividing the number of calls corresponding to each original link log by the cumulative number of calls. Then, the traffic percentage corresponding to each original link log can be multiplied by a predefined total score limit to obtain the traffic score corresponding to each original link log. Finally, the traffic score corresponding to each original link log can be multiplied by a predefined traffic weighting coefficient to obtain the traffic rating corresponding to each original link log.

[0037] Taking the root service node name "Gateway Service" as an example, assuming the Gateway Service was called 10,000,000 times in the past 24 hours, and the target object was called a total of 200 million times in the past 24 hours, we can divide the number of times the Gateway Service was called in the past 24 hours by the total number of times the target object was called in the past 24 hours to obtain a traffic percentage of 5% for the original link log corresponding to the Gateway Service. Then, with a predefined total score cap of 100 points, we can obtain a traffic score of 5 points for this original link log. Finally, with a predefined traffic weighting coefficient of 25%, we can obtain a traffic rating of 1.25 points for this original link log.

[0038] Optionally, based on the business characteristics corresponding to each original link log, a business importance score corresponding to each original link log is determined, including: based on the business type, user level, and product value corresponding to each original link log, a business importance score corresponding to each original link log is determined.

[0039] Specifically, a first business score corresponding to each original link log can be determined based on the comparison results between the business type and the set business type; a second business score corresponding to each original link log can be determined based on the comparison results between the user level and the set user level; and a third business score corresponding to each original link log can be determined based on the comparison results between the product value and the set value threshold.

[0040] Then, based on the first, second, and third service scores corresponding to each original link log, the original service score corresponding to each original link log can be determined. Next, using the original service scores corresponding to each original link log and a predefined total score cap, the weighted service score corresponding to each original link log can be determined. Finally, the weighted service score corresponding to each original link log can be multiplied by a predefined service weight coefficient to obtain the service importance score corresponding to each original link log.

[0041] Taking the business scoring rules as follows: if the business type matches the set business type, the first business score is 30 points; if the user level matches the set user level, the second business score is 40 points; and if the product value is greater than the set value threshold, the second business score is 30 points. For example, if the set business type is "order submission", the set user level is "high-priority user", and the set value threshold is 1000 yuan, then when the business type corresponding to the original link log is "order submission", the user level is "high-priority user", and the product value is 2999 yuan, the first business score, second business score, and third business score corresponding to the original link log are determined to be 30 points, 40 points, and 30 points, respectively.

[0042] Then, the first, second, and third service scores can be added together to obtain the original service score corresponding to the original link log, which is 100 points. Since the original service score equals the upper limit of the total score, the weighted service score corresponding to the original link log can be determined to be 100 points. Finally, with a predefined service weight coefficient of 25%, the service importance score corresponding to the original link log can be determined to be 25 points.

[0043] The advantage of this setup is that it allows for dynamic adjustment of business importance scores based on business characteristics, facilitating adaptive adjustments to storage strategies and balancing costs and demands.

[0044] Optionally, based on the performance characteristics corresponding to each original link log, a performance anomaly score corresponding to each original link log is determined, including: based on the number of error nodes and query performance corresponding to each original link log, a performance anomaly score corresponding to each original link log is determined.

[0045] Specifically, a first performance score can be determined based on the number of error nodes corresponding to each original link log. A second performance score can be determined based on the query performance corresponding to each original link log. For original link logs with a non-zero number of error nodes, a third performance score can be determined based on the corresponding error type.

[0046] Then, for original link logs with zero error nodes, a weighted performance score is determined based on the first and second performance scores corresponding to that original link log. For original link logs with a non-zero number of error nodes, a weighted performance score is determined based on the first, second, and third performance scores corresponding to that original link log. Finally, a performance anomaly score is determined based on the weighted performance score corresponding to that original link log and predefined performance weight coefficients.

[0047] Taking the performance anomaly scoring rules as follows: 80 points for no error nodes, 10 points for non-slow query performance, and deductions based on error type for error nodes, if the number of error nodes corresponding to an original link log is 0 and the total latency is less than 1000ms, then the original link log is determined to have a first performance score of 80 points and a second performance score of 10 points (i.e., non-slow query performance). Then, based on the first and second performance scores of the original link log, a weighted performance score of 90 points can be determined. Finally, with a predefined performance weight coefficient of 20%, the performance anomaly score corresponding to the original link log can be determined to be 18 points.

[0048] Optionally, based on the topology characteristics corresponding to each original link log, a dependency score is determined for each original link log, including: obtaining the service nodes corresponding to each original link log based on the node call paths corresponding to each original link log, and counting the number of times each service node is depended on; determining the actual average number of dependencies corresponding to each original link log based on the number of dependencies; counting the cumulative number of dependencies of all service nodes corresponding to the target object, and determining the reference average number of dependencies corresponding to the target object based on the cumulative number of dependencies and the total number of service nodes corresponding to the target object; and determining the dependency score for each original link log based on the actual average number of dependencies and the reference average number of dependencies.

[0049] The number of times a service node is depended upon can be understood as the number of times the current service node is depended upon by other service nodes in the target object.

[0050] Specifically, the dependency scoring determination operation can be performed for each original link log as follows: Calculate the average number of dependencies corresponding to each original link log to obtain the actual average number of dependencies corresponding to that original link log. Simultaneously, count the cumulative number of dependencies for all service nodes corresponding to the target object, and divide the cumulative number of dependencies by the total number of service nodes corresponding to the target object to obtain the reference average number of dependencies corresponding to the target object. Then, divide the actual average number of dependencies corresponding to that original link log by the reference average number of dependencies to obtain the dependency criticality ratio corresponding to that original link log. Next, use a ratio score conversion factor to convert the dependency criticality ratio corresponding to that original link log into an original dependency score, and determine the weighted dependency score based on the comparison between the original dependency score and the upper limit of the total score. Finally, multiply the weighted dependency score and the dependency weight coefficient of that original link log to obtain the dependency score corresponding to that original link log.

[0051] For example, assuming the service nodes corresponding to a certain original link log include a gateway service and a payment service, and the gateway service is depended on by 50 other services and the payment service is depended on by 30 other services, then the actual average number of dependencies corresponding to this original link log can be determined to be 40. Then, when the reference average number of dependencies is 15, the dependency criticality ratio corresponding to this original link log can be determined to be approximately 2.67. Subsequently, since a ratio score conversion factor of 30 allows for a reasonable distribution of scores, neither too concentrated nor too dispersed, the ratio score conversion factor can be set to 30 to ensure that most service scores are between 30 and 100. Further, based on the ratio score conversion factor, the original dependency score corresponding to this original link log can be determined to be 80.1, and since 80.1 is less than 100, the dependency score to be processed can still be determined to be 80.1. Finally, the dependency score to be processed can be rounded down to obtain a weighted dependency score of 80 corresponding to the original link log. With a predefined dependency weight coefficient of 30%, the dependency score corresponding to the original link log is determined to be 24.

[0052] The advantage of this setup is that by automatically analyzing the dependencies between service nodes and determining the dependency scores of each original link log based on these dependencies, the accuracy and efficiency of determining dependency scores can be improved.

[0053] S230. Based on the traffic score, business importance score, performance anomaly score, and dependency score corresponding to each original link log, determine the comprehensive score corresponding to each original link log.

[0054] In this step, specifically, each original link log can be used as the current link log in turn, and the traffic score, business importance score, performance anomaly score, and dependency score of the current link log can be added together to obtain the comprehensive score corresponding to the current link log. Then, it can be determined whether the processing of all original link logs is complete. If not, the process of using each original link log as the current link log in turn can be resumed until the processing of all original link logs is complete.

[0055] S240. Determine the initial sampling probability corresponding to each original link log based on the comprehensive score corresponding to each original link log.

[0056] In this step, specifically, the score range of each original link log can be determined based on the comprehensive score corresponding to each original link log, and predefined first, second, third, and fourth score ranges. The lower limit of the first score range is higher than that of the second score range, the lower limit of the second score range is higher than that of the third score range, and the lower limit of the third score range is higher than that of the fourth score range. Then, the original link logs in the first score range can be designated as core link logs, those in the second score range as important link logs, those in the third score range as ordinary link logs, and those in the fourth score range as low-frequency link logs. Finally, the initial sampling probability of the core link logs can be determined as the first probability, the initial sampling probability of the important link logs as the second probability, the initial sampling probability of the ordinary link logs as the third probability, and the initial sampling probability of the low-frequency link logs as the fourth probability. The first probability is higher than the second probability, the second probability is higher than the third probability, and the third probability is higher than the fourth probability.

[0057] Taking a score range of 80-100 points, 60-79 points, 30-59 points, and 0-29 points as an example, with an initial sampling probability of 100% for core link logs, 65% for important link logs, 15% for ordinary link logs, and 2% for low-frequency link logs, if an original link log has a comprehensive score of 68.25 points, then the original link log falls within the second score range, is an important link log, and has an initial sampling probability of 65%.

[0058] S250. Based on the storage space utilization rate corresponding to the target object, and the user level and latency time corresponding to each original link log, determine the probability adjustment factor corresponding to each original link log.

[0059] In this step, specifically, the following probability adjustment factor determination operation can be performed for each raw link log: A first adjustment component is determined based on a comparison between storage space utilization and multiple preset utilization thresholds. A second adjustment component is determined based on a comparison between user level and a set user level. A third adjustment component is determined based on a comparison between latency and a set latency threshold. Then, the first, second, and third adjustment components can be multiplied to obtain the probability adjustment factor corresponding to the current raw link log.

[0060] For example, assuming no storage is suppressed when storage utilization is less than 60%; linearly suppressing storage when storage utilization is between 60% and 70%, with a suppression ratio of 0%-10%; suppressing storage when storage utilization is between 70% and 80%, with a suppression ratio of 10%-30%; and suppressing storage when storage utilization is greater than 80%, with a suppression ratio of 30%-50%, then when the storage utilization corresponding to the target object is 65%, the formula can be used... The calculated inhibition ratio is 5%. Then, based on the above inhibition ratio, the first adjustment component is determined to be 0.95.

[0061] Furthermore, since high-priority users have high business value and are more significantly affected by failures, their user level can be set to high-priority. Then, since a 30% increase represents the optimal cost-benefit balance, when the user level corresponding to a certain original link log is a high-priority user, the second adjustment component corresponding to that original link log can be set to 1.3; when the user level corresponding to a certain original link log is not a high-priority user, the second adjustment component corresponding to that original link log can be set to 1.

[0062] Since the probability of a high-latency anomaly recurring within 24 hours is 35%, the third adjustment component can be determined based on the comparison between the latency time and the set latency time threshold. Furthermore, since a 20% improvement can effectively capture problem recurrence while controlling costs, the third adjustment component can be set to 1.2 when there is a situation within 10 minutes where the latency time exceeds the set latency time threshold.

[0063] S260. Based on the initial sampling probability and probability adjustment factor corresponding to each original link log, determine the target score corresponding to each original link log, and based on the target score corresponding to each original link log, determine the target link log in each original link log.

[0064] In this step, specifically, the initial sampling probability, probability adjustment factor, and total score upper limit corresponding to each original link log can be multiplied to obtain the target score corresponding to each original link log. Then, based on the target scores corresponding to each original link log, the original link logs can be sorted in descending order to obtain the log descending order sorting result, and the first predetermined number of original link logs in the log descending order sorting result are used as the target link logs.

[0065] Taking the sorting of all 1000 raw link logs received within the same millisecond as an example, and the preset storage throughput of the target object is 120 logs / millisecond, the 1000 raw link logs can be sorted from high to low according to the target score, and the top 120 raw link logs can be used as the target link logs.

[0066] The advantage of this setup is that it determines the target score corresponding to each original link log by using the initial sampling probability and probability adjustment factor. Based on the target score, the target link log is then selected from each original link log, avoiding the randomness of target link log selection and ensuring complete consistency of the selected target link logs under the same conditions, thus facilitating subsequent debugging and auditing. Secondly, by setting the number of target link logs to match the preset storage throughput, sampling can be automatically suppressed under high storage pressure, thereby protecting the storage system.

[0067] S270. Determine the business hash value, performance hash value, and topology hash value corresponding to each target link log respectively, and determine the comprehensive hash value corresponding to each target link log respectively based on the business hash value, performance hash value, and topology hash value corresponding to each target link log respectively.

[0068] In this step, specifically, the business hash value corresponding to each target link log can be determined based on the business type, user level, and product value associated with that log. The performance hash value can be determined based on the total latency, number of error nodes, and query performance associated with each target link log. Finally, the topology hash value can be determined based on the node call path associated with each target link log.

[0069] Then, the business hash value, performance hash value, and topology hash value corresponding to each target link log can be concatenated to obtain the comprehensive hash value corresponding to each target link log.

[0070] For example, assuming that the business type corresponding to a certain target link log is order creation, the user level is high-priority user, and the product value is large, then the business combination corresponding to the target link log can be represented as BIZ_ORDER_VIP_HIGHVALUE, and the business hash value corresponding to the target link log can be determined as BIZ_7F8E9D0A1B2C3D4.

[0071] Assuming the total latency corresponding to a certain target link log is 333ms, the number of error nodes is 0, and the query performance is a non-slow query, then the performance combination corresponding to the target link log can be determined as PERF_NORMAL_333MS_NOERROR, and the performance hash value corresponding to the target link log can be determined as PERF_A1B2C3D4E5F6G7.

[0072] Assuming the node call path corresponding to a certain target link log is gateway→order→payment &gateway→order→inventory, the topology combination corresponding to the target link log can be determined as G→O→P&G→O→I, and the topology hash value corresponding to the target link log can be determined as TP_001_A1B2C3D4E5F6.

[0073] Given that the service hash value, performance hash value, and topology hash value corresponding to the target link log are determined to be BIZ_7F8E9D0A1B2C3D4, PERF_A1B2C3D4E5F6G7, and TP_001_A1B2C3D4E5F6 respectively, the comprehensive combination representation corresponding to the target link log can be determined as TP_001_A1B2C3D4E5F6:BIZ_7F8E9D0A1B2C3D4:PERF_A1B2C3D4E5F6G7, and the comprehensive hash value corresponding to the target link log can be determined as FINGERPRINT_7D6C5B4A3E2F1E0D.

[0074] S280. Count the number of times each comprehensive hash value appears within a set time period, and determine the compression ratio of each target link log based on the number of times each comprehensive hash value appears.

[0075] In this step, specifically, the storage strategy, compression ratio, and retention time for each target link log can be determined based on the frequency of occurrence of the comprehensive hash value and the comprehensive score corresponding to each target link log. Alternatively, the storage strategy, compression ratio, and retention time for each target link log can be determined directly based on the frequency of occurrence of the comprehensive hash value corresponding to each target link log.

[0076] For example, suppose the storage policy determination rule is as follows: when the target link log belongs to the core link log or the important link log, and the number of times the comprehensive hash value of the target link log appears within a preset historical time period is greater than a preset maximum number of occurrences threshold, the storage policy for the target link log is determined to be detailed storage with medium compression, a compression ratio of 60%, and a retention time of 30 days; when the number of times the comprehensive hash value of the target link log appears within a preset historical time period is between a preset minimum number of occurrences threshold and a preset maximum number of occurrences threshold, the storage policy for the target link log is determined to be high compression storage, a compression ratio of 75%, and a retention time of 15 days; when the number of times the comprehensive hash value of the target link log appears within a preset historical time period is less than a preset minimum number of occurrences threshold, the storage policy for the target link log is determined to be low compression storage, a compression ratio of 20%, and a retention time of 7 days.

[0077] Based on the above storage strategy determination rules, if the comprehensive hash value of an important link log appears 500 times in the past 24 hours and the preset threshold for the number of occurrences is 100 times / day, the storage strategy for the important link log is determined to be detailed storage with medium compression, a compression ratio of 60%, and a retention period of 30 days.

[0078] S290. Store the logs of each target link according to the compression ratio of each target link log.

[0079] In this step, specifically, the logs of each target link can be compressed according to their compression ratio, and the compressed logs of each target link can be stored.

[0080] Taking the 97 billion raw link logs collected within 14 days as an example, assuming that 10% of the 97 billion raw link logs need to be sampled, and each raw link log occupies 2KB of memory, then the total memory occupied by the raw link logs after sampling is 1940TB.

[0081] Based on this, the link log storage method of this invention can be used to store 97 billion original link logs collected within 14 days, thereby reducing memory usage. Specifically, based on the comprehensive score corresponding to each original link log, 84.39 billion core link logs, 6.305 billion important link logs, 363.75 million ordinary link logs, and 9.7 million low-frequency link logs can be determined. Then, since the number of occurrences of the comprehensive hash value of each core link log within a preset historical time period is greater than a preset maximum frequency threshold, the memory size occupied by each core link log can be set to 0.8KB, and the compression rate can be set to 60%, resulting in a total memory size of 405.072TB for each core link log. Since the number of occurrences of the comprehensive hash value of each important link log within a preset historical time period is also greater than a preset maximum frequency threshold, the memory size occupied by each important link log can be set to 0.8KB, and the compression rate can be set to 60%, resulting in a total memory size of 30.264TB for each important link log.

[0082] Since the overall hash value of each ordinary link log appears less than the preset minimum frequency threshold within the preset historical time period, the memory size occupied by each ordinary link log can be set to 0.8KB, and the compression rate can be set to 20%, resulting in a total memory size of 0.582TB for all important link logs. Since the overall hash value of each low-frequency link log appears between the preset minimum and maximum frequency thresholds within the preset historical time period, the memory size occupied by each low-frequency link log can be set to 2KB, and the compression rate can be set to 75%, resulting in a total memory size of 0.1455TB for all low-frequency link logs.

[0083] Finally, the memory sizes occupied by each type of link log can be added together to obtain the total memory size occupied by all types of link logs, which is 436.0635TB.

[0084] The advantages of this setup are twofold: First, compared to existing technologies that use a uniform strategy to store raw link logs, resulting in insufficient storage of important data and excessive storage of irrelevant data, the technical solution in this embodiment uses different strategies to store various types of link logs. This significantly reduces storage space usage while maximizing the retention of core and important link logs. Second, by sampling and storing low-frequency link logs, important low-frequency business processes can be avoided from being missed. Finally, storing raw link logs using calculable scores and global rankings ensures that each step in the storage of raw link logs has a complete calculation process and rationale, facilitating historical decision review and auditing.

[0085] The technical solution of this embodiment determines the traffic score, service importance score, performance anomaly score, and dependency score corresponding to each original link log by using the state characteristics, service characteristics, performance characteristics, and topology characteristics corresponding to each original link log. Based on these scores, a comprehensive score is determined for each original link log, fully considering the impact of state characteristics, service characteristics, performance characteristics, and topology characteristics on the comprehensive score, thereby improving the accuracy of the comprehensive score determination. Secondly, the probability adjustment factor corresponding to each original link log is determined by the storage space utilization rate corresponding to the target object, as well as the user level and latency time corresponding to each original link log. Based on the initial sampling probability and probability adjustment factor, the target score is determined for each original link log. Finally, the target link log is identified from the original link logs based on the target score, fully considering the impact of storage space utilization rate, user level, and latency time on the target score, thus improving the accuracy of the identified target link log. Finally, by using the business hash value, performance hash value, and topology hash value corresponding to each target link log, the comprehensive hash value corresponding to each target link log is determined. Based on the frequency of occurrence of each comprehensive hash value, the compression ratio of each target link log is determined, which can further compress each target link log, thereby greatly saving storage space.

[0086] Example 3 Figure 3 This is a schematic diagram of a link log storage device according to Embodiment 3 of the present invention. This embodiment is applicable to the sampling and storage of link logs. The link log storage device can be implemented in hardware and / or software and can be configured in electronic devices such as computers.

[0087] like Figure 3 As shown, the link log storage device disclosed in this embodiment includes: The feature extraction module 31 is used to obtain all the original link logs corresponding to the target object, and extract the status features, business features, performance features and topology features from each original link log. The comprehensive scoring module 32 is used to determine the comprehensive score corresponding to each original link log based on the status characteristics, service characteristics, performance characteristics and topology characteristics corresponding to each original link log respectively; The log storage module 33 is used to determine the target link log in each original link log according to the comprehensive score corresponding to each original link log, and to store each target link log.

[0088] The technical solution in this embodiment, through the cooperation of the feature extraction module 31, the comprehensive scoring module 32 and the log storage module 33, solves the problem that directly storing all the original link logs in memory leads to low storage efficiency of the original link logs and wastes storage space, thereby improving the storage efficiency of the original link logs and saving storage space.

[0089] Optional, the comprehensive scoring module 32 includes: The traffic scoring determination unit is used to determine the traffic score corresponding to each original link log based on the status characteristics corresponding to each original link log. The business scoring determination unit is used to determine the business importance score corresponding to each original link log based on the business characteristics corresponding to each original link log. The performance scoring determination unit is used to determine the performance anomaly score corresponding to each original link log based on the performance characteristics corresponding to each original link log. The dependency scoring determination unit is used to determine the dependency score corresponding to each original link log based on the topological features corresponding to each original link log. The comprehensive scoring unit is used to determine the comprehensive score corresponding to each original link log based on the traffic score, business importance score, performance anomaly score, and dependency score corresponding to each original link log.

[0090] Optionally, the traffic scoring determination unit is specifically used for: obtaining the root service nodes corresponding to each original link log based on the root service node name corresponding to each original link log, and counting the number of times each root service node is called within a preset historical time period; obtaining all service nodes corresponding to the target object, and counting the cumulative number of times all service nodes are called within the historical time period; determining the traffic percentage corresponding to each original link log based on the number of times it is called and the cumulative number of times it is called; and determining the traffic score corresponding to each original link log based on the traffic percentage corresponding to each original link log.

[0091] Optionally, the business scoring determination unit is specifically used to: determine the business importance score corresponding to each original link log based on the business type, user level, and product value corresponding to each original link log.

[0092] Optionally, the performance scoring determination unit is specifically used to: determine the performance anomaly score corresponding to each original link log based on the number of error nodes and query performance corresponding to each original link log.

[0093] Optionally, the dependency scoring determination unit is specifically used for: obtaining the service nodes corresponding to each original link log based on the node call paths corresponding to each original link log, and counting the number of times each service node is depended on; determining the actual average number of dependencies corresponding to each original link log based on the number of dependencies corresponding to each original link log; counting the cumulative number of dependencies of all service nodes corresponding to the target object, and determining the reference average number of dependencies corresponding to the target object based on the cumulative number of dependencies and the total number of service nodes corresponding to the target object; and determining the dependency score corresponding to each original link log based on the actual average number of dependencies and the reference average number of dependencies.

[0094] Optionally, the log storage module 33 includes: The sampling probability determination unit is used to determine the initial sampling probability corresponding to each original link log based on the comprehensive score corresponding to each original link log. The adjustment factor determination unit is used to determine the probability adjustment factor corresponding to each original link log based on the storage space utilization rate corresponding to the target object, as well as the user level and latency time corresponding to each original link log. The target score determination unit is used to determine the target score corresponding to each original link log based on the initial sampling probability and probability adjustment factor corresponding to each original link log respectively; The target log determination unit is used to determine the target link log from each original link log based on the target score corresponding to each original link log. The hash value component determination unit is used to determine the business hash value, performance hash value, and topology hash value corresponding to each target link log respectively; The comprehensive hash value determination unit is used to determine the comprehensive hash value corresponding to each target link log based on the business hash value, performance hash value and topology hash value corresponding to each target link log respectively; The compression ratio determination unit is used to count the number of times each comprehensive hash value appears within a set time period, and determine the compression ratio of each target link log based on the number of times each comprehensive hash value appears; The log compression storage unit is used to store the logs of each target link according to the compression ratio of each target link log.

[0095] The link log storage device provided in this embodiment of the invention can execute the link log storage method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution. Content not described in detail in this embodiment can be referred to the description in any method embodiment of this application.

[0096] Example 4 Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the present invention is shown. For example... Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0097] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0098] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the link log storage method.

[0099] In some embodiments, the link log storage method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the link log storage method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the link log storage method by any other suitable means (e.g., by means of firmware).

[0100] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0104] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0105] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0106] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0107] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A link log storage method, characterized in that, The method includes: Obtain all original link logs corresponding to the target object, and extract the status characteristics, service characteristics, performance characteristics, and topology characteristics from each of the original link logs; Based on the status characteristics, service characteristics, performance characteristics, and topology characteristics corresponding to each of the original link logs, a comprehensive score corresponding to each of the original link logs is determined. Based on the comprehensive score corresponding to each of the original link logs, the target link logs are determined from each of the original link logs and stored.

2. The method according to claim 1, characterized in that, The step of determining a comprehensive score corresponding to each of the original link logs based on the status characteristics, service characteristics, performance characteristics, and topology characteristics corresponding to each of the original link logs includes: Based on the status characteristics corresponding to each of the original link logs, a traffic score corresponding to each of the original link logs is determined. Based on the business characteristics corresponding to each of the original link logs, a business importance score corresponding to each of the original link logs is determined. Based on the performance characteristics corresponding to each of the original link logs, a performance anomaly score corresponding to each of the original link logs is determined. Based on the topological features corresponding to each of the original link logs, a dependency score corresponding to each of the original link logs is determined. Based on the traffic score, business importance score, performance anomaly score, and dependency score corresponding to each of the original link logs, a comprehensive score corresponding to each of the original link logs is determined.

3. The method according to claim 2, characterized in that, The status features include at least the root service node name; determining the traffic score corresponding to each of the original link logs based on the status features corresponding to each of the original link logs includes: Based on the root service node name corresponding to each of the original link logs, obtain the root service node corresponding to each of the original link logs, and count the number of times each of the root service nodes is called within a preset historical time period. Obtain all service nodes corresponding to the target object, and count the cumulative number of times the all service nodes were called within the historical time period; Based on the number of times each original link log is called and the cumulative number of times it is called, the traffic percentage corresponding to each original link log is determined. Based on the traffic percentage corresponding to each of the original link logs, a traffic score is determined for each of the original link logs.

4. The method according to claim 2, characterized in that, The business characteristics include business type, user level, and product value; determining the business importance score corresponding to each of the original link logs based on the business characteristics corresponding to each of the original link logs includes: Based on the business type, user level, and product value corresponding to each of the original link logs, a business importance score is determined for each of the original link logs.

5. The method according to claim 2, characterized in that, The performance characteristics include the number of error-reporting nodes and query performance; the step of determining the performance anomaly score corresponding to each of the original link logs based on the performance characteristics corresponding to each of the original link logs includes: Based on the number of error nodes and query performance corresponding to each of the original link logs, a performance anomaly score is determined for each of the original link logs.

6. The method according to claim 2, characterized in that, The topology features include node call paths; determining the dependency score corresponding to each of the original link logs based on the topology features corresponding to each of the original link logs includes: Based on the node call path corresponding to each of the original link logs, obtain the service node corresponding to each of the original link logs, and count the number of times each service node is depended on. Based on the number of times each original link log is depended on, the actual average number of dependencies corresponding to each original link log is determined. The cumulative number of times all service nodes corresponding to the target object are depended on is counted, and a reference average number of dependencies corresponding to the target object is determined based on the cumulative number of dependencies and the total number of service nodes corresponding to the target object. Based on the actual average number of dependencies corresponding to each of the original link logs, and the reference average number of dependencies, a dependency score corresponding to each of the original link logs is determined.

7. The method according to claim 1, characterized in that, The step of determining the target link log in each of the original link logs based on the comprehensive score corresponding to each of the original link logs includes: Based on the comprehensive score corresponding to each of the original link logs, determine the initial sampling probability corresponding to each of the original link logs; Based on the storage space utilization rate corresponding to the target object, and the user level and latency time corresponding to each of the original link logs, determine the probability adjustment factor corresponding to each of the original link logs; Based on the initial sampling probability and probability adjustment factor corresponding to each of the original link logs, the target score corresponding to each of the original link logs is determined. Target link logs are determined from each of the original link logs based on the target score corresponding to each of the original link logs.

8. The method according to claim 1, characterized in that, The storage of logs for each of the target links includes: Determine the service hash value, performance hash value, and topology hash value corresponding to each of the target link logs; Based on the business hash value, performance hash value and topology hash value corresponding to each of the target link logs, a comprehensive hash value corresponding to each of the target link logs is determined. The number of occurrences of each comprehensive hash value within a set time period is counted, and the compression ratio of each target link log is determined based on the number of occurrences of each comprehensive hash value. The target link logs are stored according to their compression ratios.

9. A link log storage device, characterized in that, The device includes: The feature extraction module is used to obtain all the original link logs corresponding to the target object, and extract the status features, service features, performance features and topology features from each of the original link logs; The comprehensive scoring module is used to determine the comprehensive score corresponding to each of the original link logs based on the status characteristics, service characteristics, performance characteristics, and topology characteristics corresponding to each of the original link logs. The log storage module is used to determine the target link log in each of the original link logs based on the comprehensive score corresponding to each of the original link logs, and to store each of the target link logs.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the link log storage method according to any one of claims 1-8.

Images