Activity traffic recording method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides an activity traffic recording method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, the queue time factor and the queue size factor are calculated according to the queue capacity of the recording data queue, the data size of the current cache of the recording data queue and the time information of the earliest recording data in the recording data queue; adjusting a recording collection rate according to a preset basic collection rate, a queue time factor and a queue size factor, and recording service request data and service response data of a preset downstream service according to the recording collection rate to obtain recording data, the recording collection rate is dynamically determined according to the queue capacity of the recording data queue, the size of the data currently cached by the recording data queue and the time information, so that the influence of the fixed recording collection rate on the downstream service when the downstream service is busy is effectively reduced, and the influence of an active traffic recording scheme on the downstream service performance and stability is reduced; and the activity traffic recording effect is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and product for recording activity traffic. Background Art

[0002] On large-scale live streaming platforms, various live events (such as festivals and interactive games) carry the core business needs of the platforms. These events are usually modular in design and require frequent feature iteration and updates.

[0003] To ensure the smooth progress of the event, platform services need to be tested using traffic recording and playback. However, live events are characterized by high concurrency and strong real-time nature. Traditional event traffic recording solutions generally rely on fixed acquisition rates. This high volume of recorded traffic can easily impact the data processing efficiency of downstream services, reducing their performance and stability, resulting in poor event traffic recording results. Summary of the Invention

[0004] The embodiments of the present application provide an activity traffic recording method, apparatus, device, storage medium and product to solve the technical problems in related technologies that activity traffic recording schemes easily affect the performance and stability of downstream services and the activity traffic recording effect is poor. It can effectively reduce the impact of activity traffic recording schemes on the performance and stability of downstream services and improve the activity traffic recording effect.

[0005] In a first aspect, an embodiment of the present application provides a method for recording activity traffic, including: Determining a queue capacity of a recording data queue, a size of data currently cached in the recording data queue, and time information of the earliest recorded data in the recording data queue, and calculating a queue time factor and a queue size factor based on the queue capacity, the data size, and the time information; Adjusting the recording acquisition rate according to the preset basic acquisition rate, the queue time factor, and the queue size factor; The service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate to obtain recorded data.

[0006] In a second aspect, an embodiment of the present application provides an activity traffic recording device, including a queue analysis module, a rate determination module, and a traffic recording module, wherein: The queue analysis module is configured to determine a queue capacity of a recording data queue, a size of data currently cached in the recording data queue, and time information of the earliest recorded data in the recording data queue, and calculate a queue time factor and a queue size factor based on the queue capacity, the data size, and the time information; The rate determination module is configured to adjust the recording acquisition rate according to a preset basic acquisition rate, the queue time factor and the queue size factor; The traffic recording module is configured to record the service request data and service response data of the preset downstream service according to the recording acquisition rate to obtain recording data.

[0007] In a third aspect, an embodiment of the present application provides an activity traffic recording device, including: a memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the activity traffic recording method as described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to perform the activity traffic recording method as described in the first aspect.

[0009] In the fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the activity traffic recording method described in the first aspect.

[0010] The embodiment of the present application calculates the queue time factor and the queue size factor according to the queue capacity of the recording data queue, the data size currently cached in the recording data queue, and the time information of the earliest recorded data in the recording data queue, adjusts the recording acquisition rate according to the preset basic acquisition rate, the queue time factor, and the queue size factor, and records the service request data and service response data of the preset downstream service according to the recording acquisition rate to obtain the recording data, and dynamically determines the recording acquisition rate according to the queue capacity of the recording data queue, the data size currently cached in the recording data queue, and the time information, thereby effectively reducing the impact of the fixed recording acquisition rate on the downstream service when the downstream service is busy, reducing the impact of the active traffic recording solution on the performance and stability of the downstream service, and improving the active traffic recording effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of the first activity traffic recording method provided by an embodiment of the present application; Figure 2 This is a flow chart of a second method for recording activity traffic provided by an embodiment of the present application; Figure 3 This is a flowchart of a third method for recording activity traffic provided by an embodiment of the present application; Figure 4 This is a schematic structural diagram of an activity flow recording device provided in an embodiment of the present application; Figure 5 This is a structural diagram of an activity flow recording device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] To further clarify the objectives, technical solutions, and advantages of this application, specific embodiments of the present application are described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are intended only to illustrate this application and are not intended to limit it. It should also be noted that, for ease of description, the drawings only illustrate portions relevant to this application, not all of them. Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts depict the various operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The processes described above may terminate upon completion, but may also include additional steps not shown in the accompanying drawings. These processes may correspond to methods, functions, procedures, subroutines, subprograms, and the like.

[0013] The active traffic recording method provided in this application can be applied to the active traffic recording scenarios of downstream services. It aims to dynamically determine the recording acquisition rate based on the queue capacity of the recording data queue, the data size currently cached in the recording data queue, and the time information, thereby reducing the impact of the active traffic recording solution on the performance and stability of downstream services and improving the active traffic recording effect.

[0014] Existing activity traffic recording solutions generally record at a fixed acquisition rate. However, downstream services are characterized by high concurrency and strong real-time requirements. Traffic recording solutions based on fixed acquisition rates may affect the performance and stability of downstream services. Furthermore, during the traffic recording process, data may be lost or incomplete due to network fluctuations, service anomalies, and other reasons. Based on this, an activity traffic recording method according to an embodiment of the present application is provided to address the technical problem that existing activity traffic recording solutions easily affect the performance and stability of downstream services, resulting in poor activity traffic recording results.

[0015] Figure 1 A flowchart of the first activity traffic recording method provided in an embodiment of the present application is given. The activity traffic recording method provided in an embodiment of the present application can be executed by an activity traffic recording device, which can be implemented by hardware and / or software and integrated into the activity traffic recording equipment.

[0016] The following describes the method for recording activity traffic using an activity traffic recording device as an example. Figure 1 , the activity traffic recording method includes: S110: Determine the queue capacity of the recording data queue, the data size currently cached in the recording data queue, and the time information of the earliest recorded data in the recording data queue, and calculate the queue time factor and queue size factor according to the queue capacity, data size, and time information.

[0017] The active traffic recording method provided by the present application can, when recording traffic for downstream services, mount the recorded application package on the downstream service that needs to be recorded, create an application identifier corresponding to the application package to associate the corresponding application, and record the service request data and service response data of the application. When recording traffic for downstream services, the collected recording data can be saved to a preset recording data queue. The recording data in the recording data queue will wait to be uploaded to a preset storage location (such as a preset database). After the recording data is successfully uploaded, the corresponding recording data in the recording data queue can be deleted. When starting to record traffic for downstream services, the service request data and service response data of the preset downstream service can be recorded based on the preset basic collection rate to obtain the recording data. Among them, the queue capacity (queueCap) of the recording data queue can be a preset capacity. For example, the queue capacity of the recording data queue can be set to 2048. The unit of the queue capacity is "pieces". One recording data corresponds to one storage space in the recording data queue.

[0018] Exemplarily, the queue capacity of the recording data queue, the size of data currently cached in the recording data queue, and the time information of the earliest recorded data in the recording data queue are determined according to a preset time interval. The earliest recorded data is the earliest recorded data in the recording data queue, and the time information of the earliest recorded data is the time distance between the recording time of the earliest recorded data in the recording data queue and the current time. The recording time may be the recording start time of the recorded data.

[0019] Optionally, after determining the data size currently cached in the recording data queue, it is possible to detect whether the recording data queue is full (i.e., detect whether the data size currently cached in the recording data queue is equal to the queue capacity). If the recording data queue is full, traffic recording can be stopped until there is a vacant position in the recording data queue (i.e., the data size currently cached in the recording data queue is less than the queue capacity), thereby reducing the computing resources occupied by traffic recording and improving the smoothness of system services.

[0020] In one embodiment, a queue time factor and a queue size factor are calculated based on the queue capacity determined above, the size of the data currently cached in the recording data queue, and time information. The queue time factor can be used to reflect the impact of the time information of the earliest recorded data in the recording data queue on the recording acquisition rate. The longer the time length corresponding to the time information of the earliest recorded data in the recording data queue, the more recorded data that has been unprocessed for a long time in the recording data queue. Therefore, the recording acquisition rate needs to be appropriately reduced to reduce the accumulation of recorded data in the recording data queue and ensure sufficient time to process the recorded data in the recording data queue. Furthermore, the longer the time length corresponding to the time information of the earliest recorded data in the recording data queue, the greater the impact on the recording acquisition rate (e.g., the greater the reduction in the recording acquisition rate). The queue size factor can be used to reflect the impact of the size of the data currently cached in the recording data queue on the recording acquisition rate. The larger the size of the data currently cached in the recording data queue, the more recorded data awaiting processing in the recording data queue. Similarly, the recording acquisition rate needs to be appropriately reduced to reduce the amount of recorded data awaiting processing in the recording data queue and ensure timely processing of the recorded data in the recording data queue. Similarly, the larger the size of the data currently cached in the recording data queue, the greater the impact on the recording acquisition rate (e.g., the greater the reduction in the recording acquisition rate). The recording acquisition rate is the rate at which the service request data and service response data of the preset downstream service are recorded.

[0021] S120: Adjusting the recording acquisition rate according to the preset basic acquisition rate, the queue time factor, and the queue size factor.

[0022] Exemplarily, the recording acquisition rate is adjusted according to a preset basic acquisition rate (the value range of the preset basic acquisition rate can be set to 0~1, for example, the preset basic acquisition rate can be set to 0.5) and the queue time factor and queue size factor determined above, wherein the recording acquisition rate is negatively correlated with the queue time factor and the queue size factor.

[0023] For example, the queue time factor and the queue size factor can be multiplied to obtain a multiplication result, a preset coefficient (e.g., 1) can be subtracted from the multiplication result to obtain a subtraction result, and the subtraction result can be multiplied by a preset basic acquisition rate to obtain the recording acquisition rate. Alternatively, the ratio of the sum of the queue time factor and the queue size factor to the sum of the time factor threshold and the size factor threshold can be calculated, and the difference between the preset coefficient and the ratio can be multiplied by the preset basic acquisition rate to obtain the recording acquisition rate. The recording acquisition rate determined above is the dynamically adjusted recording acquisition rate. That is, after each determination of the latest queue time factor and queue size factor, the recording acquisition rate is dynamically adjusted based on the preset basic acquisition rate and the latest determined queue time factor and queue size factor to adapt to dynamic changes in downstream service traffic. The acquisition rate can be the amount of service request data and service response data recorded for a preset downstream service per unit time (e.g., the number of service request data and service response data collected per minute).

[0024] S130: Recording service request data and service response data of a preset downstream service according to a recording acquisition rate to obtain recorded data.

[0025] Exemplarily, the service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate determined above to obtain the recording data. In the present application, if the downstream service suddenly encounters a large traffic volume during traffic recording, the CPU usage of the traffic recording will cause the storage efficiency of the recording data queue to decrease and / or the accumulation of cached recording data to increase. At this time, the recording acquisition rate will be automatically reduced to avoid excessive CPU usage of the traffic recording due to the traffic surge, so as to allocate more computing resources to the business system, ensure the maximum availability of the business system, and reduce the impact of traffic recording on business functions.

[0026] In the above, the queue time factor and queue size factor are calculated according to the queue capacity of the recording data queue, the data size currently cached in the recording data queue, and the time information of the earliest recorded data in the recording data queue, the recording acquisition rate is adjusted according to the preset basic acquisition rate, the queue time factor, and the queue size factor, and the service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate to obtain the recording data. The recording acquisition rate is dynamically determined according to the queue capacity of the recording data queue, the data size currently cached in the recording data queue, and the time information, which effectively reduces the impact of the fixed recording acquisition rate on the downstream service when the downstream service is busy, reduces the impact of the active traffic recording solution on the performance and stability of the downstream service, and improves the active traffic recording effect.

[0027] Based on the above embodiments, Figure 2A flowchart of a second method for recording activity traffic provided by an embodiment of the present application is provided. This method is a specific implementation of the above method for recording activity traffic. Figure 2 , the activity traffic recording method includes: S210: Determine the queue capacity of the recording data queue, the data size currently cached in the recording data queue, and the time information of the earliest recorded data in the recording data queue, and calculate the queue time factor and queue size factor based on the queue capacity, data size, and time information.

[0028] In one embodiment, the queue time factor can be calculated based on the time information, and the queue size factor can be calculated based on the data size currently cached in the recording data queue and the queue capacity of the recording data queue. For example, the natural exponential function result of the natural exponential function acting on the negative number of the time information is calculated, the difference between the preset time factor base and the natural exponential function result is calculated, and the difference is used as the queue time factor. The influence of time information on the recording acquisition rate increases exponentially. Optionally, the active traffic recording method provided by the present application calculates the queue time factor and the queue size factor based on the queue capacity, the data size currently cached in the recording data queue and the time information, including: it can be the natural exponential function result of calculating the first ratio of the negative number of the natural exponential function acting on the time information to the preset time base conversion value, calculating the first difference between the preset time factor base and the natural exponential function result, and determining the first difference as the queue time factor; calculating the second ratio of the data size to the queue capacity of the recording data queue, and determining the second ratio as the queue size factor.

[0029] Optionally, the queue time factor can be determined based on the following queue time factor calculation formula: Queue Time Factor = a1 - Math.exp(-queueTime / b), where a1 is a preset time factor base, for example, a1 = 1, queueTime is time information, b is a preset time base conversion value used to convert the time information to a preset time unit, for example, b = 1000000000.0 to convert the time information to nanoseconds, and Math.exp(-queueTime / b) is the result of a natural exponential function applied to the first ratio of the negative of the time information to the preset time base conversion value. The queue size factor can be determined based on the following queue size factor calculation formula: Queue Size Factor = queueSize / queueCap, where queueSize is the size of the data currently cached in the recording data queue, and queueCap is the queue capacity. This application calculates the queue time factor based on time information, and calculates the queue size factor based on the data size currently cached in the recording data queue and the queue capacity of the recording data queue, accurately determining the queue time factor and the queue size factor. This can more accurately determine the recording acquisition rate based on the impact of the time information of the earliest recorded data in the recording data queue and the data size currently cached in the recording data queue on the recording acquisition rate, thereby improving the traffic recording effect.

[0030] S220: Adjusting the recording acquisition rate according to the preset basic acquisition rate, the queue time factor, and the queue size factor.

[0031] In one embodiment, the activity traffic recording method provided in the present application adjusts the recording acquisition rate according to a preset basic acquisition rate, a queue time factor, and a queue size factor. The rate impact factor can be determined according to the queue time factor and the queue size factor, and the rate impact factor can be multiplied by the preset basic acquisition rate to obtain the recording acquisition rate. Alternatively, the rate impact factor can be multiplied by the preset basic acquisition rate and a preset attenuation factor to obtain the recording acquisition rate, wherein the rate impact factor can be the difference between a preset coefficient (for example, 1) and the sum of the queue time factor and the queue size factor, or the difference between the preset coefficient and the product of the queue time factor and the queue size factor.

[0032] Optionally, the recording acquisition rate can be determined based on the queue time factor, the queue size factor, the preset queue factor base, and the preset basic acquisition rate. For example, a first product of the queue time factor and the queue size factor is calculated, a second difference between the preset queue factor base and the first product is calculated, a fifth product of the preset basic acquisition rate and the second difference is calculated, and the fifth product is determined as the recording acquisition rate. Optionally, the recording acquisition rate can be expressed as recording acquisition rate = preset basic acquisition rate * (preset queue factor base - queue time factor * queue size factor). Alternatively, a first product of the queue time factor and the queue size factor is calculated, a second difference between the preset queue factor base and the first product is calculated, and a second product of the preset basic acquisition rate, the second difference, and the preset attenuation factor is calculated, and the second product is determined as the recording acquisition rate, wherein the preset queue factor base can be set to 1. Optionally, the recording acquisition rate can be expressed as recording acquisition rate = preset basic acquisition rate * (preset queue factor base - queue time factor * queue size factor) * preset attenuation factor. The preset attenuation factor can be less than or equal to 1. The preset attenuation factor can be used to maintain smooth changes in the recording acquisition rate, reducing the situation where sudden changes in the recording acquisition rate cause excessive changes in traffic recording, affecting the quality of traffic recording. This application dynamically determines the recording acquisition rate based on the queue time factor, queue size factor, preset queue factor base, preset basic acquisition rate, and preset attenuation factor, thereby reducing the impact of a fixed recording acquisition rate on downstream services when downstream services are busy, thereby improving the traffic recording effect.

[0033] S230: Recording service request data and service response data of a preset downstream service according to a recording acquisition rate to obtain recorded data.

[0034] In one possible embodiment, the activity traffic recording method provided in the present application may also perform range limiting processing on the recording acquisition rate according to a preset minimum rate and a preset maximum rate before recording the service request data and service response data of the preset downstream service according to the recording acquisition rate to obtain the recording data.

[0035] Exemplarily, after adjusting the recording acquisition rate based on a preset base acquisition rate, a queue time factor, and a queue size factor, a determination is made as to whether the recording acquisition rate is within a rate range corresponding to a preset minimum rate and a preset maximum rate. If the recording acquisition rate is within the rate range corresponding to the preset minimum rate and the preset maximum rate, the recording acquisition rate is retained. If the recording acquisition rate exceeds the rate range corresponding to the preset minimum rate and the preset maximum rate, if the recording acquisition rate is less than the preset minimum rate, the recording acquisition rate is updated to the value corresponding to the preset minimum rate; if the recording acquisition rate is greater than the preset maximum rate, the recording acquisition rate is updated to the value corresponding to the preset maximum rate. For example, range limiting the recording acquisition rate can be expressed as: D = max(preset minimum rate, min(preset maximum rate, D′)), where D′ is the recording acquisition rate before range limiting, and D is the recording acquisition rate after range limiting. For example, the preset maximum rate can be set to 1 to ensure the maximum processable acquisition rate, and the preset minimum rate can be set to 0.1 to ensure real-time data acquisition.

[0036] Accordingly, the service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate to obtain the recorded data, which can be obtained by recording the service request data and service response data of the preset downstream service according to the recording acquisition rate after range restriction processing. This application performs range restriction processing on the recording acquisition rate by presetting a minimum rate and a preset maximum rate, limiting the recording acquisition rate to a reasonable range, and ensuring smooth traffic recording of the downstream service.

[0037] S240: Save the recorded data to the recorded data queue, and call the storage service to save the recorded data in the recorded data queue to a preset storage location.

[0038] In one possible embodiment, the active traffic recording method provided in the present application calls a storage service to save the recorded data in the recording data queue to a preset storage location, including: in the event that the call to the storage service fails, determining a retry time interval based on the number of storage service call retries; and re-calling the storage service based on the retry time interval to save the recorded data in the recording data queue to a preset storage location.

[0039] Exemplarily, after the recorded data is saved to the recorded data queue, the recorded data in the recorded data queue can be saved to a preset storage location by calling the storage service. After successfully saving the recorded data in the recorded data queue to the preset storage location, continue to save the next recorded data in the recorded data queue.

[0040] In one embodiment, when the call to the storage service fails, the number of storage service call retries to call the storage service to save the recorded data is determined, and the retry time interval is determined based on the number of storage service call retries. Among them, the more the storage service call retries, the longer the corresponding retry time interval. After determining the retry time interval, the storage service can be re-called according to the retry time interval to save the recorded data in the recorded data queue to a preset storage location, that is, after waiting for the currently determined retry time interval, the storage service is re-called to save the recorded data in the recorded data queue to a preset storage location. The present application determines the retry time interval according to the number of storage service call retries, and re-calls the storage service to store the recorded data according to the retry time interval. While ensuring that the recorded data is correctly stored in the preset storage location, the retry time interval is dynamically adjusted, which can effectively reduce the retry failure rate of recorded data storage, reduce the occupancy of computing resources by data storage, and improve the integrity and reliability of the storage of recorded data.

[0041] In one possible embodiment, the activity traffic recording method provided in the present application determines the retry time interval based on the number of storage service call retries. The method may be to calculate the third product of the preset basic delay time, the Nth power of the preset backoff factor, and the random jitter factor, and determine the third product as the retry time interval. Wherein, N is the number of storage service call retries. For example, the retry time interval can be determined by the following retry time interval determination formula: retry time interval = preset basic delay time * (preset backoff factor ^ number of storage service call retries) * random jitter factor, where the preset backoff factor can range from 1 to 5, for example, the preset backoff factor can be set to 1.5, which is used to control the growth rate of the retry time interval, and the random jitter factor can be set to the sum of the basic jitter coefficient (for example, 1) and the random jitter coefficient. The random jitter coefficient can be a randomly generated value between 0 and 0.2, which can reduce the situation where data storage congestion occurs due to multiple storage services calling retries at the same time, thereby improving the efficiency of recorded data storage. This application can effectively reduce the retry failure rate of recorded data storage, reduce the computing resources occupied by data storage, and improve the integrity and reliability of recorded data storage by dynamically determining the retry time interval based on the preset basic delay time, the preset backoff factor, the number of storage service call retries, and the random jitter factor.

[0042] In one possible embodiment, the active traffic recording method provided by the present application further includes, after re-calling the storage service according to the retry time interval to save the recorded data in the recorded data queue to a preset storage location: when the number of storage service call retries reaches the preset number of retries, determining the preset proportional request response time of the storage service call; when the preset proportional request response time is greater than the preset response time threshold, down-converting the recording acquisition rate, and recording the service request data and service response data of the preset downstream service according to the recording acquisition rate after the down-conversion process to obtain the recorded data.

[0043] For example, when the number of storage service call retries reaches a preset number of retries, a preset percentage of request response times for the storage service calls (e.g., the 99th percentile request response time) is determined, and a determination is made as to whether the preset percentage of request response times is within a preset response time threshold. The phrase "the preset percentage of request response times is within the preset response time threshold" may be interpreted as meaning that the response time of 99% of the multiple storage service calls does not exceed the preset response time threshold.

[0044] In one embodiment, when a preset proportion of request response time exceeds a preset response time threshold, the recording acquisition rate is down-converted, and the service request data and service response data of a preset downstream service are recorded at the down-converted recording acquisition rate to obtain recorded data. This reduces the impact of the active traffic recording solution on the performance and stability of downstream services when the storage service processing capacity decreases, thereby improving the stability of system services. Optionally, down-converting the recording acquisition rate can include calculating a fourth product of the recording acquisition rate and a preset down-converting coefficient, and determining the fourth product as the down-converted recording acquisition rate. The fourth product is less than 1, for example, the fourth product can be set to 0.8.

[0045] In one embodiment, a multi-level recovery detection strategy can be set. That is, after the recording acquisition rate is frequency-reduced, a period of time (e.g., 10 seconds, 3 minutes, 10 minutes) is observed to determine whether the preset proportion of request response time has returned to within the preset response time threshold. If the preset proportion of request response time has not returned to within the preset response time threshold, the frequency is further reduced based on the already reduced frequency (new recording acquisition rate = current recording acquisition rate * fourth product). If the preset proportion of request response time returns to within the preset response time threshold, the original recording acquisition rate is restored, and the frequency reduction of the recording acquisition rate is canceled. This application reduces the data acquisition rate when data storage efficiency is low by frequency-reducing the recording acquisition rate when the preset proportion of request response time is greater than the preset response time threshold, thereby reducing data accumulation in the recording data queue and reducing the computing resources occupied by data storage, thereby improving the stability of system services.

[0046] In the above, by calculating the queue time factor and queue size factor based on the queue capacity of the recorded data queue, the size of the data currently cached in the recorded data queue, and the time information of the earliest recorded data in the recorded data queue, adjusting the recording acquisition rate based on the preset basic acquisition rate, the queue time factor, and the queue size factor, and recording the service request data and service response data of the preset downstream service based on the recording acquisition rate to obtain recorded data, and dynamically determining the recording acquisition rate based on the queue capacity of the recorded data queue, the size of the data currently cached in the recorded data queue, and the time information, this effectively reduces the impact of a fixed recording acquisition rate on downstream services when the downstream services are busy, reduces the impact of the active traffic recording solution on the performance and stability of downstream services, and improves the effectiveness of active traffic recording. Furthermore, by determining the retry interval based on the number of storage service call retries and re-calling the storage service to store the recorded data based on the retry interval, the retry interval is dynamically adjusted while ensuring that the recorded data is correctly stored in the preset storage location. This effectively reduces the retry failure rate of recorded data storage, reduces the computing resource usage of data storage, and improves the integrity and reliability of recorded data storage.

[0047] Based on the above embodiments, Figure 3 A flowchart of a third method for recording activity traffic provided by an embodiment of the present application is provided. This method is a specific implementation of the above method for recording activity traffic. Figure 3 , the activity traffic recording method includes: S310: Determine the queue capacity of the recording data queue, the data size currently cached in the recording data queue, and the time information of the earliest recorded data in the recording data queue, and calculate the queue time factor and queue size factor based on the queue capacity, data size, and time information.

[0048] S320: Adjust the recording acquisition rate according to the preset basic acquisition rate, the queue time factor, and the queue size factor.

[0049] S330: Recording service request data and service response data of a preset downstream service according to a recording acquisition rate to obtain recording data.

[0050] S340: Save the recorded data to the recorded data queue, and call the storage service to save the recorded data in the recorded data queue to a preset storage location.

[0051] S350: Acquire test response data generated by the test environment in response to the test request data.

[0052] S360: Generate first identification information according to the data request type and the test parameter request body of the test request data, and accurately match the service request data in a preset storage location according to the first identification information.

[0053] In one embodiment, after calling the storage service to save the recorded data in the recorded data queue to a preset storage location, the stored recorded data can be used to replay traffic, thereby testing the platform service (such as the platform service after functional iteration and update), comparing the differences between the service response data (at this time, the service response data serves as the reference response data) and the test response data, and analyzing the reasons to ensure that the business system is stably iterated and launched.

[0054] For example, after a test environment for testing the platform service is built, test request data is input into the test environment so that the platform service analyzes and processes the test request data and outputs corresponding test response data. The test request data can be obtained from a preset storage service or generated based on test requirements.

[0055] Obtain multiple test request data and multiple test response data in which the test environment responds to the test request data, and determine the data request type (downType, such as the data request type for TRPC, DB, Redis and other interfaces) and the test parameter request body (requesetBody) of each test request data, and generate first identification information (such as a hash value) according to the data request type and the test parameter request body of the test request data, and accurately match the service request data in a preset storage location according to the first identification information.

[0056] Optionally, accurately matching the service request data in the preset storage location may involve traversing and calculating the second identification information (e.g., a hash value) corresponding to each piece of service request data in the preset storage location. When a match is found with the second identification information that is consistent with the first identification information, the corresponding service request data may be added to the matching result list corresponding to the test request data. After traversing each piece of service request data, if a piece of service request data exists in the matching result list corresponding to the test request data, the test request data may be considered accurately matched successfully, and the service request data that accurately matched successfully may be used as the target service request data. If no service request data exists in the matching result list corresponding to the test request data, or if multiple pieces of service request data exist, the test request data may be considered accurately matched unsuccessfully.

[0057] S370: When the exact match succeeds, the service response data corresponding to the target service request data determined by the exact match is determined as reference response data.

[0058] S380: When the exact match fails, fuzzy match the service request data in a preset storage location according to the test request data, and determine the service response data corresponding to the target service request data determined by the fuzzy match as the reference response data.

[0059] Exemplarily, if the exact match of the test request data succeeds, the service response data corresponding to the target service request data determined by the exact match is determined as the reference response data. If the exact match of the test request data fails, a fuzzy match is performed on the service request data in a preset storage location based on the test request data, and the service request data that successfully undergoes the fuzzy match is used as the target service request data, and the service response data corresponding to the target service request data determined by the fuzzy match is determined as the reference response data.

[0060] Optionally, the fuzzy matching service request data can be obtained from a preset storage location in sequence (for example, in the order of data request type and recording time), so that the platform service replays the service request data and determines the playback response. The backend personnel can determine whether the fuzzy matching is successful based on the matching of the playback response and the corresponding recorded response when the service request data was recorded. The response can be the response of the platform service interface to the service request data. When the playback response is consistent with or similar to the recorded response (for example, the similarity reaches a preset threshold), the fuzzy matching is considered successful. When the playback response is inconsistent with or not similar to the recorded response (for example, the similarity does not reach a preset threshold), the fuzzy matching is considered to have failed, and the next service request data can be obtained for fuzzy matching.

[0061] In one possible embodiment, the active traffic recording method provided in the present application fuzzy matches the service request data in a preset storage location based on the test request data, which can be: grouping the service request data according to the data request type of the service request data, and matching the service request data in the service request data group according to the recording time sequence of the service request data.

[0062] Exemplarily, the service request data is grouped and processed based on its data request type, and the service request data in the service request data group is sorted in chronological order of recording. The service request data group corresponding to the test request data is determined based on its data request type. For multiple test request data in the same service request data group, the service request data are matched within the service request data group based on the order in which the multiple test request data are processed and the order in which the service request data in the service request data group are recorded, i.e., the multiple test request data are in one-to-one correspondence with the service request data in the service request data group.

[0063] For example, the call of the same data interface is matched in the order of recording so that the first test request data matches the first service request data of the service request data group, the second test request data matches the second service request data of the service request data group, and so on, ensuring the consistency of the call order and obtaining the service request data that matches each test request data. This application performs fuzzy matching on the test request data by grouping the service request data according to the data request type and matching the service request data in the service request data group according to the recording time sequence of the service request data. When the same interface is called multiple times, the accuracy of the match is guaranteed by the time sequence, thereby improving the efficiency of the traffic playback test.

[0064] In one possible embodiment, the activity traffic recording method provided in the present application matches service request data in a service request data group according to the recording time sequence of the service request data, which can be: when service request data with the same recording time is matched in the service request data group according to the recording time sequence of the service request data, the similarity between the service request data with the same recording time and the test request data is determined, and the target service request data is determined based on the similarity.

[0065] Exemplarily, when service request data with the same recording time is matched in the service request data group according to the recording time sequence of the service request data, the similarity between the service request data with the same recording time and the test request data (for example, the similarity between the parameter request bodies corresponding to the service request data and the test request data) is determined, and the service request data with the highest similarity is used as the target service request data. Optionally, the similarity can be represented by a combination of one or more of the Euclidean distance, Hamming distance, and cosine similarity between the service request data and the test request data. The present application improves the success rate of traffic playback by accurately determining the target service request data based on the similarity between the service request data with the same recording time and the test request data when there is service request data with the same recording time.

[0066] S390: Perform traffic verification processing based on the test response data and the reference response data.

[0067] Exemplarily, the test response data and the reference response data are compared, and traffic verification processing is performed based on the differences between the test response data and the reference response data, and the reasons for the differences are analyzed to ensure that the business system is stable and iteratively launched.

[0068] What needs to be explained is that when a live broadcast activity calls a downstream service interface, such as sending service request data for basic protocols such as querying user information and querying live broadcast status, there will be a random number identifier (seq_id). When the relevant technology replays the traffic, there will be a problem of not being able to fully correspond to the parameters when regenerating it, resulting in the traffic not being able to be replayed correctly. This application generates identification information based on the data request type and the parameter request body. When the traffic is replayed, it performs precise matching based on the identification information. When the precise matching fails, it performs fuzzy matching. The fuzzy matching will first be grouped according to the data request type, and then matched according to the recording time sequence in each data request type group. If the recording time sequence is the same, the target service request data is accurately determined by combining the similarity comparison of the parameter request body to match the recorded result, ensuring accurate traffic playback. This application combines precise matching and fuzzy matching of the test request data to accurately match the reference response data corresponding to the test response data for traffic playback, ensuring stable iterative launch of the business system.

[0069] The above method calculates the queue time factor and queue size factor based on the queue capacity of the recorded data queue, the current cached data size of the recorded data queue, and the time information of the earliest recorded data in the recorded data queue. The recording acquisition rate is adjusted according to the preset basic acquisition rate, queue time factor, and queue size factor. The service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate to obtain recorded data. The recording acquisition rate is dynamically determined based on the queue capacity of the recorded data queue, the current cached data size of the recorded data queue, and time information. This effectively reduces the impact of a fixed recording acquisition rate on downstream services when the downstream services are busy, reduces the impact of the active traffic recording solution on the performance and stability of downstream services, and improves the effectiveness of active traffic recording. Furthermore, through a combination of precise matching and fuzzy matching of test request data, the reference response data corresponding to the test response data is accurately matched for traffic playback, ensuring the stable iterative launch of the business system.

[0070] Figure 4 This is a schematic diagram of the structure of an activity flow recording device provided by an embodiment of the present application. Figure 4 The activity traffic recording device includes a queue analysis module 41, a rate determination module 42 and a traffic recording module 43.

[0071] Among them, the queue analysis module 41 is configured to determine the queue capacity of the recording data queue, the data size currently cached in the recording data queue and the time information of the earliest recorded data in the recording data queue, and calculate the queue time factor and queue size factor based on the queue capacity, data size and time information; the rate determination module 42 is configured to adjust the recording acquisition rate according to the preset basic acquisition rate, queue time factor and queue size factor; the traffic recording module 43 is configured to record the service request data and service response data of the preset downstream service according to the recording acquisition rate to obtain recorded data.

[0072] In the above, the queue time factor and queue size factor are calculated according to the queue capacity of the recording data queue, the data size currently cached in the recording data queue, and the time information of the earliest recorded data in the recording data queue, the recording acquisition rate is adjusted according to the preset basic acquisition rate, the queue time factor, and the queue size factor, and the service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate to obtain the recording data. The recording acquisition rate is dynamically determined according to the queue capacity of the recording data queue, the data size currently cached in the recording data queue, and the time information, which effectively reduces the impact of the fixed recording acquisition rate on the downstream service when the downstream service is busy, reduces the impact of the active traffic recording solution on the performance and stability of the downstream service, and improves the active traffic recording effect.

[0073] In a possible embodiment, the activity traffic recording device further includes a data storage module, which is configured to save the recorded data to a recorded data queue and call a storage service to save the recorded data in the recorded data queue to a preset storage location.

[0074] In a possible embodiment, the queue analysis module 41 calculates the queue time factor and the queue size factor according to the queue capacity, data size, and time information, and is configured as follows: Calculating a natural exponential function result of a first ratio of a negative number of the natural exponential function applied to the time information to a preset time base conversion value, calculating a first difference between the preset time factor base and the natural exponential function result, and determining the first difference as the queue time factor; A second ratio of the data size to the queue capacity of the recording data queue is calculated, and the second ratio is determined as a queue size factor.

[0075] In one possible embodiment, the rate determination module 42 adjusts the recording acquisition rate according to the preset basic acquisition rate, the queue time factor, and the queue size factor, and is configured as follows: Calculating a first product of a queue time factor and a queue size factor, and calculating a second difference between a preset queue factor base and the first product; A second product of the preset basic acquisition rate, the second difference, and the preset attenuation factor is calculated, and the second product is determined as the recording acquisition rate.

[0076] In a possible embodiment, the activity traffic recording device further includes a rate limiting module, and the rate limiting module is configured to: The recording acquisition rate is limited according to the preset minimum rate and the preset maximum rate; Accordingly, the traffic recording module 43 records the service request data and service response data of the preset downstream service according to the recording acquisition rate to obtain recorded data, and is configured as follows: The service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate after range limitation processing to obtain recorded data.

[0077] In a possible embodiment, the data storage module calls the storage service to save the recorded data in the recorded data queue to a preset storage location, and is configured as follows: In the event of a storage service call failure, the retry interval is determined based on the storage service call retry count; The storage service is called again based on the retry time interval to save the recorded data in the recorded data queue to the preset storage location.

[0078] In one possible embodiment, the data storage module determines the retry time interval according to the number of storage service call retries, and is configured as follows: Calculate the third product of the preset basic delay time, the Nth power of the preset backoff factor, and the random jitter factor, and determine the third product as the retry time interval, where N is the number of storage service call retries.

[0079] In one possible embodiment, the rate determination module is further configured to: When the number of storage service call retries reaches a preset number of retries, determining a preset proportional request response time for the storage service call; When the preset proportion of request response time is greater than the preset response time threshold, the recording acquisition rate is reduced, and the service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate after the reduced frequency processing to obtain recording data.

[0080] In one possible embodiment, the rate determination module performs frequency reduction processing on the recording acquisition rate, and is configured as follows: A fourth product of the recording acquisition rate and the preset frequency reduction coefficient is calculated, and the fourth product is determined as the recording acquisition rate after the frequency reduction process.

[0081] In a possible embodiment, the activity traffic recording device further includes a traffic playback module, and the traffic playback module is configured to: Obtain test response data generated by the test environment in response to the test request data; generating first identification information according to a data request type and a test parameter request body of the test request data, and accurately matching the service request data in a preset storage location according to the first identification information; In the case of a successful exact match, the service response data corresponding to the target service request data determined by the exact match is determined as the reference response data; In the event that the exact match fails, fuzzy matching the service request data in a preset storage location is performed based on the test request data, and the service response data corresponding to the target service request data determined based on the fuzzy matching is determined as the reference response data; Perform traffic verification processing based on the test response data and reference response data.

[0082] In one possible embodiment, the traffic replay module fuzzy matches the service request data in a preset storage location according to the test request data, and is configured as follows: The service request data is grouped according to the data request type of the service request data, and the service request data is matched in the service request data group according to the recording time sequence of the service request data.

[0083] In one possible embodiment, the traffic playback module matches the service request data in the service request data group according to the recording time sequence of the service request data, and is configured as follows: When service request data with the same recording time is matched in the service request data group according to the recording time sequence of the service request data, the similarity between the service request data with the same recording time and the test request data is determined, and the target service request data is determined according to the similarity.

[0084] It is worth noting that in the embodiment of the above-mentioned activity traffic recording device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of this application.

[0085] An embodiment of the present application also provides an activity flow recording device, which can integrate the activity flow recording apparatus provided in an embodiment of the present application. Figure 5 This is a structural diagram of an activity flow recording device provided by an embodiment of the present application. Figure 5The activity flow recording device includes an input device 53, an output device 54, a memory 52, and one or more processors 51. The memory 52 is used to store one or more programs. When the one or more programs are executed by the one or more processors 51, the one or more processors 51 implement the activity flow recording method provided in the above embodiments. The activity flow recording device, equipment, and computer provided above can be used to implement the activity flow recording method provided in any of the above embodiments, and have the corresponding functions and beneficial effects.

[0086] The embodiments of the present application also provide a non-volatile storage medium that stores computer-executable instructions, which are used to execute the activity traffic recording method provided in the above embodiments when executed by a computer processor. Of course, the non-volatile storage medium that stores computer-executable instructions provided in the embodiments of the present application, whose computer-executable instructions are not limited to the activity traffic recording method provided above, can also execute the related operations in the activity traffic recording method provided in any embodiment of the present application. The activity traffic recording apparatus, equipment and storage medium provided in the above embodiments can execute the activity traffic recording method provided in any embodiment of the present application. For technical details that are not described in detail in the above embodiments, please refer to the activity traffic recording method provided in any embodiment of the present application.

[0087] Based on the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes a number of instructions for enabling a computer device, a mobile terminal or the processor therein to execute all or part of the steps of the activity traffic recording method provided in each embodiment of the present application.

Claims

1. A method for recording activity traffic, characterized in that: include: Determining a queue capacity of a recording data queue, a size of data currently cached in the recording data queue, and time information of the earliest recorded data in the recording data queue, and calculating a queue time factor and a queue size factor based on the queue capacity, the data size, and the time information; Adjusting the recording acquisition rate according to the preset basic acquisition rate, the queue time factor, and the queue size factor; The service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate to obtain recorded data.

2. The activity flow recording method according to claim 1, characterized in that: After recording the service request data and service response data of the preset downstream service according to the recording acquisition rate to obtain recorded data, the method further includes: The recorded data is saved in the recorded data queue, and a storage service is called to save the recorded data in the recorded data queue to a preset storage location.

3. The activity flow recording method according to claim 1, characterized in that: The calculating the queue time factor and the queue size factor according to the queue capacity, the data size and the time information includes: Calculating a natural exponential function result of a natural exponential function applied to a first ratio of a negative number of the time information to a preset time base conversion value, calculating a first difference between a preset time factor base and the natural exponential function result, and determining the first difference as a queue time factor; A second ratio of the data size to the queue capacity of the recording data queue is calculated, and the second ratio is determined as a queue size factor.

4. The activity flow recording method according to claim 1, characterized in that: The adjusting the recording acquisition rate according to the preset basic acquisition rate, the queue time factor, and the queue size factor includes: Calculating a first product of the queue time factor and the queue size factor, and calculating a second difference between a preset queue factor cardinality and the first product; A second product of a preset basic acquisition rate, the second difference, and a preset attenuation factor is calculated, and the second product is determined as the recording acquisition rate.

5. The activity flow recording method according to claim 1, characterized in that: Before recording the service request data and service response data of the preset downstream service according to the recording acquisition rate to obtain the recorded data, the method further includes: Performing range limiting processing on the recording acquisition rate according to a preset minimum rate and a preset maximum rate; Accordingly, recording the service request data and service response data of the preset downstream service according to the recording acquisition rate to obtain the recorded data includes: The service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate after range limitation processing to obtain recorded data.

6. The activity flow recording method according to claim 2, characterized in that: The calling storage service to save the recorded data in the recorded data queue to a preset storage location includes: In the event of a storage service call failure, the retry interval is determined based on the storage service call retry count; The storage service is re-called according to the retry time interval to save the recorded data in the recorded data queue to a preset storage location.

7. The activity flow recording method according to claim 6, characterized in that: Determining the retry time interval according to the number of storage service call retries includes: Calculate the third product of the preset basic delay time, the Nth power of the preset backoff factor, and the random jitter factor, and determine the third product as the retry time interval, where N is the number of storage service call retries.

8. The activity flow recording method according to claim 6, characterized in that: After re-calling the storage service according to the retry time interval to save the recorded data in the recorded data queue to a preset storage location, the method further includes: When the number of storage service call retries reaches a preset number of retries, determining a preset proportional request response time for the storage service call; When the preset proportion of request response time is greater than the preset response time threshold, the recording acquisition rate is down-converted, and the service request data and service response data of the preset downstream service are recorded according to the recording acquisition rate after the down-converted processing to obtain recorded data.

9. The activity flow recording method according to claim 8, characterized in that: The down-converting the recording acquisition rate includes: A fourth product of the recording acquisition rate and the preset frequency reduction coefficient is calculated, and the fourth product is determined as the recording acquisition rate after the frequency reduction process.

10. The activity flow recording method according to claim 2, characterized in that: After the calling storage service to save the recorded data in the recorded data queue to a preset storage location, the method further includes: Obtain test response data generated by the test environment in response to the test request data; generating first identification information according to a data request type and a test parameter request body of the test request data, and accurately matching the service request data in the preset storage location according to the first identification information; In the case of a successful exact match, the service response data corresponding to the target service request data determined by the exact match is determined as the reference response data; In the event that the exact match fails, fuzzy matching the service request data in the preset storage location according to the test request data, and determining the service response data corresponding to the target service request data determined by the fuzzy matching as the reference response data; A flow verification process is performed based on the test response data and the reference response data.

11. The activity flow recording method according to claim 10, characterized in that: The fuzzy matching of the service request data in the preset storage location according to the test request data includes: The service request data is grouped according to the data request type of the service request data, and the service request data is matched in the service request data group according to the recording time sequence of the service request data.

12. The activity flow recording method according to claim 11, characterized in that: The matching of the service request data in the service request data group according to the recording time sequence of the service request data includes: When service request data with the same recording time is matched in the service request data group according to the recording time sequence of the service request data, the similarity between the service request data with the same recording time and the test request data is determined, and the target service request data is determined based on the similarity.

13. An activity flow recording device, characterized in that: It includes queue analysis module, rate determination module and traffic recording module, among which: The queue analysis module is configured to determine a queue capacity of a recording data queue, a size of data currently cached in the recording data queue, and time information of the earliest recorded data in the recording data queue, and calculate a queue time factor and a queue size factor based on the queue capacity, the data size, and the time information; The rate determination module is configured to adjust the recording acquisition rate according to a preset basic acquisition rate, the queue time factor and the queue size factor; The traffic recording module is configured to record the service request data and service response data of the preset downstream service according to the recording acquisition rate to obtain recording data.

14. An activity flow recording device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the activity traffic recording method as described in any one of claims 1-12.

15. A non-volatile storage medium storing computer executable instructions, characterized in that: When executed by a computer processor, the computer executable instructions are used to perform the activity traffic recording method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the activity traffic recording method according to any one of claims 1 to 12 is implemented.

Images